Arsenal had to come from behind after Sergio Aguero scored his 30th goal of the season for City in the 62nd minute, racing away on to a pass from Yaya Toure.
Spanish international Nacho Monreal equalised nine minutes later, from a cross by Arsenal's other wing back Alex Oxlade-Chamberlain.
In the 11th minute of extra time, just after Aguero was taken off, Alexis Sanchez scored the winner.
The result gave manager Arsene Wenger, an FA Cup winner six times, a welcome boost in a season that many supporters would like to be his last at the club.
Arsenal seem likely to miss out on a top-four finish in the Premier League for the first time in 21 seasons under Wenger but winning a third FA Cup in four years could well persuade him to defy his critics and sign a new contract.
Pep Guardiola's record of having won at least one trophy in every one of his seasons as a manager is almost certain to be ended.
City trail league leader Chelsea by 11 points with six games to play going into another big game on Thursday against neighbour and rival Manchester United, who are fifth, one point behind them.
In his 50th City game, Guardiola picked an unchanged side for the first time and they were the more threatening in a dull first half despite having to replace the injured David Silva with Raheem Sterling.
Shortly before the interval they were unluckily denied a goal when an assistant referee ruled - incorrectly it appeared - that Leroy Sane's cross had drifted out of play before Aguero and then Sterling both put the ball past Petr Cech.
In a much livelier second half Toure robbed Aaron Ramsey and sent Aguero away from the halfway line to dink the ball past Cech for a 12th goal in as many games.
The equaliser arrived from a less regular scorer. It was Monreal's first goal for two years.
Toure's volley and Fernandinho's header both hit the frame of the goal in the space of four minutes but early in extra time a weary Aguero went off and within two minutes Sanchez scored from close range after Danny Welbeck missed his kick.