Police had yet to establish a motive and said there was no evidence the men were targeted because of their faith, but nothing was being ruled out. Residents demanded authorities treat the brazen daylight shooting as a hate crime.
The gunman approached the men from behind and shot both in the head at close range about 1:50pm (1750 GMT) in the Ozone Park neighbourhood of Queens, one of the city's five boroughs, police said in a statement.
The victims, identified as Imam Maulama Akonjee, 55, and Thara Uddin, 64, were wearing religious garb, police said. Police found them bleeding in the street and took them to a hospital where they were pronounced dead.
"While we do not yet know the motivation for the murders of Maulama Akonjee and Thara Uddin, we do know that our Muslim communities are in the perpetual crosshairs of bigotry," New York Mayor Bill de Blasio said in a statement. "Rest assured that our NYPD will bring this killer to justice."
The men were attacked about two blocks from the Al-Furqan Jame Mosque where they had just left afternoon prayers. Ozone Park, a diverse, largely working-class area, is home to a growing number of Muslims of Bangladeshi heritage.
Millat Uddin, 57, an Ozone Park resident not related to the imam's associate, said both men were born in Bangladesh. He said he was close to Akonjee, describing him as a "docile, calm" father of seven who was beloved in the neighbourhood.
"What matters most is harmless people have been shot dead, regardless of whether this was a hate crime," he said. "Our community's heart is broken."
Akonjee was carrying $1,000 with him at the time of the attack but the money was not taken, the New York Times reported.
"I have never felt this kind of tension," said Nizam Uddin, 57, a taxi driver who said he knew both the cleric and his associate. He also was not related to the associate.
'Makes all Muslims scared'
The shooting appeared to be the most violent attack against local Muslim leaders in recent years, said Ibrahim Hooper, national communications director for the Council on American-Islamic Relations, or CAIR, a civil rights and advocacy group.
A report by CAIR and the University of California at Berkeley released in June said the number of recorded incidents in which mosques were targeted jumped to 78 in 2015, the most since the body began tracking them in 2009.
Hooper said he could recall incidents in which an imam was pushed, called names or otherwise harassed. "Things like that, but nothing of this nature, nothing where people were killed," he said.
CAIR said it was offering a $10,000 reward for information that would lead to the arrest and conviction of the shooter.
Police released a sketch of a male suspect with dark hair, a beard and glasses. Police described him as having a medium complexion. He appeared to be in his 30s or 40s.
Witnesses told police they saw the assailant, dressed in a dark shirt and blue shorts, fleeing with a gun in his hand, police said. Surveillance footage showed the suspect tailing the victims.
Mohammed Ahmed, 22, works at his father's corner store on Liberty Avenue just two blocks from the shooting. He said he heard the shots while he was at work.
"It makes all the Muslims scared," he said. "Last time someone got shot in this neighbourhood that I know of was probably 2001."