Imam Maulama Akonjee, 55, and Thara Uddin, 64, were shot in the head on Saturday in the borough of Queens.
Some mosque-goers said the shooting was a hate crime, but police say there is as yet no evidence the men were targeted because of their faith.
Attacks on US mosques are at their highest levels since records started.
The two men, originally from Bangladesh, were approached from behind by a man who shot them both in the head, police said. They had both just left the al-Furqan Jame mosque.
A man holding a gun was seen leaving the scene of the shooting in the Ozone Park area, but no-one has been arrested.
Police said the suspect had a "medium complexion". They released a sketch on Sunday.
"While we do not yet know the motivation for the murders of Maulama Akonjee and Thara Uddin, we do know that our Muslim communities are in the perpetual crosshairs of bigotry," Mayor Bill De Blasio said in a statement.
"It remains critical that we work to bridge the divides that threaten to undermine the greatness of our city and country."
Police said that Imam Akonjee was carrying about $1,000 (£775) when he was shot, but that the money was not taken.
A video purporting to show the shooting, obtained by the New York Daily News, shows someone quickly approaching the men from behind and shooting with no warning before running away.
"He would not hurt a fly," Imam Akonjee's nephew Rahi Majid, told the newspaper.
"You would watch him come down the street and watch the peace he brings."
A service was held on Sunday to remember the two men.
On Saturday night, dozens of people from the nearby Muslim community had gathered at the scene to mourn and demonstrate, chanting "We want justice".
A report published by the University of California at Berkeley and the Council on American-Islamic
Relations showed that mosques were targeted in 78 attacks in 2015, the highest level since records were first kept in 2009.
'He stood for peace'
Last year the New York Times reported that hate crimes against US Muslims and mosques tripled in the wake of attacks in Paris and San Bernardino.
Some of those quoted in Ozone Park said they saw the consequences of Donald Trump's presidential campaign, which has seen him call for Muslims to be banned from entering the US.
"This neighbourhood is getting crazy because of this election and Trump. He hates Muslims," one man, Monir Chowdhury, told Associated Press. "I love this neighbourhood and now I'm scared."
Tributes to the two men were also paid in their native Bangladesh.
Foreign Minister Mohammed Shahriar Alam condemned what he called a "cowardly act on peace-loving people" and the US ambassador to Dhaka, Marcia Bernicat, said Imam Akonjee "stood for peace".
She added: "We join Bangladeshis to condemn his killing and honour what he stood for."