The 70-year-old former French Open and U.S. Open champion, who is captain of Romania's Fed Cup team, was sent from his courtside chair after apparently aiming abuse at the umpire, British player Johanna Konta and visiting captain Anne Keothavong during the second singles rubber.
World number seven Konta was in tears when play was briefly suspended at the tie in Constanta where she was up against Sorana Cirstea, apparently because of remarks made by Nastase.
He was eventually excluded from the tie because of "unsportsmanlike conduct".
Konta recovered her composure to win the last five games for a 6-2 6-3 victory that levelled up the tie at 1-1 after Romania's number one Simona Halep had beaten Heather Watson.
"Mr Nastase was also removed from the grounds due to his serious misconduct," the ITF said in a second statement.
"His accreditation was removed and he will play no further part in the tie.
"The ITF has launched an investigation into this matter as well as previous comments made by Mr Nastase during the week."
British captain Keothavong, who said she was made to feel "uncomfortable" by comments made by Nastase at the official pre-match dinner, said the abuse during play had been unacceptable.
"We expected a patriotic crowd for the Romanian team but we don't expect abusive language to be used ... what he said directed to both Johanna and myself is... language that is not appropriate for anyone to speak to any other human."
ITF president David Haggerty said an investigation would be launched.
"This is unacceptable behaviour by a Fed Cup captain. No player, official, member of the media or fan should have to endure any kind of abuse, and Mr Nastase will rightly play no further part in this tie," he said.
"A formal investigation is already underway and any decision or sanction will be made by the ITF's Adjudication Panel. We are unable to comment further on an ongoing investigation."
[caption id="attachment_59493" align="aligncenter" width="640"]
Britain's Johanna Konta cries while leaving the tennis court during the FedCup Group II play-off match between Romania and Great Britain, in Constanta county, Romania, Romania on Saturday Reuters
Nastase, whose behaviour on court during his career often landed him trouble and earned him the nickname "Nasty", began the tie under a cloud after remarks overheard at the official draw ceremony on Friday.
He was heard speaking in Romanian to one of his team members about Williams' baby.
"Let's see what colour it has. Chocolate with milk?" he was quoted by Romanian and some British media as saying.
Williams, who confirmed her pregnancy through a spokeswoman on Wednesday, is engaged to Reddit co-founder Alexis Ohanian, who is white.