Australian Prime Minister Kevin Rudd said in May the government was taking legal advice on launching a case against Ahmadinejad after the Iranian leader said the Jewish state should be wiped off the map.
Smith said the government would no longer proceed with the case against Ahmadinejad, whose comments he described as "appalling by any standard".
"The Government has given exhaustive consideration to international legal action against Iran for these statements," Smith said.
"Having now considered legal and other advice, the Government has decided not to pursue international legal action against Iran."
He said the government had dropped the bid because of the complexity of the issues involved and the high legal threshold required to bring forward a case.
"As well, we determined to avoid pursuing a case which would give further profile to these obscene remarks," Smith said in a statement.
"Most importantly, the Australian government would not want such legal action to complicate or distract from the international community's efforts to address the serious concerns about Iran's nuclear program and its failure to abide by binding United Nations Security Council resolutions."
Smith said in response to Iran's ongoing defiance of the UN, Australia would impose new financial and travel sanctions on 20 Iranian individuals and 18 organisations which contribute to Iran's nuclear and missile programs, or otherwise assist Iran to violate its Security Council obligations.
"These organisations include Iranian banks Melli and Saderat," he said.
Iran does not recognise the Jewish state and since becoming president in 2005 Ahmadinejad has repeatedly provoked international outrage by predicting that Israel is doomed to disappear.
