"The voting (of the envoy) was not in accordance with the procedures of the government and therefore the minister of foreign affairs has instructed the permanent representative of Malaysia to the United Nations in Vienna to return to Malaysia for consultations," the foreign ministry said in a statement.
The ministry declined to give any further details.
The envoy, Mohd Arshad Manzoor Hussain, holds the rotating chair of the 35-member IAEA's Board of Governors.
Malaysia last week was one of three countries with Cuba and Venezuela that voted against the resolution to rebuke Iran for building a second enrichment plant in secret.
The measure passed by 25 votes to three. Afghanistan, Brazil, Egypt, Pakistan, South Africa and Turkey abstained while Azerbaijan missed the ballot.
China and Russia, which have blocked global attempts to isolate Iran in the past, voted in favour of the resolution.
Diplomats said the vote indicated important developing states were souring on Iran over its nuclear defiance.
Western powers have warned Iran it is rapidly approaching an end-of-year deadline to accept a U.N.-drafted nuclear fuel deal which calls on Tehran to send 75 percent of its low-enriched uranium (LEU) from Natanz to Russia and France to be turned into fuel for a Tehran medical research reactor.
(Reporting by David Chance; Editing by Sonya Hepinstall)
