"The view of the British government is that we unreservedly condemn the Iranian President's offensive and inflammatory remarks," Brown's spokesman said.
Foreign Secretary David Miliband, meanwhile, described Ahmadinejad's comments as "offensive, inflammatory and utterly unacceptable."
Ahmadinejad sparked a walkout of dozens of diplomats from the conference in Geneva when he criticised Israel, saying he deplored the creation of a "totally racist government in occupied Palestine" in 1948 and called it "the most cruel and racist regime".
Britain's representative at the conference, its ambassador to the UN in Geneva, Peter Gooderham, was among those who left the room in protest.
Brown's spokesman said: "Such remarks have no place anywhere, least of all in a UN anti-racism forum.
"We don't think that the President of Iran's presence in itself constituted grounds to withdraw from the conference, but in the light of his remarks the decision was taken that we would not attend for the part of the speech in which he was speaking."
Miliband said the British delegation and others "rightly walked out of President Ahmadinejad's speech because such hate-filled rhetoric is an intolerable abuse of free speech and of the conference."
He added: "We will not accept an event that degenerates into racism and intimidation.
"But nor should we leave the international stage only to those, like President Ahmadinejad, who would take global efforts against racism backwards."
United Nations chief Ban Ki-moon also condemned Ahmadinejad's remarks, which prompted 23 European Union delegations to walk out.
The meeting had already been boycotted by the United States and Australia, as well as Israel, in protest at the presence of the Iranian president, who has previously called for the Jewish state to be wiped off the map.
