The High Commissioner for Human Rights warned in a statement that illegal acts by militia and security forces "could provoke a serious deterioration in the security situation."
Pillay is "concerned about reports of an increasing number of arrests, which may not be in conformity with the law, and the possible illegal use of excessive force and acts of violence by some militia members," her office said.
A spokesman for the UN rights chief, Rupert Colville, said the number of arrests of political leaders and human rights activists was "not clear", but indirect information suggested it "appears to be somewhere in the hundreds."
"There's a lack of information, especially from the provinces," he told AFP.
The UN High Commissioner for Human Rights "commended the largely peaceful and dignified conduct of the huge demonstrations" in Tehran, underlining that freedom of expression and freedom of assembly were fundamental rights.
But she took issue with reported violence by Islamic militia members, warning of possible violations of Iranian as well as international law, and queried the grounds for the arrests.
"The legal basis of the arrests that have been taking place, especially those of human rights defenders and political activists, is not clear," Pillay said.
"Why have some of those who have been arrested been denied access to lawyers and members of their families? And why is the whereabouts of others unknown?"
"These are all troubling questions, and I urge the Iranian authorities to ensure that due process is followed, since to do otherwise may fan the feelings of injustice," she added.
Reported violence by security forces and Islamic Basij militia, could also tip the balance, she warned.
The Basij have played a leading role in facing down public protests over official poll results giving hardline incumbent Mahmoud Ahmadinejad a new four-year term.
"It is the responsibility of the government to ensure that militia members and regular law enforcement agencies do not resort to illegal acts of violence," Pillay said.
"If they are perceived to be acting outside the law, it could provoke a serious deterioration in the security situation, which would be a great tragedy and is in nobody's interests."
Iran is a party to the International Covenant on Civil and Political Rights, which guarantees freedom of expression, the right to peaceful assembly and protection from arbitrary detention.
Tens of thousands of opposition supporters held a new rally in Tehran on Thursday, the sixth straight day of protests after all of three of Ahmadinejad's defeated challengers lodged complaints of vote-rigging.