They threw rocks and Molotov cocktails at Muslim-owned businesses, attacked Muslim homes and ransacked the town's main mosque.
Police added at least 13 people had been arrested over the clashes so far and a hunt was on for more suspects.
A nationwide curfew, declared for the second night running, will come into effect at 21:00 (15:30 GMT) on Tuesday.
The government imposed the island-wide curfew on Monday after the violence against the Muslim community, who account for 10 per cent of the country's population.
The UN's special advisors on the prevention of genocide and the responsibility to protect called on the government to investigate the mob violence and hold perpetrators accountable.
In a televised address, police chief Chandana Wickramaratne said constables have been asked to use maximum force against rioters.
He said, a group of people are trying to disrupt the Vesak festival and no room should be allowed to fulfil their intention.
Among those detained was Amith Weerasinghe, a man from Sri Lanka's majority Buddhist Sinhalese community and on bail for his role in similar riots in March previous year in the central Kandy district.
How has the government responded?
Police arrested a group of men for the attacks but people in the mostly Buddhist district then demanded their release, military spokesman Sumith Atapattu said.
He said the armed forces will ensure that the country will not be getting back to a situation of violent activities and the violence in terms of radicalisation and the terrorists.
Prime Minister Ranil Wickremesinghe posted this on his Twitter account on Monday.
Officials have blocked some social media platforms and messaging apps, including Facebook and Whatsapp, in an attempt to curb the violence. One Muslim businessman, who wished to remain anonymous, told the BBC he feared for his safety. They were in the street, they didn't stop anything.
Aadhil Ameez was taken into custody on 25 April, four days after the Easter Sunday attacks, for allegedly providing technical and logistical support to the suicide bombers targeting mosques and hotels. Hours later, two more bombs were detonated, with fewer casualties, as police closed in on those behind the bombings. Violence broke out in the island nation late Monday; three weeks after suicide bombings killed 258 people.
"We have already taken measures to obtain aerial photographic evidence of those involved in such activities and to direct such evidence for legal action against lawbreakers", he said.
On Sunday, Cardinal Malcolm Ranjith, the archbishop of Colombo, gave Sri Lanka's Catholic school administrators permission to reopen on an individual basis in coordination with local security officials. In a statement Tuesday, M. A. Sumanthiran, a Tamil politician and spokesman for the Tamil National Alliance, which includes political parties from the Tamil minority, drew a link between the mob attacks and Sri Lanka's civil war history.