Prime Minister Theresa May was quick to demand he apologise, along with the party chairman Brandon Lewis and Conservative Muslim Forum leader Lord Sheikh, and now the Tory high command have launched an investigation into his conduct.
Uxbridge MP Johnson made the comments in a Telegraph column and, according to the Metro, he will now face an investigation by an independent panel after complaints that his comments breached the Conservative Party's code of conduct.
He added: "All jokes about religion cause offence, so it's pointless apologising for them".
'He after all was defending the Government's position because we are not going to ban burkas, we believe in people having the right to choose'.
Under party rules, the head of the investigation may dismiss complaints that are obviously trivial or lacking in merit and those which can not fairly be investigated.
The Blackadder and Mr Bean star Atkinson said the former Foreign Secretary's comments were amusing.
In a letter to The Times, he wrote: "You should really only apologise for a bad joke".
He wrote: "As a lifelong beneficiary of the freedom to make jokes about religion, I do think that Boris Johnson's joke about wearers of the burka resembling letterboxes is a pretty good one". "On that basis, no apology is required".
But Muslim organisations said there had been "a real impact" from Johnson's remarks.
Mr Johnson also won support from a surprising source when comedian Rowan Atkinson waded into the furore.
He explained there was a complaints procedure within the Conservative Party and that the Party had a mechanism for dealing with these issues.
The ex Foreign Secretary faces being punished by party bosses and could even be kicked out of the party over the remarks, although this is very unlikely.
With a Tory party inquiry into Johnson's conduct announced on Thursday, the matter looks set to rumble on.