For Boris Johnson, facts have always been flexible. The British prime minister’s career is littered with doctored quotes, tall tales, exaggerations and mistruths. When called out, he has generally offered an apologetic shrug or a guilty grin, and moved on. Plenty of people were willing to forgive him.

At least until now. Revelations that the prime minister and his staff partied while Britain was under coronavirus restrictions have provoked public outrage and prompted many in the Conservative Party to consider dumping their leader. The Conservatives picked Johnson because his image as a cheerful rule-breaker — the naughty schoolboy of British politics gave him a rare ability to connect with voters. Now, many are having second thoughts.

“His fans would say he’s a force of nature he doesn’t let things get in his way,” said Steven Fielding, professor of political history at the University of Nottingham. “Sometimes he’s been caught out, but mostly he’s got away with it,” Fielding added. “Now the reality is becoming more apparent to more and more people.” Johnson has often been able to talk his way out of crises. The Oxford-educated politician has used words to create the image of a rumpled jokester with a mop of blond hair who doesn’t take himself too seriously. Quips and jokes tumble out of him, sometimes in Latin or ancient Greek. That persona made Johnson a popular guest on the humorous TV show “Have I Got News for You” from the late 1990s onwards, and brought him global fame as London’s boosterish mayor between 2008 and 2016. Many people thought he was too lightweight ever to become prime minister, and Johnson didn’t contradict them. He disguised his ambition with jokes, saying he had as much chance of becoming prime minister as of “finding Elvis on Mars” or being “reincarnated as an olive.” In fact, he had long dreamed of power. His sister Rachel Johnson has said his childhood ambition was to be “world king.” But his route to the top was haphazard. As a young journalist at The Times of London, he fabricated a quote about King Edward II from a historian, who also happened to be his godfather.

He was fired, but that didn’t stop him becoming Brussels correspondent for the Daily Telegraph in the early 1990s, filing exaggerated stories of EU waste and red tape. Those “Euromyths” about one-size-fits-all condoms and plans to ban “bendy bananas” helped turn British opinion against the bloc, and ultimately lead to Johnson becoming the Brexit champion who would years later bring the U.K. out of the EU.

Brexit was won in a 2016 referendum campaign that contained many questionable claims, notably the allegation often repeated by Johnson that Britain gave the EU 350 million pounds a week that could instead be spent on the U.K.’s health service.

Now, though, Johnson’s allies worry that the tide has turned. Johnson has apologised for the lockdown-breaching parties in uncharacteristically subdued and carefully worded statements. He has stopped short of admitting personal wrongdoing, saying he believed he acted within the rules.

But many Britons who stuck to lockdown rules imposed by the government cut off from friends and family, unable to visit relatives in nursing homes and hospitals have scoffed at Johnson’s “partygate” excuses, including his claim that he thought a “bring your own booze” garden party was a work event.

Chris Curtis, head of political polling at Opinium Research, said public trust in the prime minister had plummeted and Johnson’s personal approval ratings were now “pretty dire.”