Late-night comedian Stephen Colbert may have gone too far with a joke about members of the British Royal Family that has resulted in threats of legal action.

The CBS "Late Show" host was reportedly contacted by attorneys representing Sarah Rose Hanbury, the Marchioness of Cholmondeley, whom he jokingly accused of having an extramarital affair with Prince William earlier this month, Fox News reported.

Allegations of the supposed affair between Hanbury and the Prince of Wales, which have been circulating to some extent since 2019, had a recent resurgence as one of several internet-fueled conspiracy theories to explain the disappearance from the public eye of Kate Middleton, the Princess of Wales, who has been recovering from abdominal surgery in January and just revealed that she was diagnosed with and being treated for cancer.

Legal notice sent to late-night comedian

In Touch Weekly reported exclusively this week that it had received a statement from Hanbury's attorneys about a legal notice that had been sent to late-night comedian Colbert about a joke he told during a monologue earlier this month.

"The rumor appears to have entered into the mainstream media recently as a result of another joke made about it on 'The Late Show with Stephen Colbert,'" the statement said.

The lawyers added, "We have written on our client’s behalf to CBS and various other reputable media organizations to confirm that the allegation is false."

Colbert's joke about Hanbury and the Prince

On March 12, according to In Touch, Colbert devoted a few moments of his monologue to one particular "Katespiracy" about the Princess of Wales' months-long "absence" from public view and raised the rumor about an alleged affair involving the Prince of Wales.

"The kingdom has been all aflutter by the seeming disappearance of Kate Middleton. Well now, internet sleuths are guessing that Kate’s absence may be related to her husband and the future King of England William having an affair," Colbert said. "So, I think we all know who the alleged other woman is. Say it with me -- the Marchioness of Cholmondeley -- what a beautiful name!"

Further riffing on Hanbury's royal title, the comedian noted, "Now, there have been rumors [of] an affair between William and the Marching Band of Chicanery since 2019."

"According to tabloids back then, when Kate supposedly confronted him about it, he laughed it off saying there was nothing to it. Always a good response when your wife accuses you of cheating," Colbert said and added with a fake British accent, "Ha ha, imagine me having an affair! It is to laugh."

Lawyers already denied allegations as "completely false"

The statement to In Touch about the legal notice sent to Colbert wasn't the first time that Hanbury's attorneys spoke out against the comedian's joke at her expense, as Business Insider reported about a week earlier that Hanbury's lawyers stated that "the rumors are completely false."

Per the Insider, Hanbury, 39, is an extended member of the Royal Family with ancestral ties to the British monarchy; she is a former model and has been married since 2009 to David Rocksavage, the 7th Marquess of Cholmondeley, with whom she shares three children.

They used to reside in an estate next to Windsor, where William and Kate live with their children and reportedly became close friends until Rose and Kate had a reported falling out with each other around 2019, at which point the internet rumors of an extramarital affair involving Rose and William took off.

While most of the British tabloids declined to delve into those rumors -- reportedly due to quiet threats of legal action -- other European and U.S. tabloids have occasionally reported on the allegations, even as William and Kate have never displayed any signs that either of them are less than happily married.