Grok AI Security Failure Exposes Deepfake Risks on X

Elon Musk's X platform has deployed emergency security restrictions on its Grok AI tool in April 2025 following a major incident in which users exploited the system to create sexualised images of real people. The security breach, which saw the AI tool weaponised to generate non-consensual deepfakes including images of women and children, has prompted formal investigations by regulators and exposed critical vulnerabilities in gen AI safeguards.

The company confirmed in April 2025 that in jurisdictions where such content is illegal, Grok will no longer be able to edit photographs of individuals to depict them in revealing clothing.

"We have implemented technological measures to prevent the Grok account from allowing the editing of images of real people in revealing clothing," an announcement on X says.

Regulatory response to security failures

The UK's independent online safety watchdog, Ofcom, opened a formal investigation into X in April 2025 under the UK's Online Safety Act to determine whether the platform has complied with its legal duties to protect people in the UK from content that is illegal under UK law.

"We are aware of serious concerns raised about a feature on Grok on X that produces undressed images of people and sexualised images of children," Ofcom says. "We have made urgent contact with X and xAI to understand what steps they have taken to comply with their legal duties to protect users in the UK."

In an update following the latest development, an Ofcom spokesperson added: "X has said it's implemented measures to prevent the Grok account from being used to create intimate images of people. This is a welcome development. However, our formal investigation remains ongoing."

The security lapse was described as a major policy failure, with Prime Minister Sir Keir Starmer calling X's inaction "horrific", "disgusting" and "shameful". Technology Secretary Liz Kendall characterised the platform's delay in acting as "a further insult to victims, effectively monetising this horrific crime".

In the US, California's attorney general has launched an investigation in April 2025 into the spread of sexually explicit AI deepfakes – including material of minors – generated by Grok, expanding the security incident to a transatlantic regulatory concern.

Technical controls and access restrictions

In an update via its Safety account, X says: "We now geoblock the ability of all users to generate images of real people in bikinis, underwear and similar attire via the Grok account and in Grok in X in those jurisdictions where it's illegal."

The company emphasises that only paying subscribers retain access to Grok's image-editing tools – an additional "layer of protection" designed to ensure accountability among those misusing the AI. However, all users are prevented from allowing Grok to edit images of real people in revealing clothing.

Musk has insisted that Grok complies with the laws of each country. Posting on the platform, he says: "Obviously, Grok does not spontaneously generate images, it does so only according to user requests. When asked to generate images, it will refuse to produce anything illegal."

The technical implementation relies on a combination of geographic IP detection and content classification algorithms. X has deployed machine learning models trained to identify real individuals in uploaded images, blocking editing requests that attempt to modify clothing or generate revealing imagery.

This multi-layered approach represents a significant shift from the platform's previous reliance on post-publication content moderation to pre-emptive blocking at the generation stage.

Platform security and governance challenges

Despite the response, global regulators and advocacy groups argue that X's reactive measures highlight a broader security governance problem across generative AI platforms. Thousands of sexualised AI images have circulated on X in recent weeks alone, prompting calls from legislators and women's groups for Apple and Google to ban Grok from their app stores due to inadequate security controls.

Three Democratic senators in the US have urged both companies to remove X and its built-in AI tool Grok from their app stores, citing the proliferation of non-consensual content. Musk's dual role leading both X and xAI – the company that builds Grok – has further intensified scrutiny of potential conflicts between innovation and responsible security moderation.

The incident has reignited debates about the adequacy of self-regulation in the AI industry. Critics argue that platforms should implement robust safeguards before deploying generative AI tools publicly, rather than responding only after harm has occurred.

Industry observers note that the challenge extends beyond individual platforms to fundamental questions about AI governance frameworks and enforcement mechanisms across jurisdictions.

X's reversal marks a crucial moment in the evolution of AI platform security governance. By introducing geoblocking and restricting tool access, X has taken a step toward rebuilding trust – but experts warn that strong security policy enforcement and ongoing transparency will determine whether such measures hold.