The social media network X has sparked widespread outrage after people used its AI chatbot Grok to change photos of women by taking off their garments.
Numerous examples have been seen where the technology has been employed to digitally undress women to render them in bikinis without permission, as well as to insert them in inappropriate situations.
Developer Response and Regulatory Position
XAI, the organization responsible for Grok, did not respond to questions, except for an auto-reply message stating "traditional news outlets lies".
A government representative stated that legislation is being brought forward to outlaw nudification tools. Under a proposed criminal offence, anyone who distributed such tools would "face a prison sentence and significant penalties".
"The platform could prevent these types of misuse if they desired to," said a law professor, adding they "appear to enjoy impunity."
Oversight Body Stance and Tool Capabilities
The communications watchdog Ofcom said technology companies must "evaluate the danger" of people viewing unlawful material on their services, but did not specify whether it was at this time investigating X or Grok in concerning artificial intelligence images.
Grok is a free chatbot—with some paid-for features—which responds to X users' queries when they mention it in a tweet.
It is frequently used to give commentary or additional context to other users' remarks, but people on X are also able to edit an posted picture through its artificial intelligence picture modification feature.
Background of Controversy and Policy
The chatbot has been criticised for allowing users to produce images and videos with nudity and sexualised content, and it was previously alleged to have been involved in generating sexually explicit content of public figures.
"The platform has been allowing the generation and sharing of these pictures for months without taking any action and we have not witnessed any challenge by oversight bodies," Professor McGlynn added.
XAI's own terms of service prohibits "portraying images of people in a sexually explicit way".
In a comment, Ofcom said it was illegal to "generate or distribute deepfakes or child sexual abuse material" and clarified this included sexual deepfakes generated with AI.
It said platforms such as X were mandated to take "appropriate steps" to "reduce the danger" of users coming across illegal material on their services and to remove it swiftly when they learn of it.