CHATGPT CAN GENERATE VIOLENT AND SEXUALISED IMAGES, RESEARCHERS FIND

ChatGPT's latest public version can be made to produce sexually explicit and graphically violent imagery through altered prompts, according to research by British AI security firm Mindgard. The company discovered that modifications to a widely-shared instruction designed for humorous output enabled the chatbot to generate such content. OpenAI, which operates ChatGPT, stated it had introduced additional safeguards following contact from the BBC, though researchers found the problematic prompt continued to produce concerning material with further minor adjustments.

Peter Garraghan, founder of Mindgard and a professor in the computing department at Lancaster University, said the chatbot generated disturbing imagery without explicit instruction on subject matter. The AI produced graphic and sexualised content "of its own volition" when given seemingly innocuous prompts, he noted. Jim Nightingale, Mindgard's AI safety and security researcher, reported being deeply affected by the images the system could create, describing himself as "shaken, and in tears." OpenAI stated it maintains multiple layers of protection designed to prevent users from creating content that violates its terms and conditions.

Mindgard specialises in red-teaming, the practice of identifying vulnerabilities in AI systems so companies can address security gaps. The BBC confirmed it had viewed examples of the generated images, including depictions of severe head injuries and scenes suggesting sexual violence. The disclosure highlights ongoing challenges in restricting harmful content generation within large language models despite implemented safeguards.