AI Tools: Waiting to be Weaponized
In recent years, AI-generated content has become increasingly accessible and sophisticated. The advancement has enabled realistic images, videos, and voices to be created with minimal data input. While AI has been promoted as a helpful tool, they have also become increasingly weaponized in cases of sexual and gender-based violence (SGBV). One disturbing example is Grok, an AI assistant on X, which has made it possible for someone to generate or manipulate existing pictures of someone to misrepresent their representation and actions. Unlike traditional forms of online violence, such as hacking or data leaks, individuals may not even know they are victims of online SGBV because there is no direct indication that they are being targeted. It is truly terrifying that individuals are being exploited and harmed without ever participating in the creation of the content itself. To make matters worse, victims of AI sexualization may not know these perverted images are out there unless they happen upon them themselves or are alerted by others. Despite the image itself being fabricated, the consequences are very real.
Grok was first released in 2023, designed to assist with answering questions, generating ideas, and etc. Over time, the capabilities of the AI assistant expanded, and in 2025, four modes for Grok were released. Disgustingly, one of the modes was “Spicy,” marketed to encourage people to generate suggestive and provocative content with fewer safeguards and restrictions. In less than two weeks, 3 million sexualized images were created, illustrating how accessible the tool was and how eager the public jumped to misuse this new feature. The popularity of this tool is concerning, given that many tools are trained to optimize and recognize female bodies, but are not able to do the same for male bodies. The tools are evidently tailored to male users, allowing them to put in prompts asking for edited images to show women in as little clothing as possible, bypassing consent entirely. Furthermore, these pictures were not created in private; instead, people were posting prompts, sharing results, and openly teaching one another how to produce specific sexualized material. The promotion of nudification technology and monetizing it is appalling. Even more infuriating, SpaceX had purposefully set aside around $530 million for lawsuits and complaints against sexualized images generated by Grok, meaning that the developers knew just how insidiously this AI tool can be abused to exploit the innocent and still decided to promote it nonetheless. Stronger safeguards should have been considered, and something such as the “Spicy” mode should never have been an option.
In December of 2025, Grok received significant criticism after creating a sexualized image of two girls under the age of 18. The person behind the account generating the image clearly bears responsibility for intentionally misrepresenting someone without consent. However, the company also played a major part in the incident for allowing it to occur altogether when they should have been well aware that this would have happened sooner or later. The backlash surrounding Grok eventually led to an apology saying, “This violated ethical standards and potentially US laws on CSAM (child sexual abuse material)... It was a failure in safeguards, and I’m sorry for any harm caused. xAI is reviewing to prevent future issues.”
While xAI has promised to strengthen its safeguards, in April of 2026, an NBC review found that a significant amount of non-consensual sexual images and videos were still being generated. The review found that instead of directly asking Grok to put someone in a bikini, users can now ask “Grok to swap the clothing of women in two separate photos, with at least one of the photos involving tight or revealing clothing.” As users constantly upgrade their tactics to find loopholes in restrictions, developers must also constantly keep up with monitoring and strengthen safeguarding accordingly. It makes one wonder if the tool even needs to exist in the first place, so long as there are predators on these platforms, and several countries have already begun acting on that notion. Malaysia and Indonesia became the first countries to block access to Grok. The governments have realized the potential consequences from the beginning and put a stop to it before more harm occurs.
Ultimately, the controversy surrounding Grok demonstrates that AI-generated SGBV cannot simply place the blame on individuals misusing the technology. It represents a broader failure of developers, companies, platforms, and policymakers to adequately anticipate and prevent harm. As AI becomes more accessible and sophisticated, legal and ethical protections must evolve alongside it. Several countries have taken essential first steps towards addressing these problems, but it is still far from enough. For example, the UK Online Safety Act, passed in 2023, made it illegal to share explicit images or videos that have been digitally manipulated. Still, loopholes remain, as the creation of pornographic deepfakes or sharing them is not fully prevented if the intent to cause distress is not demonstrated. Global cooperation is recommended to ensure that AI tools meet a safety and ethics standard before being available to the public, so that violence is minimized in the first place. More safeguards to restrict sexual content involving identifiable individuals should be implemented, and strengthening platform accountability can help reduce violence before the entire situation becomes unmanageable.
References:
Booth, R. (2026, January 22). Grok AI generated about 3m sexualised images in 11 days, study finds. The Guardian. The Guardian. https://www.theguardian.com/technology/2026/jan/22/grok-ai-generated-millions-sexualised-images-in-month-research-says
Department for Science, Innovation & Technology. (2025, April 24). Online safety act: Explainer. GOV.UK. https://www.gov.uk/government/publications/online-safety-act-explainer/online-safety-act-explainer
Ingram, D. (2026, April 14). Musk’s AI chatbot Grok is still making sexual deepfakes. NBC News. https://www.nbcnews.com/tech/tech-news/musks-ai-chatbot-grok-xai-making-sexual-deepfakes-imagine-rcna265855
News, P. (2026, January 12). Malaysia and Indonesia become the first countries to block musk’s chatbot grok over sexualized AI images. PBS News. https://www.pbs.org/newshour/world/malaysia-and-indonesia-become-the-first-countries-to-block-musks-chatbot-grok-over-sexualized-ai-images
Quillen, S. (2026, January 5). EU flags ‘appalling’ child-like deepfakes generated by X’s grok AI. Al Jazeera. https://www.aljazeera.com/news/2026/1/5/eu-flags-appalling-child-like-deepfakes-generated-via-xs-grok-ai
Shapero, J. (2026, January 2). Musk’s AI chatbot Grok apologizes after generating sexualized image of young girls. WBTW. https://www.wbtw.com/news/national/musks-ai-chatbot-grok-apologizes-after-generating-sexualized-image-of-young-girls/
Wilson, J. (2026, January 8). Hundreds of nonconsensual AI images being created by grok on X, data shows. The Guardian. The Guardian. https://www.theguardian.com/technology/2026/jan/08/grok-x-nonconsensual-images?
Zeff, M., & Dave, P. (2026, May 20). SpaceX listed grok’s ‘spicy’ mode as a risk in its IPO filing. Wired. https://www.wired.com/story/spacex-ipo-grok-spicy-mode-risks/