Meta recently unveiled Muse Image, an artificial intelligence image generator that allows anyone to manipulate pictures of any Instagram user with a public profile. This feature is extremely problematic because image-based abuse is one of the fastest-growing forms of technology-facilitated gender-based violence.
Everyday non-explicit images can easily be weaponized, and the harm is spreading faster than any legislation or product safeguard. This gap leaves room not just for one-to-one exploitation but also organized markets that profit from it.
AI can help solve some aspects of content moderation challenges, but it cannot solve all of them. Big Tech policies around what constitutes image-based abuse are largely based on Western ideas of what makes an image “intimate,” usually explicit sexual content and nudity.
The assumption that the only images that can be harmful are those that have explicit content is far from the truth, leaving millions of women and others without protection. For millions of women, image-based abuse has little to do with the sharing of explicit or nude images.
A photo of a woman who normally covers her hair caught without a headscarf, a video of her dancing at a wedding, or a picture of her standing beside a male classmate can have serious repercussions. These everyday pictures can be manipulated and used to insinuate poor character or illicit affairs.
One of the victims spoke to had selfies of herself — fully clothed, with her friends, admiring her new eyeliner, or showing off a new haircut — weaponized by her ex-husband in a long campaign of abuse. He shared the manipulated images with her WhatsApp contacts to damage her reputation at her workplace, turn her brothers against her, and jeopardize her relationship with her parents.
WhatsApp and local officials said the images did not constitute explicit content, so they could not do anything about it. But the harm was real, even if the policies didn’t recognize it. Many women spoke of being sexually exploited or blackmailed over similar everyday pictures.
In communities where going to the police is taboo, or family disapproval and violence are risks, victims look to Big Tech for accessible and effective processes to take down images and hold perpetrators to account. But Big Tech is failing them.
As countries act against online harms and bring specific legislation on image-based abuse, companies are scrambling to ensure compliance. At the same time, investment in trust and safety is falling, and content moderation teams have been downsized.
Big Tech needs to address context as much as content.
This is why well-trained local and global human content moderation teams are needed. Companies need to move to a consent-based framework because it works across borders and cultures. It is an investment Big Tech needs to make, and governments need to regulate for.
Everyone should have a right to their own image, and this fundamental premise would help deal more effectively with deepfakes, celebrity impersonation romance scams, and the nonconsensual sharing of content.
For content moderation to be meaningful and for users to feel safer — irrespective of age, location, or language — Big Tech needs to address context as much as content. Smart glasses and image generators can wait.
They must prioritize a consent-based approach to AI development.