How did Meta allow ads for deepfake porn apps?

Meta platforms recently ran ads for Kromix, an AI-powered app that let users create deepfake videos of female politicians and other women - and it's raising some serious red flags. These ads, targeted at male users, featured a voice-over touting the app's "no restrictions" and a tagline that promised to "Unleash Next-gen AI Magic." But here's the thing: one of these ads actually included a pornographic video with a deepfake that closely resembled a US politician. It's no wonder this incident has sparked concerns about Meta's ability to regulate harmful content on its platforms, especially when it comes to nonconsensual intimate imagery.

The Kromix app, which was hosted by Apple's App Store, was pretty brazen - it invited paid users to upload photos so that it could use them in some pretty disturbing scenarios, including "bedroom rape" and "Disney love." And if that's not enough, the app also featured AI-generated pornographic videos of women wearing Spider-Man costumes. Apple's had a tough time keeping similar tools off the App Store, and the company's faced criticism for its handling of nonconsensual intimate imagery. The ads were eventually removed after WIRED asked for comment, but the incident has raised some big questions about how Meta will deal with similar content in the future.

The use of deepfake technology to target female politicians and other women is a growing concern - and it's not hard to see why. For years, politicians and elected officials (who are overwhelmingly women) have been targeted with nonconsensual deepfake images and videos. The thing is, the production and distribution of AI-generated content have become so democratized that anyone with an internet connection can create and share harmful content. Meta's ad review system, which relies mostly on automated tools, failed to catch the ads for the Kromix app - and that's a pretty big problem. It highlights the need for more effective regulation of harmful content on social media platforms. So what can you do? Well, for starters, you can report suspicious ads and advocate for stronger policies against nonconsensual intimate imagery.

Source: Ars Technica