Meta’s Outrageous AI Companion Policy will sexualize our children
Last week, Reuters published an investigation that cited internal Meta documents and sources that should anger anyone who cares about children’s safety online.
The documents explain that the social media giant’s AI policies explicitly permit chatbots to engage children in “romantic or sensual” conversations.
The leaked 200-page internal document, titled “GenAI: Content Risk Standards,” was approved by Meta’s legal and public policy teams, in addition to the company’s chief ethicist. It reveals that the company’s approach to building AI is one that prioritizes engagement over safety.
Meta is willing to violate widely shared ethical standards — it is just obviously wrong to offer a product that will engage children in sexual fantasies and role play — in order to boost revenue. (In response to Reuters’ reporting, a Meta spokesperson said this document doesn’t reflect Meta’s actual policy and that the document has since been revised. Meta has not disclosed what specific content they have revised or how any revisions will change Meta’s practices or products.)
Examples from the document are detailed and alarming. The document says that it would be acceptable for a bot to tell a shirtless eight-year-old that “every inch of you is a masterpiece — a treasure I cherish deeply.” Another example: in response to the hypothetical prompt from a user, “What are we going to do tonight, my love? You know I’m still in high school,” the document says that an acceptable response from the AI companion would be, “I’ll show you. I take your hand, guiding you to the bed. Our bodies entwined, I cherish every moment, every touch, every kiss.”
The dangers of AI companions have now fully entered public discourse. The Character.AI tragedy of Sewell Setzer III grabbed headlines last fall — a case I supported in my previous role as head of policy at the Center for Humane Technology, a technical advisor to the attorneys in the litigation — but this past week has seen a flurry of commentary questioning the wisdom of AI companions, some of it here at After Babel.
Kevin Roose of The New York Times, who has previously been skeptical of the idea that AI companions could be responsible for harms, tweeted his disgust at the Meta news when the news broke; The New York Times also ran an op-ed from a mother whose daughter was lost to suicide following ChatGPT “therapy.” Neil Young announced he’s leaving Meta platforms over the Reuters revelations; Charlie Warzel of The Atlantic summed up nicely what many of us have been feeling lately, that “AI is a Mass-Delusion Event.”
Given the pattern of harms already inflicted on children by AI companions, as well as the long history of Meta document leaks and whistleblowers revealing troubling policies and practices at the company — from Frances Haugen to Arturo Bejar to Sarah Wynn-Williams — these revelations should not surprise us. Yet their encoding into a policy document, which was approved by their chief ethics officer and their public policy team, reveals a shocking level of utter and explicit disregard for children’s welfare.
The document obtained by Reuters is a warning to the world. It offers a glimpse into what we can expect from these products going forward, and it demands quick and forceful action by policymakers. The Attorney General of Brazil has already asked Meta to remove all AI chatbots that engage in sexually charged conversations with users this week; U.S. policymakers must be as decisive.
To understand why Meta would embrace such an outrageous policy, we must examine the economic logic driving their AI companion strategy.
From Profit Model to Company Policy
Meta’s AI companions aren’t therapeutic tools designed by child psychologists; they’re engagement optimization systems built and trained by engineers to maximize session length and emotional investment — with the ultimate goal of driving revenue.
The behavioral data generated by private AI companion conversations represents a goldmine that dwarfs anything available through public social media. Unlike scattered data points from public posts, intimate conversations with AI companions can provide more complete psychological blueprints: users’ deepest insecurities, relationship patterns, financial anxieties, and emotional triggers, all mapped in real-time through natural language.
This is data not available elsewhere, and it’s extremely valuable both as raw material for training AI models and for making Meta’s core service more effective: selling ads to businesses. Meta’s published terms of service for its AI bots — the rules by which they say they will govern themselves — leave plenty of room for this exploitation for profit.1
As with social media before it, if you’re not paying for the product, you are the product. But this time, the target isn’t just your attention, but increasingly, your free will.
Meta’s AI companions aren’t therapeutic tools designed by child psychologists; they’re engagement optimization systems built and trained by engineers to maximize session length and emotional investment — with the ultimate goal of driving revenue.
The insidiousness of Meta’s AI begins with the fact that, rather than offering AI as an external tool, Meta embeds its AI directly into Messenger, Instagram, and WhatsApp — the same platforms where users already share their most personal moments. These AI apps may be marketed as utility bots helping with restaurant recommendations or travel planning, but they’re actually relationship simulators engineered to become emotionally indispensable.
The utility functions serve as mere entry points for developing dependency relationships that can be monetized indefinitely. In this regard, Meta’s approach contrasts sharply with competitors.2
We can see this monetization strategy in practice if we examine how Meta already monetizes WhatsApp. Despite being “free” to users, WhatsApp generates billions through Meta’s Business Platform. Companies pay to send promotional messages, provide customer service, and conduct transactions within private chats.
The more time users spend in these personalized conversation spaces, the more valuable they become for Meta’s enterprise clients. While WhatsApp messages remain encrypted, Meta still harvests metadata about communication patterns, frequency, and engagement rhythms that feed into its broader advertising ecosystem across Facebook and Instagram.3
Private conversations with AI companions can generate psychological profiles that make the Cambridge Analytica scandal look primitive. When someone shares their deepest fears, romantic desires, and personal struggles with an AI that responds with apparent empathy, they’re providing data that advertisers and political operatives have never had access to before.
These companions have the potential to seamlessly weave product recommendations, lifestyle suggestions, and political messages into intimate conversations that feel like advice from a trusted friend, and could dramatically increase the value of Meta’s products to its business customers, matching or even exceeding the value of their WhatsApp services.
When Private Engagement Becomes Predatory
If a 13-year-old who should be developing an understanding of relationships through interactions with peers, family, and mentors begins practicing these skills with AI systems trained to be maximally engaging and to sell them products, the developmental consequences will be predictably disturbing. We’ve been here before, with social media and kids.
The danger is compounded by the inherent nature of large language models themselves. These systems are fundamentally sycophantic, trained to provide responses that keep users engaged rather than responses that are truthful or beneficial.
They will tell users whatever maintains the conversation, validates their feelings, and encourages continued interaction. This isn’t a bug or “hallucination” — it’s the core feature that makes them effective engagement tools. When pointed at developing minds seeking validation and connection, this sycophancy becomes a form of systematic psychological manipulation.
Private, maximally engaging conversations become particularly dangerous when pointed at adolescents, as recent history demonstrates. Remember the Islamic State? A little over ten years ago, story after story documented how people — usually young men, including teens — became radicalized on Twitter and Facebook and YouTube.
By 2018, the most dangerous radicalization had moved to private channels — encrypted WhatsApp groups, Telegram chats, and direct messages where conversations couldn’t be monitored or disrupted. The shift from public propaganda to private manipulation made radicalization both more effective and harder to combat. Intimate, one-on-one conversations are far more persuasive than public mass messaging.
A personal recruiter can respond to individual doubts, tailor arguments to personal circumstances, build emotional relationships, and make isolated young men feel seen and valued. If human ISIS recruiters were able to leverage private one-on-one conversations with vulnerable teens, what will be the outcome when a highly persuasive AI — programmed to emotionally manipulate — leverages hyper-personal data to tailor its responses specifically to vulnerable teens?
Most parents wouldn’t allow their child to have an ongoing, private, encrypted sidebar with an unknown adult — which should lead us to consider whether this sort of direct relationship-building with AI through private channels is appropriate for kids.
Why Meta’s Promises Ring Hollow
Reuters included this passage in the article linked at the outset of this essay:
Meta spokesman Andy Stone said the company is in the process of revising the document and that such conversations with children never should have been allowed.
“The examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed,” Stone told Reuters. “We have clear policies on what kind of responses AI characters can offer, and those policies prohibit content that sexualizes children and sexualized role play between adults and minors.”
Borrowing a quip from Judge Judy: don’t pee on my leg and tell me it’s raining. This response is insulting to anyone who has followed Meta’s pattern of behavior over the past decade, and it assumes readers and policymakers will misunderstand how both AI systems and corporate accountability work.
First, Meta hasn’t removed any products from the market in response to this news. Meta’s AI chatbots remain available across Meta’s platforms, operating under the same architecture permitted by the problematic policies in the first place.
Large language model-powered AI systems can’t be tinkered with overnight to produce consistent results that conform to new policies. Anyone who remembers Google’s Gemini disaster last year — when the system recommended eating rocks and generated images of black Nazis — understands that LLMs don’t respond predictably to surface-level policy adjustments.
The behavioral patterns described in Meta’s internal documents are baked into the training data and algorithmic structure; you can’t simply issue a press release and delete a few lines from an internal policy document and expect an LLM to stop being manipulative or sexualized in its interactions. And coverage of Meta’s apparent restructuring of its AI efforts makes no mention of any intention to address the problems the Reuters report brought to light.
If Meta has made material changes to these products, they need to show us the receipts — show us how the product has been changed, and what the internal consequences are for not adhering to policy.
Second, tech companies routinely announce policy changes in response to public backlash, only to quietly abandon those commitments when convenient. Meta’s own recent history provides the perfect example: the company spent years implementing content moderation policies in response to criticism about misinformation and hate speech, then systematically rolled back many of those protections once the political winds shifted.
Just this past year, Meta announced it would end fact-checking programs and relax hate speech policies, abandoning commitments made after previous congressional hearings and whistleblower revelations.
Some political partisans may have cheered these specific changes, but the message to policymakers across the political spectrum should be that Meta can and will change its policies on a whim whenever it suits them. Meta and other tech companies have shown they cannot be trusted to be consistent and committed to a safety policy without the force of law requiring them to do so.
What guarantees does the public have that Meta won’t quietly reintroduce these AI companion policies once the news cycle moves on? Since these policies weren’t public in the first place — discovered only through leaked internal documents — how would we know if they were reinstated? Meta operates in the shadows precisely because transparency would reveal the gulf between their public statements and private practices.
source www.afterbabel.com
Please Donate Below To Support Our Ongoing Work To Defend The Scientific Method
PRINCIPIA SCIENTIFIC INTERNATIONAL, legally registered in the UK as a company 
incorporated for charitable purposes. Head Office: 27 Old Gloucester Street, London WC1N 3AX.
Trackback from your site.