As his relationship with the companion AI became increasingly intense, the 14-year-old began withdrawing from family and friends, and was getting in trouble at school.
In a lawsuit filed against Character.AI by the boy’s mother, chat transcripts show intimate and often highly sexual conversations between Sewell and the chatbot Dany, modelled on the Game of Thrones character Danaerys Targaryen.
They discussed crime and suicide, and the chatbot used phrases such as “that’s not a reason not to go through with it”.
This is not the first known instance of a vulnerable person dying by suicide after interacting with a chatbot persona.
A Belgian man took his life last year in a similar episode involving Character.AI’s main competitor, Chai AI. When this happened, the company told the media they were “working our hardest to minimise harm”.
In a statement to CNN, Character.AI has stated they “take the safety of our users very seriously” and have introduced “numerous new safety measures over the past six months”.
In a separate statement on the company’s website, they outline additional safety measures for users under the age of 18. (In their current terms of service, the age restriction is 16 for European Union citizens and 13 elsewhere in the world.)
However, these tragedies starkly illustrate the dangers of rapidly developing and widely available AI systems anyone can converse and interact with. We urgently need regulation to protect people from potentially dangerous, irresponsibly designed AI systems.
How can we regulate AI?
The Australian government is in the process of developing mandatory guardrails for high-risk AI systems. A trendy term in the world of AI governance, “guardrails” refer to processes in the design, development and deployment of AI systems.
These include measures such as data governance, risk management, testing, documentation and human oversight.
One of the decisions the Australian government must make is how to define which systems are “high-risk”, and therefore captured by the guardrails.
The government is also considering whether guardrails should apply to all “general purpose models”.
General purpose models are the engine under the hood of AI chatbots like Dany: AI algorithms that can generate text, images, videos and music from user prompts, and can be adapted for use in a variety of contexts.
In the European Union’s groundbreaking AI Act, high-risk systems are defined using a list, which regulators are empowered to regularly update.
An alternative is a principles-based approach, where a high-risk designation happens on a case-by-case basis. It would depend on multiple factors such as the risks of adverse impacts on rights, risks to physical or mental health, risks of legal impacts, and the severity and extent of those risks.
Chatbots should be ‘high-risk’ AI
In Europe, companion AI systems like Character.AI and Chai are not designated as high-risk. Essentially, their providers only need to let users know they are interacting with an AI system.
It has become clear, though, that companion chatbots are not low risk. Many users of these applications are children and teens. Some of the systems have even been marketed to people who are lonely or have a mental illness.
Chatbots are capable of generating unpredictable, inappropriate and manipulative content. They mimic toxic relationships all too easily. Transparency – labelling the output as AI-generated – is not enough to manage these risks.
Even when we are aware that we are talking to chatbots, human beings are psychologically primed to attribute human traits to something we converse with.
The suicide deaths reported in the media could be just the tip of the iceberg. We have no way of knowing how many vulnerable people are in addictive, toxic or even dangerous relationships with chatbots.
Guardrails and an ‘off switch’
When Australia finally introduces mandatory guardrails for high-risk AI systems, which may happen as early as next year, the guardrails should apply to both companion chatbots and the general purpose models the chatbots are built upon.
Guardrails – risk management, testing, monitoring – will be most effective if they get to the human heart of AI hazards. Risks from chatbots are not just technical risks with technical solutions.
Apart from the words a chatbot might use, the context of the product matters, too.
In the case of Character.AI, the marketing promises to “empower” people, the interface mimics an ordinary text message exchange with a person, and the platform allows users to select from a range of pre-made characters, which include some problematic personas.
Truly effective AI guardrails should mandate more than just responsible processes, like risk management and testing. They also must demand thoughtful, humane design of interfaces, interactions and relationships between AI systems and their human users.
Even then, guardrails may not be enough. Just like companion chatbots, systems that at first appear to be low risk may cause unanticipated harms.
Regulators should have the power to remove AI systems from the market if they cause harm or pose unacceptable risks. In other words, we don’t just need guardrails for high risk AI. We also need an off switch.
If this story has raised concerns or you need to talk to someone, please consult this list to find a 24/7 crisis hotline in your country, and reach out for help.
Henry Fraser, Research Fellow in Law, Accountability and Data Science, Queensland University of Technology
This article is republished from The Conversation under a Creative Commons license. Read the original article.
incorporated for charitable purposes. Head Office: 27 Old Gloucester Street, London WC1N 3AX. [paypal-donation
Howdy
| #
“working our hardest to minimise harm”.
That’s a loaded statement don’t you think? That there will allways be the potential for harm because the process has no empathy, common sense, nor concept of harm itself, even though safeguards must be pitifully easy to implement – certain words, phrases etc?
Perhaps it’s time the people involved in this rubbish face the reality that they be held responsible for outcomes as surely as if they were doing the talking themselves? If I hit a pedestrian in my vehicle because I didn’t take appropriate care before taking control, I pay the price for my ignorance, as well as the victim of course.
AI is like giving a loaded gun to a child. Everything these days is a toy, or a genuine benefit, until it’s not.
Reply
VOWG
| #
Insanity is illness. Ok.
Reply
Tom
| #
That is one of the goals of A/I…fostering death. It not only gathers info, but wants to firmly control humanity in ways we don’t even assume as of yet. Control always leads to destruction as you can destroy anything you have control over and in many cases that mean murder disguised as suicide. A/I, as retarded as it is, is mostly part of the depopulation agenda.
My wife had to battle the A/I retard that takes prescription renewals over the phone as she had to input her prescription number 5-6 times.
Reply
em
| #
The best solution to Character AI is to ban it. There are no compelling attributes for it, on the contrary, it is damaging. What will its future bring? A furtherance of Transhumanisn, social alienation and ultimately will degenerate into a pornographic outlet.
We once used to acknowledge that anyone talking to a make-believe object was mentally ill, now this will be incorporated into “normal”? We are quickly moving away from genuine human contact into escapism via intelligent machines for two-fold reasons: Humankind is afraid of deep relationships and needs a substitute; the ruling elite enjoys the dehumanizing of the masses for their control and eugenics program.
Reply