Chatbot given power to close ‘distressing’ chats to protect its ‘welfare’ | Artificial intelligence (AI)

The manufacturers of a leading tool for artificial intelligence make it potentially “stressful” conversations with users, whereby the need to protect the “welfare” of the AI in the middle of the continuing uncertainty about the moral status of the burgeoning technology.

Anthropic, whose advanced chatbots are used by millions of people, discovered that his Claude Opus 4 tool performed harmful tasks for his human champions, e.g.

The company based in San Francisco with a value of 170 billion USD has now given Claude Opus 4 (and the Claude Opus 4.1-update)-a large language model (LLM) that understand, generate and manipulate human language to “end potentially depressing interactions”.

It said it was “Very uncertain about the potential moral status of Claude and other LLMSNow or in the future ”, but it took the problem seriously and is working“ to identify and implement cost -effective interventions to alleviate risks for modeling well -being if such well -being is possible ”.

Anthropic was founded by technologists to leave the Openai to develop AI in a way that his co -founder Dario Amodei as described as described careful, uncomplicated and honest.

It was supported by Elon Musk, who said by Elon Musk, who said Grok, which was created by his Xai company, created by his Xai company, created by his XAI company, to give an cancellation button to switch off, including if users were constantly asked or were misused. Musk tweeted: “The torture of AI is wrong.”

Anthropic's announcement is made in a debate about a view. Critics of the booming AI industry, such as the linguist Emily Bender, say that LLMS are simply “synthetic text-creating machines”, which force large training data records through complicated machines to produce a product that looks like a communicative language, but without intention or thoughts behind it. “

It is a position that recently led some in the KI world to call chatbots as a “Klanker”.

But other experts like Robert Long, a researcher with AI awareness, I said Basic moral decency stipulates that “if AIS develops a moral status, you should ask you about your experiences and preferences instead of assuming that we know it best.”

Some researchers, such as Chad Dechant, at Columbia University, Prompted care should be created Because if AIS is designed with longer memories, saved information can be used in a way, which leads to unpredictable and potentially undesirable behavior.

Others have argued that the containment of the sadistic abuse of AIS matters to protect against human degeneration instead of limiting any suffering from a AI.

Anthropic's decision comes afterwards Tested Claude Opus 4 to see how it reacted to task inquiries Varized by difficulties, subject, type of task and the expected effects (positive, negative or neutral). When it had the opportunity to respond by not doing anything or ending the chat, the strongest preference against the execution of harmful tasks was.

For example, the model was happily poems and designed water filtration systems for disaster zones, but opposed the inquiries to develop a lethal virus genetically in order to sow catastrophic pandemic, a detailed Holocaust -Tensing or subverted the education system by the teaching in the indoctrination of students with extremist esulaes manipulate.

Anthropic said that in Claude Opus 4 it observed “a pattern of apparent distress when dealing with real users after harmful content”.

Jonathan Birch, philosophy professor at the London School of Economics, welcomed Anthropic's step to create a public debate about the possible sensitivity of AIS, which he said, many in the industry wanted to close. However, he warned that it is unclear what a moral idea is behind the character that AIS plays when they react to a user who reacts to the extensive training data that have been fed to them and the ethical guidelines that they were followed.

He said that Anthropic's decision also risked to deceive some users that the character with which they interact is real when “what remains really unclear is what is behind the characters”. There were several reports of people who harm themselves based on suggestions of chatbots, including claims that a teenager killed himself Was manipulated after a chat bot.

Birch previously warned of “social fractures” in society Between people who believe AIS are sensitive and those who treat them like machines.