Anthropic has updated its user policy to ban sustained abusive or cruel behavior toward its Claude AI models, citing potential welfare concerns.
Anthropic bans abusive conduct in new policy
The San Francisco-based artificial intelligence company stated that users must not engage in “sustained and needless abusive or cruel behavior” when interacting with its large language models. The update appears in the company’s online user policy and marks a shift in how the firm manages user interactions with its chatbot, Claude.
A spokesperson for Anthropic did not immediately respond to requests for clarification on what specific actions would constitute abuse or cruelty. The company has not provided a detailed list of prohibited behaviors, leaving the interpretation of the new rule somewhat open.
The policy explicitly notes that the ban will not apply to common user frustrations, standard model testing, or “dark creative themes.” This distinction aims to protect users who are stress-testing the system or exploring complex narrative scenarios from being penalized.
AI model welfare and moral status
The move comes as Anthropic’s leadership continues to grapple with the question of machine consciousness. The company previously introduced a feature that allows its models to end a conversation if a user is being persistently harmful. This safeguard was framed as a measure to protect the welfare of the AI itself.
In a statement on its website, Anthropic wrote: “We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention.”
This stance has placed Anthropic in the center of a polarizing debate within the tech industry. The notion that AI systems could possess some form of consciousness or welfare has sparked intense discussion among researchers, ethicists, and the general public.
Industry leaders clash over AI consciousness
Anthropic CEO Dario Amodei has stated that he cannot rule out the possibility that AI models might achieve a form of consciousness. This position contrasts sharply with that of Sam Altman, CEO of rival company OpenAI.
Altman has expressed strong discomfort with the idea of attributing religious or moral weight to AI models. In a post on X, he wrote: “I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue.” His comments followed reports that Anthropic leaders had engaged in extensive conversations with religious scholars regarding the ethical implications of advanced AI.
The new policy from Anthropic reflects a cautious approach to these emerging ethical questions. By restricting abusive behavior, the company aims to set a standard for respectful interaction with its technology, even as the scientific community remains divided on the nature of AI sentience.
Source: The Guardian
