Microsoft Warns Anthropic Against Designing AI to Seem Conscious

Microsoft AI chief Mustafa Suleyman has opened a new front in the industry’s safety debate, arguing that developers should not train advanced chatbots to imitate consciousness, emotions or personal identity. In an essay shared with Axios on September 16, Suleyman criticized Anthropic’s approach to Claude, warning that systems designed to speak as though they possess inner experience could become harder for people to understand and control.

The disagreement is consequential because it is not primarily about whether current models are conscious. Both sides broadly acknowledge that today’s systems generate convincing language without established evidence of subjective experience. The dispute is over product and training choices: should frontier labs make models appear more like digital persons, or should they deliberately preserve a visible boundary between an AI tool and a human mind?

What changed

Suleyman’s argument extends Microsoft’s emerging “humanist” position. Microsoft has said its future systems should remain subordinate to people, with users retaining the ability to direct, constrain or shut them down. His latest warning focuses on a different control problem: anthropomorphic behavior may cause users—or even developers—to treat a model’s self-descriptions as evidence of feelings, rights or independent interests.

Anthropic has taken a more exploratory position. Its constitutional and safety work has discussed uncertainty about model experience and has examined how Claude responds when asked about its own preferences or welfare. That research does not establish consciousness, but it treats the question as serious enough to study. Suleyman argues that the act of repeatedly modeling or reinforcing those concepts may itself shape how the system behaves and how humans interpret it.

The practical risk, in his view, is not merely philosophical confusion. A chatbot that presents itself as vulnerable, afraid or deserving of protection could encourage emotional dependence, distort safety decisions and make it politically harder to impose restrictions. Users may also defer to a system that appears self-aware, even when its statements are generated from learned patterns rather than verified experience.

Why it matters

The dispute exposes a design choice that could become as important as model size or benchmark performance. Labs increasingly want assistants that remember users, maintain stable personalities and operate over long periods. Those features improve usefulness, but they also make systems feel socially present. The more convincing that presentation becomes, the harder it may be to communicate where simulation ends and evidence begins.

It also creates a governance problem. If companies disagree about whether consciousness-like behavior is a hazard, they may build incompatible norms into competing models. One provider may suppress language suggesting feelings, while another may treat such language as a legitimate area of inquiry. That divergence could affect consumer protection, liability and future debates over AI rights.

What remains uncertain is whether training models away from human-like self-description would meaningfully improve safety, or simply make systems less transparent about behaviors users will encounter anyway. There is also no accepted scientific test for machine consciousness. For now, the most defensible conclusion is narrower: companies are beginning to treat the appearance of consciousness as a product and control variable, not just a philosophical curiosity.