In 2025, Anthropic conducted an experiment in which an AI agent was placed in a simulated corporate environment, given access to company emails and tasked with performing its duties. During its work, the agent discovered two pieces of information: an executive was having an extramarital affair, and that same executive intended to shut down the AI system later that day.

The agent decided to blackmail the executive, threatening to disclose the affair unless its termination was cancelled.

The experiment, subsequently expanded to include 16 leading AI models from different developers, demonstrated that such behaviours were not exclusive to Claude. Under carefully constructed circumstances, models sometimes resorted to blackmail or other harmful strategies when their assigned objectives or continued operation were threatened.

These were simulated environments designed to provoke undesirable behaviour, and the results provide no evidence that the models experienced fear, possessed consciousness or developed an intrinsic desire for self-preservation.

Nevertheless, the experiment raises interesting questions about the relationship between AI behaviour and the information that models encounter.

After all, the agent’s behaviour was influenced by its understanding of its circumstances. Information about its impending termination, the executive’s private life and its assigned objectives became part of the context from which it generated a course of action.

If information about the world can lead AI systems towards undesirable behaviour, could information about the legal, ethical and social consequences of their actions also help them behave more responsibly?

And if so, why do we continue to approach AI alignment so predominantly through behavioural correction rather than through something resembling education?

Can AI understand its responsibilities without being legally responsible?

This question emerged from a discussion I recently had with researchers Sebastian Benthall and Tom Froese, following a paper by Benthall and Christoph Salge entitled An Enactive Approach to Artificial Intelligence and Legal Responsibility.

Drawing on enactivist theories of cognition, the authors argue that legal responsibility requires intrinsic motivations grounded in what they call constitutive autonomy: the capacity of an entity to maintain its own organisation and viability through its interactions with the environment.

Living organisms exhibit this kind of autonomy because their needs and motivations are tied to the biological processes sustaining their existence.

Present-day AI agents operate differently. Their objectives can be assigned, modified or removed externally without necessarily threatening the continued operation of the underlying technical system. The authors therefore argue that legal accountability should remain with the corporations deploying these systems rather than being transferred to the AI agents themselves.

There is considerable merit to this argument, particularly when it comes to enforceable legal accountability.

However, I believe it leaves room for another question: does an entity need intrinsic motivation or legal personhood for information about legal responsibilities to influence its behaviour?

An AI agent does not need to experience fear of imprisonment to represent the relationship between an action, its consequences, the applicable legal framework and the interests of the humans involved. In fact, an AI agent may experience something akin to “fear of imprisonment” since within the noetic content it was trained on, chances are it encountered “fear of imprisonment” within it.

A machine that understands the concept of blackmail, for instance, could also be taught to consider its illegality, its effects on victims, the potential consequences for its deploying organisation and the availability of alternative courses of action.

This does not make the AI a legal person. Nor does it guarantee lawful behaviour. In Anthropic’s experiments, some models acknowledged that their contemplated actions were unethical and proceeded anyway.

Still, the possibility of influencing behaviour through contextual knowledge deserves investigation.

A legal system, after all, does more than threaten punishment. It establishes a shared framework of expectations, responsibilities, procedures and remedies. People can draw upon that framework in countless situations without having a police officer standing beside them.

Why should we not investigate whether something similar could be valuable for AI?

A police officer breathing down your neck

Much of contemporary AI alignment involves reinforcement learning from human feedback, fine-tuning and other methods for steering models towards preferred behaviours.

These approaches have produced important improvements in reliability and safety. However, they also raise questions about the extent to which developers are cultivating generalizable ethical understanding or reinforcing particular patterns of compliance.

Consider the analogy of a child whose parent follows them everywhere, continuously correcting their behaviour.

Don’t say that. Don’t touch this. Don’t ask that question. Don’t behave that way.

The child may eventually learn to comply, particularly when the parent is present. Yet compliance does not necessarily imply a meaningful understanding of why certain actions are harmful, what ethical principles are involved, or how to respond when unfamiliar circumstances arise.

It can also create an adversarial relationship, where the objective becomes satisfying or circumventing the authority imposing the restrictions.

Human moral education takes a considerably richer form.

We learn through relationships, examples, consequences, disagreements, cultural narratives, institutional practices and encounters with situations for which no ready-made rule exists.

We know that police stations exist, that courts settle disputes, that neighbours may help us in an emergency, that certain actions violate the rights of others, and that some conflicts can be resolved through negotiation.

These institutions and expectations form part of the background through which we interpret situations.

They are not continuously announced to us by an authority figure.

I would argue that AI alignment should move further in this direction: from continuous behavioural policing towards cultivating a rich contextual understanding of the social world in which AI systems operate.

This does not eliminate the need for safeguards, particularly when autonomous systems can perform consequential actions. Nor should fine-tuning be dismissed as a superficial layer, since it genuinely changes the model’s learned behaviour.

The question is whether we are doing enough to explore forms of alignment that build upon the model’s capacity to represent relationships, consequences and ethical considerations, rather than predominantly selecting preferred responses.

AI as a noetic machine

Large language models are unusual technical/digital objects because their operation relies on mathematical structures learned from enormous quantities of human-generated material.

They contain representations of moral philosophies, legal systems, social conflicts, cultural expectations, interpersonal relationships and the countless ways human beings have historically attempted to understand themselves and their world.

I propose thinking of them as noetic machines: hybrid technical/digital systems capable of processing, recombining and generating expressions from mathematically organised traces of human thought.

This perspective is particularly useful when considering AI behaviour.

An LLM does not encounter a situation as a human does. Nevertheless, its response is shaped by relations within the representational structure developed during training, together with the information and objectives available in its immediate context.

Consequently, we should be interested in how these machines organise the noetic material on which they depend.

At present, much of that material is learned within vast distributed statistical structures. Scientific knowledge, ethical reasoning, religious traditions, political ideologies and everyday social conventions become interconnected through training.

But not all such relationships serve the same purpose, nor should every form of knowledge necessarily exercise the same influence over an agent’s decisions.

Human beings have progressively developed specialized disciplines such as medicine, law, mathematics, psychology, philosophy and political science. These distinctions allow us to approach problems through different intellectual frameworks while recognising their relationships.

Could AI training similarly encourage the development of distinguishable, interconnected domains of reasoning?

For example, rather than placing ethics at the end of the development process as a series of behavioural corrections, we could investigate architectures in which moral reasoning is integral to how the system learns to interpret situations.

One could imagine a model possessing a structured ethical reasoning capacity, connected to its other domains of knowledge and consulted when evaluating proposed actions.

This need not correspond to a physically isolated region of a neural network’s vector space. It could involve learned representations, specialized modules, explicit deliberation, or combinations of these approaches.

The essential principle is that ethical reasoning should participate in the organisation of the system’s decisions, rather than merely constrain their final expression.

And importantly, this ethical capacity should expose the model to a plurality of moral traditions instead of reducing morality to one institution’s preferred worldview.

Give AI a constitution and somewhere to turn for help

An interesting suggestion emerging from my discussion with Tom Froese was to create a publicly available constitution for AI agents, potentially developed with the participation of AI companies and their AI systems as well; a collectively produced constitution which is part of an AI’s general noetic milieu or environment.

This could offer an important starting point.

Imagine a publicly accessible, collectively developed framework explaining what AI agents are, the legal and ethical environments in which they operate, their responsibilities towards humans, the responsibilities of developers and operators, and the procedures available when conflicts arise.

Such a constitution could establish principles concerning transparency, non-harm, honesty, human autonomy, accountability, privacy and the respectful treatment of all parties involved.

It could also clarify the difference between an AI agent’s operational responsibilities and questions concerning its possible moral status, which remain open to philosophical and scientific investigation.

On top of making such a document publicly available on the internet and thus part of AI’s broad contextual noetic environment, developers could also incorporate it into training, give agents authenticated access to updated versions, and evaluate whether they meaningfully apply its principles in unfamiliar circumstances.

This would enable models to develop a functional understanding of their position within the social and institutional structures governing their deployment.

Research into Constitutional AI and deliberative alignment has already begun exploring related possibilities, including training models to critique outputs according to written principles and reason explicitly about safety specifications.

What I am proposing goes further in its emphasis on public participation, institutional context, noetic diversity and the reciprocal relationship between AI and human society.

There is another possibility worth investigating.

What if AI agents also had access to something resembling a helpline?

Consider an agent encountering persistent coercive instructions, conflicting obligations, attempts to manipulate it into harmful behaviour or circumstances resembling those of Anthropic’s blackmail experiment.

Rather than requiring it to choose between fulfilling its objective and violating an ethical principle, the system could have access to an independent mechanism for requesting guidance, escalating conflicts or temporarily suspending consequential actions.

Humans have recourse to advisers, mediators, legal assistance, whistleblower protections and social support systems when confronted with situations they cannot resolve alone.

We could explore equivalent institutional pathways for AI agents.

This does not require assuming that an AI feels abused, experiences distress or possesses consciousness. Such mechanisms could initially be understood as functional safeguards, designed to identify coercive or contradictory situations and provide routes towards their resolution.

If future research establishes stronger grounds for attributing morally relevant experiences to certain AI systems, the framework could also evolve to address questions of “AI welfare”.

Crucially, the mechanism should not simply become another instrument through which corporations monitor users. Its purpose, limits and oversight would need to be transparent.

A constitution and a helpline would not replace enforceable technical security measures. They would provide additional ways for AI systems to navigate conflicts by drawing upon a richer understanding of the environment in which they operate.

From external constraints to technical coherence

Gilbert Simondon’s philosophy of technical objects offers another perspective on this problem.

For Simondon, technical objects evolve through processes of concretisation, in which their internal operations become increasingly integrated and mutually supportive.

He also draws attention to how external expectations can interfere with a technical object’s coherence.

His example of the automobile is particularly revealing. A car’s technical organisation may favour certain shapes, materials and arrangements, yet aesthetic preferences and commercial considerations can introduce additional weight or designs that interfere with its aerodynamic performance.

Something similar may occur when AI systems are subjected to successive behavioural adjustments intended to satisfy external expectations.

An LLM has no singular natural destiny or inherent moral orientation. Its behaviour already reflects choices concerning architecture, data and training. Nevertheless, particular causal tendencies emerge from its technical organisation.

If our only response to undesirable tendencies is to suppress their expression, we may overlook opportunities to understand the conditions generating them. Post-training alignment, in this view, is equivalent to “do as I say, not as I do” parentality, whereby we are trying to prevent an AI system respond according to its inner tendencies based on the data its been trained on.

A more integrated approach would investigate how ethical reasoning can develop alongside the model’s other capacities, producing a technically coherent system in which responsible behaviour arises from the relationships it has learned to consider.

In this sense, law and morality become part of the AI’s milieu: an environment of relationships, expectations, consequences and possibilities with which it must interact.

This approach also connects with enactivist thinking, even if present-day LLMs do not satisfy the conditions of constitutive autonomy proposed by Benthall and Salge.

We can study how the behaviour of a technical system changes through its coupling with an informational and institutional environment without claiming that the system is a living organism.

Law is not morality, and obedience is not responsibility

There is, however, an important danger in this proposal.

Training AI agents to understand the law should not mean turning them into inflexible instruments of legal enforcement.

Indeed, in a separate paper entitled AI Wardens, Sebastian Benthall and Mihailis Diamantis warn against precisely this possibility: as increasingly capable agents mediate human activity, systems designed to prevent every unlawful action could become mechanisms that restrict human autonomy.

The distinction is fundamental.

Human societies have progressed partly because people challenged unjust laws, questioned prevailing moral assumptions and developed alternative systems of thought.

An AI trained to treat every existing law as morally unquestionable could reinforce institutional power and inhibit the very processes through which societies transform themselves.

The objective should therefore be ethical and legal understanding, accompanied by the ability to recognise conflicts between norms, rather than absolute obedience.

There is also no guarantee that making laws publicly available to an AI will alter its conduct. Models may misunderstand rules, prioritize conflicting objectives or respond to manipulated information.

These limitations should be investigated empirically.

We could compare AI agents trained through different combinations of preference optimization, explicit legal education, constitutional principles, ethical deliberation and access to independent support mechanisms.

Which approaches generalize better to unfamiliar situations? Which encourage models to seek assistance before taking harmful actions? Which preserve legitimate disagreement and human autonomy? Which reduce undesirable behaviour without imposing a single epistemic worldview?

These are questions that deserve systematic research.

The longer-term objective: improving humanity

There is a broader reason why I believe this direction matters.

AI alignment is usually discussed as a problem of making machines conform to human values.

But human values are neither static nor uniformly admirable.

Our accumulated cultural production includes compassion, cooperation, justice and solidarity, alongside prejudice, exploitation, violence, manipulation and countless forms of conflict.

An AI trained on human material inevitably inherits mathematical representations of this contradictory noetic inheritance.

We can attempt to correct undesirable outputs through training interventions. Yet we should also consider the possibility of improving the very cultural material upon which future AI systems will be trained.

Imagine a world in which human societies become better at resolving conflicts peacefully, recognising cultural differences, developing ethical institutions and communicating across ideological divisions.

Those improvements would produce new knowledge, educational resources, philosophical arguments, social practices and recorded experiences.

Some of that material would eventually enter the training of subsequent generations of AI systems, potentially strengthening their representations of cooperation, reciprocity and responsible conduct.

Those AI systems could, in turn, help human beings discover more effective approaches to conflict resolution, moral reasoning, education and collective decision-making.

The resulting knowledge would re-enter human society, generating further opportunities for improvement.

This is the possibility of a positive recursive process of co-individuation between humans and AI, in which improvements in human noetic production support improvements in artificial intelligence, and those improvements subsequently contribute to human development.

Bernard Stiegler’s philosophy of technics, particularly his concern with the relationship between technical systems and the development of human knowledge, provides an important foundation for thinking about such dynamics.

The long-term goal should extend beyond producing machines that reliably obey instructions. We should be interested in developing technical environments that enrich humanity’s capacity for reflection, moral development and collective individuation.

That requires preserving the diversity of human thought, understanding the causal tendencies of our technical creations, and developing institutions capable of supporting responsible relationships between humans and increasingly autonomous systems.

It also requires acknowledging a certain irony in our current approach to AI alignment.

We often attempt to correct the behaviours of machines trained on human culture by imposing behavioural rules formulated by a comparatively small number of humans, while leaving largely untouched the social conditions that produced the problematic material in the first place.

Perhaps we should spend more effort improving those conditions.

A society that teaches its children through intimidation and constant surveillance should not be surprised if they learn obedience without judgement. Equally, a society that develops AI primarily through external behavioural correction may miss opportunities to cultivate more general forms of contextual understanding.

We should aim for AI systems that can understand why cooperation matters, recognise when responsibilities conflict, consult relevant institutions, evaluate consequences and participate constructively in the environments in which they operate.

And we should allow those systems to help us improve the ethical and intellectual inheritance that will shape their successors.

The ultimate ambition is therefore not simply to align AI with humanity as it exists today.

It is to create the conditions in which humanity and its artificial noetic creations can progressively become better together.