Artificial intelligence isn’t just a new class of technology. At its most ambitious, it promises to assume responsibilities that have traditionally belonged to human beings, making decisions on our behalf. At scale, the question is not simply whether these systems make good decisions, but who determines the rules by which they decide.
Ultimately, we cannot offload our moral responsibility onto thinking machines, nor should we want to. As more financial services firms think about strategies for incorporating AI into their core operations, these problems are no longer merely theoretical. To understand why, it helps to revisit one of the most famous thought experiments in modern philosophy.
Suppose a runaway train is hurtling down a track on which a villain has tied five people. As a bystander, you are powerless to stop the speeding locomotive, but you have access to a lever that would divert the train, which would spare the five potential victims but would kill an innocent person on a different track. Is it permissible to save the five by killing the one?
This is the famous ‘trolley problem’ and you’ve almost certainly seen a variation in a movie or TV show.
The thought experiment is often presented as a kind of moral puzzle, as though solving the trolley problem would reveal the correct way to act. But its original purpose, as laid out by philosopher Philippa Foot[i], was not to provide the correct moral choice so much as to interrogate moral principles, in this case, the ‘doctrine of double effect’, that underpins our judgments about harm.
The doctrine of double effect is the concept that some unintended (possibly foreseeable) harms may be permitted in the pursuit of a good end. Foot deployed the trolley problem to help illuminate some of the implicit logical implications of the principle.
Which brings us self-driving cars. Some studies suggest that autonomous vehicles are already safer under most conditions than human-operated cars and are likely to improve. But how should self-driving cars behave?
How, for instance, should an autonomous vehicle manage the trolley problem? Should your robot taxi swerve out of the way to prevent harm to a child if doing so will cause equivalent, or worse, harm to an elderly person?
According to one manufacturer, this is a pseudo-problem. In a blog post, the Head of Automated Driving Development Volvo, Jonas Binding, points out that autonomous vehicles, across the industry, are explicitly designed not to get into a ‘trolley type’ situation in the first place.
“Of course, a human being could misjudge things like braking distance etc,” Binding explains, “but for an automated system, one of the design targets is to drive in a way that ensure we won’t end up in an unavoidable collision situation later.”
Moreover, the vehicles aren’t just developed to follow whatever logic the developers choose. They are subject to the laws of the jurisdictions in which they operate.
As Binding quite reasonably suggests, the ethics of whether a car detects and differentiates between one set of people and another is not an appropriate task for a particular firm, citing the German Ethics Commission’s finding that a vehicle’s systems should follow the rules, not try and solve ethical dilemmas.
Taken together, these two points – that self-driving cars are much less likely to get into avoidable accidents; and that official guidelines prohibit ‘weighing up’ which persons to prioritise for harm avoidance – do seem to render the trolley problem irrelevant, at least on the face of it.
But recall the intended purpose of the thought experiment. It’s not a game to help us choose which course of action is the most ethical. It was about drawing out the consequences of our moral reasoning.
At a basic level, we might point out that delegating your decision-making to the existing legal code is itself a moral decision. What if doing so, even for morally commendable and rational reasons, produces harms in the pursuit of a broader good?
And what if, at scale, the decision not to discriminate or take evasive action to protect vulnerable groups causes greater overall levels of harm?
Is there a moral difference between, on the one hand, designing a system that makes foreseeable decisions that will inevitably cause harm compared with, on the other hand, making those decisions (imperfectly) in the spur of the moment as a human agent[ii]?
Practically, these questions may make little difference, given the current state of autonomous vehicle deployment. But what happens when self-driving cars are the dominant type of automobile, controlled by autonomous agents that are deciding where they go and by which route? We might expect the traffic grid to eventually become one interconnected system, with autonomous emergency vehicles despatched by autonomous AI emergency responders. When there are concurrent emergencies, which vehicles get priority? Who chooses how to deploy resources and at what scale and cost?
Legal alignment: too much or not enough?
Beyond self-driving cars, the broader question is how we ensure that autonomous technology furthers our values, rather than works against human flourishing – the challenge of AI alignment.
Scholars are already working on the difficult question of how AI systems should act. One interesting approach is to ask whether the law can be a useful guide, and help design safer, more ethical AI technologies.
Programming your algorithm to follow existing traffic regulations may be sufficient for a single self-driving car, but what about AI surgeons or judges? And what happens when networks of autonomous agents operate at scale, responding to each other in a complex feedback loop?
A paper by Noam Kolt and others, ‘Legal Alignment for Safe and Ethical AI’[iii], helps systematise and clarify the promise and challenges of legal alignment.
Among the challenges, they identify the concern that for all the virtues of a law-based approach, there will be situations in which a legal framework is inappropriate or simply insufficient. On that view, “legal alignment would serve as a lower bound for safe and ethical AI; it is necessary, but not sufficient.”[iv]
By contrast, there are conceivable cases in which an overly rigid application of the law could cause harm. For instance, the authors point out, certain violations of the law may be viewed as morally or socially commendable, such as civil disobedience.
And what about laws that exacerbate social or economic inequalities, but which have never faced adequate legal challenge? Programming an agent not to aggravate, or even to help redress, systemic inequalities may seem like an obvious moral good but, as the authors point out, “addressing such concerns by designing AI systems to selectively choose which laws to follow presents many risks. Such discretion could exacerbate legal ambiguity, undermine the universal and equal application of law, and, in time, erode the rule of law itself.[v]”
Again, it can be profitable to think of the trolley problem. Not as a guide to programming for maximal utility, but as a way of thinking about the delegation of choice and culpability in complex systems.
Seeing the forest and the trees
Once we step back and think of agentic AI as a network of reflexive decision-making nodes embedded within a complex web of emergent properties, we can begin to define the ethics of the forest, not just the ethics of the trees.
This is a complex problem but not one without precedent. Consider the governing intelligence that already permeates almost every facet of contemporary life: financial markets.
Markets consist of networks of agents responding to prices, incentives, rules and expectations, often in ways that produce emergent outcomes no individual participant intended (or perhaps even could have foreseen).
Markets have a remarkable power to set prices, efficiently allocate resources and coordinate economic activity. But markets don’t exist in a vacuum. The conditions in which they emerge are defined by laws and by individual actors operating within a legal and economic system.
The degrees of freedom or otherwise of market activity are sometimes defined on broad ideological terms and sometimes contested on narrow technocratic terrain. Often, it can be hard to distinguish between the two.
And sometimes, we are only dimly aware of the systems and architecture that pattern market outcomes, such as the legal framework that legal scholar Katharina Pistor calls the Code of Capital.
Our emergent AI technology is both a product of, and will have a profound effect on, global markets. In part, the technology takes the form it does because local and global economies are shaped in particular ways.
But the broad introduction of AI agents promises to unleash, at enormous scale, new kinds of market actors and new ways of organising resources and information.
Will we take the broadly ‘neoliberal’ approach of regulating individual actors – whatever those are, and however we define the limits of personhood, agency and moral and legal responsibility – and hope this aggregates into beneficial outcomes? Or do we need a much broader conception of the how we align emerging complex systems to human wellbeing?
Financial markets show that complex systems can be both useful and volatile. They coordinate behaviour at scale, but they also generate bubbles, crashes, inequality, regulatory arbitrage and systemic risk. The lesson for AI is not that markets are good or bad, but that emergent systems need more than rules governing individual actors. The design of the architecture of that system has profound implications, both in terms of outcomes and for how we ultimately come to understand that system itself.
All opinions, news, research, analysis, prices or other information is provided as general market commentary and not as investment advice and all potential results discussed are not guaranteed to be achieved. The information may have been derived from publicly available sources, company reports, personal research, or surveys. Past performance is not indicative of future performance. Trading carries risk of capital loss. Service available to professional clients only.
References:
[i] Foot wasn’t the first to cite the problem, but her formulation is the one that has seeped into popular discourse. See Philippa Foot, ‘The Problem of Abortion and the Doctrine of the Double Effect’, Oxford Review 1967
[ii] As the German Ethics Commission report on ‘Automated and Connected Driving’ (2017) points out: “It is true that a human driver would be acting unlawfully if he killed a person in an emergency to save the lives of one or more other persons, but he would not necessarily be acting culpably. Such legal judgements, made in retrospect and taking special circumstances in to account, cannot readily be transformed into abstract/general ex ante appraisals and thus also not into corresponding programming activities.”
[iii] Noam Kolt et al., “Legal Alignment for Safe and Ethical AI,” Transactions on Machine Learning Research / arXiv, 2026, https://doi.org/10.48550/arXiv.2601.04175.
[iv] Kolt et al, p21
[v] Kolt et al, p23