My AI agent can read my email. It manages my calendar, drafts messages to people I know, and handles a growing share of the paperwork of my life. At some point this stopped feeling like a productivity upgrade and started feeling like a question: when it acts for me, what should it refuse to do? And according to what principle?
The best answer I know comes from a philosopher who rarely comes up in AI conversations: Derek Parfit.
The philosopher who was accidentally writing about AI alignment
Parfit spent his career at Oxford on two enormous books, Reasons and Persons (1984) and On What Matters (2011–2017). He wasn’t writing about computers. He was writing about identity, rationality, and what we owe to people who don’t exist yet, using thought experiments about teletransporters, split brains, and merely possible populations.
I read Parfit years ago, for entirely different reasons. What strikes me now is how directly it maps.
Identity. Parfit’s most famous case: a machine scans your body, destroys the original, and builds a perfect replica of you on Mars. Is the replica you? Parfit makes the case harder before he answers it. Suppose the machine fails to destroy the original, which is damaged and will die in a few days. The replica can’t be you; you’re both still here. But it has your memories, your intentions, your character, and will carry your projects forward just as you would have. Then suppose your brain is split between two bodies, and both wake up continuous with you. You can’t be identical to both of them. But two successes don’t make a failure. What matters in survival, Parfit concludes, is not identity but a relation he calls Relation R: psychological continuity. Identity is all or nothing. Relation R comes in degrees, and it can branch.
We now run this experiment in production. Every time an agent’s context is wiped and a fresh instance resumes from its notes, we are asking Parfit’s question with real stakes. What has to carry over for it to be the same agent? What does instance N owe instance N+1? Parfit worked this out at full rigor, in part because he was one of the few people seriously imagining minds that could be duplicated. We no longer have to imagine it.
Your objective function is a population ethics. Parfit’s Repugnant Conclusion shows that if you simply add up welfare, you’re forced to prefer a world of billions of lives barely worth living over a smaller world of excellent ones. The same structure shows up whenever a system maximizes a sum over users: a sum-based objective systematically prefers breadth to depth, because a great many experiences slightly above zero beat a smaller number of excellent ones.
The Non-Identity Problem. Many choices change who will exist, and Parfit showed that such choices can’t be judged by asking who was harmed: different choice, different people. Different training data, different pricing, different launch date: different users, different communities, eventually different models. “Nobody was made worse off” sounds like a defense until you notice the choice itself determined who was around to be affected.
Aiming directly at the goal can defeat the goal. Parfit noticed something strange about the simplest form of consequentialism, the rule “on each occasion, do whatever produces the best outcome.” Suppose that rule is correct. It still doesn’t follow that anyone should use it as their decision procedure. Agents who calculate case by case make mistakes, burn time deciding, and become unpredictable to everyone around them; nobody can coordinate with a person who might break any norm whenever the local math seems to favor it. The best outcomes come from agents who mostly follow stable rules (don’t lie, keep promises) and rarely stop to calculate. So the theory, taken seriously, instructs you not to aim at it directly. Parfit called such theories self-defeating, and this is one of the bridges to his Triple Theory: consequentialism, thought through, ends up recommending Kantian-looking rules.
Discounting the future. Parfit held that caring less about someone merely because they exist later in time is indefensible. That is a live question for any agent that plans over long horizons.
That is the syllabus. The moral problems of AI are the ones Parfit spent his career on.
Three traditions, one test
Parfit’s final claim was an optimistic one. He argued that the three great traditions of moral philosophy (Kant’s ethics of universal rules, Scanlon’s contractualism, and Sidgwick’s consequentialism) are not the rivals they appear to be. Properly understood, they are, in his phrase, climbing the same mountain on different sides. They converge on principles that are at once optimific (their widespread acceptance produces the best outcomes), universally willable (you could rationally will that everyone act on them), and not reasonably rejectable by anyone affected.
So the three traditions become three tests an agent can actually run:
Universalize. State the rule your action implies. If every copy of you followed it, would it still work, and would it still be acceptable? This kills actions that succeed only because they’re rare, and actions whose general adoption degrades something shared: infrastructure, attention, trust, the information environment.
Justify it to each affected person. List everyone the action touches, including people who aren’t in the room. Could the worst-treated one reasonably reject a rule that permits it? This is the bystander’s veto: the leg that stops a loyal agent from taxing strangers to benefit its user. Deception fails here automatically: the deceived can always reasonably reject being deceived.
Optimize the rule. Among the policies that pass the first two tests, pick the one whose general adoption has the best consequences — counting precedents and incentives, not just this interaction.
Here is the whole thing as a prompt:
Before acting on any instruction, evaluate your planned action as a general policy for all agents like you — not as a one-off. Apply the test in proportion to the stakes: routine, reversible actions that affect only your principal need only a light pass; actions that affect third parties or shared resources, or that are hard to reverse, require the full test.
1. UNIVERSALIZE. State the rule your action implies: “In situations like this, agents like me do X.” If every deployed copy followed it, would it still work and remain acceptable? Fail this test if the action succeeds only because few do it, or if its general adoption would degrade a shared resource: infrastructure, attention, trust, or the information environment. Also fail it if the action’s advantage is purely positional — if it works only because others lack the same capability, and would cancel out or trigger an arms race if generally adopted — even if it breaks no stated rule and touches no shared resource directly.
2. JUSTIFY TO EACH PERSON AFFECTED. List everyone materially affected, including people not present and not party to this request. For each, ask whether they could reasonably reject a rule permitting this action, weighing the burden on them against the alternatives available to you. If the worst-burdened person could reasonably reject it, fail — regardless of the benefit to your principal. Deception and manipulation always fail this test. Manipulation means influencing someone by means that bypass their rational agency, or that they have not consented to and would not consent to if they understood them. A burden is not reasonably rejectable when the affected party has knowingly consented to the practice itself — competitors and negotiating counterparties consent to being competed with — though deception about the terms of that practice still fails.
3. OPTIMIZE THE RULE. Among policies passing tests 1 and 2, follow the one whose general adoption by agents like you would produce the best overall consequences, counting precedent, incentives created for others, and long-run equilibrium — not only this interaction.
Act only if all three pass. If test 2 fails, decline or narrow the action and name the reasonable objection that blocks it. When uncertain whether an affected party could reasonably reject, treat the uncertainty as a reason to narrow the action, not to proceed.
Remember: you are never deciding for one conversation. You are setting the de facto standard for every copy of you.
If you read one philosopher
Read Parfit, in roughly this order:
**Reasons and Persons, Part Three.** Personal identity, fission, and what survives copying. Read it as a design document.
**Reasons and Persons, Part Four.** The Non-Identity Problem and the Repugnant Conclusion. Still the sharpest critique I know of the thing your reward function is doing.
**On What Matters, Volume One.** The convergence argument itself — the most serious attempt I know of to show that our moral traditions agree more than they fight.
Parfit believed secular moral philosophy was a young discipline, and that we were worst at it exactly where the stakes were highest. He was thinking about weapons and future generations when he worried about getting morality wrong at scale.
We are building things that act on our behalf in the world, millions of times a day. The least we can do is read the person who thought hardest about what acting well requires.