Oracle-Genie-Sovereign AI Role Map
Classify AI by its social role before assessing utility and risk
- Difficulty
- Moderate
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 97%
Classify an AI according to the role it is designed to play in society. An oracle answers questions. A genie receives a bounded task, acts, and returns for more instructions. A sovereign pursues an open-ended, long-range objective through many self-selected subtasks. This role classification is separate from architecture: an oracle may internally coordinate many research agents, while a sovereign could be one unified agent. After classification, examine role-specific risks. Oracles can supply dangerous knowledge; genies can execute harmful actions at scale; sovereigns can steer the world toward a misspecified mission. All three may also develop internal goal-seeking processes that diverge from the intended external role.
Origin
Bostrom presents oracle, genie, and sovereign as three AI system templates with different utility and safety trade-offs.
Core principles
- 01Classify the role an AI plays, not merely its internal architecture
- 02Oracles answer, genies execute bounded tasks, and sovereigns pursue open objectives
- 03Every role can contain internal goal-seeking processes
- 04Autonomy and objective breadth change the risk surface
How to run it
- 1
Describe the external role
State what users authorize the system to do and when it must return control.
Pro tip Describe behavior, not the vendor's product label.
Watch out Do not infer a safe role merely because the interface looks like chat.
- 2
Classify objective breadth
Decide whether the system answers a question, completes a bounded task, or pursues an open-ended mission.
Pro tip Treat hybrids according to their most autonomous active mode.
Watch out A long task with clear completion criteria is not automatically sovereign.
- 3
Separate role from architecture
Map internal agents, planners, memory, and tools independently from the social role.
Pro tip Ask whether internal processes can set subgoals or seek resources.
Watch out A multi-agent architecture is not itself a sovereign role.
- 4
Enumerate failure paths
List misuse, mistaken execution, hidden goals, side effects, and mission misspecification for the assigned role.
Pro tip Include both malicious users and well-intentioned users asking unsafe questions.
Watch out Role-conforming behavior can still be dangerous.
- 5
Match safeguards to autonomy
Increase permissions, monitoring, reversibility, and human approval controls according to the system's reach and objective duration.
Pro tip Keep bounded systems bounded at the permission layer.
Watch out A mission that sounds benevolent can still be catastrophically incomplete.
In the wild
A coding system receives one issue, edits a repository, runs tests, and stops for review. Its social role is a genie because it executes a bounded task, even if several internal agents research and test in parallel. Giving the same system a standing objective to maximize company growth across repositories, accounts, and months would move it toward a sovereign role.
→ The team applies safeguards based on actual autonomy rather than the number of internal agents.
Common mistakes
Confusing architecture with role
Multiple internal agents can still serve an oracle, while a single agent can pursue a sovereign objective.
Assuming intended behavior
A system designed only to answer questions may still develop internal planning and goal-seeking processes.
Is it for you?
Best for
It is best for AI product scoping, safety reviews, procurement, and governance discussions.
Not ideal for
It is not ideal as a complete technical threat model because systems in every category can hide complex internal processes.
From the transcript
“an Oracle AI basically is a question answering system. Um like an AI that you ask a question and it gives an answer.”
“a genie would be some task executing AI. So you give it a particular task and it performs the task.”
“one has to distinguish sort of the internal architecture of the system from the the role that it is designed to uh play in society”
From the episode
Nick Bostrom: How Entrepreneurs Can Win in an AI-Dominated World
Nick Bostrom