Objective and Error Trade-off Design
Define the outcome and price each error before optimizing the system.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 96%
This framework moves AI design upstream from algorithms to values. First define the outcome the system should serve rather than treating an available metric as the goal. Then name the major error types, such as granting a benefit incorrectly versus denying it incorrectly, and identify who carries each cost. Make the trade-off explicit, because an optimizer requires a concrete decision boundary even when society normally expresses the issue through broad principles. Test that boundary against representative and edge cases, then revise it with affected stakeholders. The output is an objective and error policy that engineers can implement and decision-makers can defend, rather than a technically efficient system whose hidden value choices emerge only after harm occurs.
Origin
Peter Norvig describes the shift across editions of Artificial Intelligence: A Modern Approach from algorithms, to data, and finally to the harder question of choosing the objective and its error trade-offs.
Core principles
- 01Better data and algorithms cannot rescue the wrong objective.
- 02Every imperfect decision system creates different kinds of errors.
- 03Error trade-offs encode values, not merely technical settings.
- 04Stakeholders must make the objective explicit before optimization begins.
How to run it
- 1
Define the desired outcome
Describe what success means in the real world, not merely which metric is easy to optimize.
Pro tip Write the outcome in language an affected non-engineer can challenge.
Watch out Do not let the available dataset silently choose the objective.
- 2
Map the mistakes
List the important false positives, false negatives, and their practical consequences.
Pro tip Use concrete cases to expose costs hidden by aggregate accuracy.
- 3
Identify who pays
Name the users, subjects, families, institutions, and wider groups affected by each error.
Watch out The operator's inconvenience is rarely the only consequence.
- 4
Set the trade-off
Choose and document how one error should be weighed against another, including the rationale.
Pro tip Record whose values informed the boundary.
Watch out A vague principle is insufficient when the deployed system needs an exact threshold.
- 5
Stress-test and revise
Run representative and edge cases through the policy, inspect unequal effects, and revise before deployment.
Pro tip Repeat this review when the population, stakes, or operating environment changes.
In the wild
Norvig uses parole assistance to show that a system will make both kinds of mistake: releasing someone who reoffends and detaining someone who would not. The design cannot be completed by improving accuracy alone; decision-makers must state how those errors should be traded off and whose welfare counts.
→ The hidden policy choice becomes explicit before it is encoded in the model.
A payments team lists the cost of blocking legitimate customers alongside the cost of missed fraud, identifies which customers are disproportionately blocked, and agrees on a review threshold before tuning its detector.
→ The team optimizes against an approved error policy rather than raw detection rate.
Common mistakes
Optimizing the available metric
A convenient proxy can become the objective without anyone deciding whether it represents the desired outcome.
Treating trade-offs as technical
The relative cost of errors reflects human values and cannot be delegated silently to engineers or model defaults.
Using accuracy as the whole answer
Aggregate accuracy conceals which mistakes occur and who bears them.
Is it for you?
Best for
Teams designing automated or AI-assisted decisions where different errors impose materially different costs.
Not ideal for
Simple deterministic tasks with one uncontested outcome and negligible error consequences.
From the transcript
“But really the key to future progress is neither of those. The key is deciding what is it that you want? What is it that…”
“They're going to make mistakes. And so the question you have to answer is what's the trade-off between those mistakes?”
“But we don't have a history of tools for saying how do we specify the right objective? What are the trade-offs?”
From the episode
Peter Norvig: Transforming AI Into the Ultimate Human Advantage
Peter Norvig