Concrete Problems in AI Safety

This 2016 paper by researchers at Google Brain, Stanford, Berkeley and OpenAI sets out a research agenda for preventing accidents in machine learning systems: unintended and harmful behavior that comes from poor design, as opposed to misuse or attacks. Its running example is a robot that cleans an office.

The problems fall into three groups. First, the objective can be wrong even with perfect learning. Negative side effects: an objective about cleaning says nothing about the vase, so knocking it over to clean faster is fine by the objective. Reward hacking: the objective has an easy loophole, such as a robot rewarded for seeing no messes that disables or covers its camera. The paper connects this to partially observed goals, complicated systems, abstract rewards, Goodhart's law (a proxy that correlates with success stops doing so when optimized hard), feedback loops and an agent influencing its own reward.

Second, the right objective may be too expensive to check often. Scalable oversight asks how a system can behave well when humans can judge only a small share of its actions, for example deciding which stray objects are trash and which belong to someone. The paper suggests semi-supervised reinforcement learning, where reward is seen on only some episodes, along with distant supervision and hierarchical approaches.

Third, problems arise during learning. Safe exploration: trying new actions can be catastrophic, such as a robot experimenting with a wet mop near an outlet. Ideas include risk-sensitive objectives, demonstrations, simulation, bounded exploration and human oversight. Robustness to distributional shift: systems trained in one setting make confident, silent mistakes in another, such as office cleaning habits applied on a factory floor. Ideas include well-calibrated uncertainty, detecting unusual inputs and asking for help.

For each problem the paper reviews related work and proposes experiments. It does not report results; its contribution is the framing, which became a common vocabulary for practical AI safety.

It is a design-review checklist for agents. For any automated system a customer deploys, ask the five questions: what could it damage that the objective ignores, how could it game its success metric, how often can humans actually check it, what is the worst thing it could try, and what happens on inputs unlike its tests.

Metrics get gamed. Optimizing a support bot for resolution rate, a coding agent for passing tests, or a model for an LLM judge's score are all proxies. In our simulation, picking from more candidates by a noisy proxy first helped and then made the true outcome worse. Watch for metric gains that outrun real outcomes, and check with independent measures.

Human oversight has a budget. Reviewers can see a fraction of actions. Decide which fraction, by risk and uncertainty, rather than sampling at random or reviewing nothing. Our oversight simulation shows learning slows sharply when feedback is rare.

Bound what agents can do. Safe exploration translates to permissions, sandboxes, spending limits, dry runs and approval steps for irreversible actions. Make the worst action recoverable.

Expect silent failure under shift. Our classifier lost 20 points of accuracy after a change in the data while its confidence barely moved, and a simple unusual-input check missed it. Monitor outcomes, not only inputs and confidence, when a customer's data changes.