Software Engineering at Google

Fergus Henderson, a Google engineer since the mid-2000s and one of the original developers of its build tool, wrote this paper to describe how software is built at Google. It is a catalog, not a study: each section describes a practice and why it helps, drawing on his experience and on public sources such as the Site Reliability Engineering book.

Most code lives in one repository visible to every engineer. As of January 2015 it held 2 billion lines in 9 million source files, with 40,000 commits per working day. Almost everyone works at head rather than on long-lived branches. Every directory has at least two owners, and any change needs review by one of them plus a second engineer certified for "readability" in that language. Tests run before commit (presubmit) and continuously after; a rotating "build cop" keeps head green, and status lights make breakage visible.

The build system, Blaze (Bazel, in the references, is its open-source counterpart), builds from declarative BUILD files, distributes work across thousands of machines, and requires every step to be hermetic and deterministic, so results can be cached and shared between engineers. Five main languages are encouraged, with style guides, and services talk through Protocol Buffers.

Releases are frequent and automated: branch from the last green build, test, stage with mirrored production traffic, roll to canary servers, then out gradually. User-visible launches need approvals for legal, privacy, security and reliability. Outages get blameless post-mortems with action items. Most software is rewritten every few years. On the people side: 20% time for self-chosen projects, quarterly OKRs scored from 0 to 1 with a target average of 0.65, separate career ladders for engineers and managers, and peer-reviewed performance evaluation. The author admits that how projects get approved or cancelled is unclear even to him.

These practices are the default vocabulary of good engineering. Owners files, mandatory review, presubmit tests, green trunk, canary releases and blameless post-mortems show up at well-run customers and are what you recommend at the rest. Knowing where they came from and why they work makes your recommendations specific.

AI systems need the same discipline, adapted. Prompts, retrieval configs and evaluation sets are code: keep them in the repository, review them, run regression evaluations as presubmit checks, roll out behind canaries, and write post-mortems when a model change causes harm. The paper's release path maps almost one to one onto safe model rollouts.

Determinism is what makes caching and reproduction possible. Google scrubbed timestamps from build outputs so results could be cached. In our simulation, one non-deterministic step turned a single rebuilt action into 1,993. LLM pipelines have the same issue: pin model versions, seeds and prompts so results can be reproduced and cached.

Small changes get reviewed well. The review tool labeled changes over 300 lines "large" and worse. AI-generated code makes large changes cheap to produce and expensive to review; keep the norm of small, reviewable changes.

Scale changes what is needed. At 1,700 commits an hour, head would be broken most of the time without presubmit checks and fast rollback, per our model. A ten-person customer team does not need Google's apparatus; pick the practices that match their size and pain.