Build an evidence map of an unfamiliar system

This is lesson seven of Foundations of Forward Deployed Engineering. Follow a representative path, verify what revision and configuration actually run, and compare logs, data, tests, code and owner explanations. Each source answers different questions. Preserve uncertainty, instead of turning an incomplete trace into a complete architecture.

Discovery identified a candidate workflow and its prerequisites. You now need enough understanding of the existing system to judge an integration or change, without guessing about hidden behavior.

Start from the decision you need to make. You may have a repository, a warehouse, operating dashboards and documents of different ages.

Begin with the question, such as whether this source can support the required freshness, where identity changes, how a retry is handled, or what can be deployed safely within the environment. A short, targeted investigation is more useful than trying to read every file before forming a question.

That doesn't make code a last resort. Entry points, tests and schemas can be the quickest route to the relevant behavior, especially when runtime access is unavailable.

Before removing an unexplained component, seek its purpose, dependencies and evidence. It may enforce an important constraint, or it may be obsolete, and neither conclusion follows from its age, or from the absence of a person who remembers it. Record a testable removal hypothesis and the required checks, instead of treating every old component as sacred.

Now follow one record, and mark the coverage limit. Use an approved synthetic or appropriately authorized record.

Track its business identity, operation ID, transformations, stores and consumers. Avoid using a customer ID alone as a universal join key, when it's only unique within a tenant or source.

A representative path helps identify participating components, but one successful record doesn't exercise alternate branches, rare failures, retries or other tenants. Missing telemetry can also make a component invisible, so add traces for the important variations, after establishing the ordinary path.

Here is a synthetic freight path, hop by hop, with a failure question at each. From portal to order service, ask whether a client timeout left an unknown outcome, and from order service to queue, whether the message was accepted even if the acknowledgement was lost. From queue to worker, ask whether redelivery can repeat a business effect, from worker to shipment store, whether the effect and deduplication evidence are consistent, and from export to warehouse, whether the displayed result is current and complete.

And at each hop there's evidence to inspect: request identity, caller scope and response, then publish operation and broker confirmation, message and operation identity with the delivery count, the authorized state transition and its receipt, and finally the source cut, batch manifest and reconciliation.

Asynchronous work can time out too, because publishing, processing deadlines, lease expiry and polling all have time semantics. Asynchronous means the caller doesn't wait for the complete downstream workflow in the same interaction. It doesn't mean delivery or completion is guaranteed.

OpenTelemetry's trace concepts describe spans. Each span records one step of a request, with a start and an end timestamp and, except for the root span, a parent, and they include links for related asynchronous work. Instrumentation can help connect a path, but a missing span isn't by itself proof that no effect occurred.

Next, read a supplied timeline. A complete Python fixture filters a small synthetic event log by tenant, order and operation.

There are seven events. Each one carries an offset, a tenant, an order, an operation and a message. One of them belongs to a different tenant and a different operation, on the same order number. The filter keeps only rows matching all three identifiers. The times are offsets from one shared exercise clock, not wall clocks from separate machines.

Six rows survive. At zero, the request is accepted. At 100, the queue accepts the message. At 200, the client times out. At 500, the worker starts. At 550, the effect is committed with a receipt. And at 900, a redelivery returns the existing receipt.

The log records an effect after the client stopped waiting. It also records a duplicate delivery that returned the receipt. That supports this scenario's one-effect claim, assuming the log is complete and truthful. It doesn't prove the implementation is correct for all crashes or concurrent calls.

The queue wait is 400 milliseconds, between acceptance and worker start, and worker start to recorded commit is 50 milliseconds. Adding every duration in a real trace can double-count overlapping or nested work, so use the path and timestamp semantics, rather than summing arbitrary span lengths.

Now remove the commit and receipt entries. Four rows remain, and the last of them is "worker started". The log now shows work beginning after the caller gave up, and nothing about how it ended. That's exactly the state in which an unprotected retry can double-apply the effect. The client timeout doesn't establish failure without effect.

Back on the full log, filter by order alone, instead of by tenant and operation. The count becomes seven, or five on the reduced log, because tenant B's unrelated operation on the same order number joins the trace. Record the outcome as unresolved, and seek source state or another authoritative operation record. Don't replay a write merely because its success log is absent.

Now verify what actually runs. Compare the repository revision with the deployed artifact, configuration, scheduled jobs and enabled routes.

A code path in main may not be deployed, a deployed feature may be disabled, and a scheduled worker may run from a different image. Suppose the repository contains a timeout added in revision R2, while the deployment manifest identifies R1. You can't claim production has the R2 timeout.

Verify the actual release and effective configuration through authorized read paths. Conversely, a manifest alone may be stale, so connect it to the running workload or a trusted release record. Inspect configuration names and provenance, without copying secret values into your investigation notes. Distinguish source build, artifact promotion and runtime configuration, and record which evidence was available. That's enough to identify a gap, without pretending you've inspected production when you've only read source.

Use workload and cost as clues. High-volume endpoints, large tables and expensive jobs can identify important investigation targets.

They don't define importance by themselves. A low-frequency emergency path, or a small access-control table, may be critical. And an alert may be obsolete or poorly chosen. Ask what the workload serves, who relies on its output and what failure costs, then compare average and peak behavior, backlog and freshness. Don't optimize a costly query before confirming its semantics, because making the wrong report cheaper doesn't make it correct.

Finally, treat tests as executable claims. A test states an expectation for the exercised inputs and implementation.

A test might have been written from a requirement, a hypothetical failure, a bug report or a production incident. A specific date or ticket is a lead to investigate, not proof of an incident or its severity.

Read relevant tests early, when they clarify the contract, and run them only in an appropriate environment, after checking dependencies and effects. Passing tests don't prove the deployed version passes, and mocks may omit the integration boundary you need to assess.

Use tests, issue history, owner explanations and code comments together to understand reasoning. Where they disagree, write down the conflict and the next verification step. Don't assume one source always outranks all others.

Your turn. Pause the video here. Run the timeline and remove the two final outcome records, then write what remains known and unknown. Filter by order alone, and show why the unrelated tenant event contaminates the trace. Explain why business identity and operation identity both matter. Then create an evidence map with five columns: claim, source, revision or observation time, coverage limit and next check. Include the R1 and R2 deployment mismatch, and a specific test whose historical origin is unknown. For a permitted project of your choice, trace one request and one important failure branch, and compare your map with the architecture documentation. Record actual differences if present, but don't invent a contradiction because the exercise expects old documentation to be wrong.

To recap. Orient around a concrete decision, using multiple evidence sources. One trace covers one path, and asynchronous work still has timeouts and uncertain outcomes. Verify deployed revision and effective configuration before claiming source behavior is live. Cost and traffic help prioritize, but critical low-volume paths still matter. And tests demonstrate exercised expectations, not automatic production history or complete safety.

That's lesson seven. Next in the series, specify the ownership, trust, network and meaning at each boundary. The full lesson, with its exercise and drill, is on fdeinterviews.com.