Build an evidence map of an unfamiliar system

TL;DR: Follow a representative path, verify what revision and configuration actually run, and compare logs, data, tests, code and owner explanations. Each source answers different questions; preserve uncertainty instead of turning an incomplete trace into a complete architecture.

Where you are. Discovery identified a candidate workflow and its prerequisites. You now need enough understanding of the existing system to judge an integration or change without guessing about hidden behavior.

The decision you need to make comes first

You may have a repository, a warehouse, operating dashboards and documents of different ages. Begin with the question you need answered: can this source support the required freshness; where does identity change; how is a retry handled; or what can be deployed safely within the environment?

A short, targeted investigation beats trying to read every file before forming a question, though that does not make code a last resort. Entry points, tests and schemas can be the quickest route to the relevant behavior, especially when runtime access is unavailable.

Before removing an unexplained component, look for its purpose, dependencies and evidence. It may enforce an important constraint, or it may be obsolete, and neither conclusion follows from its age or from nobody remembering it. Record a testable removal hypothesis and the required checks instead of treating every old component as sacred.

One record, with its coverage limit marked

Use an approved synthetic or appropriately authorized record, and track its business identity, operation ID, transformations, stores and consumers. Avoid using a customer ID alone as a universal join key when it is only unique within a tenant or source.

A representative path helps identify the participating components, but one successful record does not exercise alternate branches, rare failures, retries or other tenants, and missing telemetry can hide a component entirely. Add traces for the important variations after establishing the ordinary path.

Asynchronous work can time out too: publishing, processing deadlines, lease expiry and polling all have time semantics. “Async” means the caller does not wait for the complete downstream workflow in the same interaction; it does not mean delivery or completion is guaranteed.

OpenTelemetry's trace concepts describe spans (each one timed step of a request, with a start and an end timestamp and, except for the root span, a parent) and their relationships, including links for related asynchronous work. Instrumentation can help connect a path, but a missing span is not by itself proof that no effect occurred.

A supplied timeline

This complete Python fixture filters a small synthetic event log by tenant, order and operation. Times are offsets from one shared exercise clock, not wall clocks from separate machines, so the fixture says nothing about whether real distributed clocks are synchronized.

events = [
    (0, 'A', 'O1', 'op-7', 'request accepted'),
    (100, 'A', 'O1', 'op-7', 'queue accepted message'),
    (200, 'A', 'O1', 'op-7', 'client timeout'),
    (300, 'B', 'O1', 'op-8', 'unrelated tenant operation'),
    (500, 'A', 'O1', 'op-7', 'worker started'),
    (550, 'A', 'O1', 'op-7', 'effect committed with receipt'),
    (900, 'A', 'O1', 'op-7', 'redelivery returned existing receipt'),
]
selected = [row for row in events if row[1:4] == ('A', 'O1', 'op-7')]
for offset, tenant, order, operation, message in sorted(selected):
    print(offset, message)
assert len(selected) == 6
assert sum(row[4] == 'effect committed with receipt' for row in selected) == 1
# 0 request accepted
# 100 queue accepted message
# 200 client timeout
# 500 worker started
# 550 effect committed with receipt
# 900 redelivery returned existing receipt

The supplied log records an effect after the client stopped waiting, and a duplicate delivery that returned the receipt. That supports this scenario's one-effect claim, assuming the log is complete and truthful; it does not prove the implementation is correct for all crashes or concurrent calls.

The queue wait is 400 ms between acceptance and worker start, and worker start to recorded commit is 50 ms. Adding every duration in a real trace can double-count overlapping or nested work, so use the path and timestamp semantics rather than summing arbitrary span lengths.

Remove the commit and receipt entries from the fixture, and the remaining client timeout no longer tells you whether any effect happened. Four rows remain and the last of them is "worker started": the log now shows work beginning after the caller gave up and nothing about how it ended, which is exactly the state in which an unprotected retry can double-apply the effect. Back on the full log, filtering by order alone instead of by tenant and operation brings the count to seven (five on the reduced log), because tenant B's unrelated operation on the same order number joins the trace. Record the outcome as unresolved and look for source state or another authoritative operation record; never replay a write just because its success log is missing.

What actually runs

Compare the repository revision with the deployed artifact, configuration, scheduled jobs and enabled routes. A code path in main may not be deployed; a deployed feature may be disabled; a scheduled worker may run from a different image.

Suppose the repository contains a timeout added in revision R2 while the deployment manifest identifies R1. You cannot claim production has the R2 timeout; verify the actual release and effective configuration through authorized read paths. Conversely, a manifest alone may be stale; connect it to the running workload or trusted release record.

Inspect configuration names and provenance without copying secret values into your investigation notes. Distinguish source build, artifact promotion and runtime configuration, and record which evidence was available. That is enough to identify a gap without pretending you inspected production when you only read source.

Workload and cost as clues

High-volume endpoints, large tables and expensive jobs can point to important investigation targets, but they do not define importance by themselves. A low-frequency emergency path or a small access-control table may be critical, and an alert may be obsolete or poorly chosen.

Ask what the workload serves, who relies on its output and what a failure costs, and compare average and peak behavior, backlog and freshness. Do not optimize a costly query before confirming its semantics; making the wrong report cheaper does not make it correct.

Tests as executable claims

A test states an expectation for the exercised inputs and implementation, and it might have been written from a requirement, a hypothetical failure, a bug report or a production incident. A specific date or ticket is a lead to investigate, not proof of an incident or its severity.

Read relevant tests early when they clarify the contract, but run them only in an appropriate environment after checking their dependencies and effects. Passing tests do not prove the deployed version passes, and mocks may omit the integration boundary you need to assess.

Use tests, issue history, owner explanations and code comments together to understand reasoning. Where they disagree, write down the conflict and the next verification step instead of assuming one source always outranks the rest.

The spine above is the investigation in order, with the coverage limit sitting as its own step because a trace that forgets it becomes an architecture nobody verified.

Do this before moving on

Run the timeline, remove the two final outcome records and write down what is still known and unknown. Then filter by order alone and show why the unrelated tenant event contaminates the trace. Explain why business identity and operation identity both matter.

Create an evidence map with five columns: claim, source, revision or observation time, coverage limit and next check. Include the R1/R2 deployment mismatch and a specific test whose historical origin is unknown.

For a permitted project of your choice, trace one request and one important failure branch. Compare your map with the architecture documentation and record any actual differences; do not invent a contradiction because the exercise expects old documentation to be wrong.

Go deeper

• Parsing messy data explains the transformations you may encounter while following a record.
• Observability for AI covers instrumenting a system so the next person does not have to do this archaeology.
• Idempotency is the property you will be checking for as you trace a record through retries.
• Change data capture explains one mechanism that maintains derived copies across systems.
• Debugging a production incident in a customer environment collects the questions that test this skill directly.

Key takeaways

• Orient around a concrete decision using multiple evidence sources.
• One trace covers one path; asynchronous work still has timeouts and uncertain outcomes.
• Verify deployed revision and effective configuration before claiming source behavior is live.
• Cost and traffic help prioritize, but critical low-volume paths still matter.
• Tests demonstrate exercised expectations, not automatic production history or complete safety.