The Tracker – Fault Finding in a Distributed Landscape

The previous posts in this series revolved around requirements. This one is about a completely different task that runs on the same architecture — and, to me, it is the most convincing evidence that a toolkit can do more than what it was built for.

The Starting Point

A fault gets reported. Not by a user, but by a pipeline — and not in prose, but as a failed scenario:

Scenario: Cancel a confirmed booking
  Given a confirmed booking exists
  When the customer triggers the cancellation
  Then the booking has the status "cancelled"

  ✗ Step 3 failed — expected "cancelled", was "confirmed"

It reads as precise, and that is exactly where the trap is. The text says exactly what is wrong — and not a word about where. A business scenario describes a contract across the entire chain. When it turns red, somebody in that chain broke their part of the contract. Which service that was is not stated — that is precisely the point of a test phrased in business terms.

As long as only one service is involved, this stays manageable. It gets expensive in the state that is the norm in a microservice landscape: a dozen services, a dozen pipelines of their own — and all of them delivering into the same integration stage. At the time of the red run, there was not one change active, but seven. From seven teams who knew nothing about each other.

That shifts the question. It is no longer "what is broken", but:

Whose change, out of which pipeline, turned this run red?

And of all people, the team that owns the test cannot answer it on its own. It sees its commit, its build, its log. It does not see that four minutes earlier another service renamed a field in a response.

The reflex is the same everywhere — and everywhere justified: "nothing changed on our side." Seven teams are right, and the run is red anyway.

What happens next is familiar to everyone: the run gets repeated. The second time it is green, because the foreign deployment has gone through by now — and the scenario quietly picks up the label "flaky". No finding, no ticket, no root cause. The time after that, nobody believes red means anything any more. That is the real damage: not the individual failure, but a suite nobody trusts.

The experience anyone running this kind of setup knows: the search takes longer than the fix. Not always, but often enough that it is the actual burden — the correction afterwards is frequently a one-liner.

And the path there is rarely analytical. It runs via the question "who might know where this comes from", via messages across several channels, via colleagues who are busy with something else — and via the timelines of several pipelines that somebody ends up lining up by hand.

What the Assistant Delivers — and What It Doesn't

It produces a diagnosis. How solid that diagnosis is, is decided not by the model but by the material.

That is the point I misjudged at the start. I had thought of the assistant as a pure signpost: a rough first suspicion, the detailed work left to a human. As soon as it could see more than the documented target state, that turned out to be a considerable understatement. The quality of the analysis is not a property of the assistant — it is a function of what the surrounding systems provide as evidence:

The ladder has a fault line in the middle. Everything above the logs is static: fault report, tests, source code and documentation say how it was meant to be. That gets you surprisingly far — the source code of the tests and the services answers the question "which components are involved in this scenario at all" more precisely than any architecture diagram, because it cannot go stale.

What the static part does not answer is the question about this run. That takes the dynamic part: a log says that a call failed. A trace says in which order, between which services and in which version — and with that, the "nothing changed on our side" from the previous section can be checked against data instead of against memory. That is the point where narrowing down actually turns into finding the cause.

The last rung of the ladder is missing in my setup, though: the deployment and pipeline history is not connected. So the assistant can reconstruct from logs and traces where the contract was broken — but not whose deployment ran through four minutes earlier. It therefore answers the question from the previous section only halfway: it names the break, somebody else has to put the trigger next to it.

I still list that rung in the table, because in my view it has the best effort-to-effect ratio of anything still open in this setup. It is a manageable data source that exists anyway — who moved what onto which stage and when — and it answers exactly the question the search hangs on the longest. It isn't connected yet; of all the expansion steps, it is the first one I would tackle.

And here is what it does not deliver even then:

It does not decide. Even a well-evidenced diagnosis is a finding that a human confirms before it turns into a ticket in somebody else's team.

It diagnoses nothing it cannot see. Whatever is not connected is missing — and has to appear in the result as missing, not be quietly left out.

It does not commit to one cause, even where the evidence would allow it. Why that of all things is the most important rule comes further down.

Take-Away: The Template for a Fault Analysis

# Fault Analysis: <Short description>

## Symptom
<How the fault was reported — verbatim, not interpreted.>

## Observable Conditions
<Since when, or since which run? Always or only sporadically?
Reproducible? "Unknown" is a valid answer.>

## Hypotheses

### H-1 · Confidence: <high | medium | low>
- **Component:** <Suspected origin>
- **Ownership:** <Team or role, if known>
- **Reasoning:** <Why is this component a candidate?>
- **Based on:** <Code location, test case, document, interface,
  log or trace location, earlier ticket — concrete enough to look up>
- **Next step to check:** <What specifically to verify>
- **Refuted if:** <How do you recognise this trail is the wrong one?>

### H-2 · Confidence: …
- …

### H-3 · Confidence: …
- …

## Not Considered
| Source | Reason |
|--------|--------|
| <e.g. deployment history> | <not connected> |

And the rulebook:

## Rules for This Assistant

**Mandatory**
- At least three hypotheses. Even when one seems obvious.
- Every hypothesis needs a refutation criterion.
- Every hypothesis needs a concrete next step to check.
- Confidence follows the evidence, not the plausibility of the
  narrative.
- Every source that was not connected or not reachable goes under
  "Not Considered".

**Forbidden**
- Committing to one cause.
- A hypothesis without evidence from a named source.
- Claiming runtime behaviour that does not follow from a log or a
  trace — code and documentation describe the target state, not
  this run.

The Refuted if field is the reason this template works. On to that now.

What Didn't Work in the First Draft

The assistant was too convincing.

The first version delivered exactly one answer, cleanly argued, in a confident tone. And that was precisely the problem.

A plausible-sounding first suspicion feels like a result. Whoever reads it checks it — and stops considering other possibilities. If the suspicion is right, that's excellent. If it is wrong, the assistant didn't shorten the search but steered it into a dead end, costing the time you would have had for the right trail without it.

That is a well-known human pattern, not a technical one. An early anchor distorts everything that comes after it — and a machine-generated anchor looks more objective than it is.

A wrong first suspicion is more dangerous than none at all.

The consequence was to forbid the assistant to commit. At least three hypotheses, each with a weight, each with a concrete next step, and each with the criterion by which you recognise it is wrong.

On top of that, the rule that separates tone from substance: the weighting hangs on the evidence, not on the phrasing. A hypothesis resting on a trace may be "high". One that only follows from the architecture documentation may not — even though it reads just as fluently. Those two cases were indistinguishable in the first version, which is why it came across as more convincing than its evidence warranted.

The side effect was the actual gain: instead of an answer you either believe or don't, you get a work list you can work off.

Limits

The diagnosis is only as good as what is connected. Where only the static part of the ladder is available — fault report, tests, source code, documentation — the assistant argues about the target state: about a system as it is meant to be, not as it ran in this particular case. Source code is the one source that cannot go stale. With documentation it is different: where the landscape has changed and the description hasn't, it argues about a system that no longer exists in that form.

That is no small thing. Anyone who wants to use this pattern should know what a given diagnosis rests on before trusting it — and take the Not Considered section seriously. It is the difference between "no trail found" and "didn't look there". In my case the deployment history is a regular entry: a known, named blind spot — documented, not hidden.

And the converse, which I had underestimated: it does not replace observability — it lives off it. I had expected an assistant like this to help most where there is no end-to-end tracing. It is the other way round. Without telemetry it stays a signpost that narrows the search space; with it, it becomes a diagnostic tool that names the cause and backs it up. So whoever wants the greater benefit invests not in the assistant, but in the data it reads from.

Conclusion

Of all the assistants in the toolkit, this is the one with the least to do with the original purpose — and the one where the feedback was clearest.

The reason, as I see it, lies in the nature of the problem: fault finding is search, and search scales badly with the number of people involved. A tool that narrows the search space doesn't save you the fix — it saves you the coordination beforehand. And the better the evidence, the less searching is left at all: "ask around in your channels" turns into "have a look at this call in this trace".

And the most important design decision was still one that makes the assistant look weaker: it must not be certain — not even where the evidence would allow it.

This closes the thread across all four assistants: the creator surfaces assumptions, the transformer surfaces origins, the critic surfaces what went unchecked, the tracker surfaces uncertainty. Four times the same principle — an assistant that makes its own limits visible.

→ The toolkit at a glance