The Critics – Checking Against Rules and Against What Already Exists

The two previous posts were about assistants that produce and convert artifacts. This one is about the ones that review them. It is the operation where it became clearest to me how much context decides over usefulness.

Three Levels of Review

Not every check costs the same. I distinguish three levels, and what sets them apart is above all how much context they need:

Levels 1 and 2 come almost for free. Level 3 is the one that actually matters — and the only one that fundamentally does not work without access to the complete inventory. Why I divide context into stages is something I describe in a post of its own.

The practical point: reviewing is more expensive than creating. Producing an artifact requires the template and a few examples. Reviewing an artifact against the inventory requires the inventory. Anyone who only budgets for the creating side when setting up an assistant system will significantly underestimate the running cost.

Take-Away: The Template for a Review Report

A review result must not be prose. Prose gets read once, nodded at, and left behind. Findings get worked off.

This template, too, is kept generic — categories, severities and language can be adapted to whatever is common practice at your place. What I would not leave out are the last two sections.

# Review Report: <Artifact>

## Summary
- **Levels checked:** Form · Internal logic · External consistency
- **Findings:** <n> Blocker · <n> Important · <n> Note

## Findings

### F-1 · <Form | Logic | Consistency | Language> · <Blocker | Important | Note>
- **Location:** <Field or section>
- **Observation:** <What specifically stands out>
- **Suggestion:** <Concrete wording or measure>
- **Based on:** <Rule, document, existing artifact>

### F-2 · … · …
- …

## Not Checked
| Check | Reason |
|-------|--------|
| <e.g. duplicate detection> | <source unreachable / outdated / not available> |

## Overall Verdict
<Only fill in if "Not Checked" is empty.
Otherwise: "No overall verdict possible — see above.">

And the rulebook that goes with it:

## Rules for This Assistant

**Mandatory**
- Every finding names a concrete location. General concerns are not
  a finding.
- Every finding names a suggestion, not just a problem.
- Every check that could not be carried out goes under
  "Not Checked" — with a reason.

**Forbidden**
- Issuing an overall verdict while "Not Checked" is non-empty.
- Softening findings in order to sound polite.
- Presenting an absence of findings as confirmation.

The Not Checked section is the core of the whole template. On to that now.

What Didn't Work in the First Draft

Problem 1: The assistant was too polite.

Language models are agreeable. The first review assistant almost always found that everything was fine in essence, and phrased criticism so cautiously that it got lost while reading. A reviewer who finds nothing is worthless — and worse than worthless, because it creates a feeling of safety.

What helped was the switch from prose to findings with a category and a severity. A field that has to be filled in is harder to water down than a paragraph. On top of that, the requirement to attach a concrete location to every finding — which rules out general concerns.

Problem 2: Missing findings looked like a clean result.

That was the more dangerous mistake. Whenever the inventory happened to be unreachable, the assistant simply checked form and logic — and reported: no findings. To the user, that looked identical to a complete review without objections.

Out of that came the rule I consider the most important one for review assistants:

A review assistant that cannot say "I couldn't check that" is more dangerous than none at all.

Hence the mandatory Not Checked section and the rule that an overall verdict is only permitted while that section stays empty. A silent blind spot becomes a visible limitation.

That is the same idea as Open Questions and Assumptions in the creation assistant. Both times it is about the system surfacing its own limits instead of glossing over them.

Problem 3: Language review against the wrong standards.

Linguistic analysis sounds harmless, and it isn't. What counts as good language in general — short sentences, few technical terms, active voice — is partly wrong for a requirement in an enterprise context. Technical terms are precision there, not a barrier. Passive constructions are sometimes exactly right, because the actor is deliberately left open.

Initially the assistant optimised towards general readability and paid for it in precision. The rules had to be switched to what is right in this context: unambiguity over readability, defined terms instead of paraphrases, no hedging.

Limits

A review assistant finds candidates, not truths. A reported duplicate is a hint that a human has to confirm — similarity in the text is not the same as sameness in substance.

And it reviews against the inventory as documented. Where documentation and reality drift apart, it reviews against the past.

Conclusion

Reviewing is the operation with the greatest benefit and the highest cost. The benefit arises almost entirely at level 3 — and that one doesn't come for free.

The rule I take away: an assistant has to be able to surface its own limits. Everything else is a false sense of safety, and a false sense of safety is worse than none.

The next post is about a task that has nothing to do with requirements any more and still runs on the same architecture: fault finding in a distributed landscape.

→ The toolkit at a glance