note 07· · 5 min

Eighteen agents audited a live curriculum. One lesson was teaching a false error.

I pointed an audit fleet at 34 published programming lessons: six auditors executing every code claim, six rewriters, six independent verifiers. 73 defects, 17 critical or high — every one anchored to a verbatim quote. Thirteen minutes of wall clock.

The headline defect makes the case by itself. Lesson one of a programming course taught that print(Hello, world!) fails because Python looks up "Hello" as a command and can't find it. Plausible. Confident. Wrong — the ! makes the line unparseable, so Python dies with a SyntaxError before any lookup happens. A beginner following the lesson's own advice ("read the error message") would watch their screen contradict their teacher. No human reviewer had caught it, because it reads fine. The auditor caught it because it doesn't run fine — and my auditors are required to run everything.

6 auditorsrun every claim73 defectsverbatim quotes6 rewritersfix + re-run6 verifiers34/34 ≥95
fig. 1 — audit → rewrite → verify, pipelined per file, no shared state

The architecture is three fleets with enforced independence. Six auditors (one per content file) execute every runnable claim with real Python — labs get run twice, because a teaching lab must fail for exactly the stated reason and pass after the stated fix. Every defect they file must carry a verbatim quote from the source; no paraphrases, because the quotes double as edit anchors. Six rewriters apply fixes — with a standing rule that they re-verify each defect themselves and are rewarded for refusing a fix that turns out to be wrong. Six verifiers then re-score everything cold, having never seen the first audit. Their score is the score of record, not the rewriters' claim about their own work.

The numbers, verbatim from the run: 34 lessons, 73 defects (17 critical or high — including nine lab starters whose indented placeholder comment would throw IndentationError at absolute beginners before indentation is taught), 47 fixes, and a final independent verdict of 34/34 lessons at the 95+ bar with zero critical and zero high remaining. Wall clock: about thirteen minutes for the fleet, plus one human hour on the residuals the verifiers surfaced. The full quoted-evidence ledger is published, red findings included — same rule as everything else I ship.

An agent that grades by reading will tell you what sounds right. An agent that grades by executing tells you what is right.

This pattern — execution-gated review, verbatim anchors, adversarially independent verification — is not specific to lessons. It audits documentation against the API it describes, runbooks against the systems they operate, marketing claims against the product. If you have a large body of content whose correctness you're currently taking on faith, this is the engagement that replaces faith with a ledger.

artifacts — public report: sageideas.dev/academy/how-we-audit · full quoted-evidence ledger in the repo · pattern available as an engagement
← All field notes Work with me →