Auditing Analytics Your Agent Wrote
Pretty dashboards are not evidence
Most of github-analysis was built with Cursor. That does not make the numbers true. On the afternoon of 2026-06-01 we ran a formal agent code audit: handoff (e015912 at 14:12), P1/P2 fixes (0b0b23d at 14:17), then deleted the transient handoff (e33e491 at 14:18). Algorithmic intent was mostly right; we still found search truncation risk, stale cache summaries, and README drift. Leaders act on polished HTML - verification has to be as fast as generation.
Here's what this piece covers: why shimmer is dangerous, the audit method, findings leaders can repeat, what we kept, and three lead habits.
Part of Field Notes: agentic delivery.
1. The leadership problem
Agent-generated code ships fast; verification discipline often does not. Metrics bugs are silent - dashboards look authoritative while joins are wrong.
Wrong: "The agent built our metrics."
Right: "e015912 → 0b0b23d: what we audited, what failed, what we kept."
The anti-pattern is trust the shimmer - pretty HTML, no reconciliation.
2. What the git history shows
| Time (EDT) | Commit | Artifact |
|---|---|---|
| 14:12 | e015912 |
Metrics review handoff for agent code audit |
| 14:17 | 0b0b23d |
Fix metrics P1/P2 issues from review findings |
| 14:18 | e33e491 |
Remove transient handoff |
Durable record kept: docs/metrics-review-findings.md. The handoff was disposable; the findings doc is the operating memory.
3. Audit methodology
Three passes - enough for an EM to demand without becoming a full-time auditor:
- Per-column verdict - correct / acceptable with caveats / wrong. Written table, not vibes.
- Cross-check against
gh searchfor sample users (we usedmonika-sbdon May 2026 caches). - Production cache replay -
--from-cacherecompute vs embedded summaries for regression.
# Spot-check shape: compare tool output to gh search for one author
def authored_matches_search(tool_count: int, gh_search_count: int) -> bool:
"""Ground truth is the search the catalog claims to encode."""
return tool_count == gh_search_count
# monika-sbd / global-services May 2026: authored=2 matched gh search (verified)
assert authored_matches_search(2, 2)Core snapshot helpers (is_open_at_month_end, half-open window_bounds_utc) matched intended semantics after the morning fixes. The audit did not invent green; it separated algorithm OK from ops risk.
4. Named pattern: audit before adopt
Audit before adopt - reconcile agent output against ground truth before leaders staff from it.
Findings that matter for leaders
| Finding | Severity | Why it matters |
|---|---|---|
| GitHub search 1,000-result cap | High on huge repos | Silent undercount if a Phase 1 query exceeds the cap - May 2026 caches were under (verified) |
prs_merged from Phase 2 rows only |
Medium | Skipped fetches undercount merges; authored/open use dedicated catalogs |
Stale _raw.json summaries |
Medium | Pre-prs_closed_unmerged caches showed zeros until re-export |
| README reconciliation formula | Doc bug | Column totals ≠ window-opened identity (article 10) |
What we kept: the pipeline shape (parallel fetch + cache + run logs), half-open windows, and the reconciliation identity from article 09. What broke organizationally: no automated tests that day - still the highest org risk. Agent audit is cheap; wrong headcount in a reorg meeting is not.
5. What I'd do as a lead
- Require an evidence bundle for any agent-built internal tool: column defs, sample reconciliation, run log.
- Schedule a quarterly re-audit when GitHub API or catalog logic changes. Yesterday's green verdict expires.
- Never trust cached Excel from before a schema change - re-export. Stale summaries are a known failure mode.
Further reading
- The afternoon we shipped three wrong metrics - same-day definition fixes
- README as contract - glossary + windows
- An 857-line honesty machine - probe before product
- Field notes: agentic delivery
- github-analysis:
e015912→0b0b23d·docs/metrics-review-findings.md
