> ## Content Index
> Fetch the complete content index at: https://contextfirst.blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Auditing Analytics Your Agent Wrote
- URL: https://contextfirst.blog/auditing-agent-wrote-analytics/
- Published: 2026-09-07T13:00:00.000Z
- Updated: 2026-09-07T13:00:00.000Z
- Description: Pretty dashboards are not evidence
- Author: Iheatu Chris Wogu
- Tags: Field Notes, Engineering Management, Agentic Delivery, Learning Journey 2026

Most of github-analysis was built with Cursor. That does not make the numbers true. On the afternoon of **2026-06-01** we ran a formal **agent code audit**: handoff (`e015912` at 14:12), P1/P2 fixes (`0b0b23d` at 14:17), then deleted the transient handoff (`e33e491` at 14:18). Algorithmic intent was mostly right; we still found search truncation risk, stale cache summaries, and README drift. Leaders act on polished HTML - verification has to be as fast as generation.

**Here's what this piece covers:** why shimmer is dangerous, the audit method, findings leaders can repeat, what we kept, and three lead habits.

Part of [Field Notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/).

---

## 1\. The leadership problem

Agent-generated code ships fast; verification discipline often does not. Metrics bugs are silent - dashboards look authoritative while joins are wrong.

**Wrong:** "The agent built our metrics."  
**Right:** "`e015912` → `0b0b23d`: what we audited, what failed, what we kept."

The anti-pattern is **trust the shimmer** \- pretty HTML, no reconciliation.

---

## 2\. What the git history shows

| Time (EDT) | Commit  | Artifact                                      |
| ---------- | ------- | --------------------------------------------- |
| 14:12      | e015912 | Metrics review handoff for agent code audit   |
| 14:17      | 0b0b23d | Fix metrics P1/P2 issues from review findings |
| 14:18      | e33e491 | Remove transient handoff                      |

Durable record kept: `docs/metrics-review-findings.md`. The handoff was disposable; the findings doc is the operating memory.

flowchart LR A\[Agent-built pipeline\] --> H\[Handoff e015912\] H --> V\[Per-column verdict + spot-check\] V --> F\[0b0b23d P1/P2 fixes\] V --> D\[metrics-review-findings.md\] H --> X\[Delete e33e491\]

---

## 3\. Audit methodology

Three passes - enough for an EM to demand without becoming a full-time auditor:

1. **Per-column verdict** \- correct / acceptable with caveats / wrong. Written table, not vibes.
2. **Cross-check against `gh search`** for sample users (we used `monika-sbd` on May 2026 caches).
3. **Production cache replay** \- `--from-cache` recompute vs embedded summaries for regression.

```python
# Spot-check shape: compare tool output to gh search for one author
def authored_matches_search(tool_count: int, gh_search_count: int) -> bool:
    """Ground truth is the search the catalog claims to encode."""
    return tool_count == gh_search_count

# monika-sbd / global-services May 2026: authored=2 matched gh search (verified)
assert authored_matches_search(2, 2)
```

Core snapshot helpers (`is_open_at_month_end`, half-open `window_bounds_utc`) matched intended semantics after the morning fixes. The audit did not invent green; it separated **algorithm OK** from **ops risk**.

---

## 4\. Named pattern: audit before adopt

> **Audit before adopt** \- reconcile agent output against ground truth before leaders staff from it.

### Findings that matter for leaders

| Finding                                | Severity           | Why it matters                                                                                                         |
| -------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| GitHub search **1,000-result cap**     | High on huge repos | Silent undercount if a Phase 1 query exceeds the cap - May 2026 caches were under (verified)                           |
| **prs\_merged from Phase 2 rows only** | Medium             | Skipped fetches undercount merges; authored/open use dedicated catalogs                                                |
| **Stale \_raw.json summaries**         | Medium             | Pre-prs\_closed\_unmerged caches showed zeros until re-export                                                          |
| **README reconciliation formula**      | Doc bug            | Column totals ≠ window-opened identity ([article 10](https://contextfirst.blog/readme-as-contract-half-open-windows/)) |

**What we kept:** the pipeline shape ([parallel fetch + cache + run logs](https://contextfirst.blog/parallel-fetch-cache-run-logs/)), half-open windows, and the reconciliation identity from [article 09](https://contextfirst.blog/afternoon-three-wrong-metrics/). **What broke organizationally:** no automated tests that day - still the highest org risk. Agent audit is cheap; wrong headcount in a reorg meeting is not.

---

## 5\. What I'd do as a lead

1. **Require an evidence bundle for any agent-built internal tool:** column defs, sample reconciliation, run log.
2. **Schedule a quarterly re-audit when GitHub API or catalog logic changes.** Yesterday's green verdict expires.
3. **Never trust cached Excel from before a schema change - re-export.** Stale summaries are a known failure mode.

---

### Further reading

- [The afternoon we shipped three wrong metrics](https://contextfirst.blog/afternoon-three-wrong-metrics/) \- same-day definition fixes
- [README as contract](https://contextfirst.blog/readme-as-contract-half-open-windows/) \- glossary + windows
- [An 857-line honesty machine](https://contextfirst.blog/pr-timeline-script-engineering-metrics/) \- probe before product
- [Field notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/)
- github-analysis: `e015912` → `0b0b23d` · `docs/metrics-review-findings.md`

---

![Iheatu Chris Wogu](https://avatars.githubusercontent.com/u/3457705?s=176&v=4)

## About the author

**Iheatu Chris Wogu** is an engineering manager focused on launch programs, honest engineering metrics, and **engineering-led, agent-accelerated** delivery - researching where work queues up when agents widen dev output, then building proof, gates, and measurement that scale.

Field notes live on [First Context](https://contextfirst.blog/). Values treatise: [engineering manager values](https://contextfirst.blog/engineering-manager-values/).

**LinkedIn:** [linkedin.com/in/iheatu-chris-wogu](https://www.linkedin.com/in/iheatu-chris-wogu/?ref=contextfirst.blog)