Field Notes

Token Compression and the Index Protocol

Compress the corpus; query the index first

Iheatu Chris Wogu

Iheatu Chris Wogu

— 3 min read
Share

Agent context is finite. Dump a monorepo wiki into the window and the agent misses the one normative paragraph while you pay for duplicated governance prose. Documentation debt shows up as wrong edits and expensive sessions - the same search exhaustion problem from the agent side.

Here's what this piece covers: the context tax, what 2026-08-22 shipped in two repos, compression principles plus the Index Protocol, the named pattern, and three habits before you scale agentic delivery across more trees.

Part of the Field Notes: agentic delivery series - series closer on making repos navigable and dense for agents.

1. The leadership problem

Agents load entire standards trees and still invent process. Humans maintain fifty copies of "how agents consume docs." Cost and quality degrade together.

Wrong: "Use smaller prompts."
Right: "dc7a0b7 / 420fa79 (2026-08-22): what we index, what we omit, retrieval order before Grep - measured corpus cuts, not vibes."

The anti-pattern is dump the whole monorepo - every prompt pays full tree tax.

2. What the git history shows

Same day, two repos closed the loop:

Repo Commit When What landed
agentic-framework dc7a0b7 2026-08-22 11:44 ET Compress corpus: invariants.md registry, router-only README, deduplicated entry docs, research archive removed
github-analysis 420fa79 2026-08-22 10:18 ET Index Protocol in graphify-search - corpus-first navigation
github-analysis ed3f446 2026-08-22 10:19 ET Tighten Index Protocol; block loading the full graphify skill as a dump

dc7a0b7 recorded approximate savings in token-compression-plan.md: research archive ~110 KB, graphify kit stubs ~50 KB, README ~12 KB, plus smaller methodology trims. Coarse byte deltas - re-index after changes; do not treat KB as token-perfect without a fresh graphify estimate.

flowchart TD P[Prompt] --> Q[graphify query / path / explain] Q -->|hit| A[Scoped subgraph] Q -->|miss| G[Grep / Read fallback] H[beforeSubmitPrompt] --> U[graphify update AST-only] U --> Q C[Token compression] --> I[invariants single source] I --> Q

3. What I built

Token compression principles (agentic-framework):

  1. Single source - governance in invariants.md; entry docs link, do not duplicate.
  2. Reference, don't restate - skills cite canonical sections.
  3. Normative density - RFC-2119 for requirements; cut marketing repetition.
  4. Archive excision - superseded research leaves the tree; history stays in git.
  5. Kit stubs - portable copies redirect to canonical docs/graphify/.

Index Protocol (github-analysis graphify-search, mirrored across Graphify repos):

  1. Prefer live graph (graphify update . via Cursor beforeSubmitPrompt - AST-only, no API cost).
  2. Retrieval order: query → path → explain → lexical fallback only on miss.
  3. Subagents must propagate the rule - no bypass because paths feel "familiar."
# Index Protocol - cheap structure before expensive reads
graphify query "where is recurrence median computed?"
graphify path "detect_regressions" "write_summary_md"
graphify explain "Index Protocol"
# Grep/Read only if the graph returns nothing useful

Backlog after the Aug 22 pass: still-large skills (gh-stack, spec-builder) and duplicated standards intros - compression is a quarterly habit, not a one-commit event. Complements Graphify for monorepo navigation (portable kit) with density of what gets indexed.

4. Named pattern: Index Protocol (+ compression)

Index Protocol - structured repo index first; token compression keeps the indexed corpus dense - omit noise, measure context cost, fall back to Grep only on miss.
Artifact Who uses it Decision it supports
invariants.md Agents + humans Single normative lattice
token-compression-plan.md Doc owners What to cut next
graphify-search skill Every agent session Retrieval order
Hook graphify-refresh.sh Cursor users Fresh AST graph per prompt

What broke: skills that restate the whole operating model burn tokens every load. Skipping graphify because "I know the file" recreates search exhaustion. Measured % token savings beyond the plan's KB table still need a re-index comparison when you claim interview-grade precision.

5. What I'd do as a lead

Three habits - steal one this month:

  1. Assign a doc owner for the agent-facing corpus - same accountability as code ownership.
  2. Enforce Index Protocol in AGENTS.md for repos past a size threshold. Query before Grep is a team rule, not a preference.
  3. Run a quarterly compression pass. Delete duplicated "how agents consume" intros; keep one invariants registry.

Further reading

Iheatu Chris Wogu

About the author

Iheatu Chris Wogu is an engineering manager focused on launch programs, honest engineering metrics, and engineering-led, agent-accelerated delivery - researching where work queues up when agents widen dev output, then building proof, gates, and measurement that scale.

Field notes live on First Context. Values treatise: engineering manager values.

LinkedIn: linkedin.com/in/iheatu-chris-wogu

Read more