> ## Content Index
> Fetch the complete content index at: https://contextfirst.blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Token Compression and the Index Protocol
- URL: https://contextfirst.blog/token-compression-index-protocol/
- Published: 2026-09-20T13:00:00.000Z
- Updated: 2026-09-20T13:00:00.000Z
- Description: Compress the corpus; query the index first
- Author: Iheatu Chris Wogu
- Tags: Field Notes, Engineering Management, Agentic Delivery, Learning Journey 2026

Agent context is finite. Dump a monorepo wiki into the window and the agent misses the one normative paragraph while you pay for duplicated governance prose. Documentation debt shows up as wrong edits and expensive sessions - the same [search exhaustion](https://contextfirst.blog/graphify-monorepo-navigation/) problem from the agent side.

**Here's what this piece covers:** the context tax, what 2026-08-22 shipped in two repos, compression principles plus the Index Protocol, the named pattern, and three habits before you scale agentic delivery across more trees.

Part of the [Field Notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) series - series closer on making repos navigable and dense for agents.

---

## 1\. The leadership problem

Agents load entire standards trees and still invent process. Humans maintain fifty copies of "how agents consume docs." Cost and quality degrade together.

**Wrong:** "Use smaller prompts."  
**Right:** "`dc7a0b7` / `420fa79` (2026-08-22): what we index, what we omit, retrieval order before Grep - measured corpus cuts, not vibes."

The anti-pattern is **dump the whole monorepo** \- every prompt pays full tree tax.

---

## 2\. What the git history shows

Same day, two repos closed the loop:

| Repo                  | Commit  | When                | What landed                                                                                                    |
| --------------------- | ------- | ------------------- | -------------------------------------------------------------------------------------------------------------- |
| **agentic-framework** | dc7a0b7 | 2026-08-22 11:44 ET | Compress corpus: invariants.md registry, router-only README, deduplicated entry docs, research archive removed |
| **github-analysis**   | 420fa79 | 2026-08-22 10:18 ET | Index Protocol in graphify-search \- corpus-first navigation                                                   |
| **github-analysis**   | ed3f446 | 2026-08-22 10:19 ET | Tighten Index Protocol; block loading the full graphify skill as a dump                                        |

`dc7a0b7` recorded approximate savings in `token-compression-plan.md`: research archive \~110 KB, graphify kit stubs \~50 KB, README \~12 KB, plus smaller methodology trims. Coarse byte deltas - re-index after changes; do not treat KB as token-perfect without a fresh graphify estimate.

flowchart TD P\[Prompt\] --> Q\[graphify query / path / explain\] Q -->|hit| A\[Scoped subgraph\] Q -->|miss| G\[Grep / Read fallback\] H\[beforeSubmitPrompt\] --> U\[graphify update AST-only\] U --> Q C\[Token compression\] --> I\[invariants single source\] I --> Q

---

## 3\. What I built

**Token compression principles** (agentic-framework):

1. **Single source** \- governance in `invariants.md`; entry docs link, do not duplicate.
2. **Reference, don't restate** \- skills cite canonical sections.
3. **Normative density** \- RFC-2119 for requirements; cut marketing repetition.
4. **Archive excision** \- superseded research leaves the tree; history stays in git.
5. **Kit stubs** \- portable copies redirect to canonical `docs/graphify/`.

**Index Protocol** (github-analysis `graphify-search`, mirrored across Graphify repos):

1. Prefer live graph (`graphify update .` via Cursor `beforeSubmitPrompt` \- AST-only, no API cost).
2. Retrieval order: `query` → `path` → `explain` → lexical fallback **only on miss**.
3. Subagents must propagate the rule - no bypass because paths feel "familiar."

```bash
# Index Protocol - cheap structure before expensive reads
graphify query "where is recurrence median computed?"
graphify path "detect_regressions" "write_summary_md"
graphify explain "Index Protocol"
# Grep/Read only if the graph returns nothing useful
```

Backlog after the Aug 22 pass: still-large skills (`gh-stack`, `spec-builder`) and duplicated standards intros - compression is a quarterly habit, not a one-commit event. Complements [Graphify for monorepo navigation](https://contextfirst.blog/graphify-monorepo-navigation/) (portable kit) with **density** of what gets indexed.

---

## 4\. Named pattern: Index Protocol (+ compression)

> **Index Protocol** \- structured repo index first; **token compression** keeps the indexed corpus dense - omit noise, measure context cost, fall back to Grep only on miss.

| Artifact                  | Who uses it         | Decision it supports       |
| ------------------------- | ------------------- | -------------------------- |
| invariants.md             | Agents + humans     | Single normative lattice   |
| token-compression-plan.md | Doc owners          | What to cut next           |
| graphify-search skill     | Every agent session | Retrieval order            |
| Hook graphify-refresh.sh  | Cursor users        | Fresh AST graph per prompt |

**What broke:** skills that restate the whole operating model burn tokens every load. Skipping graphify because "I know the file" recreates search exhaustion. Measured % token savings beyond the plan's KB table still need a re-index comparison when you claim interview-grade precision.

---

## 5\. What I'd do as a lead

Three habits - steal one this month:

1. **Assign a doc owner for the agent-facing corpus** \- same accountability as code ownership.
2. **Enforce Index Protocol in `AGENTS.md` for repos past a size threshold.** Query before Grep is a team rule, not a preference.
3. **Run a quarterly compression pass.** Delete duplicated "how agents consume" intros; keep one invariants registry.

---

### Further reading

- [Field notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) \- series intro
- Prior: [Blocker matrix](https://contextfirst.blog/blocker-matrix-fix-what-unblocks/) · [Graphify monorepo navigation](https://contextfirst.blog/graphify-monorepo-navigation/)
- Related: [Shipping the agentic framework](https://contextfirst.blog/shipping-agentic-software-delivery-framework/) · [Specifications as contracts](https://contextfirst.blog/specifications-as-execution-contracts/)
- Anchors: agentic-framework `dc7a0b7` · github-analysis `420fa79` / `ed3f446` · 2026-08-22

---

![Iheatu Chris Wogu](https://avatars.githubusercontent.com/u/3457705?s=176&v=4)

## About the author

**Iheatu Chris Wogu** is an engineering manager focused on launch programs, honest engineering metrics, and **engineering-led, agent-accelerated** delivery - researching where work queues up when agents widen dev output, then building proof, gates, and measurement that scale.

Field notes live on [First Context](https://contextfirst.blog/). Values treatise: [engineering manager values](https://contextfirst.blog/engineering-manager-values/).

**LinkedIn:** [linkedin.com/in/iheatu-chris-wogu](https://www.linkedin.com/in/iheatu-chris-wogu/?ref=contextfirst.blog)