> ## Content Index
> Fetch the complete content index at: https://contextfirst.blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Parallel Fetch, Raw Cache, and Run Logs
- URL: https://contextfirst.blog/parallel-fetch-cache-run-logs/
- Published: 2026-09-04T13:00:00.000Z
- Updated: 2026-09-04T13:00:01.000Z
- Description: Reproducible pipelines beat live API roulette
- Author: Iheatu Chris Wogu
- Tags: Field Notes, Engineering Management, Agentic Delivery, Learning Journey 2026

Twenty-six minutes after the [CLI refactor](https://contextfirst.blog/script-to-cli-spreadsheet-product/), a month-end report that took the better part of an hour still failed the only test that matters: **can you re-run it and defend the number?** On **2026-06-01 at 09:58**, commit `336332d` added parallel workers, a `_raw.json` cache, run logging, and individual production sheets - treating metrics like a small data pipeline, not a hero script.

**Here's what this piece covers:** why slow untrusted reruns kill staffing conversations, what `336332d` shipped, how Phase 1 → Phase 2 → cache → run log fits together, the named pattern, and three lead habits.

Part of [Field Notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/).

---

## 1\. The leadership problem

Slow reports do not get run. Stale reports get argued about. When a VP asks why `prs_merged` moved week over week, "the API felt different" is not an answer - you need an audit trail.

**Wrong:** "We parallelized for speed."  
**Right:** "`336332d` same day as the CLI refactor: parallel fetch, cached raw JSON, run log so we can diff two staff meetings."

The anti-pattern is the **stateless hero run** \- re-fetch everything live, lose the trail, then blame GitHub when the spreadsheet disagrees with last Tuesday.

---

## 2\. What the git history shows

| Time (EDT) | Commit  | What landed                                                                     |
| ---------- | ------- | ------------------------------------------------------------------------------- |
| 09:32      | 9d47215 | Modular CLI (analyze / export / run)                                            |
| 09:58      | 336332d | \--workers, \--pr-resource-workers, \_raw.json, \_run.log, person summary sheet |

Verified: `336332d` message is *Add parallel fetch, run logging, cache, and individual production metrics.* Same morning as productization - operating concerns, not a later polish pass.

flowchart LR P1\[Phase 1 catalogs\] --> P2\[Phase 2 PR detail\] P2 --> C\["\_raw.json cache"\] P2 --> S\[Person summaries\] C --> R\["--from-cache rebuild"\] P1 --> L\["\_run.log"\] P2 --> L

---

## 3\. What I built

**Phase 1** \- GitHub search builds PR number catalogs (created, merged, activity, open-at-end candidates) in parallel where the queries allow.

**Phase 2** \- Per-PR detail fetch (\~6 `gh api` calls each: metadata, commits, files, events, reviews, comments). `ThreadPoolExecutor` in `pipeline/runner.py`; defaults are `--workers 4` and `--pr-resource-workers 5`. Serial baseline uses `pr-resource-workers 1` when you need to prove rate-limit behavior.

**Cache** \- `_raw.json` holds the raw payload so `--from-cache` rebuilds TSV/Excel without re-hitting the API. Filename convention: `{repo}_{start}_to_{end}.*`.

**Run log** \- auth checks, progress, skips, errors. First file to open when a run fails mid-staff-prep.

Shape of the contract (manager-safe sketch):

```python
# pipeline idea - fetch once, recompute many times
def run_month(repo: str, start: str, end: str, *, from_cache: bool) -> None:
    log = RunLog(f"{repo}_{start}_to_{end}_run.log")
    if from_cache:
        payload = load_raw_cache(repo, start, end)
        log.info("loaded _raw.json; skipping Phase 1/2")
    else:
        catalogs = discover_prs(repo, start, end)          # Phase 1
        rows = fetch_pr_details(catalogs, workers=4)       # Phase 2
        payload = save_raw_cache(repo, start, end, rows)
    write_person_summary(payload)  # rollups always recomputed on export
```

Later the same day, `f19d1fc` documented search date filters and API rate limits after large-repo runs hit limits - document the ceiling; do not silent-truncate. Cache stale summaries bit us when column schema moved ([auditing agent analytics](https://contextfirst.blog/auditing-agent-wrote-analytics/)).

---

## 4\. Named pattern: raw cache + run logs

> **Raw cache + run logs** \- parallel fetch for speed; cached raw JSON for reproducibility; a run log so two staff meetings can be diffed.

| Artifact            | Role                                        |
| ------------------- | ------------------------------------------- |
| Phase 1 catalogs    | Scope of the window                         |
| Phase 2 detail rows | Expensive facts                             |
| \_raw.json          | Accelerant for rebuilds - not eternal truth |
| \_run.log           | Evidence when someone challenges the sheet  |

**What broke:** treating cache as source of truth after schema change. Rollups must recompute on export; the raw payload is the durable layer.

---

## 5\. What I'd do as a lead

1. **Require a `_run.log` attachment when someone challenges a report.** Memory is not an audit trail.
2. **Treat cache as accelerant, not source of truth.** Recompute rollups on export; re-fetch when the column contract changes.
3. **Cap parallel workers in CI; full parallelism for local month-end runs.** Rate limits are an ops problem - tune them explicitly.

---

### Further reading

- [Your Slack script just shipped](https://contextfirst.blog/script-to-cli-spreadsheet-product/) \- `9d47215` · CLI threshold
- [The afternoon we shipped three wrong metrics](https://contextfirst.blog/afternoon-three-wrong-metrics/) \- same-day definition fixes
- [Field notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) \- series intro
- github-analysis: commit `336332d` · 2026-06-01 · `docs/architecture.md`

---

![Iheatu Chris Wogu](https://avatars.githubusercontent.com/u/3457705?s=176&v=4)

## About the author

**Iheatu Chris Wogu** is an engineering manager focused on launch programs, honest engineering metrics, and **engineering-led, agent-accelerated** delivery - researching where work queues up when agents widen dev output, then building proof, gates, and measurement that scale.

Field notes live on [First Context](https://contextfirst.blog/). Values treatise: [engineering manager values](https://contextfirst.blog/engineering-manager-values/).

**LinkedIn:** [linkedin.com/in/iheatu-chris-wogu](https://www.linkedin.com/in/iheatu-chris-wogu/?ref=contextfirst.blog)