Panama/user/agents/skills/diagnosing-bugs/SKILL.md

---
name: diagnosing-bugs
description: Use when something is broken, throwing, failing, flaky, or slow, and before proposing any fix. Covers hard bugs, intermittent failures, regressions between two known-good states, and performance problems.
---

# Diagnosing Bugs

A discipline for hard bugs. Skip a phase only with an explicit reason.

```
NO FIX WITHOUT A LOOP THAT GOES RED ON THIS BUG
```

Reading code to build a theory before that loop exists is the exact failure this skill prevents.

Where the repo has a `CONTEXT.md`, read it first for a mental model of the modules involved, and
check any ADRs covering the area you are about to touch.

## Redact

This skill has you show commands, outputs, and captured artifacts. **Redact every secret before
showing it**, writing `<REDACTED>` in its place. Build loops against environment variables so the
credential stays in the environment rather than in the transcript. Captured artifacts carry auth
headers: quote only the lines carrying signal.

If the redacted output is not enough to diagnose the bug, say so and ask.

## Phase 1: Build a feedback loop

**This is the skill. Everything after it is mechanical.** With a **tight** pass/fail signal that goes
red on *this* bug, you will find the cause: bisection, hypothesis testing, and instrumentation all
just consume it. Without one, no amount of staring at code will save you.

Spend disproportionate effort here. Be aggressive, be creative, refuse to give up.

### Ways to build one, in roughly this order

1. **A failing test** at whatever seam reaches the bug: unit, integration, or end-to-end.
2. **An HTTP script** against a running dev server: `curl` with the exact payload.
3. **A single function call in isolation.** For a Convex backend, `bunx convex run <function>` with a
   fixture argument; for a CLI, an invocation with a fixture input diffed against known-good output.
4. **A headless browser script** (Playwright) that drives the UI and asserts on DOM, console, or
   network.
5. **Replay a captured trace.** Save a real request, payload, or event log to disk and replay it
   through the code path in isolation.
6. **A throwaway harness.** The minimum subset of the system that exercises the bug in one call.
7. **A property or fuzz loop.** For "sometimes wrong output", run a thousand random inputs and look
   for the failure mode.
8. **A bisection harness.** Where the bug appeared between two known-good states (commit, dataset,
   dependency version), automate "boot at state X, check, repeat" so `git bisect run` can drive it.
9. **A differential loop.** Same input through old versus new, or through two configs, diffing the
   outputs.
10. **A human-in-the-loop script.** Last resort, where a human must click. Drive *them* with
    [scripts/hitl-loop.template.sh](scripts/hitl-loop.template.sh) so the loop stays structured and
    its captured output still feeds back to you.

### Tighten it

Treat the loop as a product. Once you have *a* loop, make it **tight**:

- **Faster**: cache setup, skip unrelated init, narrow the scope.
- **Sharper**: assert on the specific symptom, not on "did not crash".
- **More deterministic**: pin time, seed randomness, isolate the filesystem, freeze the network.

A flaky 30-second loop is barely better than nothing. A deterministic 2-second one is a superpower.

### Non-deterministic bugs

The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger a hundred times,
parallelise, add stress, narrow the timing window, inject sleeps. A 50% flake is debuggable; 1% is
not. Keep raising the rate until it is.

### When you genuinely cannot build one

Stop and say so explicitly. List what you tried, then ask for one of: access to an environment that
reproduces it, a redacted captured artifact (HAR file, log dump, screen recording with timestamps),
or permission to add temporary production instrumentation. **Do not proceed to hypothesise without a
loop.**

### Completion criterion

Phase 1 is done when you can name **one command** that you have **already run at least once**,
showing the invocation and its redacted output, and that is:

- [ ] **Red-capable**: drives the real code path and asserts the user's exact symptom, so it goes red
      on this bug and green once fixed. Not "runs without erroring".
- [ ] **Deterministic**: the same verdict every run, or a pinned high reproduction rate.
- [ ] **Fast**: seconds, not minutes.
- [ ] **Agent-runnable**: you can run it unattended, with a human involved only through the HITL
      script.

No red-capable command, no Phase 2.

## Phase 2: Reproduce, then minimise

Run the loop and watch it go red. Confirm the failure is the one the **user** described rather than a
different failure that happens to live nearby, that it reproduces across runs, and that you have
captured the exact symptom so later phases can prove the fix addressed it.

Then **minimise**: shrink to the smallest scenario that still goes red. Cut inputs, callers, config,
data, and steps **one at a time**, re-running after each cut. Done when every remaining element is
load-bearing, meaning removing any one of them turns the loop green.

A minimal repro shrinks the hypothesis space in Phase 3 and becomes the clean regression test in
Phase 5. Do not proceed until you have reproduced **and** minimised.

## Phase 3: Hypothesise

Generate **3-5 ranked hypotheses before testing any of them.** Generating one at a time anchors you
on the first plausible idea, which is the most common way a debugging session goes long.

Each must be **falsifiable**, stating the prediction it makes:

> "If `<X>` is the cause, then `<changing Y>` makes the bug disappear / `<changing Z>` makes it worse."

Cannot state the prediction? The hypothesis is a vibe. Discard or sharpen it.

Two cheap sources of hypotheses before you start guessing:

- **What changed recently?** `git log`, recent commits, new dependencies, config or environment
  differences. A regression usually has a commit attached to it.
- **Where does the bad value come from?** Where the error surfaces deep in a call chain, trace
  backward to the original trigger rather than fixing where it appears. See
  [TRACING.md](TRACING.md).

**Show the ranked list before testing.** Domain knowledge re-ranks it instantly ("we deployed a
change to #3 yesterday"), or rules one out entirely. Cheap checkpoint, big saving. Do not block on
it: proceed with your ranking if nobody is around.

## Phase 4: Instrument

Each probe maps to a specific prediction from Phase 3. **Change one variable at a time.**

1. **A debugger or REPL** where the environment supports it. One breakpoint beats ten logs.
2. **Targeted logs** at the boundaries that distinguish the hypotheses.
3. Never "log everything and grep".

**Tag every debug log** with a unique prefix, `[DEBUG-a4f2]`, so cleanup later is a single grep.
Untagged logs survive forever; tagged logs die.

**Multi-component systems.** Where the failure crosses boundaries (client → server function →
database, or CI → build → deploy), instrument *each boundary* before theorising about any one
component: log what enters, log what exits, verify config propagation. One run then shows **which
layer** breaks, which turns a whole-system mystery into a single-component bug.

**Performance branch.** For a regression in speed, logs are usually the wrong tool. Establish a
baseline measurement (a timing harness, a profiler, a query plan), then bisect against it. Measure
first, fix second.

## Phase 5: Fix, with a regression test

Write the regression test **before the fix**, but only where a **correct seam** exists for it.

A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call
site. Where the only available seam is too shallow (a single-caller test when the bug needs several
callers, a unit test that cannot reproduce the chain that triggered it), a test there gives false
confidence.

**Where no correct seam exists, that is itself the finding.** Note it: the architecture is preventing
the bug from being locked down. Call the Skill tool with "codebase-design" to name what would have to
change.

Where a correct seam does exist:

1. Turn the minimised repro into a failing test at that seam
2. Watch it fail
3. Apply the fix, addressing the root cause rather than the symptom
4. Watch it pass
5. Re-run the Phase 1 loop against the original, un-minimised scenario

**One fix at a time.** No "while I'm here" improvements, no bundled refactoring: you will not know
which change worked.

## When three fixes have failed, question the architecture

Count your attempts. Under three, return to Phase 1 and re-analyse with what you now know. **At three
or more, stop.** Do not attempt a fourth.

The pattern that says the architecture is wrong rather than the hypothesis:

- Each fix reveals new shared state or coupling somewhere else
- Each fix creates a new symptom elsewhere
- Fixing it "properly" would require massive refactoring

That is not a failed hypothesis, it is a wrong shape. Raise it and discuss before touching anything
else.

## Phase 6: Cleanup

Required before saying it is done:

- [ ] The original repro no longer reproduces: re-run the Phase 1 loop
- [ ] The regression test passes, or the absence of a correct seam is documented
- [ ] Every `[DEBUG-...]` probe is removed, verified by grepping the prefix
- [ ] Throwaway harnesses are deleted, or moved somewhere clearly marked
- [ ] The hypothesis that turned out correct is stated in the commit message, so the next person
      learns what you learned

## Red flags

| Thought | Reality |
|---|---|
| "Quick fix now, investigate later" | The first fix sets the pattern. Do it right from the start. |
| "It's probably X, let me just change it" | Seeing a symptom is not understanding a cause. |
| "Let me try changing this and see" | That is guessing. Build the loop. |
| "This bug is simple, it doesn't need the process" | Simple bugs have root causes too, and the process is fast on them. |
| "It's an emergency, no time for this" | Systematic is faster than guess-and-check thrashing. Always. |
| "I'll write the test after I confirm the fix works" | Untested fixes do not stick. |
| "I'll change these three things and re-run" | Then you cannot tell which one mattered. |
| "One more fix attempt" (after two) | Three failures means the architecture, not the hypothesis. |

Being told "stop guessing", "is that not actually happening?", or "are we stuck?" means you are
already here. Return to Phase 1.