|
|
|
@@ -0,0 +1,213 @@
|
|
|
|
|
---
|
|
|
|
|
name: diagnosing-bugs
|
|
|
|
|
description: Use when something is broken, throwing, failing, flaky, or slow, and before proposing any fix. Covers hard bugs, intermittent failures, regressions between two known-good states, and performance problems.
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Diagnosing Bugs
|
|
|
|
|
|
|
|
|
|
A discipline for hard bugs. Skip a phase only with an explicit reason.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
NO FIX WITHOUT A LOOP THAT GOES RED ON THIS BUG
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Reading code to build a theory before that loop exists is the exact failure this skill prevents.
|
|
|
|
|
|
|
|
|
|
Where the repo has a `CONTEXT.md`, read it first for a mental model of the modules involved, and
|
|
|
|
|
check any ADRs covering the area you are about to touch.
|
|
|
|
|
|
|
|
|
|
## Redact
|
|
|
|
|
|
|
|
|
|
This skill has you show commands, outputs, and captured artifacts. **Redact every secret before
|
|
|
|
|
showing it**, writing `<REDACTED>` in its place. Build loops against environment variables so the
|
|
|
|
|
credential stays in the environment rather than in the transcript. Captured artifacts carry auth
|
|
|
|
|
headers: quote only the lines carrying signal.
|
|
|
|
|
|
|
|
|
|
If the redacted output is not enough to diagnose the bug, say so and ask.
|
|
|
|
|
|
|
|
|
|
## Phase 1: Build a feedback loop
|
|
|
|
|
|
|
|
|
|
**This is the skill. Everything after it is mechanical.** With a **tight** pass/fail signal that goes
|
|
|
|
|
red on *this* bug, you will find the cause: bisection, hypothesis testing, and instrumentation all
|
|
|
|
|
just consume it. Without one, no amount of staring at code will save you.
|
|
|
|
|
|
|
|
|
|
Spend disproportionate effort here. Be aggressive, be creative, refuse to give up.
|
|
|
|
|
|
|
|
|
|
### Ways to build one, in roughly this order
|
|
|
|
|
|
|
|
|
|
1. **A failing test** at whatever seam reaches the bug: unit, integration, or end-to-end.
|
|
|
|
|
2. **An HTTP script** against a running dev server: `curl` with the exact payload.
|
|
|
|
|
3. **A single function call in isolation.** For a Convex backend, `bunx convex run <function>` with a
|
|
|
|
|
fixture argument; for a CLI, an invocation with a fixture input diffed against known-good output.
|
|
|
|
|
4. **A headless browser script** (Playwright) that drives the UI and asserts on DOM, console, or
|
|
|
|
|
network.
|
|
|
|
|
5. **Replay a captured trace.** Save a real request, payload, or event log to disk and replay it
|
|
|
|
|
through the code path in isolation.
|
|
|
|
|
6. **A throwaway harness.** The minimum subset of the system that exercises the bug in one call.
|
|
|
|
|
7. **A property or fuzz loop.** For "sometimes wrong output", run a thousand random inputs and look
|
|
|
|
|
for the failure mode.
|
|
|
|
|
8. **A bisection harness.** Where the bug appeared between two known-good states (commit, dataset,
|
|
|
|
|
dependency version), automate "boot at state X, check, repeat" so `git bisect run` can drive it.
|
|
|
|
|
9. **A differential loop.** Same input through old versus new, or through two configs, diffing the
|
|
|
|
|
outputs.
|
|
|
|
|
10. **A human-in-the-loop script.** Last resort, where a human must click. Drive *them* with
|
|
|
|
|
[scripts/hitl-loop.template.sh](scripts/hitl-loop.template.sh) so the loop stays structured and
|
|
|
|
|
its captured output still feeds back to you.
|
|
|
|
|
|
|
|
|
|
### Tighten it
|
|
|
|
|
|
|
|
|
|
Treat the loop as a product. Once you have *a* loop, make it **tight**:
|
|
|
|
|
|
|
|
|
|
- **Faster**: cache setup, skip unrelated init, narrow the scope.
|
|
|
|
|
- **Sharper**: assert on the specific symptom, not on "did not crash".
|
|
|
|
|
- **More deterministic**: pin time, seed randomness, isolate the filesystem, freeze the network.
|
|
|
|
|
|
|
|
|
|
A flaky 30-second loop is barely better than nothing. A deterministic 2-second one is a superpower.
|
|
|
|
|
|
|
|
|
|
### Non-deterministic bugs
|
|
|
|
|
|
|
|
|
|
The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger a hundred times,
|
|
|
|
|
parallelise, add stress, narrow the timing window, inject sleeps. A 50% flake is debuggable; 1% is
|
|
|
|
|
not. Keep raising the rate until it is.
|
|
|
|
|
|
|
|
|
|
### When you genuinely cannot build one
|
|
|
|
|
|
|
|
|
|
Stop and say so explicitly. List what you tried, then ask for one of: access to an environment that
|
|
|
|
|
reproduces it, a redacted captured artifact (HAR file, log dump, screen recording with timestamps),
|
|
|
|
|
or permission to add temporary production instrumentation. **Do not proceed to hypothesise without a
|
|
|
|
|
loop.**
|
|
|
|
|
|
|
|
|
|
### Completion criterion
|
|
|
|
|
|
|
|
|
|
Phase 1 is done when you can name **one command** that you have **already run at least once**,
|
|
|
|
|
showing the invocation and its redacted output, and that is:
|
|
|
|
|
|
|
|
|
|
- [ ] **Red-capable**: drives the real code path and asserts the user's exact symptom, so it goes red
|
|
|
|
|
on this bug and green once fixed. Not "runs without erroring".
|
|
|
|
|
- [ ] **Deterministic**: the same verdict every run, or a pinned high reproduction rate.
|
|
|
|
|
- [ ] **Fast**: seconds, not minutes.
|
|
|
|
|
- [ ] **Agent-runnable**: you can run it unattended, with a human involved only through the HITL
|
|
|
|
|
script.
|
|
|
|
|
|
|
|
|
|
No red-capable command, no Phase 2.
|
|
|
|
|
|
|
|
|
|
## Phase 2: Reproduce, then minimise
|
|
|
|
|
|
|
|
|
|
Run the loop and watch it go red. Confirm the failure is the one the **user** described rather than a
|
|
|
|
|
different failure that happens to live nearby, that it reproduces across runs, and that you have
|
|
|
|
|
captured the exact symptom so later phases can prove the fix addressed it.
|
|
|
|
|
|
|
|
|
|
Then **minimise**: shrink to the smallest scenario that still goes red. Cut inputs, callers, config,
|
|
|
|
|
data, and steps **one at a time**, re-running after each cut. Done when every remaining element is
|
|
|
|
|
load-bearing, meaning removing any one of them turns the loop green.
|
|
|
|
|
|
|
|
|
|
A minimal repro shrinks the hypothesis space in Phase 3 and becomes the clean regression test in
|
|
|
|
|
Phase 5. Do not proceed until you have reproduced **and** minimised.
|
|
|
|
|
|
|
|
|
|
## Phase 3: Hypothesise
|
|
|
|
|
|
|
|
|
|
Generate **3-5 ranked hypotheses before testing any of them.** Generating one at a time anchors you
|
|
|
|
|
on the first plausible idea, which is the most common way a debugging session goes long.
|
|
|
|
|
|
|
|
|
|
Each must be **falsifiable**, stating the prediction it makes:
|
|
|
|
|
|
|
|
|
|
> "If `<X>` is the cause, then `<changing Y>` makes the bug disappear / `<changing Z>` makes it worse."
|
|
|
|
|
|
|
|
|
|
Cannot state the prediction? The hypothesis is a vibe. Discard or sharpen it.
|
|
|
|
|
|
|
|
|
|
Two cheap sources of hypotheses before you start guessing:
|
|
|
|
|
|
|
|
|
|
- **What changed recently?** `git log`, recent commits, new dependencies, config or environment
|
|
|
|
|
differences. A regression usually has a commit attached to it.
|
|
|
|
|
- **Where does the bad value come from?** Where the error surfaces deep in a call chain, trace
|
|
|
|
|
backward to the original trigger rather than fixing where it appears. See
|
|
|
|
|
[TRACING.md](TRACING.md).
|
|
|
|
|
|
|
|
|
|
**Show the ranked list before testing.** Domain knowledge re-ranks it instantly ("we deployed a
|
|
|
|
|
change to #3 yesterday"), or rules one out entirely. Cheap checkpoint, big saving. Do not block on
|
|
|
|
|
it: proceed with your ranking if nobody is around.
|
|
|
|
|
|
|
|
|
|
## Phase 4: Instrument
|
|
|
|
|
|
|
|
|
|
Each probe maps to a specific prediction from Phase 3. **Change one variable at a time.**
|
|
|
|
|
|
|
|
|
|
1. **A debugger or REPL** where the environment supports it. One breakpoint beats ten logs.
|
|
|
|
|
2. **Targeted logs** at the boundaries that distinguish the hypotheses.
|
|
|
|
|
3. Never "log everything and grep".
|
|
|
|
|
|
|
|
|
|
**Tag every debug log** with a unique prefix, `[DEBUG-a4f2]`, so cleanup later is a single grep.
|
|
|
|
|
Untagged logs survive forever; tagged logs die.
|
|
|
|
|
|
|
|
|
|
**Multi-component systems.** Where the failure crosses boundaries (client → server function →
|
|
|
|
|
database, or CI → build → deploy), instrument *each boundary* before theorising about any one
|
|
|
|
|
component: log what enters, log what exits, verify config propagation. One run then shows **which
|
|
|
|
|
layer** breaks, which turns a whole-system mystery into a single-component bug.
|
|
|
|
|
|
|
|
|
|
**Performance branch.** For a regression in speed, logs are usually the wrong tool. Establish a
|
|
|
|
|
baseline measurement (a timing harness, a profiler, a query plan), then bisect against it. Measure
|
|
|
|
|
first, fix second.
|
|
|
|
|
|
|
|
|
|
## Phase 5: Fix, with a regression test
|
|
|
|
|
|
|
|
|
|
Write the regression test **before the fix**, but only where a **correct seam** exists for it.
|
|
|
|
|
|
|
|
|
|
A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call
|
|
|
|
|
site. Where the only available seam is too shallow (a single-caller test when the bug needs several
|
|
|
|
|
callers, a unit test that cannot reproduce the chain that triggered it), a test there gives false
|
|
|
|
|
confidence.
|
|
|
|
|
|
|
|
|
|
**Where no correct seam exists, that is itself the finding.** Note it: the architecture is preventing
|
|
|
|
|
the bug from being locked down. Call the Skill tool with "codebase-design" to name what would have to
|
|
|
|
|
change.
|
|
|
|
|
|
|
|
|
|
Where a correct seam does exist:
|
|
|
|
|
|
|
|
|
|
1. Turn the minimised repro into a failing test at that seam
|
|
|
|
|
2. Watch it fail
|
|
|
|
|
3. Apply the fix, addressing the root cause rather than the symptom
|
|
|
|
|
4. Watch it pass
|
|
|
|
|
5. Re-run the Phase 1 loop against the original, un-minimised scenario
|
|
|
|
|
|
|
|
|
|
**One fix at a time.** No "while I'm here" improvements, no bundled refactoring: you will not know
|
|
|
|
|
which change worked.
|
|
|
|
|
|
|
|
|
|
## When three fixes have failed, question the architecture
|
|
|
|
|
|
|
|
|
|
Count your attempts. Under three, return to Phase 1 and re-analyse with what you now know. **At three
|
|
|
|
|
or more, stop.** Do not attempt a fourth.
|
|
|
|
|
|
|
|
|
|
The pattern that says the architecture is wrong rather than the hypothesis:
|
|
|
|
|
|
|
|
|
|
- Each fix reveals new shared state or coupling somewhere else
|
|
|
|
|
- Each fix creates a new symptom elsewhere
|
|
|
|
|
- Fixing it "properly" would require massive refactoring
|
|
|
|
|
|
|
|
|
|
That is not a failed hypothesis, it is a wrong shape. Raise it and discuss before touching anything
|
|
|
|
|
else.
|
|
|
|
|
|
|
|
|
|
## Phase 6: Cleanup
|
|
|
|
|
|
|
|
|
|
Required before saying it is done:
|
|
|
|
|
|
|
|
|
|
- [ ] The original repro no longer reproduces: re-run the Phase 1 loop
|
|
|
|
|
- [ ] The regression test passes, or the absence of a correct seam is documented
|
|
|
|
|
- [ ] Every `[DEBUG-...]` probe is removed, verified by grepping the prefix
|
|
|
|
|
- [ ] Throwaway harnesses are deleted, or moved somewhere clearly marked
|
|
|
|
|
- [ ] The hypothesis that turned out correct is stated in the commit message, so the next person
|
|
|
|
|
learns what you learned
|
|
|
|
|
|
|
|
|
|
## Red flags
|
|
|
|
|
|
|
|
|
|
| Thought | Reality |
|
|
|
|
|
|---|---|
|
|
|
|
|
| "Quick fix now, investigate later" | The first fix sets the pattern. Do it right from the start. |
|
|
|
|
|
| "It's probably X, let me just change it" | Seeing a symptom is not understanding a cause. |
|
|
|
|
|
| "Let me try changing this and see" | That is guessing. Build the loop. |
|
|
|
|
|
| "This bug is simple, it doesn't need the process" | Simple bugs have root causes too, and the process is fast on them. |
|
|
|
|
|
| "It's an emergency, no time for this" | Systematic is faster than guess-and-check thrashing. Always. |
|
|
|
|
|
| "I'll write the test after I confirm the fix works" | Untested fixes do not stick. |
|
|
|
|
|
| "I'll change these three things and re-run" | Then you cannot tell which one mattered. |
|
|
|
|
|
| "One more fix attempt" (after two) | Three failures means the architecture, not the hypothesis. |
|
|
|
|
|
|
|
|
|
|
Being told "stop guessing", "is that not actually happening?", or "are we stuck?" means you are
|
|
|
|
|
already here. Return to Phase 1.
|