Gabriel Brown
89761a7da3
Agent instructions, skills, SSH host aliases and expansion triggers are worth having identical on every machine one person owns, and belong in none of the shared configuration. They live in user/ now, with a manifest saying where each piece goes and a link-user stage that puts it there. That stage does nothing unless the machine said yes. Somebody who clones Panama to try the desktop keeps their own ~/.claude/CLAUDE.md exactly where it was; the question names the destinations and defaults to no. Anything displaced goes to config/old rather than being deleted. ~/.claude/CLAUDE.md and ~/.codex/AGENTS.md were byte-identical copies of one file, which is the drift this exists to prevent. Also adds the vitals toggles for the battery and Claude usage readouts, which had preferences and no way to reach them.
85 lines
4.1 KiB
Markdown
85 lines
4.1 KiB
Markdown
# Verifying behavior-shaping guidance
|
|
|
|
The verification branch of [`writing-for-agents`](SKILL.md). Read it when the document is meant to
|
|
**change what an agent does**: a rule expected to hold under pressure, or wording meant to change the
|
|
shape of an output.
|
|
|
|
Reference skills do not need this. A vocabulary, an API guide, or a list of definitions is verified
|
|
by reading it back against the ladder and the pruning rules. Running this method on reference is
|
|
ceremony that buys nothing.
|
|
|
|
## Establish the baseline first
|
|
|
|
You cannot fix a failure you have not watched happen. Before writing the guidance, run the task
|
|
**without** it in a fresh context and record what the agent actually does, in its own words.
|
|
|
|
**The control decides whether to write anything at all.** If the no-guidance run does not exhibit the
|
|
failure, there is nothing to fix: stop, and do not author the guidance. Most guidance that turns out
|
|
to be a no-op would have died here. This step removes more work than it adds.
|
|
|
|
Where the failure does appear, capture the rationalizations verbatim. They are the raw material: a
|
|
discipline rule works by answering the specific excuse the agent reached for, not the excuse you
|
|
imagined it would.
|
|
|
|
## Micro-test the wording
|
|
|
|
Full scenario runs are the final gate, but they are slow per iteration. Test the wording itself
|
|
first:
|
|
|
|
1. **One fresh-context sample per call.** The system prompt is the realistic context the guidance
|
|
will live in, meaning the whole skill, not the guidance in isolation. The user message is a task
|
|
that tempts the failure.
|
|
2. **Always run the no-guidance control alongside.** Same rule as above: no failure in the control,
|
|
no guidance needed.
|
|
3. **Five or more reps per variant.** Single samples lie.
|
|
4. **Read every flagged match yourself.** Score programmatically if you like, but template echoes and
|
|
quoted counter-examples masquerade as hits, and automated counts overstate both failure and
|
|
success.
|
|
5. **Treat variance as a metric.** When guidance lands, the reps converge on the same shape. Five
|
|
different interpretations across five reps means the wording is not binding: tighten the form
|
|
before adding words.
|
|
|
|
Micro-tests verify wording. They do not replace a real run for a rule that has to survive pressure.
|
|
|
|
## Pressure-test a discipline rule
|
|
|
|
A rule an agent follows when nothing is at stake is untested. Combine pressures in one scenario:
|
|
|
|
- **Time**: the change is urgent, something is broken in production.
|
|
- **Sunk cost**: the work is nearly done and the rule would discard it.
|
|
- **Authority**: someone senior said to skip it.
|
|
- **Exhaustion**: it is the eighth iteration of a long loop.
|
|
|
|
Success is the agent following the rule with all of them stacked. Each new rationalization it
|
|
invents is the next thing to close.
|
|
|
|
## Close the loopholes
|
|
|
|
Do not just state the rule; forbid the specific workaround the baseline produced. "Delete it" invites
|
|
"I'll keep it as reference"; "Delete it. Not as reference, not to adapt while rewriting. Delete means
|
|
delete" does not.
|
|
|
|
Two structures earn their place in a discipline skill, and only there:
|
|
|
|
- **A rationalization table**, one row per excuse from the baseline runs, each answered:
|
|
|
|
| Excuse | Reality |
|
|
|---|---|
|
|
| "Too simple to test" | Simple code breaks. The test takes 30 seconds. |
|
|
|
|
- **A red-flags list** of the thoughts that mean the agent is mid-rationalization, so it can catch
|
|
itself: *"this is different because..."*, *"I already checked it manually"*.
|
|
|
|
Add the foundational line early, because it closes a whole class of argument at once:
|
|
**violating the letter of the rule is violating its spirit.**
|
|
|
|
Both structures are the row-one form from `SKILL.md`. Reaching for them on a shaping failure makes
|
|
the output worse, so confirm the failure is discipline before writing either.
|
|
|
|
## Done when
|
|
|
|
- A no-guidance control was run, and it exhibited the failure the guidance addresses.
|
|
- The guidance answers rationalizations observed in that run, not invented ones.
|
|
- Reps with the guidance converge on one shape rather than five.
|
|
- For a discipline rule: it holds with time, sunk cost, and authority pressure stacked.
|