Debugging on Nick Liu - Software Engineer

The installer stamped an ID it never read. It reinstalled over itself.

nickboy@users.noreply.github.com (Nick Liu) — Sun, 06 Sep 2026 00:00:00 +0000

An installer writes a script. It stamps that script with `HERDR_INTEGRATION_ID=claude`, an identifier that exists for no reason other than to say "I made this". Then, to decide whether it has already run, it ignores that stamp entirely and compares the rendered command string in a config file. Rewrite the command to an equivalent form and the installer no longer recognises its own work, so it installs a second copy alongside the first.

I hit three bugs in three days. They looked unrelated until I wrote them down next to each other: one in someone else’s tool, two in a plugin of mine. They share a shape, and the shape is worth more than any of the three fixes.

State inferred from a derived value rather than read from the component that owns it.

Case one: the installer that did not recognise its own work
#

herdr integration install claude wires herdr into Claude Code by adding a hook to settings.json. To be re-runnable, it has to answer one question first: have I already done this?

It answers it by rendering the command string it would write, and checking whether that exact string is present in settings.json.

That works for exactly as long as nobody touches the file. Rewrite the command to an equivalent portable form, say because you want it to work on a machine where the binary lives somewhere else, and the string no longer matches. The installer concludes it has never run, and appends a duplicate hook. Now every event fires the integration twice.

The frustrating part is that the correct answer was already in the file. The installer generates a script and stamps it:

HERDR_INTEGRATION_ID=claude

That is an unambiguous identity, authored by the installer, for the installer, surviving any amount of reformatting of the command that invokes it. And the check does not consult it.

I reproduced this first-hand in a sandboxed CLAUDE_CONFIG_DIR rather than against my real config, which I recommend for anything that mutates a settings file you care about, and reported it upstream.

Case two: the rename check that could never repair anything
#

I wrote a herdr plugin during my trial of it called tab-smart-rename, which renames each tab after whatever the agent in it is currently working on.

To avoid renaming a tab that is already correctly named, it compared the new name against a cached session name it had stored the last time it ran.

Consider what happens the moment anything else changes the tab label. Another plugin, a manual rename, herdr itself. The cache still holds the last name this plugin set. The new name matches the cache. So the plugin concludes nothing has changed and does nothing, and the label stays wrong forever.

The check is not comparing “what the tab says” to “what the tab should say”. It is comparing “what I last decided” to “what I would decide now”, which are two views of the same opinion. From its own point of view nothing had changed, and it was right. It just was not looking at the tab.

Case three: the tab id that came from the environment
#

Same plugin. To know which tab to rename, it read a tab id from an inherited environment variable.

Environment variables are inherited by children, which is the entire point of them, and it makes them a terrible source of truth for “where am I right now”. Launch a session from one pane while working in another and the new process carries the launching pane’s id. The plugin then confidently renames a tab that has nothing to do with the agent it is watching.

This is a bug class I have hit before and written about, in a different costume. The CLAUDECODE=1 leak had the same root: an environment flag treated as a statement about the current process when it is really a statement about some ancestor. The reliable answer there was to walk the process tree and let ancestry decide, and the reliable answer here is to ask the multiplexer which pane this is.

The shape
#

Bug	It compared	The owner of the truth	Result
Integration installer	rendered command string	`HERDR_INTEGRATION_ID` it wrote itself	duplicate hook, fires twice
Rename check	its own cached name	the tab’s actual label	can never repair a drifted label
Tab id resolution	inherited env var	the pane the process is really in	renames a tab it does not own

In each case the derivative drifts from the thing it stands for, and nothing notices, because the code is comparing the derivative to itself.

Why this class is so quiet
#

The tell is that the failure is silent and self-consistent. Every component reports success.

The installer succeeded. It wrote a valid hook to a valid file and exited zero. The rename check succeeded, and it was correct given its premises. The tab id resolution succeeded, and it renamed a tab that genuinely exists.

There is no exception, no error path, no log line, and nothing to alert on, because from the inside nothing went wrong. You find these by noticing an outcome that is wrong in the world, not by noticing a failure in the system, and that is a much slower loop. It is the same reason a shell function that swallows commands with exit 0 is so much worse than one that crashes.

There is a second-order effect that makes it worse still. A derived value that drifts tends to drift further over time, because the mechanism that would correct it is the mechanism that is broken. The rename check is the clearest version: the longer the label is wrong, the more certain the plugin is that everything is fine.

What to do instead
#

The fix in all three cases is the same sentence: ask the component that owns the state.

The installer should read its own stamp. It went to the trouble of writing an identity; identity is what identity is for.
The rename check should read the tab’s current label, which is one call away, rather than what it remembers setting.
The tab id should come from the multiplexer’s notion of the current pane, not from an environment variable that any parent may have set.

The general test is a question you can ask about any comparison in a codebase: if the world changed underneath this check, would it notice? If both sides of the comparison come from the same place, the answer is no, and you have written a check that can only ever agree with itself.

A related smell, and the one that caught the installer for me: if a component authors an identifier and then does not read it, either the identifier is dead code or the reader is looking in the wrong place. It is almost never the former.

Lessons
#

Read state from the component that owns it. A derived value is a copy, and copies drift.
If a check compares two things that both come from you, it can only agree with itself. Ask whether the check would notice a change made by anything else.
An identity you write and never read is a bug waiting to be filed. The installer stamped HERDR_INTEGRATION_ID and then decided idempotency by string-matching a command it also wrote.
Idempotency means recognising your own work after someone reformats it. Matching on rendered output makes any equivalent rewrite look like a fresh install.
Environment variables describe an ancestor, not the current process. For “where am I now”, ask the thing that knows.
This class fails silently and self-consistently, so every component reports success. You will find it by noticing a wrong outcome in the world, never by noticing a failure in the system.
Reproduce config-mutating bugs in a sandboxed config directory. CLAUDE_CONFIG_DIR cost me nothing and meant the duplicate hook landed somewhere disposable.

References
#

herdr: https://herdr.dev (source: https://github.com/herdrdev/herdr)
Claude Code settings and hooks reference: https://code.claude.com/docs/en/hooks
The trial the two plugin bugs came out of: Running six agents made tab patrol my biggest time sink. So: a herdr trial.
The same silence, different costume: My commands vanished with exit 0. The culprit was a file named env.
The project that surfaced the installer bug: I deleted six subsystems by swapping one dependency.

Every context ring read 10%. Two bugs, and fixing one hid the other.

nickboy@users.noreply.github.com (Nick Liu) — Tue, 01 Sep 2026 00:00:00 +0000

Each agent key on the Stream Deck draws a ring showing how full that session's context window is. Mine sat at 10% for days. Every session, regardless of activity, regardless of how long the agent had been grinding. I found the bug, fixed it, watched agents start reporting correct percentages every single turn, and the ring still said 10%. That was the good part, because it meant there were two.

Tested with Claude Code 2.1.x · macOS

Ten percent is a suspicious number. Not zero, which would say “nothing ever arrived”. Not a plausible-looking 37%, which would say “this works and your session is small”. Ten percent is round, and round numbers in a display that should be noisy mean the display is not reading anything.

It turned out to be an orphan. Something wrote 10% into the cache a long time ago and nothing had overwritten it since, on any session, for days.

Symptom

day 1

Ring pinned at 10% on every agent slot. Suspiciously round.
Bug one found

day 1

The statusline delegate reads .context_window.percentUsed. Claude Code emits .context_window.used_percentage. The extraction silently yields nothing, every turn.
Fixed, and wrong

day 2

Agents now report a correct percentage every turn. The ring still reads 10%.
Three live probes

day 2

pane.report_metadata emits nothing. pane.agent_status_changed carries no tokens. Tokens are assigned in exactly one place.
Root cause

day 2

The daemon snapshotted once on connect. Every ring froze at whatever it read at that instant.

Bug one: a field name that does not exist
#

HerdDeck ships a statusline delegate. It sits in front of whatever statusline you already have, forwards the per-turn JSON payload on to herdr, and passes your original output through untouched. It read this:

.context_window.percentUsed

Claude Code emits .context_window.used_percentage.

Getting the field name wrong is unremarkable. How it failed is the part worth keeping. A statusline script runs on every turn of every session, and a statusline script that throws takes your prompt down with it, so the whole thing is written to fall through silently on every path. Wrong field name, malformed JSON, jq missing, all of it produces the same result: no error, no output, no report, forever.

Silent-by-design is correct for a statusline. It is also a category of bug where nothing anywhere tells you the value never arrived, and the only symptom is a number somewhere else that stops changing.

There is a small joke at my expense here. I wrote a post about this exact payload in May, and that post prints the schema with the correct field name in it. My own published documentation had it right. The code I shipped three months later did not.

The fix keeps the wrong name deliberately, which was not my first instinct:

# Field name: Claude Code emits `.context_window.used_percentage`. An
# earlier revision of this script read `.percentUsed`, which does not
# exist — the extraction silently yielded nothing and the donut never
# lit up. `percentUsed` is kept only as a fallback so the script also
# works against any build that ever used it; `used_percentage` wins.
PCT="$(printf '%s' "$INPUT" | jq -r '.context_window | (.used_percentage // .percentUsed // empty)' 2>/dev/null)"

Two tests pin the behaviour: one that the fallback works, one that used_percentage wins when both are present. A silent fallback needs a test asserting the ordering, or it is just a second way to be quietly wrong.

The fix worked and the ring did not move
#

This is the part of the debugging that went wrong, and it is why the post exists.

After the field-name fix, every agent reported a correct, changing percentage on every turn. I could watch the numbers arrive. And the ring still sat at 10%.

For an hour or so I was debugging the wrong thing, because the evidence pointed backwards. A fix that produces correct data and no visible change reads as “the fix did not work”. I went back over the delegate. I checked the JSON. I checked that the daemon was receiving it. All fine, all pointless, because the second bug had been there the entire time and the first one had been hiding it perfectly.

A cosmetic symptom with two independent causes is much harder than a symptom with one hard failure. With one cause, fixing it resolves the symptom and you are done. With two, fixing the first one changes nothing observable, and the natural inference is that you were wrong about the first one.

Three probes against a live server
#

So I stopped reading code and started asking the server questions. Three probes, against herdr 0.8.0, with a real agent working in a real pane.

Probe	Result
Report metadata to a pane while subscribed to `pane.updated`	zero events for that pane. `pane.report_metadata` emits nothing at all
Inspect `pane.agent_status_changed` payload	carries only `{pane_id, agent_status}`. No tokens
Grep the cache for token assignment	exactly one site: the `session.snapshot` seed

Put those together and the answer is unavoidable. The daemon snapshotted once on connect, and every ring froze at whatever the snapshot happened to contain at that instant. Nothing in the protocol was ever going to push a token count at it.

`pane.updated` looked like the answer
#

There is an event called pane.updated and it does carry tokens. It looked like the fix for about ten minutes.

It is not, for two reasons, and the second one is decisive.

It is chatty. It fires on scroll offsets and status flicker, and the measurement recorded in the source is roughly 25 events in 2.5 seconds from a single active pane. Subscribing to that to catch a number that changes once per turn is a poor trade.

And it does not fire for a metadata report at all. That is the case that matters. A pane whose context moved while it sat idle would simply never be announced, which is precisely the situation where you want the ring to be honest, because you are looking at a key rather than at the terminal.

Rejecting an event on measurement rather than on taste is worth doing explicitly. pane.updated is the obvious answer, it is in the schema, and it would have shipped a ring that worked most of the time and lied when you were not watching.

Re-read, merge, do not re-seed
#

The fix is a periodic snapshot on a 10-second default, and the merge is the load-bearing part:

/** Token re-read period; 0 disables it. Default 10s — roughly
 * twice Claude Code's default statusline refresh, so the donut
 * trails the real percentage by at most one report. */
tokensRefreshMs?: number;
// ...
this.tokensRefreshMs = opts?.tokensRefreshMs ?? 10_000;

The timer’s snapshot can be older than pushed events the cache has already applied. Re-seeding from it would let a stale read undo a pane close that a newer event had already processed, resurrecting a pane that is gone. There is a test pinning exactly that, and its comment says why:

// The refresh re-reads a full snapshot but must merge tokens ONLY:
// re-seeding would let a stale snapshot undo a close that a newer
// pushed event already applied.

One correction to my own framing, because I described this as “tokens only” for a while and it stopped being true. The merge now also carries the focused flag, deliberately and with its own justification comment. If you go looking at that function expecting the name to be the specification, it will not be.

Lessons
#

A suspiciously round number in a noisy display means the display is not reading anything. Zero says “nothing arrived”, and a plausible number says “this works”. Round says “orphan”.
Silent-by-design is right for a statusline and it removes your only failure signal. A wrong field name in a script built never to break a prompt produces no error, no output, and no report, on every turn, indefinitely.
When a correct fix produces no visible change, consider that you fixed one of two bugs before concluding you fixed none. Two independent causes of one cosmetic symptom is the case where evidence points backwards.
Probe the live server instead of reading its schema. “Does this call emit an event at all” is a question a schema cannot answer and a five-minute experiment can.
Reject the obvious event on measurement, not taste, and write down the measurement. pane.updated carried the field I wanted and did not fire for the case I needed.
A periodic re-read must merge, never re-seed. A refresh that is older than the events you have already applied will happily undo them.
If a fallback exists, test the ordering, not just the fallback. Otherwise it is a second way to be quietly wrong.

References
#

Claude Code statusline docs, source of truth for the payload schema: https://code.claude.com/docs/en/statusline
The post that printed the correct field name three months before I got it wrong: The Claude Code statusline is a per-turn telemetry side channel
The rebuild this ring belongs to: I deleted six subsystems by swapping one dependency.
HerdDeck statusline delegate: scripts/herddeck-statusline.sh
HerdDeck state cache, including mergeTokensFromSnapshot: packages/daemon/src/stateCache.ts
HerdDeck target monitor, the refresh timer: packages/daemon/src/herdr/monitor.ts

Five wrong answers in one day. One nearly deleted 10 GB of coursework.

nickboy@users.noreply.github.com (Nick Liu) — Mon, 17 Aug 2026 00:00:00 +0000

`mdls kMDItemLastUsedDate` returned `(null)` for Microsoft Word. I read the null as "never opened" and put Office on a removal list: 10.1 GB, four apps. One last check saved me. My home directory held 150 Office documents, a conference presentation edited two weeks earlier, and a PowerPoint lock file, which only exists while the file is open. The proof that the null was misleading had been sitting in my own diagnostic report for an hour.

That was one of five. In a single day of hardening this machine, five different tools told me things that were not true. None of the answers looked like an error. Each one arrived as a clean, confident finding, and under each one a check had quietly failed or asked the wrong question. All five had the same shape underneath. Once I could name the shape, I stopped falling for it.

Case 1: null means “I don’t know”, not “no”
#

The plan was simple: free disk space by removing apps that had never been used. The evidence came from Spotlight metadata:

$ mdls -name kMDItemLastUsedDate -raw "/Applications/Google Chrome.app"
2026-08-15 20:18:18 +0000
$ mdls -name kMDItemLastUsedDate -raw "/Applications/Microsoft Word.app"
(null)

Chrome had a date. Word did not. The conclusion looked solid, because the same command told the two apps apart: Chrome is in use, Word is not.

It was not solid. kMDItemLastUsedDate is null when an app has never been opened, and when Spotlight’s metadata store has been reset. My store had been reset. I even knew that, because four hours earlier I had written this line into my own diagnostic report:

12:14:16.629 lsd _CSStoreCreateWithURL failed with error ... "The file
 com.apple.LaunchServices-33554481-v2.csstore couldn't be opened
 because there is no such file."

The LaunchServices database was rebuilding at boot. In one section of my report, this was a strange log line. In another section, its consequences became hard evidence for deletion. I never connected the two.

What answers “is this app still needed” is user data:

$ find ~/Documents ~/Desktop ~/Downloads -iname '*.docx' -o -iname '*.pptx' | wc -l
 150
$ ls -t ~/Documents/**/*.pptx | head -3
cs6460-final-GT.pptx
~$cs6460-milestone2-GT.pptx # lock file: exists only while the file is open
cs6460-milestone2-GT.pptx

Documents, lock files, container contents. Things a person made on purpose. An index field is different: the OS is free to throw it away and rebuild it, and while that happens the field tells you nothing.

Case 2: a query that never ran is not a query that found nothing
#

My environment audit reported “no git remotes configured” for a repository that had pushed four signed commits and opened a pull request earlier the same day.

The check behind that sentence was this:

$ cd ~ && git remote -v
fatal: not a git repository (or any of the parent directories): .git

These dotfiles are managed with yadm, and yadm keeps its bare repository in ~/.local/share/yadm/repo.git. $HOME is the work tree, not a git directory. Plain git in $HOME cannot answer the question at all. Then “cannot answer” got recorded as “the answer is none”.

$ yadm remote -v
origin git@github.com:nickboy/dotfiles.git (fetch)
origin git@github.com:nickboy/dotfiles.git (push)

The tool did not lie. fatal: not a git repository and empty output are two different results, but the code reading them treated both as “nothing there”.

Case 3: a 404 from the wrong endpoint
#

Same audit, same day: “branch protection could not be determined”. The check had asked the classic API:

$ gh api repos/nickboy/dotfiles/branches/master/protection
{"message":"Branch not protected","status":"404"}

That is a real 404, and the message reads like a finding. But GitHub has two protection mechanisms, and this repo uses the newer one:

$ gh api repos/nickboy/dotfiles/rulesets --jq '.[] | "\(.name) \(.enforcement)"'
Main Branch Protection active

A branch protected by a ruleset returns 404 from /branches/*/protection. Any tool that only checks the old endpoint will report every ruleset-protected branch as unprotected, in a sentence that sounds like it actually checked.

Asking the right endpoint paid off twice, because the true answer held a detail the false one would have hidden. The ruleset requires a pull request and four status checks, with no bypass actors. But required_approving_review_count is 0, and there is no signature requirement. “Protected” here means “must go through a PR and CI”. It does not mean “must be reviewed”. My signed commits are a habit; the server never checks them. Recording how strong a control is beats recording that one exists.

Case 4: the number was real, the units were not
#

My health check started with a headline number: WindowServer was eating 62.4% CPU, the biggest single CPU user on the machine. The number came from ps:

$ ps -Ao pid,%cpu,comm | grep WindowServer
 436 62.4 .../SkyLight.framework/Resources/WindowServer

On macOS, ps %cpu is a decaying average. It leans hard toward the last minute, not toward the life of the process. The cumulative figure is CPU time divided by elapsed time:

$ ps -o time=,etime= -p 436
 26:03.57 01:02:14 # 1563s CPU / 3734s elapsed = 41.9%

So 42%, not 62%. The next morning brought a bigger correction, after sixteen hours of normal use:

WindowServer 6.6% (16.4 hours)
ghostty 3.3% (16.4 hours)

Six point six. My original 62% came from an hour in which I was the load. I was running diagnostics nonstop, with Activity Monitor open and a backup copying files in the background. I had measured the observer and blamed the system. The whole “WindowServer is the problem” section of that report was an accident of timing.

Case 5: the benchmark that measured nothing happening
#

ll felt slow. The alias behind it is eza with a long list of flags, including --git. I benchmarked it and got a number that ended the discussion:

eza --all --long --git --icons ... 8.4 ms

Eight milliseconds. Nothing to fix. Except the same command, in the same directory, also reported this:

$ eza --all --long --git ... ~/Downloads | wc -lc
 0 0

Zero lines, zero bytes, exit 0, and 128 ms of system time. The benchmark had run inside the sandboxed shell my coding agent uses, and that sandbox blocks the metadata and git reads that --long --git needs. eza was making syscalls, and the sandbox refused every one of them. My benchmark timed the result with great precision: a program doing nothing, very fast.

I re-ran everything in a real terminal:

directory	size	with `--git`	without
`nickboy.github.io`	47 entries, 263 tracked	68.8 ms	13.9 ms
`claudedeck`	125 tracked	49.0 ms	20.8 ms
`Downloads`	622 entries, not a repo	49.6 ms	39.8 ms

Inside a repo, --git costs 2x to 5x, and the cost follows the repository, not the directory being listed: eza resolves git status for every tracked file to fill one column. Listing a 47-entry folder inside a 263-file repo pays for all 263 files. My everyday alias paid that price in every project directory, which is exactly where I type it most.

The flag moved to an opt-in alias, llg. It also came out of ls, l, lt and tree completely, because the git column only renders in long view. For those four, the output is byte-identical with and without the flag. I verified that instead of assuming it:

$ [[ $(eza --git ... | md5) == $(eza ... | md5) ]] && echo "no-op"
no-op

Dead config, carried for months, costing a git query on every call.

The common shape
#

Five cases, one sentence:

A query that failed and a query that found nothing are different results.

Every case was a tool reporting an absence. In none of them had anyone checked whether the question ever reached the thing it was asking about.

“Absence of evidence is not evidence of absence” is an old phrase. The working version is more direct: evidence gathered by a broken method is not evidence at all, and a broken method usually exits 0.

The test that could not fail
#

The same day turned up a bug that had survived for a year because the test guarding it was written backwards.

Ghostty on macOS reads both ~/.config/ghostty/config and ~/Library/Application Support/com.mitchellh.ghostty/config. My bootstrap script had symlinked the second to the first, so the same file was parsed twice. For single-value keys the last value wins, so nothing looked wrong. But repeatable keys append:

$ ghostty +show-config | grep -c '^custom-shader = '
4 # the file declares 2

Two extra full-screen shader passes on every frame, for a year. And the test suite stayed green the whole time, because it contained this:

GHOSTTY_LINK="$HOME/Library/Application Support/com.mitchellh.ghostty/config"
if [ -e "$GHOSTTY_LINK" ] || [ -L "$GHOSTTY_LINK" ]; then
 run_test "Ghostty config is symlink" "[ -L \"$GHOSTTY_LINK\" ]"
fi

Two problems in four lines. The assertion checks the opposite of the correct rule, because the symlink should not exist at all. And the assertion sits inside a guard, so when I deleted the symlink, the test did not fail. It silently stopped running. The first post in this series was a catalog of safety nets that had quietly died. This test is the sharpest example I have found since: it disappeared at the exact moment it finally had something to say.

The replacement tests the property instead of the mechanism:

run_test "Ghostty config is not parsed twice (custom-shader not duplicated)" \
 "[ '$LOADED_SHADERS' -eq '$CFG_SHADERS' ]"

Whatever causes a double parse next time (this symlink, a different symlink, an include directive, some future Ghostty change), the count stops matching and the test goes red. I put the symlink back once, on purpose, and watched the test fail before I trusted it.

Lessons
#

Null, empty, and 404 are answers about the query, not always about the world. Before treating one as a finding, confirm the query reached the right place.
Decide “is this app still needed” from data a person created. An index the OS can reset is not a record of anything.
Know which average your tool reports. ps %cpu on macOS leans toward the last minute; cumulative load is CPU time divided by elapsed time.
Measure where the work really happens. A sandbox that blocks the syscalls under test produces a fast, precise, meaningless number.
Never measure a system while you are the heaviest thing running on it.
Test the property, not the mechanism. The symlink was one way to get a double parse; the count catches all of them.

References
#

eza (--git cost measured on v0.23.5)
yadm (bare repo location: ~/.local/share/yadm/repo.git)
GitHub rulesets API (rulesets are invisible to the older /branches/*/protection endpoint)
Ghostty configuration (macOS reads both the XDG path and the Application Support path)
ps(1) on macOS (%cpu is documented as a decaying average)
The repo and its test suite: nickboy/dotfiles

My commands vanished with exit 0. The culprit was a file named env.

nickboy@users.noreply.github.com (Nick Liu) — Thu, 13 Aug 2026 00:00:00 +0000

`env -u VAR command` did nothing. Exit code 0, no output, no error. A different command, same thing. A different variable, same thing. Any invocation that started with `env` just quietly evaporated. What finally made the problem visible was a `git init` that reported success while creating no `.git` directory at all.

That was a year ago. The case closed last month, and the culprit was not uv, not some third-party installer, not anything exotic. It was this repo’s own bootstrap script. The thing that caught it was the regression test I had written for the original incident.

This post is three silent failures from the same repo, plus that ending. They share a shape worth naming.

Case 1: the executable that should have been sourced
#

uv’s installer drops a file at ~/.local/bin/env. It is a PATH snippet, a few export lines meant to be sourced by your shell profile. It is not a program.

At some point, something gave it an executable bit. I could not figure out what at the time, so that part of the case stayed open. But the consequences were mechanical from there:

Every env -u X cmd, every shebang-adjacent env trick, every script that spelled out env for portability: all of them ran a five-line PATH snippet and stopped. With exit 0, because the snippet ran fine. It was just never asked to run your command, since sourcing semantics and executing semantics are different contracts.

The fix was one line:

chmod -x ~/.local/bin/env

Finding the line took the better part of a debugging session. So the incident became a permanent test, with the story written into the comment:

# uv's installer drops a PATH snippet at ~/.local/bin/env meant to be
# SOURCED; if it ever becomes executable it shadows /usr/bin/env (PATH
# puts ~/.local/bin second) and silently swallows every 'env ... cmd'
# invocation with exit 0. That exact trap cost a debugging session.
run_test "env resolves to /usr/bin/env (no executable shim)" \
 "[ \"\$(command -v env)\" = /usr/bin/env ]"

There is also a generalized version: a scan that flags any executable in ~/.local/bin shadowing a system command. The specific trap gets a named test; the class of trap gets a sweep.

Case 2: the fixture that reformatted my real repo
#

I was writing a fixture test for the secret scanner: create a throwaway git repo in a temp directory, plant a fake credential, confirm the scanner catches it. Standard stuff.

The test runs inside the yadm pre-commit hook. yadm exports GIT_DIR and GIT_WORK_TREE for its own purposes, and here is the part I did not respect enough: those environment variables outrank git -C <path>. The -C flag changes the working directory; the environment still decides which repository you are talking to.

So the fixture’s git init reinitialized the real yadm repository. It flipped core.bare and broke the repo’s configuration. Recovery turned out to be one setting, with zero history lost:

yadm gitconfig core.bare false

But the lesson went into the suite in capital letters:

# CRITICAL: every call strips GIT_DIR/GIT_WORK_TREE — the yadm
# pre_commit hook exports them, and an ambient GIT_DIR silently
# redirects fixture git commands at the REAL yadm repo (this once
# flipped its core.bare).
GL_GIT="env -u GIT_DIR -u GIT_WORK_TREE git"
$GL_GIT -C "$GL_TMP" init -q

The suite now also runs a hostile-environment simulation: it deliberately plants a decoy GIT_DIR and confirms the fixtures ignore it. And the fake credential is assembled at runtime, so no secret-shaped literal ever exists in the repo for a scanner to trip on:

printf 'aws_key = "%s%s"\n' "AKIA" "ZZZQK9X2M4P7L3TQ" \
 > "$GL_TMP/leak.txt"

Case 3: the file manager that shrugged
#

yazi 26 renamed a fetcher config field from id to group. My config still said id. When yazi hits a config it cannot parse, it does not error; it falls back to its presets and carries on looking almost normal. The git and mactag columns just stopped being enriched, and I only noticed weeks later, by accident.

The canary here is two words:

yazi --version # exits 1 when the config fails to parse

That exit-code behavior is something I tested, not something I read somewhere. It now runs in the test suite and again after upgrades, as a schema check. When yazi next renames a field, the failure will be loud and same-day instead of silent and discovered by luck.

The ending: a year later, the case closed itself
#

Here is where case 1 gets its resolution, and it did not come from detective work.

An unrelated idempotency audit of the bootstrap script added one final step to it: after bootstrapping, run the full test suite as self-verification. The very first self-verification run came back red.

The bootstrap script contained a blanket chmod +x ~/.local/bin/*. The “scripts must be executable” convention, applied to a whole directory. It had just re-armed the uv trap, a year after I defused it, and it had presumably been the original armer all along. One line took down three tests at once: the env resolution test, the PATH shadow sweep, and the gitleaks fixtures, whose env -u calls were being swallowed again.

# before: the whole directory gets the executable bit.
# uv's source-me env snippet is in there, so the trap gets
# re-armed on every bootstrap run.
chmod +x ~/.local/bin/*

# after: only scripts yadm actually tracks get chmod'd.
# untracked files (including uv's env) never gain the bit.

The full lifecycle reads like this: symptom, defusal, regression test, then a year of quiet, then the test catches the original root cause in the act of reoffending, and the root cause gets fixed. The gap between step three and step four was twelve months. The gap between “bootstrap gained self-verification” and “cold case closed” was a few minutes.

The common shape
#

All three cases had no error message. All three returned success. All three were a system helpfully falling back to something instead of failing: a snippet standing in for a binary, an environment variable standing in for a flag, presets standing in for a config.

Silent fallback is the expensive kind of bug. A loud failure costs you five minutes. A quiet substitution costs you weeks, because nothing marks the moment where reality diverged from your model of it.

Institutionalizing the paranoia
#

Every incident in this post exists in the test suite as a named check. The suite grew from 74 checks to 105 across this overhaul, and almost all of that growth is “we got bitten here” items. Case 1 is the proof that the habit pays twice: the test written to protect the future is the same one that closed the past.

The other byproduct is a small document, docs/upgrade-watch.md, pairing every config file that tracks an upstream schema with a canary that fails loudly:

| Config file | Canary |
| --------------------------- | ----------------------- |
| ~/.config/yazi/yazi.toml | yazi --version |
| ~/.config/atuin/config.toml | atuin doctor |
| ~/.tmux.conf | throwaway-server parse |
| ~/.local/bin/env (uv) | suite `env` check |

Lessons
#

Environment variables outrank the flags on your command line. git -C does not protect you from an ambient GIT_DIR.
A file that is meant to be sourced becomes a command-swallowing trap the moment it gains an executable bit and a good PATH position.
Tools that silently fall back to defaults on a bad config need a canary with a real exit code. Test the canary’s behavior; do not assume it.
Write the regression test even when the root cause is still unknown. Mine waited a year and then caught the culprit red-handed.
Bootstrap scripts should only touch files they own. A wildcard chmod is a loaded gun pointed at the future.

References
#

uv (the installer that ships the ~/.local/bin/env snippet)
yazi (the config-schema rename landed in yazi 26; exit-code canary behavior verified on this machine)
yadm hooks documentation (the pre-commit hook environment where GIT_DIR/GIT_WORK_TREE are exported)
The repo and its test suite: nickboy/dotfiles

My diagrams rendered on refresh and vanished on click. The head never loaded.

nickboy@users.noreply.github.com (Nick Liu) — Tue, 11 Aug 2026 00:00:00 +0000

A reader clicking from my homepage to a post with a diagram got a block of raw mermaid source. The same reader pasting that post's URL directly got a rendered diagram. Same page, same build, same browser. The difference was the click, and the bug had been live on this site for months across every diagram, every math formula, and every chart, because I had only ever tested pages by loading them directly.

Tested with Blowfish 2.10x · htmx 2.0.10

Two features, both reasonable, one collision
#

This site has htmx’s hx-boost on the body: internal navigation swaps page content in place instead of doing full page loads, which keeps transitions smooth. Separately, the Blowfish theme is smart about heavy libraries: mermaid, KaTeX, and Chart.js bundles are only included on pages that use them, injected into the <head> of exactly those pages.

Each feature is sensible alone. Together they have a hole in the middle:

hx-boost replaces the body and updates the title; it does not merge the incoming head. So when you navigate from a page without diagrams to a page with them, the one script tag that would have loaded the mermaid bundle is in the part of the document that gets thrown away. My re-render code even had a guard, if (typeof mermaid !== 'undefined'), which is exactly the kind of guard that turns a loud failure into a silent one. It skipped, politely, every time.

Direct loads worked because the full document parses normally. Which is why months of my own testing missed it: I test pages by URL. Readers arrive by link.

The fix: steal the bundle URL from the response you already have
#

The swapped-in page’s HTML passes through htmx’s hands, head included, before the body is extracted. The fix reads the bundle URL out of that response and injects the script once:

function ensureVendorThenRender(evt, srcPattern, isReady, render) {
 if (isReady()) { render(); return; }
 var xhr = evt && evt.detail && evt.detail.xhr;
 var html = xhr && xhr.responseText;
 var match = html && html.match(srcPattern);
 if (!match || injectedVendors[match[0]]) return;
 injectedVendors[match[0]] = true;
 var s = document.createElement("script");
 s.src = match[0];
 s.onload = render;
 document.head.appendChild(s);
}

Called from the htmx:afterSettle handler with a pattern like /\/js\/mermaid\.bundle\.[^"']+\.js/, once per library. Fingerprinted filenames come along for free since the pattern matches whatever hash the current build produced.

Charts needed a second layer. The theme’s chart shortcode wraps its new Chart(...) call in a DOMContentLoaded listener, and after an htmx swap that event is ancient history; the listener registers and waits forever. So the chart path re-inserts each inline chart script with addEventListener temporarily patched to invoke DOMContentLoaded callbacks immediately, plus a Chart.getChart() check so nothing draws twice. It is not elegant. It is honest about the constraint: the shortcode’s markup is the theme’s, and the fix has to meet it where it is.

If you want the pattern’s shape as a takeaway: conditionally loaded assets and partial navigation are individually fine and jointly broken, and neither feature’s documentation will warn you about the other. Any site combining per-page bundles with hx-boost, Turbo, or SPA-style routing has this class of bug available to it.

The tests were their own debugging story
#

I added three Playwright regression tests, each walking the failing path on purpose: land on a listing page without the library, click through to a post with it, assert the rendered artifact exists (pre.mermaid svg, .katex, a live Chart.getChart() instance). Two traps surfaced on the way, and they cost more time than the fix itself.

First, the clicks kept landing on the wrong page. Blowfish’s article cards stretch the post link across the whole card and stack tag links on top of it, and Playwright’s coordinate-based click aimed at the card’s center, which happened to be a tag chip. The katex test spent several runs waiting for math on /tags/georgia-tech/. The cure is locator.dispatchEvent('click'), which delivers the event to the exact anchor regardless of what floats above it, plus a waitForURL assertion so a wrong destination fails fast with a readable error instead of a timeout.

Second, my shell habit was eating the exit code. npx playwright test | tail reports tail’s status, which is always zero, and a failing suite sailed through my terminal looking green twice before I noticed. Redirect to a file and check the real exit code. CI was never fooled, because CI does not pipe; only my eyeballs were.

The affected pages, for the record, included every post in my ML math series and the diagram in the hooks post. All fixed by the same three functions, all guarded by the same three tests.

Lessons
#

Per-page asset loading plus body-swap navigation is a bug class, not a bug. If your head is conditional and your navigation is partial, budget for the collision.
A typeof x !== 'undefined' guard converts a crash you would notice into a skip you will not. Guards need a fallback path, not just an exit.
Test the click path, not only the URL path. Direct loads exercise the document pipeline; in-site navigation exercises a completely different one.
In tests, dispatch events to the element you mean. Coordinate clicks trust the visual stack, and the visual stack lies on layered layouts.
Never pipe a test runner’s output when you care about its exit code.

References
#

htmx: hx-boost attribute
Chart.js: Chart.getChart API
Playwright: dispatchEvent
The fix and tests: layouts/partials/extend-footer.html and tests/site-enhancements.spec.ts in this site’s repo

Debugging on Nick Liu - Software Engineer

The installer stamped an ID it never read. It reinstalled over itself.

Case one: the installer that did not recognise its own work #

Case two: the rename check that could never repair anything #

Case three: the tab id that came from the environment #

The shape #

Why this class is so quiet #

What to do instead #

Lessons #

References #

Every context ring read 10%. Two bugs, and fixing one hid the other.

Symptom

day 1

Bug one found

day 1

Fixed, and wrong

day 2

Three live probes

day 2

Root cause

day 2

Bug one: a field name that does not exist #

The fix worked and the ring did not move #

Three probes against a live server #

pane.updated looked like the answer #

Re-read, merge, do not re-seed #

Lessons #

References #

Five wrong answers in one day. One nearly deleted 10 GB of coursework.

Case 1: null means “I don’t know”, not “no” #

Case 2: a query that never ran is not a query that found nothing #

Case 3: a 404 from the wrong endpoint #

Case 4: the number was real, the units were not #

Case 5: the benchmark that measured nothing happening #

The common shape #

The test that could not fail #

Lessons #

References #

My commands vanished with exit 0. The culprit was a file named env.

Case 1: the executable that should have been sourced #

Case 2: the fixture that reformatted my real repo #

Case 3: the file manager that shrugged #

The ending: a year later, the case closed itself #

The common shape #

Institutionalizing the paranoia #

Lessons #

References #

My diagrams rendered on refresh and vanished on click. The head never loaded.

Two features, both reasonable, one collision #

The fix: steal the bundle URL from the response you already have #

The tests were their own debugging story #

Lessons #

References #

Case one: the installer that did not recognise its own work
#

Case two: the rename check that could never repair anything
#

Case three: the tab id that came from the environment
#

The shape
#

Why this class is so quiet
#

What to do instead
#

Lessons
#

References
#

Bug one: a field name that does not exist
#

The fix worked and the ring did not move
#

Three probes against a live server
#

`pane.updated` looked like the answer
#

Re-read, merge, do not re-seed
#

Lessons
#

References
#

Case 1: null means “I don’t know”, not “no”
#

Case 2: a query that never ran is not a query that found nothing
#

Case 3: a 404 from the wrong endpoint
#

Case 4: the number was real, the units were not
#

Case 5: the benchmark that measured nothing happening
#

The common shape
#

The test that could not fail
#

Lessons
#

References
#

Case 1: the executable that should have been sourced
#

Case 2: the fixture that reformatted my real repo
#

Case 3: the file manager that shrugged
#

The ending: a year later, the case closed itself
#

The common shape
#

Institutionalizing the paranoia
#

Lessons
#

References
#

Two features, both reasonable, one collision
#

The fix: steal the bundle URL from the response you already have
#

The tests were their own debugging story
#

Lessons
#

References
#