Testing on Nick Liu - Software Engineer

My dotfiles had a no-exceptions test gate. It had never run once.

nickboy@users.noreply.github.com (Nick Liu) — Wed, 12 Aug 2026 00:00:00 +0000

My dotfiles repo has a CLAUDE.md, and the CLAUDE.md has a rule in bold: every commit must pass the test suite, no exceptions. Within the first hour of an audit this July, I learned that this rule had been enforced exactly zero times since the day it was written. The hook file existed, its contents were correct, it even had its executable bit. It was just sitting at a path that yadm stopped reading a major version ago.

No error message. No warning. To yadm, a hook in the wrong place and no hook at all are the same thing.

This was supposed to be a cleanup
#

The plan was modest: go through all 82 tracked files and delete some stale configuration. What actually happened was closer to an archaeology dig, and every layer had something dead in it that looked alive from the surface. Five findings, in the order they turned up.

Finding 1: the hook on the dead path
#

yadm 2.x read hooks from ~/.yadm/hooks/pre-commit, with a hyphen. yadm 3.x reads them from $YADM_DIR/hooks/pre_<command>, so a commit hook lives at ~/.config/yadm/hooks/pre_commit, with an underscore. My machine runs yadm 3.5. My hook was still parked at the 2.x path.

# yadm 2.x (dead, but the file was still there)
~/.yadm/hooks/pre-commit

# what yadm 3.x actually reads: $YADM_DIR/hooks/pre_<command>
~/.config/yadm/hooks/pre_commit

A directory move and one character, hyphen to underscore, and a rule labeled “no exceptions” died quietly for about a year. The fix took a minute. The part worth keeping is the comment that now sits at the top of the hook, so the path contract is written where the next person (me) will trip over it:

#!/bin/bash

# yadm pre-commit hook (yadm 3.x reads $YADM_DIR/hooks/pre_<command>,
# i.e. this file must live at ~/.config/yadm/hooks/pre_commit).
# Runs the full test suite (which includes markdownlint) before
# allowing a commit. Bypass in emergencies: yadm commit --no-verify

Finding 2: thirteen CI jobs, zero teeth
#

The repo’s CI had 13 jobs. Thirteen green circles is a comforting sight. Then I looked at which ones could actually stop a bad commit, and the answer was none that mattered: the critical steps had continue-on-error or a trailing || true scattered on them.

Thirteen jobs bought me a lot of reassurance and almost no protection. Whether a pipeline will actually refuse a bad commit is a separate property from how much of it there is, and nothing about the green circles tells you which one you have.

Finding 3: launchctl said yes and did nothing
#

The daily-maintenance launch agent had been silently stopped since July 3. The root cause is a genuinely nasty little contract: launchctl load exits 0 even when the agent is disabled in the override database, so the command reports success while the service stays down.

# exits 0 even when the agent is disabled in the override database
launchctl load ~/Library/LaunchAgents/com.daily-maintenance.plist

# what actually works: enable explicitly, then bootstrap
launchctl enable gui/$UID/com.daily-maintenance
bash ~/install-daily-maintenance.sh

“The command succeeded” and “the service is running” are separate claims. I now treat any launchd interaction that only checks exit codes as unverified.

Finding 4: the zombie updater
#

A duplicate brew-update automation had been running since February 2023. Two and a half years of doing the same work twice, and nobody noticed, because both copies succeeded. Redundant success is invisible in a way redundant failure never is.

Finding 5: the editor frozen in time, with a full paper trail
#

This one is the reason the post exists. Neovim had been frozen on the June 30 nightly build for five weeks before any symptom appeared. When it finally surfaced, it did so sideways: neo-tree called nvim_win_resize, an API added in a newer nightly than the one I was running, and blew up with “attempt to call a nil value”.

The causal chain took a while to untangle:

The wedged install I confirmed against bob’s own dev-branch source rather than guessing. And there is a general trap in here for anyone on nightlies:

" trap: every 0.13-dev nightly satisfies this condition,
" so it gates nothing, including the weeks before the API existed
if has("nvim-0.13")
 " ...
endif

has("nvim-0.13") is true for every 0.13-dev build, first to last. As a version gate for a feature that landed mid-cycle, it is no gate at all.

The immediate fix was three parts: unwedge the stuck install, point PATH back at the real bob, and add a boot canary to the test suite:

# if the editor cannot start, this goes red today, not in five weeks
# when some plugin happens to call an API that is not there
nvim --headless -c q

But the part that actually stings is this: the failure was in the maintenance log every day for a month, described accurately each time. It just landed in a file nobody reads. So the structural fix went to the output side. The maintenance summary now sends a desktop notification whenever any task fails, and the message layout assumes it will be truncated: log location first, at most three failures listed, the rest folded into a count.

# log location first (notifications get truncated, so the most
# important information leads), max 3 failures, rest folded
body="Failures logged in $LOG"
[ "$total" -gt 3 ] && body="$body (+$((total - 3)) more)"

What shipped
#

The cleanup turned into 22 single-concern commits, one problem per commit so that any of them can be reverted alone.

Metric	Before	After	Verified how
CI jobs	13	5	coverage went up, not down: fed the gates a broken `.sh`, a broken `.md`, and a broken KDL config; all three rejected
Local test suite	none	74 checks	born in this audit; at 105 today
zsh startup	398 ms	254 ms	`hyperfine --warmup 3 'zsh -i -c exit'`
Working pre-commit gate	no	yes	commits with failing tests are refused

Two of those rows deserve a sentence. The CI shrink is the point of finding 2: fewer jobs, strictly more protection, and the protection is proven by deliberately feeding the pipeline broken files rather than by looking at green circles. And the startup number comes from hyperfine because an unreproducible number is a mood, not a measurement.

Lessons
#

Automation you have never seen fail is indistinguishable from automation that does not exist. Their observable behavior is identical.
Validate a gate by deliberately breaking something it should catch. A gate that has only ever seen good inputs is unproven by construction.
Failures must arrive somewhere you actually look. A log line is not an alert; it is a diary entry.
Counting CI jobs measures reassurance. Counting what CI refuses measures protection.
When a tool renames its config path across a major version, every consumer of the old path fails silently. Write the path contract into the file itself.

References
#

yadm hooks documentation (the pre_<command> naming and $YADM_DIR lookup)
hyperfine (startup-time benchmarking)
bob (the Neovim version manager; wedge behavior confirmed against its dev-branch source)
The repo this all happened in: nickboy/dotfiles
launchctl behavior observed on macOS on this machine; see man launchctl for the enable/bootstrap subcommands

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

nickboy@users.noreply.github.com (Nick Liu) — Wed, 05 Aug 2026 00:00:00 +0000

The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules.

Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:

Honesty rules:
 * The model is shown only `item.spec`; the hidden tests are written into the sandbox
 alongside the candidate but never given to the model.
 * A pre-flight check runs the *reference* solution against the hidden tests. If the
 reference does not pass, the item is malformed and the audit aborts (a HARNESS
 problem, never silently scored). This guarantees the test file is sound, so during
 the audit any candidate error (syntax, import, exception, failing assertion) is a
 legitimate FAIL for pass@k.
 * `HARNESS_ERROR` is reserved for runs where the sandbox produced no usable verdict
 (no report and no failing exit code). Such attempts are excluded from pass@k and
 reported separately, so they masquerade as neither pass nor fail.

Three rules. Each one exists because of a specific way a harness can lie.

Rule 1: run the reference solution first
#

Before any model attempt is scored, the harness runs the problem’s own reference solution against the hidden tests:

def preflight(item: Item, sandbox: Sandbox, limits: Limits) -> int:
 """Verify the reference solution passes the hidden tests. Returns the test count."""
 run = _run(item, item.reference_code, sandbox, limits)
 outcome, _passed, total, detail = _classify(run)
 if outcome is not Outcome.PASS:
 raise HarnessError(
 "pre-flight failed: the reference solution did not pass the hidden tests "
 f"({detail}). Fix the item before auditing.\n"
 f"stderr:\n{run.stderr[:2000]}"
 )
 return total

Without this gate, a broken test file is indistinguishable from a model that cannot solve the problem. You would conclude the problem is AI-resistant when your own tests do not run. In a project whose whole point was hunting for AI-resistant problems, this is the exact wrong conclusion to hand out for free. The gate turns that scenario into a loud abort instead of a quiet score of zero.

The gate also buys something subtle. Once the reference is known to pass, any error from a candidate solution (syntax error, import error, exception, failing assertion) is a legitimate FAIL. I do not have to wonder whose fault a stack trace is. The test file was proven sound minutes earlier.

Rule 2: three outcomes, not two
#

Most harnesses classify an attempt as pass or fail. Mine has a third verdict, and the classification code shows where it comes from:

raw = run.files.get(_JSON_REPORT)
if raw is None:
 # No machine-readable report. The test file is known-good (pre-flight), so a
 # non-zero exit means the candidate broke collection/execution -> FAIL. A clean
 # exit with no report is a genuine harness anomaly.
 if run.exit_code not in (0, None):
 return Outcome.FAIL, 0, 0, f"no json report; pytest exit {run.exit_code}"
 return Outcome.HARNESS_ERROR, 0, 0, "no json report produced"

With only two buckets, a crash has to land somewhere, and both landings are lies. Count it as a pass and your scores inflate. Count it as a fail and infrastructure flakes become model failures. HARNESS_ERROR attempts are excluded from the pass@k denominator and reported separately, so they masquerade as neither.

Across all 1,095 attempts, the harness error counter finished at zero. I want to be careful about what that proves. It does not prove the third bucket was unnecessary. It proves the infrastructure held, and I can only say that because the bucket existed to catch anything that did not. A counter that stays at zero is not vestigial. It is the evidence.

The scoring itself uses the unbiased pass@k estimator from the Codex paper rather than the naive “did any of the first k pass”, which is biased. That function validates k against the sample count and returns 1.0 when a pass is guaranteed by counting alone.

Rule 3: report what the sandbox does not guarantee
#

Every run record carries a capability string:

"capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False"

The honest part is the False. macOS does not allow a hard memory cap the usual way, so the tool bounds memory by polling. It would have been easy to leave that field out, or worse, to print True because polling exists. Instead the record says: this boundary is best-effort. If a reviewer ever asks whether a candidate could have exceeded memory limits undetected, the answer is in the data, not in my memory of how the sandbox worked.

A full record, for shape:

{
 "item_id": "count-multiples-01",
 "model": "claude-haiku-4-5",
 "date": "2026-07-16",
 "n_scored": 20,
 "n_pass": 0,
 "n_harness_error": 0,
 "sandbox_backend": "local",
 "capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False",
 "passk": { "1": 0.0, "5": 0.0 },
 "gate_k": 1,
 "theta": 0.2,
 "resistant": true,
 "construct_clean": true,
 "combined_verdict": "AI-RESISTANT (construct-valid)"
}

Model and date on every record. Part 1 showed the same problem scoring 0.00, 0.05, and 0.20 on three different days. Without the stamp, those would have been one contradictory number. With it, they are three measurements.

Lessons
#

A harness problem must never be silently scored. Abort loudly or record it as its own outcome, but never let it wear the model’s jersey.
Prove your test file with the reference solution before scoring anything. After that, every candidate error is legitimately the candidate’s.
Pass or fail is not enough. The third bucket exists so that crashes and flakes stop laundering themselves into your metrics.
Write down what your sandbox does not enforce. The embarrassing field in the record is the one that makes the record credible.

Part 3 covers the statistic that came back 0.00 when the agreement was 6 out of 8, and why the zero was correct.

References
#

Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (the unbiased pass@k estimator)
Code shown above: src/cs1_auditor/harness.py and src/cs1_auditor/metrics.py in the cs1-auditor project, excerpted verbatim

Testing on Nick Liu - Software Engineer

My dotfiles had a no-exceptions test gate. It had never run once.

This was supposed to be a cleanup #

Finding 1: the hook on the dead path #

Finding 2: thirteen CI jobs, zero teeth #

Finding 3: launchctl said yes and did nothing #

Finding 4: the zombie updater #

Finding 5: the editor frozen in time, with a full paper trail #

What shipped #

Lessons #

References #

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

Rule 1: run the reference solution first #

Rule 2: three outcomes, not two #

Rule 3: report what the sandbox does not guarantee #

Lessons #

References #

This was supposed to be a cleanup
#

Finding 1: the hook on the dead path
#

Finding 2: thirteen CI jobs, zero teeth
#

Finding 3: launchctl said yes and did nothing
#

Finding 4: the zombie updater
#

Finding 5: the editor frozen in time, with a full paper trail
#

What shipped
#

Lessons
#

References
#

Rule 1: run the reference solution first
#

Rule 2: three outcomes, not two
#

Rule 3: report what the sandbox does not guarantee
#

Lessons
#

References
#