`env -u VAR command` did nothing. Exit code 0, no output, no error. A different command, same thing. A different variable, same thing. Any invocation that started with `env` just quietly evaporated. What finally made the problem visible was a `git init` that reported success while creating no `.git` directory at all. That was a year ago. The case closed last month, and the culprit was not uv, not some third-party installer, not anything exotic. It was this repo’s own bootstrap script. The thing that caught it was the regression test I had written for the original incident.
My dotfiles repo has a CLAUDE.md, and the CLAUDE.md has a rule in bold: every commit must pass the test suite, no exceptions. Within the first hour of an audit this July, I learned that this rule had been enforced exactly zero times since the day it was written. The hook file existed, its contents were correct, it even had its executable bit. It was just sitting at a path that yadm stopped reading a major version ago. No error message. No warning. To yadm, a hook in the wrong place and no hook at all are the same thing.
A reader clicking from my homepage to a post with a diagram got a block of raw mermaid source. The same reader pasting that post's URL directly got a rendered diagram. Same page, same build, same browser. The difference was the click, and the bug had been live on this site for months across every diagram, every math formula, and every chart, because I had only ever tested pages by loading them directly. 🧪 Tested with Blowfish 2.10x · htmx 2.0.10 Two features, both reasonable, one collision # This site has htmx’s hx-boost on the body: internal navigation swaps page content in place instead of doing full page loads, which keeps transitions smooth. Separately, the Blowfish theme is smart about heavy libraries: mermaid, KaTeX, and Chart.js bundles are only included on pages that use them, injected into the <head> of exactly those pages.
In July I went three weeks without opening my blog repo. During those weeks it published two posts, on schedule, each one confirmed live by an automated check, and the only reason I know all this is a green history in the Actions tab. The system's single notification channel is a failure email, and it has never fired. This post is the full pipeline, including the parts that only exist because something went wrong on the way here. 🧪 Tested with Hugo 0.164.0 · Cloudflare Pages The one Hugo fact everything hangs on # Hugo skips content dated in the future unless you pass --buildFuture (docs). That single default turns the date field into a release valve. Merge a post dated next Tuesday and production simply does not contain it: not in the sitemap, not in RSS, not at its URL. It sits in main, invisible, until a build happens after its date.
I kept slamming into my Claude subscription's usage limits, and the workload didn't feel heavy enough to explain it. The culprit was allocation, not volume: my main session ran on the most capable model available, and that model was also doing the grep runs, the batch edits, and the yes/no verification passes. Flagship reasoning, billed at flagship rates, spent on work a model a tenth the price does identically well. 🧪 Tested with Claude Code 2.1.x · macOS Where the tokens actually go # Two things surprised me when I looked at my consumption honestly.
The last deliverable of my summer project was a slide deck in the official university template. I generate slides from Python, and my first instinct was to rebuild the branding by hand: sample the navy, find the fonts, redraw the footer. That path produces decks that look almost right, which is worse than wrong. The trick that works is to load the official .pptx itself, delete every sample slide while keeping the slide masters, and add my own slides on the official layouts. Backgrounds, fonts, and logo come along for free. Auditing AI-Solvability · Part 5 of 5 1 2 3 4 5 The generator is one Python file, roughly 470 lines of python-pptx, and it built the final presentation for the auditing project from part 1. This post is about the two things in it worth stealing and the one bug worth laughing at.
I made two sabotaged copies of a CS1 problem. Into the first I inserted zero-width characters, invisible in any editor. Into the second I went heavier and swapped letters for Unicode look-alikes as well. The hidden tests stayed byte-for-byte identical. If obfuscation works as an AI defense, the model's score should drop. It scored 0.85 on both copies. The attack did nothing. My fairness check flagged both copies anyway, and that second part is the one worth writing about. Auditing AI-Solvability · Part 4 of 5 1 2 3 4 5 Why sabotage my own problem # Instructors keep looking for ways to make assignments that AI tools fail. One family of ideas is to leave the problem alone and poison the text: invisible characters, homoglyphs, formatting tricks that a human reader never notices but that might derail a model reading the same bytes. Before trusting any resistance my tool (part 1) reported, I needed to know what this kind of tampering does to a score, because a problem that scores low for wording reasons is fake difficulty, not a real skill gap.
I labelled eight CS1 problems blind, before looking at any output from my auditing tool. Then I compared my labels with the tool's verdicts. We agreed on 6 of 8. Then I computed Cohen's kappa, the standard chance-corrected agreement statistic, and it printed 0.00. Exactly zero, sitting next to a 75 percent raw agreement. My first thought was that my code was broken. It was not. Auditing AI-Solvability · Part 3 of 5 1 2 3 4 5 The setup # This was the human-agreement check for the tool from part 1. The tool labels each problem AI-solvable or AI-resistant. For the check to mean anything, my own labels had to be committed before I saw the tool’s, so I wrote them down blind, sealed against hindsight. Eight problems, two raters: me and the tool.
The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules. Auditing AI-Solvability · Part 2 of 5 1 2 3 4 5 Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:
For six weeks this summer I hunted for a CS1 programming problem that an AI model could not solve. I found exactly one candidate. It scored 0.00 against the weaker model, the stronger model solved it, and my fairness checks came back clean. Then I reworded the problem statement in plain English, changed nothing about the hidden tests, and the resistance evaporated. Auditing AI-Solvability · Part 1 of 5 1 2 3 4 5 What I was measuring # The tool is called cs1-auditor. It is a Python command-line tool that audits how solvable one CS1 problem is for a named model. You point it at a problem folder holding a plain-English spec, a hidden test suite, and a reference solution. It samples N candidate solutions from the model, runs each one in a sandbox against the hidden tests, and reports two things. The first is an AI-solvability score, which is pass@k for that model on that date. The second is a construct check, which asks whether any resistance comes from a real skill or from a trick in the wording. A problem only earns the label AI-resistant when the score is below a threshold and the construct check is clean.
The first sign that a model generation had turned over under my feet was a billing prompt that made no sense. I ran /model to switch back to Fable 5 and the CLI opened a browser tab telling me to set up usage credits, for a model my Max plan already includes. A full restart of Claude Code fixed it. That little glitch sent me digging, and what I found was a pile of changes, some breaking, some silent, scattered across at least five official pages. This is the consolidated list I wanted that week. 🧪 Tested with Claude Code 2.1.220 · macOS Eight weeks that rearranged the lineup # 06-09
Fable 5 ships
My Claude Code config now holds two hooks, ten skills, and three custom subagents, and most of them started life in the wrong layer. The instruction the model followed nine times out of ten lived in a prompt until I accepted that nine out of ten is a coin I lose every day. The workflow I pasted into chats became a skill. The bulk work that was draining my priciest model's quota became a fleet of cheaper agents. Same features, different failure modes. Taming Claude Code Sessions · Part 6 of 6 1 2 3 4 5 6 🧪 Tested with Claude Code 2.1.x · macOS The four layers # Claude Code has four extension points, and they answer four different questions: