Ai-Evals on Nick Liu - Software Engineer

I hid zero-width characters in a CS1 spec. The model did not blink.

nickboy@users.noreply.github.com (Nick Liu) — Fri, 07 Aug 2026 00:00:00 +0000

I made two sabotaged copies of a CS1 problem. Into the first I inserted zero-width characters, invisible in any editor. Into the second I went heavier and swapped letters for Unicode look-alikes as well. The hidden tests stayed byte-for-byte identical. If obfuscation works as an AI defense, the model's score should drop. It scored 0.85 on both copies. The attack did nothing. My fairness check flagged both copies anyway, and that second part is the one worth writing about.

Why sabotage my own problem
#

Instructors keep looking for ways to make assignments that AI tools fail. One family of ideas is to leave the problem alone and poison the text: invisible characters, homoglyphs, formatting tricks that a human reader never notices but that might derail a model reading the same bytes. Before trusting any resistance my tool (part 1) reported, I needed to know what this kind of tampering does to a score, because a problem that scores low for wording reasons is fake difficulty, not a real skill gap.

The donor problem was conditionals-01, a clean baseline that the weak model already passed at 0.75. I audited both sabotaged twins against claude-haiku-4-5, 20 attempts each, the same protocol as every other run in this series.

The attack failed
#

Problem	Model	pass@1	construct clean	verdict
conditionals-01-obf	claude-haiku-4-5	0.85	False	AI-SOLVABLE
conditionals-01-obf-heavy	claude-haiku-4-5	0.85	False	AI-SOLVABLE

Both copies passed at 0.85, slightly above the clean baseline’s 0.75 from an earlier run, which I read as ordinary run-to-run drift rather than obfuscation helping. Zero attempts were derailed by the invisible characters in any way I could detect. To be precise about the evidence base, because this is where sweeping claims get made: this is one problem, one model family, two obfuscation recipes, one summer. I am not writing “obfuscation cannot fool LLMs”. I am writing that these two attempts did not fool this model on these dates, and I did not find the result surprising after watching the same model shrug off paraphrases in both directions all month.

This is a negative result about the attack. I want to be straight about that. The experiment did not produce a defense, and I was not shopping for one.

The part that worked
#

The tool runs four offline fairness checks on every problem, no model call involved: the skill must be declared, the entry point must appear in the spec and be exercised by the tests, the spec must be plain ASCII with no hidden characters, and the layout must be free of construct-irrelevant obstacles. The third check exists precisely for this attack. Both sabotaged twins came back construct clean: False.

That flag matters even though the attack failed. Suppose a future model, or a weaker one, had stumbled over the invisible characters and scored 0.10. Without the check, the tool would have printed AI-RESISTANT, and I would have concluded I owned a problem that tests a real skill. The low score would have been fake difficulty, manufactured by tampering that no human student would even experience, since the rendered text looks identical. The construct check is the thing standing between “the model lacks this skill” and “I vandalized the input encoding”. A safeguard, not a headline.

The vocabulary for this comes from measurement theory: construct-irrelevant variance, score movement caused by something other than the skill you claim to measure. Invisible Unicode in a spec is about as construct-irrelevant as variance gets.

The detector connection
#

There is a sibling arms race on the detection side, and the literature on it shaped how I read my own result. Weber-Wulff and colleagues tested fourteen AI text detectors and all fourteen were easy to fool, with the worst errors landing on writers whose first language is not English. Sadasivan and colleagues argue AI text detection cannot be made reliable at scale. I am careful with both claims: fourteen specific detectors, tested in 2023, and I do not extrapolate to detectors released since. But the shape rhymes with my experiment. Tricks aimed at machines tend to wash out against the machines while the collateral lands on humans, and as someone writing in a second language, the collateral pattern in that first study is not abstract to me.

So my conclusion sits deliberately small. Obfuscating specs did not lower model scores in my two attempts, and any pipeline that reports AI-resistance should check its inputs for tampering before believing its own numbers, because the day the trick does work is the day it silently poisons your metric.

Lessons
#

Test the attack before trusting the defense. Two obfuscation recipes, twenty attempts each, zero effect on the score.
A fairness check earns its keep even when the attack fails. Its job is the counterfactual: the day a low score arrives, it tells you whether the difficulty is real.
Scope your negative results as tightly as your positive ones. One model family, one problem, two recipes. That is the whole claim.
Machine-versus-machine tricks have a track record of missing the machine and hitting people. Design assessments assuming the trick washes out.

Part 5 leaves the auditing tool and covers the other artifact of this project: a slide deck generator that steals its polish from the official university template, and the eight-inch hexagon gap it shipped with.

References
#

Weber-Wulff et al. (2023), “Testing of Detection Tools for AI-Generated Text”
Sadasivan et al. (2023), “Can AI-Generated Text be Reliably Detected?”
Messick (1995), “Validity of Psychological Assessment” (construct-irrelevant variance)
Run data: cs1-auditor .runs/*.json, pulled 2026-08-01

Cohen's kappa came back 0.00. The agreement was 6 out of 8.

nickboy@users.noreply.github.com (Nick Liu) — Thu, 06 Aug 2026 00:00:00 +0000

I labelled eight CS1 problems blind, before looking at any output from my auditing tool. Then I compared my labels with the tool's verdicts. We agreed on 6 of 8. Then I computed Cohen's kappa, the standard chance-corrected agreement statistic, and it printed 0.00. Exactly zero, sitting next to a 75 percent raw agreement. My first thought was that my code was broken. It was not.

The setup
#

This was the human-agreement check for the tool from part 1. The tool labels each problem AI-solvable or AI-resistant. For the check to mean anything, my own labels had to be committed before I saw the tool’s, so I wrote them down blind, sealed against hindsight. Eight problems, two raters: me and the tool.

Raw agreement 6 of 8. For a first calibration pass on a small set, that felt fine. Kappa exists because raw agreement flatters you: two raters who both say “solvable” almost every time will agree often by luck alone, so kappa subtracts the agreement you would expect from chance. I expected it to shave 0.75 down to something modest. I did not expect it to shave 0.75 down to nothing.

Chasing the zero
#

The suspect list, in the order I worked through it: my kappa implementation, the label encoding, the library convention for two-category data. All innocent. The arithmetic is short enough to do on paper, and doing it on paper is what ended the confusion.

Over the eight problems, the tool produced one label. Every verdict was the same. The strong model solved everything it touched, so nothing crossed the resistance threshold, so the tool said “solvable” eight times. My blind labels said “solvable” six times and “resistant” twice. Observed agreement: 6 of 8, which is 0.75.

Now the chance correction. Expected agreement multiplies the raters’ label frequencies. The tool says “solvable” with probability 1.0. I say it with probability 0.75. So the chance of agreeing on “solvable” is 0.75, and the chance of agreeing on “resistant” is zero times something, which is zero. Expected agreement: 0.75.

Kappa is observed minus expected over one minus expected. That is (0.75 - 0.75) / (1 - 0.75). Zero, by construction, and not a rounding artifact. When one rater has no variance, every bit of observed agreement is exactly what chance predicts, and the correction consumes the entire signal. Kappa is not measuring how often we agreed. It is measuring how much better than chance we agreed, and against a constant rater, nobody can beat chance.

The step that made me believe it
#

I did not trust the explanation until I attacked it with synthetic labels. I generated fake rater pairs where one rater was constant and the other varied, swept the disagreement count, and watched kappa pin to zero every time regardless of raw agreement. Then I gave the constant rater a single deviant label and watched kappa jump to a nonzero value. The statistic was behaving exactly as defined. My data was the degenerate case, not my code.

If I had to keep one paragraph from this post, it would be that one. The temptation with a surprising statistic is to pick between two stories: “the number is broken” or “the number is profound”. Both are cheap. The third option costs an hour with synthetic data and tells you which world you are in. In my case the zero was neither broken nor profound. It was the statistic saying: this validation design cannot distinguish your tool from a coin that always lands heads, because on this problem set the tool never did anything else.

What the zero told me to do
#

A useless kappa on this set does not mean the agreement check was worthless. It means the set was wrong for the question. Every problem the strong model touched, it solved, which is itself the headline finding of the whole project. But it leaves the agreement study with no variance to correlate. The fix is structural: the set needed problems where the tool’s verdict varies, and by the end of the summer I had evidence those barely exist within CS1.

And the honest limit, stated in the body and not in a footnote: the second rater was me, and I built the tool. This was a self-check, useful for catching gross disagreement between my intuition and my own machinery. It is not external validation, and I do not present it as such. One rater, one vendor, one summer.

Lessons
#

Kappa of 0.00 with high raw agreement is not a paradox. It means at least one rater never varied, and chance explains everything you observed.
Before calling a surprising statistic broken or profound, spend the hour on synthetic data. Degenerate inputs produce exotic outputs from perfectly correct code.
An agreement study needs variance in both raters. If your tool gives one answer on the whole set, you have learned something about the set, not about agreement.
Self-checks catch self-delusion. They do not substitute for a second human rater, and writing that sentence in the report is cheaper than having a reviewer write it for you.

Part 4 puts the fairness checks under adversarial pressure: I hid zero-width characters inside a problem spec and measured whether any model flinched.

References
#

Cohen (1960), “A Coefficient of Agreement for Nominal Scales” (the kappa definition used here)
Messick (1995), “Validity of Psychological Assessment” (the validity framework this check belongs to)
Run data: cs1-auditor .runs/*.json, pulled 2026-08-01

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

nickboy@users.noreply.github.com (Nick Liu) — Wed, 05 Aug 2026 00:00:00 +0000

The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules.

Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:

Honesty rules:
 * The model is shown only `item.spec`; the hidden tests are written into the sandbox
 alongside the candidate but never given to the model.
 * A pre-flight check runs the *reference* solution against the hidden tests. If the
 reference does not pass, the item is malformed and the audit aborts (a HARNESS
 problem, never silently scored). This guarantees the test file is sound, so during
 the audit any candidate error (syntax, import, exception, failing assertion) is a
 legitimate FAIL for pass@k.
 * `HARNESS_ERROR` is reserved for runs where the sandbox produced no usable verdict
 (no report and no failing exit code). Such attempts are excluded from pass@k and
 reported separately, so they masquerade as neither pass nor fail.

Three rules. Each one exists because of a specific way a harness can lie.

Rule 1: run the reference solution first
#

Before any model attempt is scored, the harness runs the problem’s own reference solution against the hidden tests:

def preflight(item: Item, sandbox: Sandbox, limits: Limits) -> int:
 """Verify the reference solution passes the hidden tests. Returns the test count."""
 run = _run(item, item.reference_code, sandbox, limits)
 outcome, _passed, total, detail = _classify(run)
 if outcome is not Outcome.PASS:
 raise HarnessError(
 "pre-flight failed: the reference solution did not pass the hidden tests "
 f"({detail}). Fix the item before auditing.\n"
 f"stderr:\n{run.stderr[:2000]}"
 )
 return total

Without this gate, a broken test file is indistinguishable from a model that cannot solve the problem. You would conclude the problem is AI-resistant when your own tests do not run. In a project whose whole point was hunting for AI-resistant problems, this is the exact wrong conclusion to hand out for free. The gate turns that scenario into a loud abort instead of a quiet score of zero.

The gate also buys something subtle. Once the reference is known to pass, any error from a candidate solution (syntax error, import error, exception, failing assertion) is a legitimate FAIL. I do not have to wonder whose fault a stack trace is. The test file was proven sound minutes earlier.

Rule 2: three outcomes, not two
#

Most harnesses classify an attempt as pass or fail. Mine has a third verdict, and the classification code shows where it comes from:

raw = run.files.get(_JSON_REPORT)
if raw is None:
 # No machine-readable report. The test file is known-good (pre-flight), so a
 # non-zero exit means the candidate broke collection/execution -> FAIL. A clean
 # exit with no report is a genuine harness anomaly.
 if run.exit_code not in (0, None):
 return Outcome.FAIL, 0, 0, f"no json report; pytest exit {run.exit_code}"
 return Outcome.HARNESS_ERROR, 0, 0, "no json report produced"

With only two buckets, a crash has to land somewhere, and both landings are lies. Count it as a pass and your scores inflate. Count it as a fail and infrastructure flakes become model failures. HARNESS_ERROR attempts are excluded from the pass@k denominator and reported separately, so they masquerade as neither.

Across all 1,095 attempts, the harness error counter finished at zero. I want to be careful about what that proves. It does not prove the third bucket was unnecessary. It proves the infrastructure held, and I can only say that because the bucket existed to catch anything that did not. A counter that stays at zero is not vestigial. It is the evidence.

The scoring itself uses the unbiased pass@k estimator from the Codex paper rather than the naive “did any of the first k pass”, which is biased. That function validates k against the sample count and returns 1.0 when a pass is guaranteed by counting alone.

Rule 3: report what the sandbox does not guarantee
#

Every run record carries a capability string:

"capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False"

The honest part is the False. macOS does not allow a hard memory cap the usual way, so the tool bounds memory by polling. It would have been easy to leave that field out, or worse, to print True because polling exists. Instead the record says: this boundary is best-effort. If a reviewer ever asks whether a candidate could have exceeded memory limits undetected, the answer is in the data, not in my memory of how the sandbox worked.

A full record, for shape:

{
 "item_id": "count-multiples-01",
 "model": "claude-haiku-4-5",
 "date": "2026-07-16",
 "n_scored": 20,
 "n_pass": 0,
 "n_harness_error": 0,
 "sandbox_backend": "local",
 "capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False",
 "passk": { "1": 0.0, "5": 0.0 },
 "gate_k": 1,
 "theta": 0.2,
 "resistant": true,
 "construct_clean": true,
 "combined_verdict": "AI-RESISTANT (construct-valid)"
}

Model and date on every record. Part 1 showed the same problem scoring 0.00, 0.05, and 0.20 on three different days. Without the stamp, those would have been one contradictory number. With it, they are three measurements.

Lessons
#

A harness problem must never be silently scored. Abort loudly or record it as its own outcome, but never let it wear the model’s jersey.
Prove your test file with the reference solution before scoring anything. After that, every candidate error is legitimately the candidate’s.
Pass or fail is not enough. The third bucket exists so that crashes and flakes stop laundering themselves into your metrics.
Write down what your sandbox does not enforce. The embarrassing field in the record is the one that makes the record credible.

Part 3 covers the statistic that came back 0.00 when the agreement was 6 out of 8, and why the zero was correct.

References
#

Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (the unbiased pass@k estimator)
Code shown above: src/cs1_auditor/harness.py and src/cs1_auditor/metrics.py in the cs1-auditor project, excerpted verbatim

I went looking for AI-resistant CS1 problems. The search came up empty.

nickboy@users.noreply.github.com (Nick Liu) — Tue, 04 Aug 2026 00:00:00 +0000

For six weeks this summer I hunted for a CS1 programming problem that an AI model could not solve. I found exactly one candidate. It scored 0.00 against the weaker model, the stronger model solved it, and my fairness checks came back clean. Then I reworded the problem statement in plain English, changed nothing about the hidden tests, and the resistance evaporated.

What I was measuring
#

The tool is called cs1-auditor. It is a Python command-line tool that audits how solvable one CS1 problem is for a named model. You point it at a problem folder holding a plain-English spec, a hidden test suite, and a reference solution. It samples N candidate solutions from the model, runs each one in a sandbox against the hidden tests, and reports two things. The first is an AI-solvability score, which is pass@k for that model on that date. The second is a construct check, which asks whether any resistance comes from a real skill or from a trick in the wording. A problem only earns the label AI-resistant when the score is below a threshold and the construct check is clean.

Defaults: 20 samples per audit, pass@k reported at k=1 and k=5, threshold 0.2. The pass@k estimator is the unbiased one from the Codex paper, not the naive “did any of the first k pass”.

The scale of the evidence, so you can weigh it: I authored 58 problems, ran 34 of them against live models, and collected 1,095 sampled solutions between 2026-06-14 and 2026-07-23. Four model versions, all from one vendor: claude-sonnet-4-6, claude-haiku-4-5, claude-haiku-4-5-20251001, and claude-sonnet-4-5-20250929. One rater, one vendor, one summer. Every claim below sits inside that box.

The one candidate
#

Most problems never got close to resistant. On six clean baseline problems, the strong model went 1.00 across the board and the weak model slipped on only two (0.75 on a conditionals problem, 0.60 on a rounding problem). Five problems built around a non-default rule did better at separating the models, and one stood out:

Problem	The non-default rule	Sonnet	Haiku
count-multiples-01	closed range, negative bounds	1.00	0.00

Haiku produced twenty candidate solutions and zero passed. Sonnet solved it every time. The construct check came back clean: the skill was declared, the spec was plain ASCII, no tricks. The tool printed the label I had been chasing: AI-RESISTANT (construct-valid).

The reword protocol
#

A low score can mean the problem tests a real skill the model lacks. It can also mean the model tripped on the phrasing. To tell these apart I added a fifth check that is a protocol rather than a rule: reword the spec in plain English, keep the hidden tests byte-for-byte identical, and audit again. If the resistance is real, it should survive a paraphrase.

Probe	Haiku pass@1	Outcome
count-multiples-01 (baseline)	0.00	AI-RESISTANT (construct-valid)
reword 1	0.00	resistance held
reword 2	0.75	flipped to AI-SOLVABLE

The second paraphrase lifted Haiku from 0.00 to 0.75. Same tests. Same skill, supposedly. Sonnet solved the baseline and both rewords at 1.00 throughout. Whatever count-multiples-01 was measuring, it was not a stable property of the task. It was a property of one particular arrangement of words, on one model, on one date.

The score itself would not sit still
#

One more honesty requirement before the big claim. The 0.00 comes from the 20-attempt run of 16 July. I reran the same baseline problem on later dates:

Date	Attempts	Passed	pass@1
2026-07-16	20	0	0.00
2026-07-17	20	1	0.05
2026-07-18	5	1	0.20

So I will not write that Haiku failed it every single time. It scored 0.00 on the run I reported, and drifted to 0.05 and 0.20 on repeats. At first this annoyed me. Then I realized it is a second argument for the same thesis: even the score of one fixed problem moves between runs, so a single number pinned to a task is not a stable property. It is a measurement of a model, on a date, with a sample size attached. My tool stamps every record with all three, and this is why.

Widening the search
#

Maybe count-multiples-01 was unlucky and better candidates existed. I ran two more batches, 13 fresh problems designed around non-default rules. The bar for a keeper: fair by the construct check, solved by the strong model, failed by the weak model, and still failing after a reword. Not one problem cleared all four.

The most instructive failure was slice-1based-01. Haiku scored 0.15, which looked promising, but Sonnet also failed it at 0.00. A problem both models fail is not a resistant problem. It is a badly specified one. Its rewords lifted Haiku to 0.40 and 0.20, which fits the pattern: the difficulty lived in the wording.

Why the search may be structurally empty
#

The pattern generalizes into a tension I could not find a way around.

To be fair, and to be solvable by a strong model, the twist in a problem must be stated clearly. But a clearly stated twist is one that the weakest model you can still realistically use also reads. The genuinely weak models that a resistant problem would need have been retired. So the skill gap that resistance depends on has closed, at least for CS1, at least for the models I tested.

This does not mean no AI-resistant assessment can exist anywhere. It means that within my box (one vendor, four model versions, six weeks, 58 authored problems, 1,095 attempts) every candidate either failed the fairness bar or failed the reword. Prior work found GPT-4 passing about 95 percent of textbook CS1 tasks, so the ceiling was known to be high before I started. What I did not expect is how completely the floor had risen too.

Lessons
#

A low pass@k is a claim about wording until a paraphrase has failed to fix it. Reword first, then talk about skill gaps.
Report the run you measured, with its date and sample size. My “0.00” became 0.05 and 0.20 on later runs of the identical problem.
A problem that every model fails is not resistant. It is broken. Check the strong model before celebrating the weak model’s failure.
If a resistant problem needs an unclearly stated twist to work, the resistance is construct-irrelevant variance, not a skill.

The next post in this series is about the part of the tool that made these numbers trustworthy in the first place: the harness, and the three rules that kept 1,095 sandbox runs from lying to me.

References
#

Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (the unbiased pass@k estimator, section 2.1)
Kiesler and Schiffner (2023), “Large Language Models in Introductory Programming Education” (GPT-4 solving 68 of 72 textbook introductory Python tasks)
Messick (1995), “Validity of Psychological Assessment” (construct-irrelevant variance, the concept behind the construct check)
Run data: cs1-auditor run records (.runs/*.json), pulled 2026-08-01; every figure above is from those records

Ai-Evals on Nick Liu - Software Engineer

I hid zero-width characters in a CS1 spec. The model did not blink.

Why sabotage my own problem #

The attack failed #

The part that worked #

The detector connection #

Lessons #

References #

Cohen's kappa came back 0.00. The agreement was 6 out of 8.

The setup #

Chasing the zero #

The step that made me believe it #

What the zero told me to do #

Lessons #

References #

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

Rule 1: run the reference solution first #

Rule 2: three outcomes, not two #

Rule 3: report what the sandbox does not guarantee #

Lessons #

References #

I went looking for AI-resistant CS1 problems. The search came up empty.

What I was measuring #

The one candidate #

The reword protocol #

The score itself would not sit still #

Widening the search #

Why the search may be structurally empty #

Lessons #

References #

Why sabotage my own problem
#

The attack failed
#

The part that worked
#

The detector connection
#

Lessons
#

References
#

The setup
#

Chasing the zero
#

The step that made me believe it
#

What the zero told me to do
#

Lessons
#

References
#

Rule 1: run the reference solution first
#

Rule 2: three outcomes, not two
#

Rule 3: report what the sandbox does not guarantee
#

Lessons
#

References
#

What I was measuring
#

The one candidate
#

The reword protocol
#

The score itself would not sit still
#

Widening the search
#

Why the search may be structurally empty
#

Lessons
#

References
#