Python on Nick Liu - Software Engineer

Do not restyle a deck to look official. Wipe the template and keep its masters.

nickboy@users.noreply.github.com (Nick Liu) — Sat, 08 Aug 2026 00:00:00 +0000

The last deliverable of my summer project was a slide deck in the official university template. I generate slides from Python, and my first instinct was to rebuild the branding by hand: sample the navy, find the fonts, redraw the footer. That path produces decks that look almost right, which is worse than wrong. The trick that works is to load the official .pptx itself, delete every sample slide while keeping the slide masters, and add my own slides on the official layouts. Backgrounds, fonts, and logo come along for free.

The generator is one Python file, roughly 470 lines of python-pptx, and it built the final presentation for the auditing project from part 1. This post is about the two things in it worth stealing and the one bug worth laughing at.

Wiping slides without leaving corpses
#

python-pptx will happily open the official template as a presentation. The sample slides then need to go, and this is where the subtlety lives:

def wipe_slides(prs):
 lst = prs.slides._sldIdLst
 for sldId in list(lst):
 prs.part.drop_rel(sldId.get(qn("r:id"))) # drop rel too, or you get orphan parts
 lst.remove(sldId)

A .pptx is a zip of XML parts wired together by relationships. Removing a slide’s id from the slide list makes it disappear from the deck, but the underlying part is still referenced by a relationship from the presentation part. Skip the drop_rel line and the file still opens, which is the trap: you ship a package carrying orphaned parts, and whether anything downstream chokes on them is up to tooling you do not control. Drop the relationship and the id together and the package stays clean.

What survives the wipe is the point. Slide masters and layouts are separate parts, untouched by this loop. Every slide I add afterwards goes onto an official layout, which means the background art, the footer, the logo placement, and the typography all come from the university’s own files. Nothing to sample, nothing to eyeball, nothing to drift out of date when the brand team ships a new template. Load, wipe, rebuild.

Colors from the theme, not from the eyedropper
#

The template’s theme defines its palette, so the constants in my file are transcriptions with their theme slots noted, not guesses from a color picker:

NAVY = RGBColor(0x00, 0x30, 0x57) # Tech Navy (dk1)
GOLD = RGBColor(0xB3, 0xA3, 0x69) # Tech Gold (accent1)
GREY = RGBColor(0x54, 0x58, 0x59) # Grey Matter (dk2)
BODYFONT = "Roboto"
HEADFONT = "Roboto Slab"

The comments carry the important part. When a shape needs explicit coloring, the value traces back to a named slot in the official theme, so a reviewer can check it and a future me can update it.

The eight-inch hexagon gap
#

Now the bug. Content slides lay out a row of hexagons, and the gap between them was computed from the full slide width:

# before (buggy): with n=2 this gives gap = 8.03 inches
gap = (13.33 - 0.9 - n * w) / max(n - 1, 1) if n > 1 else 0

The formula spreads n hexagons across the entire 13.33 inch slide. With five hexagons that looks intentional. With two, the leftover width all becomes gap: 8.03 inches of it, so the two hexagons were flung to opposite edges of the slide with a hole in the middle, like a divorce rendered in vector graphics. The layout code was not wrong for the case I wrote it for. It was wrong for the case I never tested.

That render is real: for this post I re-ran the generator with the buggy formula restored and two placeholder stats, and the template did exactly what it did in June.

The fix abandons full-width justification for a fixed gap with true centering, and draws the hexagons larger when there are only one or two, since a sparse row can afford the space:

# after
w, h, y = (2.9, 2.5, 2.35) if n <= 2 else (2.2, 1.95, 2.55)
gap = 0.7
x0 = (13.33 - (n * w + gap * (n - 1))) / 2

Same slide source, current code. Justified layout distributes leftover space, and leftover space grows as items shrink in number. Any layout formula that divides remaining width by item count has this failure mode waiting at n=2 and n=1. Centered layout with a fixed gap degrades gracefully instead.

Lessons
#

Inherit branding, never imitate it. Load the official file, wipe its slides, keep its masters, and build on its layouts.
When you remove a part from an OOXML package, remove its relationship in the same breath. A file that still opens is not evidence the package is clean.
Hardcoded colors are debt unless each one names the theme slot it came from.
Justified layouts fail at low item counts because leftover space has nowhere to go but the gaps. Test layout math at n=1 and n=2, not only at the count in your mockup.

That closes the series. The auditor, the harness rules, the degenerate kappa, the failed sabotage, and the deck it was all presented on. The run records stay stamped with model and date, so when someone reruns these problems against next year’s models, disagreement with my numbers will be a finding and not a contradiction.

References
#

python-pptx documentation (presentations, slide masters, and layout objects)
Code shown above: gt-slides/build_deck.py in the project repo, excerpted verbatim

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

nickboy@users.noreply.github.com (Nick Liu) — Wed, 05 Aug 2026 00:00:00 +0000

The scariest failure mode in an eval harness is not a wrong answer. It is a harness problem wearing a model problem's clothes. A test file that does not run looks exactly like a problem no model can solve. A sandbox flake looks exactly like a failed attempt. Before I trusted any number from my CS1 auditing tool, I had to make those confusions impossible, and it came down to three rules.

Part 1 of this series reported scores from 1,095 sampled solutions. This post is about why I believe those scores. The harness lives in one Python module, and its docstring is a contract I wrote before the code:

Honesty rules:
 * The model is shown only `item.spec`; the hidden tests are written into the sandbox
 alongside the candidate but never given to the model.
 * A pre-flight check runs the *reference* solution against the hidden tests. If the
 reference does not pass, the item is malformed and the audit aborts (a HARNESS
 problem, never silently scored). This guarantees the test file is sound, so during
 the audit any candidate error (syntax, import, exception, failing assertion) is a
 legitimate FAIL for pass@k.
 * `HARNESS_ERROR` is reserved for runs where the sandbox produced no usable verdict
 (no report and no failing exit code). Such attempts are excluded from pass@k and
 reported separately, so they masquerade as neither pass nor fail.

Three rules. Each one exists because of a specific way a harness can lie.

Rule 1: run the reference solution first
#

Before any model attempt is scored, the harness runs the problem’s own reference solution against the hidden tests:

def preflight(item: Item, sandbox: Sandbox, limits: Limits) -> int:
 """Verify the reference solution passes the hidden tests. Returns the test count."""
 run = _run(item, item.reference_code, sandbox, limits)
 outcome, _passed, total, detail = _classify(run)
 if outcome is not Outcome.PASS:
 raise HarnessError(
 "pre-flight failed: the reference solution did not pass the hidden tests "
 f"({detail}). Fix the item before auditing.\n"
 f"stderr:\n{run.stderr[:2000]}"
 )
 return total

Without this gate, a broken test file is indistinguishable from a model that cannot solve the problem. You would conclude the problem is AI-resistant when your own tests do not run. In a project whose whole point was hunting for AI-resistant problems, this is the exact wrong conclusion to hand out for free. The gate turns that scenario into a loud abort instead of a quiet score of zero.

The gate also buys something subtle. Once the reference is known to pass, any error from a candidate solution (syntax error, import error, exception, failing assertion) is a legitimate FAIL. I do not have to wonder whose fault a stack trace is. The test file was proven sound minutes earlier.

Rule 2: three outcomes, not two
#

Most harnesses classify an attempt as pass or fail. Mine has a third verdict, and the classification code shows where it comes from:

raw = run.files.get(_JSON_REPORT)
if raw is None:
 # No machine-readable report. The test file is known-good (pre-flight), so a
 # non-zero exit means the candidate broke collection/execution -> FAIL. A clean
 # exit with no report is a genuine harness anomaly.
 if run.exit_code not in (0, None):
 return Outcome.FAIL, 0, 0, f"no json report; pytest exit {run.exit_code}"
 return Outcome.HARNESS_ERROR, 0, 0, "no json report produced"

With only two buckets, a crash has to land somewhere, and both landings are lies. Count it as a pass and your scores inflate. Count it as a fail and infrastructure flakes become model failures. HARNESS_ERROR attempts are excluded from the pass@k denominator and reported separately, so they masquerade as neither.

Across all 1,095 attempts, the harness error counter finished at zero. I want to be careful about what that proves. It does not prove the third bucket was unnecessary. It proves the infrastructure held, and I can only say that because the bucket existed to catch anything that did not. A counter that stays at zero is not vestigial. It is the evidence.

The scoring itself uses the unbiased pass@k estimator from the Codex paper rather than the naive “did any of the first k pass”, which is biased. That function validates k against the sample count and returns 1.0 when a pass is guaranteed by counting alone.

Rule 3: report what the sandbox does not guarantee
#

Every run record carries a capability string:

"capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False"

The honest part is the False. macOS does not allow a hard memory cap the usual way, so the tool bounds memory by polling. It would have been easy to leave that field out, or worse, to print True because polling exists. Instead the record says: this boundary is best-effort. If a reviewer ever asks whether a candidate could have exceeded memory limits undetected, the answer is in the data, not in my memory of how the sandbox worked.

A full record, for shape:

{
 "item_id": "count-multiples-01",
 "model": "claude-haiku-4-5",
 "date": "2026-07-16",
 "n_scored": 20,
 "n_pass": 0,
 "n_harness_error": 0,
 "sandbox_backend": "local",
 "capabilities": "network_isolated=True filesystem_confined=True memory_enforced=False",
 "passk": { "1": 0.0, "5": 0.0 },
 "gate_k": 1,
 "theta": 0.2,
 "resistant": true,
 "construct_clean": true,
 "combined_verdict": "AI-RESISTANT (construct-valid)"
}

Model and date on every record. Part 1 showed the same problem scoring 0.00, 0.05, and 0.20 on three different days. Without the stamp, those would have been one contradictory number. With it, they are three measurements.

Lessons
#

A harness problem must never be silently scored. Abort loudly or record it as its own outcome, but never let it wear the model’s jersey.
Prove your test file with the reference solution before scoring anything. After that, every candidate error is legitimately the candidate’s.
Pass or fail is not enough. The third bucket exists so that crashes and flakes stop laundering themselves into your metrics.
Write down what your sandbox does not enforce. The embarrassing field in the record is the one that makes the record credible.

Part 3 covers the statistic that came back 0.00 when the agreement was 6 out of 8, and why the zero was correct.

References
#

Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (the unbiased pass@k estimator)
Code shown above: src/cs1_auditor/harness.py and src/cs1_auditor/metrics.py in the cs1-auditor project, excerpted verbatim

Python on Nick Liu - Software Engineer

Do not restyle a deck to look official. Wipe the template and keep its masters.

Wiping slides without leaving corpses #

Colors from the theme, not from the eyedropper #

The eight-inch hexagon gap #

Lessons #

References #

A crash is never a pass. Three rules that kept 1,095 eval runs honest.

Rule 1: run the reference solution first #

Rule 2: three outcomes, not two #

Rule 3: report what the sandbox does not guarantee #

Lessons #

References #

Wiping slides without leaving corpses
#

Colors from the theme, not from the eyedropper
#

The eight-inch hexagon gap
#

Lessons
#

References
#

Rule 1: run the reference solution first
#

Rule 2: three outcomes, not two
#

Rule 3: report what the sandbox does not guarantee
#

Lessons
#

References
#