Benchmarks
Benchmarks, research maps, and inspectable tools.
I look for important capabilities and research directions before they become obvious. SpatialBench and MeloText-ID preserve measurements from before large capability jumps. The expanded Ulam atlas, the OpenAI mathematics assessment, and CostFloor turn that same research judgment into tools other people can examine and use.
Featured benchmark · measured before the capability jump
SpatialBench
I collaborated with Geby Jaff (@spicey_lemonade) on this public 50-problem benchmark for 2D path tracing and 3D mental rotation. I developed its probability-equivalence scoring method and the structural mathematical framing for the two task families, building a measurement for a capability frontier that had substantial room to move.
50 spatial problems2D tracing + 3D rotation67.07% without tools91.78% with tools
The published leaderboard records 67.07% for GPT-6 Astra without tools and 91.78% with Code Interpreter, compared with earlier listed scores of 2.95–13.5%. Creating the measurement before that jump was a directional forecast about visual capabilities, with a frozen public artifact against which that judgment can be assessed. Its 50 questions measure these particular tasks.
Expanded research infrastructure · OPDP v1.8
Ulam Open Problem Difficulty Atlas
A research map covering 102,819 records, with separate dimensions for intrinsic and AI-relative difficulty, tractability, verification, formalization, prerequisites, and tool use. The atlas now combines the original full-statement cohorts with MathDB excerpts and normalized ProofAtlas intake.
102,819 records3 intake blocksProvenance + confidenceUsed for OpenMath selection
The total includes 15,458 statement-bearing records, 87,105 excerpt-only MathDB records, and 256 source-normalized ProofAtlas records. The latter two cohorts retain C0 provisional confidence. Source-linked estimates support research selection, and the official September 27 Sundai event used the atlas as the basis for its six problems.
October 2026 · independent research assessment
OpenAI mathematics assessment
I assessed a frozen 722-manuscript snapshot of the OpenAI mathematics release against the atlas, organizing 372 result families across 17 disciplines. A 254-page report, executive brief, and structured data explain how each manuscript relates to a research target and where its contribution sits.
722 manuscripts372 result families17 disciplines254-page report
The assessment distinguishes related targets, partial or special cases, lexical candidates, and same-target scope matches. The 59 same-target matches are scope classifications, not certified solutions. This is my independent assessment, with a separate overlay preserving the canonical atlas.
Audited benchmark · MeloText-ID v0.2.0
MeloText-ID
A benchmark I created for recognizing familiar melodies from sparse symbolic notes, across transposition and with or without explicit durations. Twenty models choose among 25 titles in four matched conditions, with repeated balanced trials, exact grading, and frozen task commitments.
20 models20,000 scored decisions25 melodies × 4 conditions71.38% best / 4% chance
The public audit retains 20,000 scored decisions across two sealed runs. GPT-6 Astra leads at 71.38% against 4% chance, 30.02 points above the pilot's 41.36% best. Public protocol, aggregate results, and audit artifacts make that cross-representation capability jump inspectable.
Applied model · automation and physical costs
CostFloor
An auditable scenario engine for exploring product costs when labor and capital are recursively replaced with physical inputs under automation. It separates energy, materials, scarcity, taxes, and margins, with editable assumptions and visible replacement steps.
6 reconciled fixtures3–20 recursion depthLow / base / high ranges
The prototype includes six reconciled fixtures and low/base/high scenarios with recursion controls. Its purpose is to expose the assumptions behind post-automation cost claims and make them editable, rather than hide them inside a single forecast.
OpenMath, from problem selection to proof review
Turning the atlas into curated research tasks, formalization requirements, evaluation criteria, and reproducible submission reviews. This includes separating a correct partial result from the larger open problem it contributes to.
Benchmarks that preserve the before-and-after
Developing inspectable measurements of capabilities that were weak when the tasks were created, so later progress can be compared against frozen protocols, scoring, and evidence. SpatialBench and MeloText-ID preserve that record across visual and musical representations.