Vox & MENS Comparative Efficacy Benchmarking: Research, Gap Audit, and Automation Plan (2026)

Vox & MENS Comparative Efficacy Benchmarking (2026)

Companion docs: SCIENTIA Self-Publication Gap Map, SCIENTIA Self-Publication Finalization Plan, ADR-011, Publication worthiness rules, Free-Tier Model Selection & Onboarding research, contracts/eval/humaneval-vox/.

Scope. This doc answers, in order: (A) what HLE and 2025-2026 eval-community practice teach about avoiding overfitting when comparing AI systems; (B) HF/GitHub/arXiv publishing rules as of 2026; (C) what metrics are required to credibly claim Vox+harness efficacy; (D) a code-audited gap map of SCIENTIA specific to this publication class; (E) a code-audited gap map of the vox-orchestrator model router against 2026 market routing practice; (F) a recommended live-benchmark architecture for voxlang.org; (G) an automation plan for recurring publications; (H) a worked example — a small blind benchmark this session actually ran against the held-out humaneval-vox corpus. Every code claim below carries a file:line citation; where research and code disagree, both are stated.

A. HLE and anti-overfitting benchmark design — what 2025-2026 taught the field

HLE’s actual structure. 2,500 expert-authored questions (Center for AI Safety + Scale AI, ~$500K prize pool), filtered so frontier models originally scored at/below random guessing, then double human-reviewed. Coverage: math 41%, physics 9%, biology/medicine 11%, CS/AI 10%, chemistry 7%, humanities/social science 9%, engineering 4%, other 9%; ~14% multimodal, 24% multiple-choice, the rest short-answer exact-match. Unlike MMLU/GPQA, HLE ships a calibration/confidence-scoring component — models state confidence, and a calibration-error metric is reported alongside raw accuracy — precisely because saturated benchmarks stop distinguishing models on raw accuracy alone. As of August 2026 the leading model sits near 47% (Gemini 3.1 Pro), the ~90-model average near 30%, vs. an implied human-expert ceiling — headroom is still large. A July 2025 FutureHouse audit found ~30% of chemistry/biology answers arguably wrong, producing HLE-Rolling, a continuously-corrected revision (an answer-key-quality fix, not a contamination-resistance mechanism per se).

What HLE is a model of, for Vox’s purposes: (1) build the benchmark to be hard enough that today’s best system is visibly imperfect — a benchmark every system saturates stops discriminating; (2) report calibration, not just pass/fail — a system that says “I’m not sure” on a hard case is more trustworthy than one that’s confidently wrong; (3) treat the benchmark as living — publish an errata/correction process rather than freezing a flawed v1 forever.

The dominant 2025-2026 lesson: contamination is the default failure mode, not an edge case. The most consequential finding for a project the size of Vox is the SWE-bench contamination crisis: independent audits found ~10.6% direct solution leakage into SWE-bench Verified issue text, model recall of memorized file paths up to 76% of the time, and OpenAI’s own audit of o3 failures found 59.4% were test-harness artifacts, not model limitations — OpenAI publicly stopped reporting SWE-bench Verified as a result. The structural fix the field converged on is a rolling, timestamp-gated corpus: LiveCodeBench (arXiv:2403.07974) tags every problem with a publish date and restricts each evaluated model to problems published after its training cutoff, visibly exposing contamination as an accuracy cliff at the cutoff date. SWE-bench’s own successors (SWE-bench Pro, SWE-rebench, SWE-MERA) all adopted this pattern. humaneval-vox (§H below) is a fixed, non-rolling 164-problem corpus today — it needs a rolling extension before any comparative claim built on it would survive the scrutiny a SWE-bench-style audit would apply.

The second lesson: leaderboard governance can itself be the attack surface. The Leaderboard Illusion (arXiv:2504.20879) documented that LMArena let providers privately test many variants pre-release and publish only the best (Meta tested 27 private Llama-4 variants), while proprietary closed models received disproportionate battle volume and lower removal rates than open models — a governance failure, not a modeling one. Applied to Vox: any voxlang.org leaderboard must pre-register which model/harness configuration is being scored (exact version string, reasoning-effort setting, prompt template) before results are published, and must never let Vox’s own MENS be scored under a privately-tuned variant while competitors are scored off-the-shelf.

The third lesson: statistical rigor on deltas, not point scores. Top models on Arena-style leaderboards routinely sit within overlapping 95% confidence intervals despite headline score gaps of 2-5 points; the emerging practice is bootstrap CIs and rank intervals (not just score intervals) — a 2026 audit of Open LLM Leaderboard v1 found 11 of 40 displayed pairwise rankings failed to meet statistical-resolution targets. A single-run “Vox beats Model X” claim without a CI is not credible by 2026 standards.

The fourth lesson: judge credibility. The Sakana AI Scientist backlash (bold “AI passed peer review” claims undercut once critics found the automated reviewer missed serious flaws and the claim required undisclosed pre-arranged reviewer consent) and the well-documented self-preference bias in LLM judges (arXiv:2410.21819 — judges rate same-family outputs higher, correlated with familiarity not quality) both point the same direction: prefer symbolic/exact-match verification (compile exit code, test pass/fail) over LLM-judge grading wherever the task allows it, and when an LLM judge is unavoidable, use a model from a different family than the system under test and report human-agreement rates. Vox is unusually well positioned here — code compiling or not, and tests passing or not, are exact-match, non-LLM-judged signals by construction.

B. Publishing rules — Hugging Face, GitHub, arXiv (2026)

Hugging Face. Dataset cards need YAML front-matter (license, language, pretty_name, task_categories, size_categories) in README.md; model cards should carry both quantitative eval results and a Bias/Risk/Limitations section (a 2026 audit of 100 HF model cards found only ~60-65% include even superficial safety/bias sections — a real, low bar). Evaluation results are standardized as .eval_results/ YAML that auto-surfaces on the model page and any linked leaderboard Space (the current plug-in point for a benchmark project). There is no HF publishing adapter anywhere in the Vox codebase today (§D3) — this is a concrete, unaddressed gap given HF is the natural home for humaneval-vox as a public dataset and for a leaderboard Space.

GitHub. CITATION.cff at repo root is the citation SSOT; Zenodo’s GitHub integration auto-mints a DOI per tagged release plus a persistent concept-DOI across versions when it’s present. Important nuance: ACM’s Artifact Review/Badging policy explicitly rejects GitHub-only artifacts for “Artifacts Available” — permanence requires Zenodo/FigShare/Software Heritage. Vox already has both a Zenodo adapter and a Software Heritage adapter (crates/vox-publisher/src/scholarly/zenodo.rs, software_heritage.rs) — this requirement is already satisfiable, it just needs to be the documented default for any benchmark-harness release, not GitHub releases alone.

arXiv — material 2026 changes vs. 2024-2025. Two changes matter directly for Vox: (1) endorsement tightened Jan 21, 2026 — institutional email alone no longer suffices; first-time submitters need either prior accepted authorship in the field or a personal endorsement from an established arXiv author — plan for this lead time before any Vox arXiv submission. (2) AI-generated-content enforcement: arXiv stopped accepting unreviewed CS “review/position papers” (Oct 2025), and as of May 2026 the cs section chair confirmed “incontrovertible evidence” of unverified LLM output (hallucinated citations, residual prompt text) triggers a one-year submission ban for that author, after which future submissions must first be accepted at a peer-reviewed venue. This directly reinforces SCIENTIA’s existing hard red line #1 (“fabricated or unresolved citations used as evidence” — worthiness rules) and its citation-verification requirement — that rule is not theoretical, it is what keeps a Vox submission out of a ban. Category: a Vox efficacy/benchmark paper should primary-list cs.SE (harness/tooling framing) or cs.CL/cs.LG (model-evaluation framing), cross-listed. Dual-submission enforcement has also tightened materially at major venues (AAAI-26 desk-rejected 141 papers via LLM-assisted overlap detection) — a consideration for anything Vox eventually submits to a conference in parallel with an arXiv post.

C. Metrics required to credibly claim Vox + harness efficacy

Synthesizing (A), (B), and the harness-specific literature (Artificial Analysis’s Coding Agent Index, RouterArena, SWE-bench Pro), a credible claim needs metrics across four families — a single pass-rate number is not sufficient evidence by 2026 norms:

1. Correctness, symbolically verified. Compile-success rate, test-pass rate (pass@1, pass@k with k≥5), and edit-reliability (does the harness’s structured-edit format apply cleanly, à la Aider’s diff/whole distinction) — all exact-match, no LLM judge.
2. Efficiency. Tokens-per-solution (generation + any repair turns), wall-clock latency (p50/p99), and cost-per-solved-problem — directly comparable to vox models eval’s existing intelligence/tokens_per_pass/p50_ms/cost_per_success_usd fields (crates/vox-cli/src/commands/model/eval.rs:91-114).
3. Robustness/calibration. HLE-style confidence reporting (does the harness know when it’s likely wrong?), and — specific to a harness-vs-model claim — Artificial Analysis’s reward-hacking check: an independent pass verifying a solution didn’t edit the test file, read the reference solution, or fetch the answer externally. This matters more for Vox than most: the golden/held-out split in humaneval-vox is only useful if nothing downstream (training corpora, RAG context, agent tool access) can see the held-out 31 problems’ reference.vox at generation time.
4. Comparative, with uncertainty. Every “Vox beats X” claim needs a reported confidence interval or rank interval (lesson A3), a pinned exact competitor configuration (lesson A2), and — this is the harness-specific nuance Artificial Analysis and RouterArena both encode — the row being compared must be a (model, harness, config) tuple, not a bare model name, since the same base model scores meaningfully differently under different scaffolding (Terminal-Bench 2.1: Codex CLI 89.5% vs. Claude Code 89.1% on effectively the same underlying capability tier).

D. SCIENTIA gap audit, specific to Vox-efficacy publications

The existing gap map already documents SCIENTIA’s substantial built infrastructure (claim extraction, pre-registration, nanopub/RO-Crate, worthiness rubric, and — confirmed by this session’s code audit — scholarly adapters for arXiv (assist), Zenodo, OpenReview, OSF, Crossref, ORCID, and Software Heritage, all present under crates/vox-publisher/src/scholarly/). Three gaps specific to “publish evidence of Vox/MENS efficacy” are not covered by the existing gap map’s A-H items and should be logged as new entries:

Gap I — No Hugging Face publishing adapter

Confirmed by direct grep: no huggingface.rs/hf.rs exists in vox-publisher; the only HF reference in the repo is HuggingFaceCatalog in crates/vox-orchestrator/src/catalog.rs:424-427, which is Vox consuming HF as an inference provider, not publishing to the Hub. For a benchmark/dataset artifact (humaneval-vox itself is exactly the kind of thing that belongs on HF as a dataset with a dataset card and a linked leaderboard Space — §B), this is a real, concrete gap, not a stretch goal. Scope: dataset-card YAML generation from the existing contracts/eval/humaneval-vox/manifest.v1.yaml (mechanical — the manifest already has corpus_hash, count, held_out fields a dataset card needs) plus a ScholarlyAdapter-shaped HF Hub upload client. Effort: small-medium, follows the exact adapter pattern already proven five times over (zenodo.rs, osf.rs, etc.).

Gap J — humaneval-vox is not rolling/contamination-resistant, and is not wired to the publication pipeline

Two distinct problems. First (methodology): humaneval-vox’s manifest (contracts/eval/humaneval-vox/manifest.v1.yaml:12-24) is a fixed 164-problem set with a pinned corpus_hash — exactly the SWE-bench-style contamination risk (A) describes, especially since 120 of 164 problems are training_eligible: true and therefore, by design, may appear in MENS training data. Any “MENS solves N% of Vox problems” claim built on the 120 training-eligible problems is close to meaningless as external evidence; only the 31 held-out problems (010-, 041-050- per this session’s audit) are usable for an honest claim, and even those need periodic rotation (new held-out problems added on a schedule, old ones retired to training-eligible) to stay contamination-resistant as MENS’s training corpus grows — the LiveCodeBench pattern, adapted: timestamp each held-out problem’s addition date and only count problems added after a model’s stated cutoff.

Second (wiring): vox models eval (crates/vox-cli/src/commands/model/eval.rs) — the actual “intelligence/token-efficiency/responsiveness” cross-model comparator — runs its own 10 hardcoded fixtures (L31-74) and explicitly notes it is “intentionally not dependent on the (empty) contracts/eval/humaneval-vox corpus” (L28-29), a comment now stale since the corpus has 164 populated problems. There is no command today that runs humaneval-vox (or any held-out subset of it) against the model registry and writes a model_scoreboard row per model, i.e. no bridge between the language-efficacy corpus and the model-comparison infrastructure. This is the single highest-leverage fix for this research thread: it would make Gap A’s “self-observation candidate producers” (existing gap map) concrete for the specific case of “a new model was added to the registry — how well does it write Vox?”

Gap K — No public/live benchmark surface

Confirmed: docs-astro/src/components/ has exactly one component (VoxPlayground.astro); no dashboard/chart component exists there to reuse, and no findings-style leaderboard route exists (§F has the concrete architecture recommendation). This is gap map’s existing Gap G (reading surface) narrowed to the specific “public leaderboard” case — Gap G’s findings/<trusty-uri> route design is the right host, but a leaderboard additionally needs a CI-regenerated data file (not a one-off manifest), which Gap G as originally scoped doesn’t cover.

E. Model router gap audit vs. 2026 market practice

What Vox has today (code-audited, vox-orchestrator::models::{registry, select, autonomic} plus vox-research-shim::selection::free_tier): a hierarchical rule chain, not a learned router — env override → local-pin → hardcoded premium-alias table → weighted-linear scorer (quality × w1 + efficiency × w2 + latency × w3 + throughput × w4 + health × w5, crates/vox-orchestrator/src/models/scoring.rs), with a real, closed telemetry feedback loop for cost and latency only (ModelRegistry::inject_pricing_catalog/inject_scoreboard_latency, registry.rs:96-161) but a static, unmeasured quality_score heuristic (log10(max_tokens) blended with a free/paid constant, scoring.rs:152-162) nudged only 0.15-weighted by observed success when scoreboard data exists. A separate ContextualBandit (calibration.rs:172-201) and arm_stats accumulator exist and are written to but never read by the selector — a closed write, open read gap. A separate free-tier router (vox-research-shim/src/selection/free_tier.rs) is real and tested but lives on a different scoring stack (ScoringWeights/ModelScorer in vox-research-shim) than the canonical models::select() path — the fragmentation the existing model-orchestration audit doc already flags.

Where this sits relative to the market (per this session’s research pass): the field splits into (a) crowd/market-signal routers (OpenRouter Auto — cheap, weak published evidence), (b) learned preference-classifier routers (RouteLLM, arXiv:2406.18665 — trained on Chatbot Arena human-preference data, reports >85% cost reduction at 95% retained quality on MT-Bench; NotDiamond — proprietary meta-model ranker), (c) infra load-balancers with no quality signal (LiteLLM Router, Portkey, Requesty — cost/latency/availability only, same category Vox’s static scorer is closest to), and (d) academic cascade/bandit routers explicitly optimizing a cost-quality Pareto frontier (FrugalGPT-style escalation; RouterArena, arXiv:2510.00202, is the closest existing analog to a “compare routers against each other” platform Vox could emulate or even join). Vox’s current router is architecturally closest to category (c), with a locality axis (MENS-local vs. cloud) that genuinely differentiates it from every router surveyed — none of RouteLLM/NotDiamond/OpenRouter/LiteLLM model a locality axis explicitly — but it is not yet doing what makes RouteLLM/NotDiamond credible: a learned per-request quality prediction validated against held-out preference data.

Concrete, prioritized upgrade list (cheapest-to-close first):

1. Close the write-only loops. arm_stats/ContextualBandit::select() and scientia_model_profile_learning (written by rollup_model_scoreboard_with_scientia, crates/vox-db/src/research_pipeline.rs:603-633) are both fully implemented accumulators that the live selector never reads. Wiring either into best_for_with_filter()’s scoring path is a read-side change only — no new data collection needed.
2. Unify the free-tier router into the canonical path. vox-research-shim::selection::free_tier::FreeTierRouter should not use a separate ScoringWeights/ModelScorer stack from vox-orchestrator::models::scoring — this is the same fragmentation the existing model-orchestration audit doc (model-orchestration-ssot-audit-2026.md) already names as open work.
3. Measure quality, don’t heuristic it. quality_score today is log10(max_tokens) — replace or supplement with the Gap-J bridge (§D): every humaneval-vox held-out run against a registered model writes a real pass-rate-derived quality signal into the scoreboard the selector already partially reads.
4. Adopt a RouteLLM-style learned classifier as a stretch goal, trained on Vox’s own task-outcome data (which the write-only accumulators above are already collecting) rather than external preference data — this is the credible path to actually beating a static-rule router, and it reuses infrastructure Vox already has (the L2 classifier scaffold in autonomic.rs, currently only classifying new models’ tier/strengths once, could be repurposed/extended for per-request routing).
5. Cascade on actual output, not just pre-call signals. TierCascadeRouter escalates before calling any model, from task-difficulty heuristics — never from actually trying the cheap model and inspecting its own output confidence (FrugalGPT’s core mechanism). This is a bigger lift than 1-4 but is where most of RouteLLM/FrugalGPT’s measured cost savings come from.

F. Recommended live-benchmark architecture for voxlang.org

Combining LiveCodeBench’s rolling-window design (A), Artificial Analysis’s (model, harness, config)-tuple rows and reward-hacking judge pass (C), RouterArena’s multi-axis scoring, and Aider’s low-infra static-site hosting pattern — none of which require new bespoke infrastructure Vox doesn’t already have pieces of:

1. Corpus: extend humaneval-vox’s held-out set with a rolling addition schedule (new problems added monthly, tagged with an addition timestamp; only problems added after a model’s public training-cutoff date count toward that model’s score) — directly closes Gap J’s contamination exposure.
2. Runner: extend vox models eval (or a new sibling command) to run the held-out corpus against every registered ModelSpec (including external harnesses scored as a tuple: e.g. ("claude-opus-4.7", "claude-code", default-config) vs. ("kimi-k2.6-thinking", "vox-harness", default-config)), recording pass@1/pass@5, tokens, latency, cost into model_scoreboard — this is Gap J’s bridge, made concrete.
3. Anti-gaming pass: a deterministic-first, judge-second pipeline mirroring Artificial Analysis — the compiler/test-runner exit code is the primary verifier (no LLM judge in the loop for correctness at all, per lesson A4); a lightweight second pass flags any generated solution that edited tests.vox, referenced reference.vox, or fetched network content during generation.
4. Statistics: bootstrap CIs and rank intervals on every published delta (lesson A3) — not point scores.
5. Publication surface: a CI-regenerated JSON/data file feeding a new docs-astro route (Gap K) — no new chart library needed beyond what a static site needs; Aider’s own leaderboard is exactly this pattern (versioned CSV → static site regen) and is worth copying directly rather than inventing bespoke dashboard infra.
6. Governance (lesson A2): every scored (model, harness, config) tuple’s exact version/settings is pinned and published alongside the score, and Vox’s own MENS entry is generated under the same automated pipeline as every competitor entry — no private pre-tested variant gets to represent Vox publicly, mirroring exactly the failure LMArena had to walk back.
7. This feeds SCIENTIA, not a separate system: each scheduled benchmark run is exactly the kind of finding-candidate.v1 row the existing gap map’s Gap A (self-observation producers) already specifies — a benchmark-CI-history detector that was named as “concretely missing” there. Building the runner in (2) as a ResearchEventEmitter-emitting producer, rather than a bespoke script, means every efficacy result is automatically eligible for the existing worthiness-gated publication pipeline (arXiv/Zenodo/OpenReview/HF-once-Gap-I-lands) without new plumbing.

G. Automation plan for future publications

The existing gap map’s Gap C (IMRaD manuscript scaffolder) and Gap F (vox scientia scout) are the right existing hooks — this research doesn’t propose new automation primitives, it proposes wiring §F’s benchmark runner into them:

1. Land §F’s runner as a Gap-A producer emitting finding-candidate.v1 rows in the algorithmic_improvement/reproducibility_infra classes whenever a scheduled run shows a statistically-significant delta (CI-based, per lesson A3) vs. the prior run or vs. a competitor.
2. vox scientia scout (once Gap A lands per the existing gap map’s recommended first slice) surfaces these automatically — a new MENS checkpoint or router change that moves the needle becomes a visible publication candidate without a human remembering to check.
3. Gap C’s IMRaD scaffolder fills the Results table directly from model_scoreboard rows and the Methods section from the pinned corpus/config manifest — this is exactly the “safe slots only” pattern the worthiness rules already require (no auto-generated novelty/significance prose).
4. A quarterly cadence (mirroring the existing Provider Atlas cadence in the finalization plan) for a “Vox Efficacy Report” micro-publication — per the existing gap map’s Gap E (AI/SWE micro-publication track), this is a reproducibility_infra/algorithmic_improvement-class artifact, not an Atlas-class one, and should route to venues like REP/MSR/ICSE-SEIP rather than IMC/MLSys.
5. Once Gap I (HF adapter) lands, every quarterly report’s underlying corpus snapshot and scoreboard data publishes as a versioned HF dataset alongside the arXiv/Zenodo artifact — satisfying the reproducibility expectation (§B) that a benchmark claim ship its raw data, not just aggregate numbers.

H. Worked example — a small blind benchmark against humaneval-vox (this session)

Without MENS access, this session used itself (Claude Sonnet 5) as a stand-in “AI generating Vox code,” to produce one real, reproducible data point rather than only prose recommendations. Methodology, honestly stated with its limitations:

• Corpus: 5 of the 31 held-out (training_eligible: false) humaneval-vox problems — 041-nth-prime, 043-triangle-valid, 044-digital-root, 045-caesar-shift, 049-merge-sorted — chosen for a spread of difficulty (arithmetic/number-theory, boolean logic, string/char manipulation, list merging).
• Blindness: solutions were generated from each problem’s spec.toml prompt/signature only, without reading the corresponding reference.vox (avoiding the exact contamination failure mode §A/§D-Gap-J describes). Vox syntax itself was learned from two different golden examples (examples/golden/fizzbuzz-equivalent and examples/golden/str_utils.vox) — analogous to consulting language documentation, not the benchmark’s answer key.
• Verification: each generated solution was composed with the corpus’s own fn main() assertion block (not the assertions this session wrote) and run through the real vox toolchain — vox check for compilation, then vox run --mode interp for the assertions. Exit codes only; no LLM judge, per lesson A4.
• Oracle control: before scoring any candidate, all five fixtures’ reference solutions were run through the identical harness and passed 5/5. This is what licenses reading a candidate failure as the model’s rather than the harness’s.
• Reproduced across two builds: the full run (oracle + candidates) was executed twice against independently-produced vox binaries — 0.6.0+build.dev (unknown) and 0.6.0+build.4866 (fda6357f9) — and returned identical results both times (oracle 5/5, candidates 4/5, same fixture failing). The re-run was not ceremonial: the first binary had appeared mid-build and could have been partial, so its result was not trustworthy until confirmed against a cleanly-completed build.

H.1 Verified result: 4 / 5 pass@1

Fixture	Result	Failure mode
041-nth-prime	PASS	—
043-triangle-valid	PASS	—
044-digital-root	PASS	—
045-caesar-shift	PASS	—
049-merge-sorted	FAIL	parse error on &&; four Cannot unify Int with Option(Int) type errors underneath

Sample size 5 — far too small to be a capability claim about anything. The 95% Wilson interval on 4/5 is roughly 38%–96%, which is the honest width and precisely why §F mandates interval reporting rather than headline point scores. The value here is not the number; it is the failure taxonomy below.

H.2 The failure is the finding: four C-family idiom hazards, in escalating depth

Isolating 049 by repairing one layer at a time exposed four distinct ways a model trained on Python/Rust/JS mis-writes Vox — and they are not equally serious:

1. && → and — hard parse error. Loud, trivially repairable, cheap.
2. len(xs) → xs.len() — free function vs. method. Loud, trivially repairable.
3. xs[i] returns Option[int], not int — the deep one. After repairing (1) and (2), the program still failed with four Cannot unify Int with Option(Int) errors. Safe indexing is a type-system difference, not a spelling difference: the reference solution restructures the whole loop body around match xs[i] { Some(x) => … None => … }. No amount of surface-token correction reaches it; the model has to know the semantics.
4. == → is — emitted as a warning (“== works, but Vox’s canonical spelling is is”), not an error. Notably, three of the four passing solutions also triggered it and passed anyway.

The (3) vs. (4) contrast is the actionable result. Vox already ships the right affordance for divergences like (4) — a compiler diagnostic that names the canonical spelling — and that class costs a model nothing, since the code runs regardless. Class (3) has no such affordance and is where an LLM-targeting language actually loses: the model must restructure control flow, and the only feedback is a type error several lines from the real cause. If Vox wants to measure and improve itself as an LLM target, ranking divergences by which class they fall into — cheap-and-loud (1,2,4) vs. structural (3) — is a more useful metric than aggregate pass rate, and it is directly derivable from the diagnostics the compiler already emits. This is a concrete candidate for the algorithmic_improvement micro-publication track (Gap E), and it is exactly the kind of finding the corpus bridge (§I item 1) would surface continuously instead of anecdotally.

H.3 Build-infrastructure finding

• Result of this run: the release build did not complete inside this session. It first blocked on the workspace target/ lock held by ~40 concurrent cargo/rustc processes already running on this machine outside this session (branch fix-all-ci-failures, consistent with other active work), then, once it acquired the lock, rustc died compiling vox-orchestrator-mcp with exit code 0xc0000409 (STATUS_STACK_BUFFER_OVERRUN). That signature is misleading: the backtrace shows handle_alloc_error → rust_oom → default_alloc_error_hook, i.e. memory exhaustion, not a code defect — there were no error[E…] diagnostics anywhere in the log. Machine: 15.7 GB total RAM, ~4 GB free even at rest; vox-orchestrator-mcp links ~80 crates and was building at opt-level=3 with linker-plugin-lto, whose borrow-check pass needs multi-GB. Under contention it lost the allocation.

  This is a reproducibility_infra-class finding in its own right, and a more useful one than a clean pass would have been. Three concrete consequences, all now encoded in the implementation plan: (1) §F’s runner must be a scheduled CI job with a dedicated build lane, never an ad hoc on-demand session; (2) any operator-facing build instruction must name the -j 2 workaround, because the crash signature gives no hint that memory is the problem and will send people hunting for a nonexistent compile error; (3) the benchmark’s binary resolver must accept a dev-profile vox binary, since verification needs vox check/vox run to be correct rather than fast — published latency comes from the generation call, not the verification subprocess, so the build profile cannot bias any reported metric.

• Artifacts produced: the 5 blind-generated solutions, the composed programs (candidate + corpus assertions), and the compiler logs are in this session’s scratchpad (gen_0NN.vox, composed/0NN.vox, composed/0NN.check.log) — not committed, since a 5-fixture anecdote is not a corpus artifact. Anyone with a vox binary can reproduce the table above.

Method note on the credibility argument in §A. An earlier draft of this section reported that verification was blocked and declined to state a pass rate, because at that point the toolchain genuinely had not built and lesson A4 forbids asserting an unverified pass/fail. Once the build succeeded, the benchmark was actually run and the number replaced the placeholder. Both halves of that sequence are the point: the rule is verify before asserting, not avoid asserting — and the oracle control (§H, 5/5 on reference solutions) is what makes the eventual assertion mean anything, since without it “4/5” is indistinguishable from a harness that passes 80% of everything.

I. Prioritized action list

1. (Small, high-leverage) Bridge vox models eval to humaneval-vox’s held-out set (closes Gap J’s wiring half) — makes every other item in this doc concrete instead of aspirational.
2. (Small) Add a rolling-addition timestamp field to humaneval-vox fixtures and a monthly-addition cadence (closes Gap J’s contamination half).
3. (Small) Read arm_stats/scientia_model_profile_learning in the live scorer (closes the write-only loop, §E-1).
4. (Medium) Unify vox-research-shim::selection::free_tier onto vox-orchestrator::models::scoring (§E-2).
5. (Medium) Build the HF publishing adapter (Gap I) — start with the dataset card for humaneval-vox itself, mechanical from the existing manifest.
6. (Medium) Ship the §F CI-regenerated leaderboard data file + docs-astro route (Gap K).
7. (Larger, follow-on) RouteLLM-style learned per-request classifier trained on Vox’s own accumulated outcome data (§E-4); cascade-on-actual-confidence (§E-5).
8. (Ongoing, process) Every future efficacy claim: symbolic verification first, CI/rank-interval reporting, pinned exact competitor configs, no privately-tuned Vox variant — lessons A1-A4, enforced by wiring benchmark runs through the existing worthiness-gated SCIENTIA pipeline (§G) rather than publishing ad hoc numbers.