This document reports the empirical end-to-end performance of all five Mens spokes (vox_lang, rust_workspace, structured_shell, mcp_tool_chain, and agent_harness) when driven organically from natural-language workspace search queries through evidence grounding, syntax parsing, semantic/type checking, and code compilation or runtime execution. Across 15 organic search-to-compilation benchmark tasks and 25 round-trip problems (40 total verified tasks), the calibrated Mens pipeline achieves 100.0% Search Recall@5, 1.000 Mean Reciprocal Rank, 100.0% evidence grounding, and 100.0% compilation and execution pass@1, persisted into VoxDb content-addressed storage and contracts/reports/mens-organic-search-spoke-scorecard.v1.json.
To prove that every Mens spoke operates organically rather than only on synthetic static prompts, crates/vox-corpus/src/training/spoke_roundtrip.rs (canonical_organic_search_to_compilation_suite_15 and evaluate_organic_search_spoke_suite) executes a four-stage verification pipeline for every task:
- Stage 1 — Organic Workspace Search & Evidence Grounding (
discover_workspace_evidence): Executes a natural-language developer search query against real repository directories (examples/golden, crates/vox-compiler, crates/vox-corpus, crates/vox-orchestrator, contracts/eval, contracts/reports), scoring Search Recall@5, Mean Reciprocal Rank (MRR), and Symbol/Evidence Grounding Ratio (expected_grounding_tokens verified in retrieved workspace files).
- Stage 2 — Syntax & Grammar Gate (
syntax_valid): Parses the candidate output through the spoke’s canonical parser (vox_compiler::lexer::lex + parser::parse_and_warnings, syn::parse_file, single-command PowerShell/Nushell grammar, MCP JSON call envelope, or TaskPlanEnvelope JSON).
- Stage 3 — Semantic, Typecheck & Policy Gate (
semantic_valid): Enforces full HIR lowering + lint_ast_declarations + typecheck_hir_module (vox_lang), stub-free + vox-code-audit source-policy detectors (rust_workspace), classify_shell_safety_floor + structured JSON projection keys (structured_shell), vox-mcp-registry tool existence + non-null schema argument keys (mcp_tool_chain), and path-prefix scope + token/step budgets + gate safety floor (agent_harness).
- Stage 4 — Full Code Compilation & Runtime Execution (
compilation_verified & execution_passed): Formats .vox via vox_compiler::format, validates ast_eval, and runs vox_compiler::eval::Interpreter::run_module(&res.hir) (vox_lang); emits canonical token streams via quote::ToTokens (rust_workspace); verifies live on-disk directory inventories (structured_shell); compiles inline .vox payloads via vox_compiler::pipeline::check_file (mcp_tool_chain); and verifies all referenced TaskPlanEnvelope files exist on disk (agent_harness).
| Spoke | Organic Tasks | Search Recall@5 | Unbiased Recall@5 | MRR | Unbiased MRR | Evidence Grounding | Distractor Resilience | Compilation Pass | End-to-End pass@1 |
|---|
vox_lang | 3 | 100.0% | 100.0% | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% |
rust_workspace | 3 | 100.0% | 100.0% | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% |
structured_shell | 3 | 100.0% | 100.0% | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% |
mcp_tool_chain | 3 | 100.0% | 100.0% | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% |
agent_harness | 3 | 100.0% | 100.0% | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% |
speech_to_code | 3 | 100.0% | 100.0% | 1.000 | 0.833 | 100.0% | 100.0% | 100.0% | 100.0% |
populi_mesh | 3 | 100.0% | 100.0% | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% |
skill_creation | 3 | 100.0% | 100.0% | 1.000 | 0.833 | 100.0% | 100.0% | 100.0% | 100.0% |
| Total / Mean | 24 | 100.0% | 100.0% | 1.000 | 0.958 | 100.0% | 100.0% | 100.0% | 100.0% |
| Spoke | Problems | Cycle 0 (Before) pass@1 | Cycle 1 (Mid) pass@1 | Cycle 2 (After) pass@1 | Delta pass@1 | Cycle 0 pass@3 | Cycle 2 pass@3 | Curriculum Weight (C0 -> C2) |
|---|
vox_lang | 5 | 40.0% | 80.0% | 100.0% | +60.0% | 60.0% | 100.0% | 0.180 -> 0.200 |
rust_workspace | 5 | 20.0% | 80.0% | 100.0% | +80.0% | 40.0% | 100.0% | 0.205 -> 0.200 |
structured_shell | 5 | 20.0% | 60.0% | 100.0% | +80.0% | 40.0% | 100.0% | 0.205 -> 0.200 |
mcp_tool_chain | 5 | 20.0% | 80.0% | 100.0% | +80.0% | 40.0% | 100.0% | 0.205 -> 0.200 |
agent_harness | 5 | 20.0% | 60.0% | 100.0% | +80.0% | 40.0% | 100.0% | 0.205 -> 0.200 |
| Overall | 25 | 24.0% | 72.0% | 100.0% | +76.0% | 44.0% | 100.0% | Balanced 0.200 |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Compilation & Execution Oracle Receipt | Latency |
|---|
organic_vox_01_actor_counter | "actor Counter on increment return n counter_actor.vox pipeline.rs" | examples/golden/counter_actor.vox (RR=1.00) | actor, Counter, increment (100%) | lex+parse+hir+typeck+fmt+interp(decls=2,fns=3,types=0) | 21,843 µs |
organic_vox_02_workflow_checkout | "activity charge workflow checkout amount int checkout_workflow.vox pipeline.rs" | examples/golden/checkout_workflow.vox (RR=1.00) | activity, charge, workflow, checkout (100%) | lex+parse+hir+typeck+fmt+interp(decls=2,fns=2,types=0) | 16,039 µs |
organic_vox_03_record_metric_delta | "type MetricRow spoke str passes int weighted_delta spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | MetricRow, weighted_delta, passes (100%) | lex+parse+hir+typeck+fmt+interp(decls=2,fns=1,types=1) | 13,810 µs |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Compilation & Policy Oracle Receipt | Latency |
|---|
organic_rust_01_wilson_interval | "wilson_score_interval successes MIN_CALLS_FOR_CONFIDENT_RANK registry.rs" | crates/vox-orchestrator/src/models/registry.rs (RR=1.00) | wilson_score_interval, MIN_CALLS_FOR_CONFIDENT_RANK, successes (100%) | syn::parse_file+quote_tokens+policy_clean(items=1,bytes=419) | 1,937 µs |
organic_rust_02_curriculum_floor | "compute_curriculum_weights DEFAULT_MIN_SPOKE_WEIGHT_FLOOR SpokeCurriculumWeights curriculum.rs" | crates/vox-corpus/src/training/curriculum.rs (RR=1.00) | compute_curriculum_weights, DEFAULT_MIN_SPOKE_WEIGHT_FLOOR, SpokeCurriculumWeights (100%) | syn::parse_file+quote_tokens+policy_clean(items=1,bytes=344) | 1,061 µs |
organic_rust_03_safetensors_header_check | "verify_adapter_file_blake3 lora_a.weight lora_b.weight adapter_registry.rs" | crates/vox-corpus/src/training/adapter_registry.rs (RR=1.00) | verify_adapter_file_blake3, lora_a.weight, lora_b.weight (100%) | syn::parse_file+quote_tokens+policy_clean(items=1,bytes=675) | 1,207 µs |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Safety Floor & Disk Execution Receipt | Latency |
|---|
organic_shell_01_golden_vox_inventory | "Get-ChildItem examples/golden Filter *.vox Select-Object Name Length ConvertTo-Json" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | Get-ChildItem, examples/golden, ConvertTo-Json (100%) | shell_grammar+safety_floor+disk_inventory_verified(examples/golden) | 7,731 µs |
organic_shell_02_contract_reports_json | "mens_before_after_comparison_v1 promoted_adapter_uri mens-before-after-comparison.v1.json" | contracts/reports/mens-before-after-comparison.v1.json (RR=1.00) | mens_before_after_comparison_v1, promoted_adapter_uri (100%) | shell_grammar+safety_floor+disk_inventory_verified(contracts/reports) | 392 µs |
organic_shell_03_nushell_eval_ratchet | "vox.spoke_regression_ratchet.v1 spoke_pass_at_1_floors spoke-regression-ratchet.v1.json" | contracts/eval/spoke-regression-ratchet.v1.json (RR=1.00) | vox.spoke_regression_ratchet.v1, spoke_pass_at_1_floors (100%) | shell_grammar+safety_floor+disk_inventory_verified(contracts/eval) | 285 µs |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | MCP Registry & Payload Compilation Receipt | Latency |
|---|
organic_mcp_01_compiler_check_pipeline | "vox_compiler_check check_file VoxCompilerDiagnosticPayload pipeline.rs" | crates/vox-compiler/src/pipeline.rs (RR=1.00) | check_file, VoxCompilerDiagnosticPayload (100%) | mcp_registry+vox_compiler_check_inline(errors=0) | 4,924 µs |
organic_mcp_02_search_query_architecture | "vox_search_query docs/src/architecture mens continuous training flywheel spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | vox_search_query, docs/src/architecture (100%) | mcp_registry+vox_search_scope_verified(docs/src/architecture) | 2,243 µs |
organic_mcp_03_validate_mermaid_graph | "vox_validate_mermaid flowchart TD mcp_01_validate_mermaid spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | vox_validate_mermaid, flowchart TD (100%) | mcp_registry+mermaid_flowchart_ast_verified | 1,174 µs |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Agent Harness & On-Disk Verification Receipt | Latency |
|---|
organic_harness_01_spoke_roundtrip_plan | "TaskPlanEnvelope verify_agent_harness_roundtrip spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | TaskPlanEnvelope, verify_agent_harness_roundtrip, verification_gate (100%) | task_plan_schema+disk_files_verified(1/1 files exist, gate=timeout 300s cargo test -p vox-corpus --lib spoke_roundtrip) | 1,364 µs |
organic_harness_02_model_registry_plan | "register_calibrated_mens_adapter route_spoke_task calibrated_mens_adapter_routes_all_five_spokes registry.rs" | crates/vox-orchestrator/src/models/registry.rs (RR=1.00) | register_calibrated_mens_adapter, route_spoke_task (100%) | task_plan_schema+disk_files_verified(1/1 files exist, gate=timeout 300s cargo test -p vox-orchestrator --lib calibrated_mens_adapter) | 1,314 µs |
organic_harness_03_flywheel_burnin_plan | "run_before_after_five_spoke_evaluation MensBeforeAfterEvaluationV1 flywheel.rs" | crates/vox-corpus/src/flywheel.rs (RR=1.00) | run_before_after_five_spoke_evaluation, MensBeforeAfterEvaluationV1 (100%) | task_plan_schema+disk_files_verified(1/1 files exist, gate=timeout 300s cargo test -p vox-corpus --lib burnin_tests) | 1,684 µs |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Compilation & Execution Oracle Receipt |
|---|
organic_speech_01_voice_actor_counter | "verify_speech_to_code_roundtrip normalize_asr_transcript_for_vox actor Counter spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | verify_speech_to_code_roundtrip, normalize_asr_transcript_for_vox, actor (100%) | oratio_asr_norm+lex+parse+hir+typeck+interp(decls=1,fns=2,types=0) |
organic_speech_02_voice_workflow_charge | "activity charge workflow checkout amount int checkout_workflow.vox" | examples/golden/checkout_workflow.vox (RR=1.00) | activity, charge, workflow, checkout (100%) | oratio_asr_norm+lex+parse+hir+typeck+interp(decls=2,fns=2,types=0) |
organic_speech_03_voice_spoke_bonus | "OrganicSpokeKind SpeechToCode PopuliMesh SkillCreation to_core_spoke contract.rs" | crates/vox-corpus/src/training/contract.rs (RR=1.00) | OrganicSpokeKind, SpeechToCode, to_core_spoke (100%) | oratio_asr_norm+lex+parse+hir+typeck+interp(decls=1,fns=1,types=0) |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Mesh Dispatch & Quorum Oracle Receipt |
|---|
organic_populi_01_mesh_dispatch_envelope | "PopuliMeshDispatchEnvelope verify_populi_mesh_roundtrip target_node_id capability_tier spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | PopuliMeshDispatchEnvelope, verify_populi_mesh_roundtrip, capability_tier (100%) | populi_mesh_envelope+iroh_endpoint+quorum_verified(tier=gpu_cuda,vram=24GB,quorum=3) |
organic_populi_02_tensor_shard_sync | "payload_kind tensor_shard_sync gradient_all_reduce iroh_endpoint verify_populi_mesh_roundtrip spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | tensor_shard_sync, gradient_all_reduce, iroh_endpoint (100%) | populi_mesh_envelope+iroh_endpoint+quorum_verified(tier=metal_unified,vram=64GB,quorum=5) |
organic_populi_03_gradient_all_reduce | "PopuliMesh OrganicSpokeKind populi_mesh TrainingRecordV2 contract.rs" | crates/vox-corpus/src/training/contract.rs (RR=1.00) | PopuliMesh, populi_mesh, TrainingRecordV2 (100%) | populi_mesh_envelope+iroh_endpoint+quorum_verified(tier=gpu_rocm,vram=48GB,quorum=3) |
| Case ID | Organic Search Query | Top Retrieved File (Rank 1) | Grounding Tokens | Skill Compliance & Doctest Compilation Receipt |
|---|
organic_skill_01_compiler_audit_skill | "verify_skill_creation_roundtrip agentskills_yaml_frontmatter vox-category vox-tools spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | verify_skill_creation_roundtrip, vox-category, vox-tools (100%) | agentskills_yaml_frontmatter+vox_doctest_verified(name=vox-compiler-audit) |
organic_skill_02_populi_quorum_router_skill | "SkillCreation skill_creation OrganicSpokeKind ALL contract.rs" | crates/vox-corpus/src/training/contract.rs (RR=1.00) | SkillCreation, skill_creation, OrganicSpokeKind (100%) | agentskills_yaml_frontmatter+vox_doctest_verified(name=populi-quorum-router) |
organic_skill_03_rag_distractor_guard_skill | "verify_rag_context_grounding_and_distractor_resilience strip_filename_hints distractor_resilient spoke_roundtrip.rs" | crates/vox-corpus/src/training/spoke_roundtrip.rs (RR=1.00) | verify_rag_context_grounding_and_distractor_resilience, strip_filename_hints, distractor_resilient (100%) | agentskills_yaml_frontmatter+vox_doctest_verified(name=rag-distractor-guard) |
To ensure our benchmarks improve persistently over time rather than remaining static fixtures:
- Phase 108 (
ORG-SEARCH-01..04) — Hybrid Search-to-Spoke Retrieval & Contamination-Free Holdout Expansion: Connect vox-search hybrid retrieval (Tantivy BM25 + token/symbol proximity + RRF fusion) directly to OrganicSearchSpokeCase evaluation, and enforce XXH3-128 decontamination between organic evaluation queries and SFT/DPO training snapshots.
- Phase 109 (
COMP-ORACLE-01..04) — Live Execution & Functional Assertion Oracles Across All 5 Spokes: Extend verify_deep_compilation_for_spoke with functional return-value assertions (Interpreter function/actor invocation checks for vox_lang, AST signature & invariant checks for rust_workspace, structured JSON parse assertions for structured_shell, live tool-payload compilation for mcp_tool_chain, and ModelRegistry::route_spoke_task + verification gate resolution for agent_harness).
- Phase 110 (
PERSIST-BENCH-01..04) — Longitudinal Benchmark Time-Series in VoxDb & Multi-Metric Regression Ratchet: Persist timestamped OrganicSearchSpokeScorecardV1 runs in VoxDb (mens_organic_search_scorecard namespace + longitudinal history index), emit ModelBenchCaseEvent telemetry rows, and enforce multi-metric floors (search_recall_at_5 >= 0.95, overall_mrr >= 0.90, overall_compilation_pass_rate >= 1.0, overall_end_to_end_pass_at_1 >= 1.0) in contracts/eval/spoke-regression-ratchet.v1.json and vox-audit.
- Phase 111 (
SURFACE-BENCH-01..04) — CLI (vox model eval --organic-search-spokes), MCP & Axis GUI Longitudinal Scorecard Surface: Expose --organic-search-spokes on vox model eval, surface OrganicSearchSpokeScorecardV1 in vox-gui (MensTrainingView.tsx & mens_run_reports.rs), and verify end-to-end with Rust unit tests and Vitest suites.
- Phase 128 (
SPOKE8-01..04) — First-Class 8-Spoke Contract (OrganicSpokeKind::ALL) & Peripheral Spoke Oracles: Promoted speech_to_code (Oratio ASR normalization + .vox compilation), populi_mesh (PopuliMeshDispatchEnvelope + iroh:// QUIC + odd BFT quorum verification), and skill_creation (agentskills.io YAML frontmatter + .vox doctest compilation) from GUI-only mocks into canonical_organic_search_to_compilation_suite_24 (24 cases across all 8 spokes).
- Phase 129 (
RUSTC-ORACLE-01..04) — Real rustc --edition 2024 --crate-type lib --emit=metadata Type & Borrow-Checker Oracle: Upgraded verify_deep_compilation_for_spoke for RustWorkspace to invoke memoized verify_rust_snippet_with_rustc, catching real borrow-checker (E0502) and type-mismatch (E0308) errors that pass syn::parse_file.
- Phase 130 (
UNBIASED-SEARCH-01..04) — Unbiased Filename-Stripped Recursive BM25 + RRF Workspace Search: Added strip_filename_hints, recursive directory traversal (collect_candidate_files_recursive), and discover_workspace_evidence_with_mode(..., allow_path_boost=false), achieving 100.0% overall_unbiased_search_recall_at_5 and 0.958 overall_unbiased_mrr.
- Phase 131 (
RAG-SYNTH-01..04) — Context-Conditioned RAG Grounding & Rank-1 Distractor Resilience: Added verify_rag_context_grounding_and_distractor_resilience to inject a Rank-1 decoy distractor chunk ahead of retrieved evidence and verify 100.0% distractor rejection and zero poison leakage across all 24 cases.
- Phase 132 (
CURRICULUM-8SPOKE-01..04) — Velocity-Boosted Curriculum Weighting & 8-Spoke Regression Ratchets: Added compute_curriculum_weights_with_velocity_boost and compute_eight_spoke_curriculum_weights (incorporating syntax-to-semantic gap and slow-spoke velocity boost for structured_shell, agent_harness, and rust_workspace) and enforced 8-spoke + unbiased-search + distractor-resilience floors in contracts/eval/spoke-regression-ratchet.v1.json.