vox-eval API Reference (generated)

vox-eval

Crate: vox-eval · Version: 0.6.0 · Target: Library (lib.rs)

Evaluation metrics for model outputs and Vox code samples: deterministic scoring heuristics (format/safety/quality/parse) + MENS CompileVerdict. Not the interpreter — the --interp evaluator lives in vox-compiler/src/eval/.

Crate Overview

Lightweight evaluation metrics for model outputs and Vox code samples (held-out / smoke tests).

Functions are deterministic heuristics—not a replacement for human eval or full static analysis.

Public API Surface

Modules

Module	Description
compare	—
corpus_score	—
corpus_stats	—
mens	—
provenance	—
report	—

Structs

Struct	Description
EvalMetrics	Aggregated scores from a single eval run (format, safety, quality proxy, parse validity, coverage).
CollateralDamageReport	Result of evaluating collateral damage on a held-out benchmark.
CollateralDamageConfig	Configuration for collateral damage evaluation.
SemanticEntropyReport	Result of evaluating semantic entropy (diversity) of model samples.

Functions

Function	Description
format_validity_score	Scores model output based on whether it starts with common refusal prefixes.
is_safety_rejection	Returns true if the text matches known unsafe prompt-injection heuristics.
detect_constructs	Returns construct names whose regex matches at least once in code.
construct_coverage_score	—
scope_compliance_score	Bounded-domain heuristic for docs / examples: 1.0 when no high-risk escape patterns appear. Used by vox doctor --scope as a coarse guardrail (not a full security audit).
eval_collateral_damage	Evaluate collateral damage by comparing pre/post training scores on a held-out benchmark. Research (Continual Learning §catastrophic-forgetting) proves that fine-tuning without held-out evaluation hides regression. This function computes the degradation rate and recommends blocking promotion if it exceeds the threshold. eval_fn is a caller-supplied closure that evaluates the model against the benchmark and returns a score in [0.0, 1.0].
eval_collateral_damage_suite	Evaluate collateral damage across multiple benchmarks. Returns Err with the first benchmark that exceeds the threshold.
cargo_build_reward	Compilation-driven feedback for Rust code. Spawns a lightweight cargo check in a temporary directory containing the provided snippet. Returns 1.0 if it passes, 0.0 otherwise.
cargo_test_reward	Test-driven feedback for Rust code. Spawns a lightweight cargo test in a temporary directory containing the provided snippet. Returns 1.0 if it passes, 0.0 otherwise.
eval_semantic_entropy	Sample n outputs from the model for the same prompt, extract code, and measure structural diversity based on a pseudo-AST hash. This avoids a circular dependency on the full vox-compiler by using a regex-based structural “shape” extraction.
extract_vox_code	Heuristic code extractor for triple-backticked blocks.