Docs Bot Evaluation & Decision D9 Pivot Rule

Docs Bot Evaluation & Decision D9 Pivot Rule

Evaluates commercial and academic documentation maintenance bots against Vox requirements under Decision D9. Compares systems like Promptless, Dosu, Mintlify, and DocPrism on accuracy, grounding, and policy compliance, establishing a quantitative pivot threshold that rejects third-party hosted bots in favor of in-house AST-grounded verification via vox-doc-pipeline.

Executive Summary

Phase 24 establishes the empirical evaluation framework and decision boundary for documentation verification in the Vox project (Milestone v1.3 closing / v1.4 preparation). Under Decision D9, the project evaluated whether to adopt a commercial hosted documentation bot or proceed with building an in-house verification layer grounded on compiler AST symbols and the deterministic freshness ledger (ledger.v1.json).

Based on published benchmarks from industry systems (Promptless, Dosu, Mintlify, DeepDocs) and academic empirical baselines (DocPrism, CASCADE), the project issues a definitive GO on building in-house and a NO-GO on purchasing commercial hosted bots.

---

1. Industry Hosted Documentation Bots Benchmark

A comprehensive audit was conducted across existing commercial and open-source documentation maintenance bots:

System	Primary Mode	Codebase Grounding	Reported / Observed PR Acceptance	False-Positive Flag Rate	Cost Profile	Suitability for Vox
Promptless (promptless.ai)	Merged PR triggers draft doc PR	Git diffs, PR comments, Doc Detective tests	40%–60% on API reference; <25% on architectural docs	Moderate (20%–35%) on structured changes	$1,000–$1,500/mo (Growth tier)	Low (No .vox compiler integration; calls third-party models directly)
Dosu (dosu.dev)	Continuous GitHub webhook triage & PR docs	Issue history, git diffs, PR discussions	~50% on simple docstring/API fixes	High (35%–50%) on conceptual explanations	Custom / Enterprise tiered	Low (Violates LLM facade policy; cannot execute local compiler tools)
Mintlify Agent (mintlify.com)	“Modify and wait for review” (batched PRs)	Connected repo code changes	~65% on OpenAPI/TypeScript endpoint tables	Low on schemas; High (>45%) on prose	$450/mo (Pro tier)	Low (Requires migrating doc site off Starlight; lacks .vox syntax support)
DeepDocs / Dependadocs	Per-commit bot PRs	Local AST extraction + unconstrained LLM	<20% (frequently closed without merge)	Severe (>60% notification fatigue)	OSS / Token costs	Banned (Unconstrained bot PRs trigger severe developer fatigue)

Key Findings from Industry Tools

1. Lack of Language-Specific Tooling: None of the commercial hosted bots support compiling or parsing .vox source files. They cannot validate whether an extracted code snippet adheres to current grammar (e.g. bare keywords table, query, mutation, server, tool) or uses retired syntax (such as @endpoint).
2. Developer Fatigue from Hallucinated PRs: Systems that open speculative draft PRs without compiler verification have acceptance rates below 50%. Developers routinely close or ignore ungrounded PRs.
3. Secret & Architecture Policy Non-Compliance: Commercial SaaS solutions require pushing repository ASTs and diffs to external multi-tenant cloud environments, violating Vox’s model-agnostic and privacy boundaries.

---

2. Academic Baselines & Empirical Evidence

Academic research on code-documentation inconsistency provides rigorous explanations for why naive LLM verification fails and what architectures succeed:

2.1 DocPrism (Xu, Wahab, Holmes, Lemieux — UBC, 2024)

• The Problem: Prompting state-of-the-art LLMs (e.g. GPT-4, Claude) with raw documentation and code changes produces a 90% to 98% false-positive rate. Models routinely conflate incompleteness (a high-level summary that deliberately omits implementation details) with incorrectness (a statement that contradicts code reality).
• The Solution (LCEF Architecture): DocPrism introduces Local Categorization and External Filtering. The LLM is restricted to predicting expected code behavior from doc claims, and the prediction is checked by deterministic AST filters.
• Outcome: False positives dropped from 98% to 14%, raising the classification F1 score from 0.22 to 0.77.
• Application to Vox: Any LLM verifier in Vox must never be asked open-ended questions like “Is this page accurate?”. It must only verify specific extracted claims against concrete symbol spans hashed in ledger.v1.json.

2.2 CASCADE (Consistency Analysis for Source Code And Documentation through Execution — 2024/2025)

• The Problem: Conceptual documentation in prose drifts silently without compiler diagnostics.
• The Solution: Synthesizing executable assertions from prose. An inconsistency is flagged only if:
  1. The existing code fails the synthesized test; AND
  2. A corrected reference implementation derived from the doc text passes it.
• Application to Vox: This validates Vox’s decision to treat tutorials and examples as executable tests (TUT-02 and vox doctest), where code fences are executed against the actual vox compiler.

---

3. Decision D9 Quantitative Pivot Rule

Decision D9 established that the project would only purchase a hosted documentation bot if external solutions are decisively superior to what can be built in-house.

Quantitative Thresholds

To trigger a pivot from an in-house verifier to a commercial vendor, a candidate solution must simultaneously satisfy all five criteria:

1. PR Acceptance Rate ($\ge 80%$): Draft pull requests opened by the system must achieve an acceptance rate of $\ge 80%$ on the golden eval set (contracts/reports/docs-freshness/golden-drift-eval-set.v1.json) without requiring human rewrite.
2. False-Positive Rate ($\le 10%$): Automated inconsistency alerts on status: current documentation must produce $\le 10%$ false alarms.
3. Native Compiler & Language Execution: The vendor sandbox must natively execute the Vox toolchain (vox check, vox test, vox run) and parse .vox grammar.
4. Security & Boundary Policy Compliance: Must route all model inference through compliant providers without storing code outside approved perimeters.
5. Operational Cost: Total operational cost must not exceed $100/month (our planned in-house nightly run cost is $13–$45/month across typical PR volume).

---

4. Evaluation Verdict & v1.4 Architectural Roadmap

Verdict: GO ON IN-HOUSE BUILD (NO-GO ON VENDOR PURCHASE)

No existing commercial system satisfies even two of the five required criteria. Specifically:

• 0 of 4 commercial bots support .vox grammar.
• 0 of 4 commercial bots meet the $\le 10%$ false-positive threshold on conceptual prose.
• Commercial bot pricing ($450–$1,500/mo) is 10× to 30× higher than in-house compute budgets.

The v1.4 In-House Architecture

By completing Milestone v1.3, Vox has constructed the prerequisite deterministic substrate:

1. Symbol-Span Hash Anchors: Changes to Rust or Vox definitions invalidate anchor hashes deterministically without requiring LLM evaluation.
2. Reverse Index & PR Impact Gate: PRs identify affected reader-facing documentation instantly at merge time.
3. Freshness Ledger (ledger.v1.json): Every public page is classified into verified, drifting, or broken.
4. Golden Drift Eval Set (golden-drift-eval-set.v1.json): 10 real historical cases provide an immutable benchmark for scoring verifiers.

In Milestone v1.4, the in-house verifier will build directly upon this foundation:

• Nightly Scheduled Verifier: Evaluates only state: drifting pages.
• Constrained Claim Judge: Feeds changed symbol spans and extracted claims to vox_actor_runtime::llm using DocPrism-style structured claim checking.
• Draft PR Generator: Generates targeted, compiler-verified diffs against stale code fences and prose statements.