Crate: vox-llm-egress · Version: 0.6.0 · Target: Library (lib.rs)
Sanctioned low-layer LLM provider egress: chat_once/stream_once/embed_once + per-provider AIMD throttle. Pure wire — no config/secret resolution (callers pass a resolved EgressRequest). The single path all OpenAI-compatible inference egress goes through.
Sanctioned low-layer LLM provider egress. Pure wire: callers pass a fully-resolved
[EgressRequest] (resolution lives in vox_config::resolve_egress); this crate does
throttle + HTTP + 429 handling + response parsing. It owns NO config/secret resolution
and NO telemetry-to-db (both pull higher layers) — chat_once returns the tokens/cost
callers need to record telemetry themselves.
| Module | Description |
|---|
throttle | — |
| Struct | Description |
|---|
EgressRequest | A fully-resolved provider request. No resolution happens in this crate. Debug is hand-written to redact api_key so the bearer token can never leak into logs/traces/error messages. headers here carry only non-secret attribution (HTTP-Referer / X-Title); the bearer is applied separately from api_key. |
ChatMessage | One chat message on the wire (OpenAI-compatible). The three trailing fields are additive plumbing for a full tool-calling turn (Task 1.3b): an assistant message MAY carry tool_calls (the calls it requested), and a role: "tool" result message carries tool_call_id (and optionally name) to correlate the result back to the specific call. All three are skip_serializing_if = "Option::is_none" so every existing plain {role, content} caller (ghost_text, inline_edit, plan, eval, judge, route, …) continues to serialize an unchanged wire payload — no tool_calls/tool_call_id/name keys at all, not even null, when left None. |
LlmImageUrl | — |
ToolDef | Tool definition passed through to the provider. |
EgressToolCall | A single tool invocation the model requested, parsed from the OpenAI-compatible message.tool_calls wire shape: {"id": "...", "type": "function", "function": {"name": "...", "arguments": "<json-encoded string>"}}. This crate defines its own minimal type here rather than reusing vox_openai::chat_completion::ChatCompletionToolCall (which this crate already depends on for other reasons) for two reasons: (1) that type has no id field, which a future tool-dispatch loop needs to correlate a call with its result message, and (2) chat_once parses the whole response body as a raw serde_json::Value rather than through vox_openai’s typed ChatCompletionResponse — pulling in a typed sub-struct for just this one field would be inconsistent with the rest of this file. A future consolidation task may want to unify these; not done here to keep this change to pure additive plumbing. |
ChatParams | Per-call generation parameters. |
EgressChatResponse | Parsed chat result. Carries usage/cost/latency so callers record telemetry. |
| Enum | Description |
|---|
LlmContentPart | — |
EgressError | Structured egress failure so callers map to their own error types. |
| Function | Description |
|---|
is_context_exceeded | True when this error’s underlying HTTP response body looks like a provider-reported “prompt exceeded the model’s context window” failure. See [is_context_exceeded_error] for the matching rules and their caveats. Only [EgressError::Status] carries a response body, so every other variant is false (a rate-limit, transport failure, or JSON-decode failure is never a context-overflow condition). |
is_context_exceeded_error | Best-effort classifier for “the provider rejected this request because the prompt (plus requested completion) exceeded the model’s context window.” Vox routes chat through 5 provider lanes (see ChatProviderRouteKind in vox_actor_runtime::model_resolution) — OpenRouter, a generic OpenAI-compatible endpoint, HuggingFace Router, HuggingFace Dedicated, and Ollama — and each reports context overflow with different HTTP-body phrasing. This mirrors the pattern LiteLLM uses to normalize the same class of provider-specific error strings into a single ContextWindowExceededError (see docs/src/architecture/multi-provider-local-cloud-routing-research-2026-07-30.md). # Matching approach This does a case-insensitive substring match of body against a curated phrase list covering common OpenAI-compatible, OpenRouter, HuggingFace, and Ollama overflow phrasings. It is deliberately not status-code-gated: providers use a mix of 400, 413, and even 422 for this condition (and proxies like OpenRouter may relay a wrapped upstream body under its own status), so gating on a fixed status code would silently miss real cases. The tradeoff is the inverse risk — a body that happens to contain one of these phrases for an unrelated reason would be misclassified — but that is judged unlikely for a technical error string. # Completeness caveat This is a best-effort, non-exhaustive phrase list, not a guaranteed-complete provider taxonomy. Provider error text changes over time and new providers may phrase this differently again. Extend the phrase list as new provider phrasings are observed in the wild; do not treat a false result as proof the request wasn’t a context-overflow failure. |
estimate_cost | The ONE cost estimate: (prompt+completion tokens)/1000 * cost_per_1k. Used only when the provider reports no cost. Callers MUST NOT re-implement this math (single source). |
| Type | Description |
|---|
ChatStream | Streaming item type for [stream_once]. |