feat(inference): allow local embeddings route#1774
Conversation
shiju-nv
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
Route OpenAI-compatible embeddings through the local inference proxy so sandboxed vector workloads reach a configured provider via the same route classification and auth path that chat, completion, and model discovery already use. - Add openai_embeddings to the OpenAI-compatible protocol set so providers (openai, nvidia) advertise embeddings routing. - Classify POST /v1/embeddings as the openai_embeddings protocol in the sandbox L7 patterns. - Serve embeddings buffered with an accurate Content-Length, since the response is a single JSON object rather than an SSE token stream. The streaming path appends an SSE error frame on a size-cap or idle-timeout truncation, which would corrupt a one-object body the client parses whole. protocol_returns_buffered_body() selects the path. - Probe an embeddings-only backend against /v1/embeddings during validation, after the chat and completion protocols so a multi-protocol route still prefers those. - Extract two shared helpers. http_status_text() backs both response formatters and adds 401/422/429/503 for embeddings passthrough and router error mapping; write_inference_router_error() backs the streaming and buffered routing paths. - Return an OpenAI-shaped embeddings body from the mock route. Tests cover profile lookup, L7 pattern detection, the mock body, and buffered Content-Length framing with no chunked transfer-encoding and no SSE error frame. Signed-off-by: Shiju <shiju@nvidia.com>
The buffered proxy path read the whole upstream response into memory with
no size bound. The route timeout bounds elapsed time but not memory, so a
misbehaving or oversized upstream could force unbounded allocation in the
sandbox proxy. The streaming path already caps each response at 32 MiB;
the buffered path did not.
Cap the buffered read at the same 32 MiB. An advertised over-cap body is
rejected from its Content-Length before any bytes are read, and chunks
accumulate under the same bound so a chunked or mislabeled body cannot
slip past. An over-cap response fails as an upstream protocol error,
surfaced as HTTP 502 at the proxy boundary, and is never partially
returned.
Tests
- cargo test -p openshell-router \
proxy_to_backend_rejects_over_cap_response_body
Signed-off-by: Shiju <shiju@nvidia.com>
…tocols
A managed route resolves to its provider profile's full protocol set, so
an embeddings model such as text-embedding-3-small lists
openai_chat_completions alongside openai_embeddings. Route verification
probed only the first writable protocol and stopped on its failure. It
sent a chat probe with the embedding model, the provider rejected it as
wrong-shape, and the route failed validation before the embeddings probe
ran. Embeddings-only configs could not be verified.
Try the advertised protocols in preference order. A request-shape
rejection (HTTP 400, 404, 405, 422) falls through to the next protocol,
so an embeddings model validates against /v1/embeddings even when the
chat probe rejects it. Credential, rate-limit, connectivity, and
upstream-health failures stay terminal and stop validation at the first
probe, so a bad key or a down backend is reported as itself rather than
masked by a later probe.
validation_probe becomes validation_probes, which returns the ordered
list, and the per-probe fallback retry (max_completion_tokens versus
max_tokens) moves into a shared helper.
Tests
- cargo test -p openshell-router \
verify_embeddings_model_falls_through_chat_probe
- cargo test -p openshell-router verify_stops_on_credentials_failure
Signed-off-by: Shiju <shiju@nvidia.com>
GET /v1/models returns a single JSON model list, the same response shape
as embeddings. The sandbox inference proxy was routing it through the SSE
streaming path. A streaming size-cap or idle-timeout truncation appends
an SSE error frame to the body, which corrupts a payload the client
parses as one JSON object.
Make response framing a property of the protocol. A new ResponseFraming
field on InferenceApiPattern is set once per pattern in default_patterns.
model_discovery and openai_embeddings are now Buffered, while chat
completions, completions, responses, and Anthropic messages stay
Streaming. The proxy dispatch gates on pattern.is_buffered(), which
replaces the stringly-typed protocol_returns_buffered_body predicate so
the streaming-versus-buffered decision lives in one place and cannot
drift across the sites that read it.
Model discovery now flows through the same buffered path as embeddings,
framed with an accurate Content-Length and bounded by the buffered-read
size cap that path already enforces.
Tests
- cargo test -p openshell-sandbox protocol_framing_classification
- cargo test -p openshell-sandbox \
inference_model_discovery_served_buffered_with_content_length
Signed-off-by: Shiju <shiju@nvidia.com>
shiju-nv
force-pushed
the
fix/inference-embeddings-route
branch
from
6f2d8ab to
5f66c3c
Compare
Gator validation: this PR is project-valid and linked issue #1771 is valid for the same work. The change is concentrated in the inference routing/proxy path and addresses a clear sandbox UX gap for Independent code review found blocking feedback, so this remains in
Current CI is green for branch checks, DCO, Markdown, Rust, Python, Helm Lint, and published gate statuses. E2E is currently skipped because no No local tests were run by gator. |
johntmyers
added
the
gator:in-review
PR Review StatusValidation: this PR is project-valid and linked issue #1771 is valid for the same work. The change is concentrated in the inference routing/proxy path and addresses a clear sandbox UX gap for Review findings:
Docs: missing for a direct user-facing capability change. Please update the existing Next state: |
Signed-off-by: Shiju <shiju@nvidia.com>
johntmyers
added
gator:watch-pipeline
Re-check After Author UpdateI re-evaluated latest head Disposition: resolved for gator review. Remaining items:
Next state: |
|
Label |
johntmyers
added
gator:approval-needed
Maintainer Approval NeededGator validation and PR monitoring are complete. Validation: PR #1774 is project-valid and linked issue #1771 is already Human maintainer approval or merge decision is now required. |
2 participants
Summary
Adds
openai_embeddingsas a first-class OpenAI-compatible protocol so sandboxed workloads can reachPOST /v1/embeddingsthrough the local inference proxy, the same way chat, completion, responses, and model discovery already route. The response is served buffered with an accurateContent-Lengthrather than through the SSE streaming path, which would corrupt a single-JSON-object body if the stream were truncated.Related Issue
Closes #1771.
Changes
openai_embeddingsto the OpenAI-compatible protocol set so providers (openai, nvidia) advertise embeddings routing.POST /v1/embeddingsas theopenai_embeddingsprotocol in the sandbox L7 inference patterns.Content-Length. An embeddings response is one JSON object; the streaming path appends an SSE error frame on a size-cap or idle-timeout truncation, which would corrupt a body the client parses whole./v1/embeddings, ordered after the chat and completion probes so a multi-protocol route still prefers those.http_status_text()(adding 401/422/429/503 for embeddings passthrough and router error mapping) andwrite_inference_router_error()across the streaming and buffered paths.Testing
mise run pre-commitpassesChecklist
--
Follow-up PRs (stacked on this branch)
This is the first commit in a stacked series; each later PR builds on the one before it, so they are easiest to review and merge in order: