feat: OpenAI-compatible API server example (Network#run)#137
Merged
Conversation
examples/openai_server.cr serves a loaded SHAInet model over a subset of the OpenAI REST API so existing OpenAI clients (openai Python pkg, curl, LangChain) work as a drop-in local endpoint: - POST /v1/chat/completions — streaming (SSE) and non-streaming, OpenAI JSON with usage (prompt/completion/total tokens) and finish_reason. - GET /v1/models — lists the loaded model. - GET /health — liveness probe. Built entirely on Network#run + BPETokenizer + Sampler, reusing llama_chat's load path and chat-template detection (ChatML / LLaMA3 / plain). Honors temperature, max_tokens, stop (string or array), seed, and maps frequency/presence penalty onto the repetition penalty. Generation is mutex-serialized because the KV cache is shared mutable state. Security: binds 127.0.0.1 with no auth by default; set SHAINET_API_KEY to require an Authorization: Bearer header. Same quantization/offload env vars as llama_chat (SHAINET_FP32 / SHAINET_Q4 / SHAINET_MOE_OFFLOAD).
Some OpenAI clients are configured with a base URL that already includes /v1 (so they request /v1/models), others without it (requesting /models directly, which 404'd). Strip an optional leading /v1 before routing so both forms hit the same handlers.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an example OpenAI-compatible HTTP API server that exposes a locally loaded SHAInet model via a small subset of the OpenAI REST API, making it easy to point existing OpenAI client tooling at a local endpoint.
Changes:
- Introduces
examples/openai_server.cr, implementing/v1/chat/completions(streaming SSE + non-stream),/v1/models, and/health. - Documents the new example server and its default local/no-auth security posture in
README.md.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| README.md | Adds a README entry describing the OpenAI-compatible example server and how to run/secure it. |
| examples/openai_server.cr | New example: OpenAI-shaped chat completions API (streaming + non-streaming) backed by Network#run, with optional bearer auth. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/openai_server.cr— a local HTTP server that exposes a loaded SHAInet model over a subset of the OpenAI REST API, so existing OpenAI clients (theopenaiPython package,curl, LangChain, …) can point at it as a drop-in endpoint. Built entirely onNetwork#run.Endpoints
POST /v1/chat/completions— streaming (SSE) and non-streaming. OpenAI-shaped JSON:choices[].message/delta,finish_reason, andusage(prompt/completion/total tokens).GET /v1/models— lists the loaded model.GET /health— liveness probe.Reuse / behavior
HFLoader.load+BPETokenizer.from_hf; same quantization/offload env vars asllama_chat(SHAINET_FP32/SHAINET_Q4/SHAINET_MOE_OFFLOAD).temperature,max_tokens,stop(string or array),seed; mapsfrequency_penalty/presence_penaltyonto SHAInet's repetition penalty (top_pis approximated by a fixedtop_k, documented inline).Security
Binds
127.0.0.1with no auth by default. SetSHAINET_API_KEYto requireAuthorization: Bearer <key>; pass an explicit host (e.g.0.0.0.0) only deliberately.Verified
Smoke-tested against
Qwen3-0.6B:/healthand/v1/modelsreturn correct JSON.stream:true→ role delta, per-token content deltas, finish chunk, thendata: [DONE].crystal tool formatclean,ameba0.Usage: