Provider config
The provider is selected and configured by the runtime, not the program — the .syn never names a host or key, so it can't redirect the call or leak the key. Each knob resolves process environ > .env > default.
.env (gitignored)
SYNSEMA_LLM_PROVIDER=deepseek
DEEPSEEK_API_KEY=sk-...
The key reaches the runtime without entering the process environment — no child process or program can read it (the program would need require secret, and even then sees it redacted). No export/source needed.
Knobs
Env var / .env entry | Purpose | Default |
|---|---|---|
ANTHROPIC_API_KEY / OPENAI_API_KEY / MINIMAX_API_KEY / DEEPSEEK_API_KEY | API key; presence auto-selects the provider | — (offline if absent) |
SYNSEMA_LLM_PROVIDER | Force anthropic / openai / minimax / deepseek | auto |
SYNSEMA_LLM_MODEL | Model id | claude-sonnet-4-6 / gpt-4o / MiniMax-M3 / deepseek-chat |
SYNSEMA_LLM_MAX_TOKENS | Output cap | 4096 |
SYNSEMA_LLM_BASE_URL | Endpoint base (point at a local/compatible server) | official |
SYNSEMA_LLM_TIMEOUT | HTTP timeout in seconds for the network providers. With the default streaming transport it measures silence between bytes — each chunk renews it — so a minutes-long generation flows and a dead host still fails fast. Invalid/≤0 → default | 60 |
SYNSEMA_LLM_HTTP_STREAM | Internal SSE transport for the network providers (the language ops still return complete text; llm_stream emits real chunks). 0/false → classic non-stream path, escape hatch for odd proxies | 1 (on) |
SYNSEMA_LLM_BUDGET | Per-process LLM token budget (input + output, all ops). At the ceiling, every op degrades to the marker [llm budget exceeded: used N of M tokens] — no error, no network call, one stderr notice. Consumption is readable with llm_usage() (see LLM primitives). Invalid/0 → no budget, with a warning | — (unlimited) |
You can also force the provider per-run: synsema run app.syn --provider anthropic (flag > env > .env).
Timeouts no longer cap long generations. The network providers request the API response as a stream and reassemble it internally, so the read-timeout detects dead connections (60s of real silence) instead of limiting total generation time. Verified live: a 177s / 68 KB generation (MiniMax-M3, a reasoning model) completes where the old fixed 60s window killed it. With SYNSEMA_LLM_HTTP_STREAM=0 the timeout caps the whole call, as before. Local / on-prem (100% private)
Any OpenAI-compatible server (Ollama, LM Studio, vLLM):
SYNSEMA_LLM_PROVIDER=openai
SYNSEMA_LLM_BASE_URL=http://localhost:11434/v1 # Ollama
SYNSEMA_LLM_MODEL=llama3.1
OPENAI_API_KEY=ollama # any non-empty value
Embedded local provider (local) — GGUF in-process, zero network
With a binary compiled with the llm-local feature, the runtime runs a quantized GGUF inside the Synsema process (candle, CPU): no server, no API key, no socket at all — the only provider that works under a total deny net. Always explicit, never auto-selected:
SYNSEMA_LLM_PROVIDER=local
SYNSEMA_LLM_MODEL=/models/qwen2.5-0.5b-instruct-q4_k_m.gguf # path to the .gguf (required)
| Knob | Purpose | Default |
|---|---|---|
SYNSEMA_LLM_CTX | Context window (capped to the GGUF's own limit) | 4096 |
SYNSEMA_LLM_THREADS | CPU threads for inference | all cores |
SYNSEMA_LLM_TEMPERATURE | 0 = greedy/deterministic; >0 = sampling (fixed seed) | 0 |
SYNSEMA_LLM_MAX_CONCURRENT | Max model instances; 1 serializes concurrent calls under serve | 1 |
SYNSEMA_LLM_STREAM_BUFFER | Chunks in flight between generation and llm_stream emission | 32 |
SYNSEMA_LLM_MAX_TOKENS applies as usual. Supported architectures: llama, qwen2, qwen3 — this is the general.architecture string in the GGUF header, not the brand name; anything else fails with a clear [local error: …]. Popular families convert to GGUF declaring one of those three, so real coverage is wider than the three names suggest. Families verified live (probe = load + question + coherent answer + clean stop):
| Model (GGUF probed) | declares | Verified |
|---|---|---|
| Qwen2.5 Instruct 0.5B/3B | qwen2 | ✅ |
| Qwen3 0.6B | qwen3 | ✅ thinking model — emits raw <think>…</think>; budget SYNSEMA_LLM_MAX_TOKENS for it |
| Qwen3 4B Instruct-2507 | qwen3 | ✅ (2.5 GB GGUF, ran in 8 GB RAM). The 4B Thinking-2507 also runs but thinks for thousands of tokens per answer — impractical on CPU; prefer the Instruct variant |
| Mistral 7B Instruct v0.3 | llama | ✅ its [INST] template is auto-detected |
| Llama 3.2 1B Instruct | llama | ✅ llama3 template auto-detected |
| SmolLM2 135M Instruct | llama | ✅ chatml |
| DeepSeek-R1-Distill-Qwen 1.5B | qwen2 | ✅ runs in plain mode; strip the <think> tags in your code |
| TinyLlama 1.1B Chat | llama | ⚠️ loads and runs, but its zephyr chat template isn't recognized → plain fallback, behaves like a base model |
A supported arch loads and runs; chat usability also needs a recognized chat template (chatml / llama3 / [INST]; otherwise plain fallback). gemma/phi/glm GGUFs are rejected on purpose: candle exposes no public KV-cache reset for them yet, and request isolation comes first.
Tip — models pulled with ollama are plain GGUF blobs (and its CDN is far faster than single-stream HF downloads): ollama pull llama3.2:1b, then point SYNSEMA_LLM_MODEL at the blob under ~/.ollama/models/blobs/sha256-… (the digest is in the manifest under ~/.ollama/models/manifests/…; ollama does not need to be running).
On a binary without the feature, SYNSEMA_LLM_PROVIDER=local prints a stderr notice and stays offline — it never silently falls back to another provider.
Build it — and build it fast
# plain build (works everywhere, but candle's AVX2 kernels stay OFF — slow prefill):
cargo install --path crates/synsema-cli --features llm-local --force
# ~3× faster prefill: candle picks its AVX2 quantized kernels at COMPILE time,
# and Rust's default x86-64 target does not enable them:
RUSTFLAGS="-C target-cpu=native" cargo install --path crates/synsema-cli --features llm-local --force
Measured (Qwen2.5 Q4_K_M): generation ~13 tok/s (0.5B) / ~5 tok/s (3B); prefill with the flag ~35 tok/s (0.5B) — built for short prompts (hundreds of tokens, not thousands). Model load is paid once per process (~6s / ~19s): under serve, the first request loads, the rest reuse. RAM: ~1GB (0.5B) / ~2.4GB (3B). native ties the binary to that machine's CPU — right for your own VPS; for a binary you distribute, use -C target-cpu=x86-64-v3 (AVX2+FMA, x86 CPUs from ~2015).
---
The egress to the configured host is part of require llm — not a separate net grant (the embedded provider needs no egress at all). Offline (no key), the ops return placeholders; branch on llm_available(). To bypass the built-in ops entirely and hit the API yourself, see Provider API directly.