Self-hosted models
Running with no external calls at all.
For instances that must not talk to the internet, both model workloads can run on your own hardware.
Two workloads
| Default | Runs locally | |
|---|---|---|
| Embedding — indexing content for search | Bundled, local | Already, out of the box |
| Chat — conversation, tagging, summarising | Your provider | With vLLM or similar |
Embedding is local by default and needs no configuration. Only chat needs changing.
Point chat at your own server
Run any OpenAI-compatible server — vLLM is a common choice — then:
LLM_PROVIDER=vllm
LLM_BASE_URL=http://your-model-host:8000/v1
LLM_CHAT_MODEL=your-model-nameWith both local, the instance makes no external calls for any purpose.
Choosing a model
The tasks are bounded and verifiable — classify into a known set, summarise material that has already been retrieved, answer from provided context. A mid-sized instruction-tuned model handles all of it.
Save the large models for the agent, which is a separate system running wherever you choose.
Hardware
Embedding runs on CPU and works everywhere. For a large corpus a GPU makes the initial indexing substantially faster; a single mid-range card is enough. Chat models are the ones with real requirements — size those against whichever model you pick.
Air-gapped installs also need the container images mirrored to a registry you can reach. See Requirements.