Run your own AI server

Set up a computer of your own, such as a Mac Studio with LM Studio or Ollama, as ChatPilot's private AI: the server, the network, Tailscale and which model fits its memory.

ChatPilot can use an AI that runs on a computer of yours instead of an online one: a Mac Studio on your desk, a PC with a big graphics card, a server in the cupboard. Your messages then never leave your own machines, and chats set to Only my own AI can use it too. This page sets up that computer. Connecting ChatPilot to it is in Choose the AI.

What you need

The AI computer. A Mac with Apple silicon (M1 or newer) is the easiest: its memory is shared with the graphics chip, so a Mac Studio with lots of memory runs models no graphics card of the same price can hold. A PC or Linux server with an NVIDIA graphics card works as well.
A server program that speaks the OpenAI-compatible language ChatPilot uses. On a Mac: LM Studio (an app, with fast MLX models made for Apple silicon) or Ollama (an app and a command line). On Linux also vLLM or llama.cpp's server.
One network between that computer and the one ChatPilot runs on: your home or office network, or Tailscale when ChatPilot runs elsewhere (a laptop on the road, a server in a data centre).

1. Choose a model that fits the memory

A model needs about 0.6 GB of memory per billion parameters at the usual 4-bit size (Q4, or "4bit" for MLX), plus room for the conversation (the context) and for the computer itself. Keep about a quarter of the memory free.

Memory	Models that run well	Good for
16 GB	up to about 8B	sorting, translation, short summaries
32 GB	up to about 14B–24B	the above, plus summaries and simple drafts
64 GB	up to about 32B, or a mixture-of-experts model of that size	drafts, discussions, auto-answer
128 GB	up to about 70B, or a larger mixture-of-experts model	everything, close to the online models
192 GB or more	the biggest open models	everything, several models at once

What to look for:

An instruct (chat) model that handles tool calls and structured output (JSON). ChatPilot's sorting and drafts need both. Model pages in LM Studio and Ollama say whether a model supports tools.
Not an image-only or code-only model. A small vision model writes poor drafts and loses track in a discussion, however fast it is.
A mixture-of-experts model (its name or page says "MoE", e.g. "30B-A3B") answers about as fast as its small active part but knows as much as its full size: a good choice on a Mac with much memory.
Two models work well together: a big one as the main AI for drafts, discussions and auto-answer, a small fast one as the second model for sorting and summaries.

Then let the fitness test decide: it runs ChatPilot's own tasks on the model and says what it is good for.

2. Set up the server program

LM Studio

Install LM Studio and download a model under Discover (on a Mac, pick an MLX version).
Open the Developer tab and switch the server on (Status: Running). It listens on port 1234.
In the server's Settings, switch on Serve on Local Network, so other computers can reach it. Leave Just-in-Time Model Loading on: LM Studio then loads the model ChatPilot asks for.
Set the Context Length of the model (in its load settings) to at least 16384. A shorter context cuts long chats off at the start; the fitness test's Long conversation fails then.
Optional: under the server's settings, require an API key. Put that key in ChatPilot under API key.

The address for ChatPilot: http://<computer name>:1234/v1.

Ollama

Install Ollama and fetch a model, e.g. ollama pull <model> in a terminal.
Let it listen on the network: in the Ollama app's settings switch on Expose Ollama to the network, or set OLLAMA_HOST=0.0.0.0 for it (on a Mac: launchctl setenv OLLAMA_HOST 0.0.0.0, then quit and reopen Ollama). Out of the box it only answers on its own computer.
Give it a longer context: its default is too short for ChatPilot. Set OLLAMA_CONTEXT_LENGTH=16384 the same way (or the context slider in the app's settings).
Optional: OLLAMA_KEEP_ALIVE=-1 keeps a loaded model in memory instead of unloading it after 5 quiet minutes; ChatPilot's Keep the model loaded box does the same from ChatPilot's side. OLLAMA_MAX_LOADED_MODELS sets how many models stay loaded at once when you use two.

The address for ChatPilot: http://<computer name>:11434/v1.

Check it from ChatPilot's computer

Open http://<computer name>:1234/v1/models (LM Studio) or …:11434/v1/models (Ollama) in a browser on the computer ChatPilot runs on. A list of models means ChatPilot can reach it too. Nothing comes back: see When it doesn't connect.

3. Keep the computer awake and reachable

Sleep. On a Mac: System Settings → Energy (or Energy Saver), switch on Prevent automatic sleeping when the display is off and Wake for network access. Also handy: Start up automatically after a power failure. On Linux, switch suspend off on a server.
Wake-on-LAN, if you'd rather let it sleep: ChatPilot can wake it before each try. Put its MAC address under Where → Wake-on-LAN in ChatPilot (on a Mac: System Settings → Network → the connection → Details → Hardware). This only works when both computers are on the same network, not over the internet or Tailscale.
A fixed name. Use the computer's name (mac-studio, mac-studio.local) or give it a fixed address in your router, so the address in ChatPilot keeps working.
The firewall. When macOS asks whether LM Studio or Ollama may accept incoming connections, allow it. With the firewall on and no question asked, add the app under System Settings → Network → Firewall → Options.

Reach it from anywhere: Tailscale

When ChatPilot runs on a laptop you take along, or on a server outside your network, use Tailscale: a private network between your own devices, encrypted end to end, with nothing opened to the internet.

Install Tailscale on the AI computer and on the computer ChatPilot runs on, and sign in to the same account on both.
In Tailscale's admin console, keep MagicDNS on: every computer gets a name like mac-studio.your-tailnet.ts.net.
In ChatPilot, press Look for servers (it also looks at your Tailscale computers), or type the address yourself: http://mac-studio.your-tailnet.ts.net:1234/v1.

http:// is fine over Tailscale: the connection is encrypted underneath. Never open the server's port on your router to the internet; anyone could use your AI, and your messages would travel readable. If you must reach it over the internet without Tailscale, put a reverse proxy with a certificate (https://) and a key in front of it. ChatPilot warns under Where when an address on the internet starts with http://.

4. Connect ChatPilot

In ChatPilot, Settings → AI:

As the main AI: under Main AI choose Other OpenAI-compatible service or local model.
As the second model: fill in the Second model tab and tick under Use it for what it does.

Press Look for servers or type the address, tick This server is mine (private), pick the model, run Speed and fitness test, and Save. All of these are explained in Choose the AI, together with what happens when the server sleeps or a model fails.

When it doesn't connect

Nothing at /v1/models from another computer, but it works on the AI computer itself: the server only listens on its own computer. Switch on Serve on Local Network (LM Studio) or set OLLAMA_HOST=0.0.0.0 and restart Ollama.
Nothing at all: the firewall blocks it, the server program isn't running, or the name doesn't resolve. Try the IP address instead of the name.
"needs a key" in Look for servers, or 401: the server wants an API key; fill in API key (and Key header, in the Advanced view, if a proxy wants its own header).
Slow first answer: the model is loaded on the first request. ChatPilot warms it up when it starts and after Save; tick Keep the model loaded to keep it there.
Timeouts on sorting or drafts: the model is too big or too slow for that task. The fitness test shows the speed per task; give that task to a smaller model, or leave it with the main AI.
Long chats come out wrong, Long conversation fails: the context is too short. Raise it as in step 2.

Checked against ChatPilot 0.1.0b1. Report a problem with this page