# System-1 Gateway: full documentation > Fast multiple-choice decisions for Claude Code and other MCP clients, from a local engine you pick. --- Page: Home URL: https://thematthub.github.io/Sys1-AI-Gateway/index.html # System-1 Gateway The System-1 Gateway gives Claude Code fast multiple-choice decisions from a small local engine, through one MCP server with three stable tools. You pick the engine (Laya, Von, Liquid AI d1 through llama.cpp, any Ollama chat model, a Hugging Face reranker through a small local server, a hosted Claude model, or one you add), and Claude sees the same tools whichever one answers. Python 3.11 or later and the standard library only; version 0.2.0, released 2026-10-20. In Claude Code, three commands install it: ``` /plugin marketplace add TheMattHub/Sys1-AI-Gateway /plugin install system1@system1-gateway /system1:setup ``` After a restart Claude can ask it a judgement call. One `system1_decide` call and its answer, here from the built-in `mock` provider (latencies set to 0): ```json {"question": "Rerun the flaky pipeline or block the merge?", "options": ["rerun the pipeline", "block the merge"]} ``` ```json {"schema_version": "1", "request_id": "demo-1", "status": "decided", "decision": "rerun the pipeline", "confidence": 0.9, "rationale": "mock provider: always picks the first option", "provider": "mock", "probabilities": {"rerun the pipeline": 0.9, "block the merge": 0.09999999999999998}, "calibrated": false, "latency_ms": 0, "attempts": [{"provider": "mock", "outcome": "ok", "latency_ms": 0, "detail": null}], "error": null} ``` Three things set it apart. It abstains instead of guessing: below the confidence threshold the answer is `abstained` and Claude reasons it through itself. The threshold means something, because each engine's probabilities are calibrated on labelled decisions. And the choice of engine rests on measurement: the [naga decisions study](studies/naga-decisions.md) compares every engine we tried on frozen blind test sets.
- **Claude Code user** --- Install it, pick an engine, make a first decision, then turn on model routing. [Your first decision](tutorial/first-decision.md), [Install an engine](how-to/engines.md) and [Turn on model routing](how-to/routing.md) - **Choosing a decision model** --- Speed against accuracy on real decisions, on one laptop GPU, with the caveats. [Which model for naga decisions](studies/naga-decisions.md) and [Providers](reference/providers.md) - **Contributor** --- Add an engine adapter: one file and one registry line; the tools stay the same. [Add a provider](how-to/add-provider.md) and [CONTRIBUTING.md](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/CONTRIBUTING.md) - **AI agent** --- Read the site as plain text and validate against the published schema. [llms.txt](https://thematthub.github.io/Sys1-AI-Gateway/llms.txt), [llms-full.txt](https://thematthub.github.io/Sys1-AI-Gateway/llms-full.txt) and [Tools and schema](reference/tools.md)
Every page also has a Markdown copy: replace `.html` with `.md` in its address. --- Page: Your first decision URL: https://thematthub.github.io/Sys1-AI-Gateway/tutorial/first-decision.html # Your first decision This tutorial takes you from nothing to one `system1_decide` call in Claude Code, in about five minutes plus the engine download. You need Claude Code and Python 3.11 or later; you do not need a clone of this repository. At the end you will know which engine you run and how to check it. ## 1. Install the plugin The repository is both a Claude Code plugin (`system1`) and its own marketplace (`system1-gateway`). In Claude Code, run: ``` /plugin marketplace add TheMattHub/Sys1-AI-Gateway /plugin install system1@system1-gateway ``` The next session greets you with a one-line pointer to `/system1:setup`. ## 2. Run the setup ``` /system1:setup ``` The setup checks for Python 3.11+, installs the gateway into `~/.system1-gateway/venv`, runs the tests and registers the `system1` MCP server at user scope, so it works in every project. It asks before each change outside the plugin. When it asks for a provider, you have three ways on: - **You already run Laya:** give its endpoint (default `http://127.0.0.1:8765`). - **No engine yet:** let it hand over to `/system1:choose-model`, which profiles your hardware, recommends an engine and installs it on a yes. The [study](../studies/naga-decisions.md) helps you choose. - **Just trying it:** pick `mock`, a deterministic stand-in that always picks the first option. Setup can also start a fake Laya for one smoke test and stop it again. Setup ends with a smoke test: it starts the gateway exactly as Claude Code does and asks one question. ## 3. Restart Claude Code The MCP server loads in new sessions only. After the restart, `/mcp` lists `system1` as connected. ## 4. Ask it something Ask Claude for a judgement call, for example: "Use system1_decide: rerun the flaky pipeline or block the merge?" Claude calls the tool with a question and the options: ```json {"question": "Rerun the flaky pipeline or block the merge?", "options": ["rerun the pipeline", "block the merge"]} ``` With the `mock` provider the answer is (latencies set to 0): ```json {"schema_version": "1", "request_id": "demo-1", "status": "decided", "decision": "rerun the pipeline", "confidence": 0.9, "rationale": "mock provider: always picks the first option", "provider": "mock", "probabilities": {"rerun the pipeline": 0.9, "block the merge": 0.09999999999999998}, "calibrated": false, "latency_ms": 0, "attempts": [{"provider": "mock", "outcome": "ok", "latency_ms": 0, "detail": null}], "error": null} ``` With a real engine, `provider`, `confidence` and `rationale` come from it, and `calibrated` is `true` when a fitted calibration matches the engine and checkpoint. If the status is `abstained`, the engine was not sure enough, and Claude decides itself; that is a valid answer, not a failure. ## Next - Install a real engine: [Install an engine](../how-to/engines.md). - Let System-1 suggest a Claude model for subagents: [Turn on model routing](../how-to/routing.md). - Something did not connect: [Troubleshoot](../how-to/troubleshoot.md). --- Page: Install the gateway URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/install.html # Install the gateway The gateway installs either as a Claude Code plugin with guided setup or with one script from a clone. Both put the same stdlib-only package into a user-level venv, `~/.system1-gateway/venv`, and register it once at user scope, so nothing is added to your projects (no `.mcp.json`). Pick the plugin unless you want the gateway in another MCP client only. ## With the Claude Code plugin ``` /plugin marketplace add TheMattHub/Sys1-AI-Gateway # GitHub, or a local clone path /plugin install system1@system1-gateway /system1:setup ``` `/system1:setup` installs, runs the tests, sets the provider and registers the server; then restart Claude Code. The [tutorial](../tutorial/first-decision.md) walks through it. - **Upgrade:** `/plugin marketplace update system1-gateway`, then `/plugin update system1@system1-gateway`, then `/system1:setup` to reinstall. - **Remove:** `/system1:setup` (uninstall) first, then `/plugin uninstall system1@system1-gateway`. The plugin goes last because the uninstall scripts ship inside it. - **Plugin development:** `claude --plugin-dir ` loads the plugin straight from a clone. ## With the install script ```powershell # Windows (PowerShell 5.1+ or 7+) git clone https://github.com/TheMattHub/Sys1-AI-Gateway ; cd Sys1-AI-Gateway powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` ```sh # Linux / macOS git clone https://github.com/TheMattHub/Sys1-AI-Gateway && cd Sys1-AI-Gateway sh install.sh ``` The script finds Python 3.11+, creates `~/.system1-gateway/venv`, copies the package into it, creates the user config from [config.example.toml](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/config.example.toml) if there is none, and prints the registration command. The installed copy does not depend on the clone. Re-run it to upgrade; set `SYSTEM1_INSTALL_DIR` to install somewhere else. ## Register with Claude Code Run the command the installer printed: ```sh claude mcp add --scope user --transport stdio system1 -- "" -I -m system1_gateway ``` - `` is `%USERPROFILE%\.system1-gateway\venv\Scripts\python.exe` on Windows and `~/.system1-gateway/venv/bin/python` on Linux and macOS. - `-I` runs Python in isolated mode, so a project file such as `json.py` cannot shadow the gateway's imports. - **In PowerShell, quote the separator as `'--'`.** When `claude` is the npm `claude.ps1` shim, PowerShell swallows a bare `--` and Claude fails with `unknown option '-I'`. The command `install.ps1` prints is already quoted. Check it with `claude mcp list`, or `/mcp` inside Claude Code; remove it with `claude mcp remove --scope user system1`. ## Use with other MCP clients The gateway is a standard stdio MCP server, and every client launches the same command: the venv Python with `-I -m system1_gateway`. On Windows, JSON configs accept forward slashes (`C:/Users//.system1-gateway/venv/Scripts/python.exe`). ```json { "mcpServers": { "system1": { "command": "", "args": ["-I", "-m", "system1_gateway"] } } } ``` | Client | Where it goes | |---|---| | Claude Desktop | `claude_desktop_config.json` (Settings, Developer, Edit Config), shape above | | Cursor | `~/.cursor/mcp.json`, shape above | | Gemini CLI | `~/.gemini/settings.json`, shape above | | Windsurf | `~/.codeium/windsurf/mcp_config.json`, shape above | | VS Code (Copilot) | user `mcp.json` (command *MCP: Open User Configuration*): `{"servers": {"system1": {"type": "stdio", "command": "", "args": ["-I", "-m", "system1_gateway"]}}}` | | Codex CLI | `~/.codex/config.toml`: `[mcp_servers.system1]` with `command = ""` and `args = ["-I", "-m", "system1_gateway"]` | These locations change between client versions, so check your client's MCP docs if one does not work. The skills and hooks are Claude Code only; the tools behave the same in every client. ## Uninstall ```powershell powershell -ExecutionPolicy Bypass -File .\uninstall.ps1 # add -Purge to delete the config too ``` ```sh sh uninstall.sh # add --purge to delete the config too ``` Uninstalling unregisters the MCP server (if `claude` is on PATH), deletes `~/.system1-gateway/venv` and removes the Windows start-at-login entry. Your config, the engines (Laya is about 6 GB) and the logs stay unless you pass `-Purge` or `--purge`. Stop the engine before purging, because Windows cannot delete a running `laya-serve.exe`. --- Page: Install an engine URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/engines.html # Install an engine An engine is the local model that actually answers; the gateway only talks to it. The easiest path is `/system1:choose-model`: it profiles your hardware and preferences, checks the current models online, recommends one checkpoint and installs it on a yes, each engine in its own venv under `~/.system1-gateway/engines/`. This page shows what it sets up, for when you want to do it by hand. Which engine to pick is a question of speed, accuracy and hardware, measured in the [naga decisions study](../studies/naga-decisions.md). The gateway has adapters for every engine the study recommends: | Provider | Engine | Start it | Notes | |---|---|---|---| | `laya` | Laya `typed-decisions` | `~/.system1-gateway/engines/laya/start.ps1` or `start.sh` | one pass for several questions; at most 20 options | | `von` | Von 1.3 (`pip install von-sdk`) | `von serve --host 127.0.0.1 --port 8777` | one pass; 8,192-token context; no option limit declared | | `ollama` | any local chat model | Ollama 0.12 or later (for logprobs) | one call per question; at most 26 options | | `systemone` | Liquid AI d1 through llama.cpp, or a Hugging Face-only engine through [engines/systemone/serve.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/engines/systemone/serve.py) (gte reranker, Qwen3-Reranker, AnyJev, OpenThai-SystemOne) | `llama-server -m .gguf --port 8790 ...` or `python engines/systemone/serve.py --engine gte` | the same `/v1/systemone` contract as Von | | `anthropic` | a hosted Claude model (default Claude Haiku 5.5) | nothing to start; needs `ANTHROPIC_API_KEY` | requests leave your machine and are billed per token | For any other engine, see [Add a provider](add-provider.md). ## Laya [Laya](https://huggingface.co/convaiinnovations/laya) (Convai, Apache-2.0) runs as its own local server, `laya-serve`, in a separate venv. `/system1:choose-model` installs it into `~/.system1-gateway/engines/laya/` with a launcher copied from [engines/laya/](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/engines/laya): ```powershell powershell -ExecutionPolicy Bypass -File "$HOME\.system1-gateway\engines\laya\start.ps1" # -Model, -Port optional ``` ```sh ~/.system1-gateway/engines/laya/start.sh # LAYA_MODELS / LAYA_PORT env vars optional ``` The launcher binds to `127.0.0.1` (laya-serve's own default is `0.0.0.0`), preloads `typed-decisions`, uses CUDA or MPS when available and the CPU otherwise, logs to `laya-serve.log` next to it, and does nothing if the port is already in use. Then point the gateway at it: ```toml [gateway] provider = "laya" [providers.laya] endpoint = "http://127.0.0.1:8765" model = "typed-decisions" # english | multilingual | typed-decisions | "" to let Laya auto-route ``` If laya-serve runs with `LAYA_API_KEY`, set the same value in the `SYSTEM1_LAYA_API_KEY` environment variable, never in the config file. ## Von Install `von-sdk` into its own venv, start `von serve --host 127.0.0.1 --port 8777`, and set: ```toml [gateway] provider = "von" [providers.von] endpoint = "http://127.0.0.1:8777" model = "von-1.3" ``` ## A decision model through llama.cpp (Liquid AI d1) llama.cpp's `llama-server` serves native decision models such as Liquid AI's d1 at `POST /v1/systemone` (build b11757 or later). Download a GGUF (for example `LiquidAI/d1-3B-GGUF`, file `d1-3B-Q8_0.gguf`, which fits a 4 GB GPU), and start it on loopback with one slot and a batch that holds the whole input: ```sh llama-server -m d1-3B-Q8_0.gguf --host 127.0.0.1 --port 8790 -ngl 99 -np 1 -c 16384 -b 16384 -ub 16384 ``` ```toml [gateway] provider = "systemone" [providers.systemone] endpoint = "http://127.0.0.1:8790" ``` The d1 models carry the LFM Open License v1.0, which licenses commercial use only below USD 10 M annual revenue. ## A Hugging Face-only engine The gte reranker, Qwen3-Reranker, AnyJev and OpenThai-SystemOne ship as Python weights with no server of their own. [engines/systemone/serve.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/engines/systemone/serve.py) wraps one of them in the same `/v1/systemone` contract on `127.0.0.1:8790`, so the `systemone` provider above reaches it unchanged. Run it in its own venv with `torch` and `transformers` (plus `sentence-transformers` for gte, `anyjev` and `bitsandbytes` for AnyJev, `openthai-systemone` for OpenThai): ```sh python engines/systemone/serve.py --engine gte # gte | qwen3-rerank | anyjev | openthai; --model, --port optional ``` The server binds `127.0.0.1` only and refuses requests whose `Host` is not `127.0.0.1` or `localhost` on its port, bodies that are not JSON or over 1 MB, and more than 16 questions or 256 options, so a web page in your browser cannot drive it. Set `SYSTEM1_SYSTEMONE_API_KEY` in both the server's and the gateway's environment to require a bearer token too. The provider sends each option as its own description, so a reranker scores the option text alone; options that say what they mean ("billing: charges, refunds") work much better than bare names. ## A hosted Claude model The `anthropic` provider calls the Anthropic Messages API over HTTPS and reads the key from the `ANTHROPIC_API_KEY` environment variable, never from the config file: ```toml [gateway] provider = "anthropic" [providers.anthropic] model = "claude-haiku-5-5" ``` It was the most accurate engine in the study, but every request leaves your machine, needs a network and is billed per token. The API gives no answer probabilities, so `confidence` is the model's own estimate and the calibration layer cannot rescale it. ## A chat model through Ollama Pull a model with Ollama (0.12 or later), then name it; `model` is required: ```toml [gateway] provider = "ollama" [providers.ollama] endpoint = "http://127.0.0.1:11434" model = "gemma4:e4b" # see `ollama list` ``` An LLM's letter probabilities are over-confident, so [calibrate](calibrate.md) before you trust a threshold. Decision models built for Ollama's own System One endpoint may do worse through this letter prompt: the Ollama build of OpenThai-SystemOne got 5 of 40 block questions this way, against 23 through its own client. ## Start at login The engine does not start on its own, and after a reboot the gateway reports `down` until it does. `/system1:choose-model` offers a user-level entry, no admin rights needed: - **Windows:** `system1-laya.cmd` in your Startup folder; `uninstall.ps1` removes it. - **Linux:** a `systemd --user` unit running `start.sh` (`systemctl --user enable --now system1-laya`). - **macOS:** a LaunchAgent in `~/Library/LaunchAgents/` running `start.sh` at load. ## Try it without an engine Set `provider = "mock"` for a deterministic stand-in, or run `python tests/fake_laya.py` from a clone: a fake that speaks Laya's API on `127.0.0.1:8765`. Both are for checking the plumbing, not for real decisions. --- Page: Configure URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/configure.html # Configure The gateway reads one TOML file, and every key in it is optional. It lives at `%APPDATA%\system1\config.toml` on Windows and `~/.config/system1/config.toml` on Linux and macOS, or at any path in `SYSTEM1_CONFIG`; the installer creates it from [config.example.toml](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/config.example.toml). Environment variables override the file, and the file is read each time the server starts, so restart Claude Code sessions after a change. ## Choose the provider and a fallback ```toml [gateway] provider = "laya" # the primary engine fallback = ["mock"] # tried in order on a timeout, an error or a malformed answer; [] by default timeout_s = 5.0 # per attempt ``` A fallback runs only when the primary fails. It never runs because the primary abstained: an abstention is a valid answer. ## Set the confidence threshold ```toml [gateway] min_confidence = 0.6 # abstain below this ``` A request can override it with its own `min_confidence`. The number only means what it says once a calibration matches your engine and checkpoint; see [Calibrate and evaluate](calibrate.md). ## Provider settings and secrets Provider settings go in `[providers.]`, or in environment variables named `SYSTEM1__` (for example `SYSTEM1_LAYA_ENDPOINT`). Secrets such as API keys go only in environment variables, never in the file. A non-loopback endpoint is refused unless you set `allow_remote = true` for that provider. ## Logs Logs always go to stderr, because stdout carries the protocol. Set `log_file` to also write a file, and `log_level = "DEBUG"` to include the question text. Every line carries the request id. ## Pause System-1 `/system1:toggle off` pauses the routing hook, the skill picker and `system1_decide` from the next prompt, with no restart; `/system1:toggle on` resumes them and `/system1:toggle status` shows the state. A broken config never stops the server: `system1_health` reports `down` and `system1_decide` returns `config_error` with the reason. Every key, default and variable is in [Config keys](../reference/config.md). --- Page: Turn on model routing URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/routing.html # Turn on model routing Model routing lets System-1 suggest a Claude model (haiku, sonnet or opus) for the subagents Claude spawns. On every prompt the plugin's hook asks the engine four yes/no questions about the text, a small rule turns the answers into a tier, and Claude gets a one-line hint. It works in Claude Code only, needs a working engine, and is off by default. ## 1. Turn it on, record-only Start by collecting data without hints, so you can check the engine before it influences anything: ```toml [routing] enabled = true hint = false # log only; no hint yet log_dirs = ["~/src/my-project"] # repos whose prompts may be logged (default: none) ``` `hint` defaults to `true`; set it to `false` until the engine is shown to be ready (step 3). The log is opt-in per repo: a prompt is logged only when Claude Code's working directory is in one of `log_dirs` or below it. It keeps prompt text on this machine, in `~/.system1-gateway/logs/routing.jsonl`, so leave out repos with confidential work. ## 2. Collect and label Work as usual. The hook also records each Agent spawn next to the model the orchestrator chose, without changing it. When 30 logged prompts are waiting, the session-start hook suggests `/system1:calibrate`, which labels them with you and fits the calibration ([Calibrate and evaluate](calibrate.md)). ## 3. Check the engine is ready ```sh scripts/evaluate.py --provider laya # or von, ollama; one engine at a time on a small GPU ``` The verdict is `READY` only if every feature is at least 85% accurate at the threshold, the rule's safety is at least 95%, and at least 30 failed spawns are labelled. Then set `hint = true` (or delete the line), and optionally `[routing] min_confidence` to the threshold calibration recommends. ## 4. See whether it pays off `/system1:report` reads your Claude Code transcripts and reports, as totals only, how often hints appear, whether Claude follows them, what subagents cost by tier, and the estimated savings minus the hint overhead. ## What to expect - **Advisory only:** the hook never blocks or rewrites a prompt. - **Silent** when the engine abstains, fails or is not installed, for slash commands, for prompts under 10 words (usually follow-ups whose size depends on the conversation), and for subagent hand-backs. - **Cost:** about 0.5 s per prompt with Laya warm on a GPU, mostly Python startup; capped at 1.5 s when the engine is down. - **Debugging:** the hook hides its own errors. Run ` -I -m system1_gateway.route_hook` and paste `{"prompt": "..."}` to see what it does. ## The skill picker The same hook can also name the skill that fits a prompt. It is off and record-only by default: ```toml [skills] enabled = true hint = false min_confidence = 0.8 ``` With Laya it stays dormant and costs nothing: a typical setup has about 100 loadable skills, more than Laya's 20 options. It wakes on its own for a provider whose `max_options` holds the whole list. Routing and the picker share the hook's 1.5 s budget, and the picker is skipped when routing uses it up. Why the hook asks about text features instead of "which model?" is in [How routing and calibration work](../background/routing-and-calibration.md). --- Page: Calibrate and evaluate URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/calibrate.html # Calibrate and evaluate Calibration makes an engine's confidence mean what it says, so that `min_confidence` is a real threshold. The repository ships fits for Laya `typed-decisions`, so a fresh install starts calibrated for that checkpoint; any other engine, or your own prompts, needs a fit of its own. The guided way is `/system1:calibrate`; this page shows the scripts it runs. All scripts run with the gateway venv Python (`` below), from a clone or from the plugin copy. ## 1. Label decisions `/system1:calibrate` labels the routing log with you: Claude proposes a label for each entry and you correct only the ones you disagree with. It also proposes yes/no labels for the four routing features. Seed sets labelled by a teacher model are in [calibration/](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/calibration), and you can add your own decisions as JSONL lines: `{"question", "options", "context", "label"}`. For Agent spawns, the outcome is the better label: did the model that ran it get the job done? ```sh scripts/outcomes.py --apply ``` It proposes `failed` or `succeeded` for every spawn from your transcripts, writes it into the log, and lists the doubtful ones for you to confirm. ## 2. Fit and measure ```sh scripts/calibrate.py DATA.jsonl --target-accuracy 0.8 ``` It re-asks every example live, fits one temperature per question, and reports accuracy, calibration error (ECE) before and after, AUROC, and a threshold table of coverage against accuracy. Read it before you save: - `--write` saves the fit for this machine (`~/.system1-gateway/calibration.json`, which takes precedence). - `--ship` saves it into the repository's `system1_gateway/calibration.json`, for every install. Then re-run the installer so the venv gets a shipped fit, and set `[routing] min_confidence` (routing only) or `[gateway] min_confidence` (all decisions) to the recommended threshold. ## 3. Evaluate an engine for routing ```sh scripts/evaluate.py --provider laya ``` It is read-only and reports per-feature accuracy and AUROC, the rule's safety and savings, and a verdict, `READY` or `NOT READY`. The bars can be changed with `--min-feature-accuracy`, `--min-safety` and `--min-failures`. ## When accuracy is the problem A temperature never changes which answer wins, so it cannot fix a wrong engine. Try clearer question and option wording first, then the engine's known limits (for Laya: at most 20 options, English-only checkpoints on non-Latin text), then fine-tuning once you have a few hundred labelled examples. Fits are measured on the data they were fitted on and from small samples, so treat the numbers as rough. How the calibration itself works is in [How routing and calibration work](../background/routing-and-calibration.md). --- Page: Add a provider URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/add-provider.html # Add a provider A provider is one small adapter file that maps the gateway's neutral request to one engine's API. Only the adapter knows the engine: the MCP tools, the schema and the gateway stay the same. In Claude Code, `/system1:add-provider` walks through these steps in a git clone; here `myengine` stands for the engine you add. ## 1. Write the adapter Create `system1_gateway/providers/myengine.py` with a subclass of `System1Provider` (see [base.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/providers/base.py)). The existing adapters are the examples to copy: [laya.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/providers/laya.py) for a local HTTP engine, [mock.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/providers/mock.py) for an in-process one. - `name = "myengine"`, and the engine's native contract in the module docstring. - The constructor validates the settings and raises `ValueError` for unusable ones; the gateway reports those as `config_error`. - `decide(request, timeout_s)` returns `{"decision", "confidence", "rationale"}` (confidence in 0..1) or `{"abstain": True, "rationale"}`. Raise `ProviderError` for transport failures and `MalformedResponse` for a reply you cannot read; the gateway checks the rest. - Pass `timeout_s` to every I/O call: a timed-out thread cannot be killed. - `health(timeout_s)` returns `{"ok": bool, "detail": str}`; `capabilities()` returns non-secret facts, including `max_options` if the engine has a limit. - For HTTP engines, [http.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/providers/http.py) gives you the loopback guard, the `allow_remote` opt-in, no proxy for local calls, and typed errors. Keep it stdlib only. Secrets come from environment variables (`SYSTEM1_MYENGINE_API_KEY`), never the config. ## 2. Register it Add `"myengine": MyEngineProvider` to `REGISTRY` in [providers/\_\_init\_\_.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/providers/__init__.py), and a commented `[providers.myengine]` section to [config.example.toml](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/config.example.toml). ## 3. Test it Add `tests/test_myengine.py`. For an HTTP engine, serve a fake on `127.0.0.1` (port 0), modelled on [tests/fake_laya.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/tests/fake_laya.py); tests make no real network calls. Cover the mapping, an abstention, a transport error, a bad body, a timeout, an invalid setting and `health`. Then run the suite: ```sh python -m unittest discover -s tests -v ``` ## 4. Use it Re-run the installer, set `provider = "myengine"` or add it to `fallback`, and check it: ```sh scripts/smoke.py --question "Rerun or block?" --options rerun block ``` Contributions are welcome; see [CONTRIBUTING.md](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/CONTRIBUTING.md). --- Page: Troubleshoot URL: https://thematthub.github.io/Sys1-AI-Gateway/how-to/troubleshoot.html # Troubleshoot When the gateway misbehaves, check one layer at a time from the top and stop at the first that fails. In Claude Code, `/system1:doctor` does exactly this, shows the evidence and asks before fixing anything. The session-start hook also speaks up when the gateway is not installed, the config is broken, or routing is on but the engine is not running. ## 1. Is it registered? `claude mcp get system1` must show the venv Python followed by `-I -m system1_gateway`, at user scope. If it is missing, run `/system1:setup` or [register it](install.md#register-with-claude-code). A project-scope or `.mcp.json` entry can shadow it. ## 2. Does the command start? ```sh scripts/smoke.py ``` `smoke.py` launches the installed gateway exactly as an MCP client does and checks the handshake and health; add `--question "..." --options a b` for one decision, or `--python ` to test another interpreter. It runs on any Python 3.8+. | Output | Cause | Fix | |---|---|---| | `NOT INSTALLED` | the venv is missing | re-run the installer | | `No module named system1_gateway` | the package was not copied into the venv | re-run the installer | | Python will not start | the base Python was upgraded or removed under the venv | re-run the installer; it rebuilds the venv | | `HANDSHAKE FAILED` with a traceback | a code bug | run the tests and read the traceback | ## 3. What does health say? - **`config_errors`:** the message names the file and the key: a TOML syntax error, an unknown key, `timeout_s` not above 0, `min_confidence` outside [0, 1], an unknown provider, or a non-local endpoint without `allow_remote`. `SYSTEM1_*` environment variables override the file, so check those too. - **A provider with `ok: false`:** is the engine running, on the configured port? Laya answers `GET /health`. To tell a gateway problem from an engine problem, run `python tests/fake_laya.py --port ` from a clone; if smoke then passes, the problem is on the engine's side. - **`degraded`:** the primary is down and a fallback answered. Fix the primary. - **`paused`:** `/system1:toggle on` resumes it. ## 4. What do the decisions say? Run smoke with a question and read `attempts[].outcome` and `detail`: - **`timeout`:** compare the engine's latency with `timeout_s`; raise it only if the engine really is that slow. - **`malformed`:** the reply does not match what the adapter expects, often a newer engine version. Compare a raw reply with the adapter's docstring, and update it with [Add a provider](add-provider.md). - **Always `abstained`:** confidence is below `min_confidence`. That is correct behaviour; lower the threshold only if less certain answers are acceptable. - **`error`:** read `detail` (HTTP status, connection refused, ...). ## Routing gives no hints Check that `[routing] enabled = true`, that `hint` is not `false`, and that the engine is up. Then run the hook by hand: `echo '{"prompt": "Write unit tests for the parser module"}' | -I -m system1_gateway.route_hook`. JSON output means it works; no output means the engine abstained or failed. On Windows the hook runs through Git Bash, so paths in environment variables must be Windows-style (`C:/...`). For more detail, set `SYSTEM1_LOG_LEVEL=DEBUG` (it also logs question text) or `log_file`, and start Claude Code with `claude --debug` to see the server's stderr. --- Page: Tools and schema URL: https://thematthub.github.io/Sys1-AI-Gateway/reference/tools.html # Tools and schema The gateway exposes three MCP tools over stdio, and their contract is schema version `"1"`. New optional fields can be added without a version bump; a breaking change bumps it. The request and response JSON Schemas are published at [schema/request.json](https://thematthub.github.io/Sys1-AI-Gateway/schema/request.json) and [schema/response.json](https://thematthub.github.io/Sys1-AI-Gateway/schema/response.json), written from [schema.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/schema.py) at build time, and `system1_capabilities` returns the same. ## system1_decide Asks the configured provider for one decision. | Field | Required | Notes | |---|---|---| | `question` | yes | non-empty string, at most 8,000 characters | | `options` | no | 2 to 256 unique non-empty strings of at most 500 characters; the decision is exactly one of them. Without options the question is answered yes or no. Each provider may set a lower limit (Laya: 20) | | `context` | no | object, passed to the provider | | `min_confidence` | no | number in [0, 1]; overrides the configured threshold | | `request_id` | no | 1 to 128 characters, echoed back; generated if omitted | | `schema_version` | no | `"1"` | Unknown fields are rejected with `invalid_request`. | Response field | Meaning | |---|---| | `status` | `decided`, `abstained` or `error` | | `decision` | one of the options, or `null` when abstained or on error | | `confidence` | probability of the decision, after calibration when one matches | | `probabilities` | per-option probabilities, when the provider reports them | | `calibrated` | `true` when a fitted temperature was applied | | `rationale`, `provider` | the provider's explanation and name | | `latency_ms` | total time | | `attempts` | every provider tried: `provider`, `outcome` (`ok`, `abstained`, `timeout`, `error`, `malformed`), `latency_ms`, `detail` | | `error` | `null`, or `{"code", "message"}` | | `schema_version`, `request_id` | as sent, or generated | Error codes: `invalid_request`, `config_error`, `provider_timeout`, `provider_error`, `malformed_response`. ## system1_health Returns `status` (`ok`; `degraded` when the primary is down and a fallback is up; `down`; or `paused`), the gateway and schema versions, the config file read, any `config_errors`, and a probe for each provider in the chain (`name`, `role`, `ok`, `latency_ms`, `detail`). ## system1_capabilities Returns the gateway version, schema versions, tool names, provider chain, timeout and threshold, each provider's capabilities (for example `max_options`), the calibration fits loaded, the available providers, any config errors, and the full request and response JSON Schemas. ## Behaviour - **Validation on both sides:** requests are checked before any provider runs, and provider answers are checked by the gateway: a non-object, a confidence outside [0, 1], or a decision not in `options` is `malformed_response`. - **Abstention:** if the provider abstains, or its confidence is below the threshold, `status` is `abstained` and `decision` is `null`. It means "reason it through yourself", and it does not trigger a fallback. - **Fallback:** runs only on a timeout, an error or a malformed answer, trying providers in order; every try is recorded in `attempts`. - **Timeouts** apply to each attempt. - **Paused:** while `/system1:toggle off` is in effect, `system1_decide` abstains with a rationale saying so. - **Logs** go to stderr, since stdout carries the protocol; every line carries the request id. --- Page: Config keys URL: https://thematthub.github.io/Sys1-AI-Gateway/reference/config.html # Config keys The config file is `%APPDATA%\system1\config.toml` on Windows and `~/.config/system1/config.toml` (or `$XDG_CONFIG_HOME/system1/config.toml`) on Linux and macOS, or the path in `SYSTEM1_CONFIG`. Every key is optional, environment variables override the file, and an unknown key in `[gateway]`, `[routing]` or `[skills]` is a config error. The commented template is [config.example.toml](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/config.example.toml). ## [gateway] | Key | Default | Env var | Meaning | |---|---|---|---| | `provider` | `"laya"` | `SYSTEM1_PROVIDER` | primary provider | | `fallback` | `[]` | `SYSTEM1_FALLBACK` (comma list) | providers tried in order on a timeout, error or malformed answer | | `timeout_s` | `5.0` | `SYSTEM1_TIMEOUT_S` | limit per attempt, in (0, 300] | | `min_confidence` | `0.6` | `SYSTEM1_MIN_CONFIDENCE` | abstain below this, in [0, 1]; a request can override it | | `log_level` | `"INFO"` | `SYSTEM1_LOG_LEVEL` | `DEBUG` also logs question text | | `log_file` | `""` | `SYSTEM1_LOG_FILE` | logs always go to stderr; set this to also write a file | ## [routing] | Key | Default | Env var | Meaning | |---|---|---|---| | `enabled` | `false` | `SYSTEM1_ROUTING_ENABLED` | the model-routing hook | | `hint` | `true` | `SYSTEM1_ROUTING_HINT` | `false` logs only and adds no hint | | `log_dirs` | `[]` | `SYSTEM1_ROUTING_LOG_DIRS` (`;` on Windows, `:` elsewhere) | repos whose prompts may be logged; none by default | | `log_file` | `"~/.system1-gateway/logs/routing.jsonl"` | `SYSTEM1_ROUTING_LOG_FILE` | the routing log | | `min_confidence` | the `[gateway]` value | `SYSTEM1_ROUTING_MIN_CONFIDENCE` | routing's own threshold, from calibration | ## [skills] | Key | Default | Env var | Meaning | |---|---|---|---| | `enabled` | `false` | `SYSTEM1_SKILLS_ENABLED` | the skill picker | | `hint` | `false` | `SYSTEM1_SKILLS_HINT` | `false` records to the routing log with `"source": "skill"` | | `min_confidence` | `0.8` | `SYSTEM1_SKILLS_MIN_CONFIDENCE` | the picker's threshold | ## [providers.<name>] Each provider reads its own table, and every key can also be set as `SYSTEM1__` (for example `SYSTEM1_LAYA_ENDPOINT`). The keys of each provider are in [Providers](providers.md). ## Other environment variables | Variable | Meaning | |---|---| | `SYSTEM1_CONFIG` | path of the config file; a missing file is a config error | | `SYSTEM1_INSTALL_DIR` | install root instead of `~/.system1-gateway` (venv, engines, logs, pause file) | --- Page: Slash commands and scripts URL: https://thematthub.github.io/Sys1-AI-Gateway/reference/commands.html # Slash commands and scripts The plugin adds seven slash commands to Claude Code, and each one drives scripts that you can also run yourself with the gateway venv Python. The scripts are in [scripts/](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/scripts) and run from a clone or from the plugin copy. ## Slash commands | Command | Use it to | |---|---| | `/system1:setup` | install, upgrade, reconfigure, register or uninstall | | `/system1:choose-model` | pick an engine for your hardware, languages and privacy or cost constraints, using live web research and your Claude memory, then install it on a yes | | `/system1:report` | see whether System-1 is used and pays off: reach, hint adherence, subagent cost by tier, realised and missed savings, overhead (totals only); then offer a focused calibration | | `/system1:calibrate` | label collected decisions, fit the calibration, measure accuracy and set thresholds | | `/system1:doctor` | diagnose a gateway that will not connect, is `down` or `degraded`, times out, answers malformed or always abstains | | `/system1:add-provider` | add or update an engine adapter; needs a git clone | | `/system1:toggle on\|off\|status` | pause or resume System-1 everywhere from the next prompt, with no restart, or show its state | | `/system1:bench` | run, extend or rerun the naga decisions bench (`bench/naga`): add an engine, a test set or a trained student, then rebuild the study's tables and charts; needs a git clone | `setup`, `choose-model`, `add-provider`, `toggle` and `bench` are user-only: you type them, so their descriptions stay out of every session's context. `doctor`, `report` and `calibrate` also trigger from plain words ("why is system1 down?", "is System-1 worth it?"). ## Scripts | Script | What it does | |---|---| | `smoke.py [--python PATH] [--question Q --options A B ...]` | starts the installed gateway as an MCP client does; checks the handshake, health and optionally one decision. Exit 0 when all pass | | `toggle.py on\|off\|status` | pauses or resumes System-1 (a pause file), or shows its state | | `calibrate.py DATA.jsonl [--target-accuracy 0.8] [--write \| --ship] [--json]` | fits a temperature per question from labelled decisions and reports accuracy, ECE, AUROC and a threshold table | | `evaluate.py [--provider NAME] [--log PATH] [--min-feature-accuracy 0.85] [--min-safety 0.95] [--min-failures 30] [--json]` | read-only: is an engine ready to route? Verdict `READY` or `NOT READY` | | `outcomes.py [--days 30] [--projects-dir DIR] [--apply] [--json]` | proposes `failed` or `succeeded` for every Agent spawn from your transcripts; `--apply` writes it into the routing log | | `usage_report.py [--days 14] [--projects-dir DIR] [--prices FILE] [--json]` | read-only: how often hints appear, adherence, subagent cost by tier and estimated savings, from your transcripts | The transcript readers print aggregates only, never prompt text. Costs come from [scripts/prices.json](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/scripts/prices.json), API list prices used to compare tiers; a Claude Code subscription bills differently. ## Install scripts | Script | What it does | |---|---| | `install.ps1`, `install.sh` | create or reuse `~/.system1-gateway/venv`, copy the package in, create the config if missing, print the registration command | | `uninstall.ps1 [-Purge]`, `uninstall.sh [--purge]` | unregister, delete the venv and the Windows login entry; `-Purge` also deletes config, engines and logs | ## Tests ```sh python -m unittest discover -s tests -v # from the repo root, Python 3.11+ ``` The tests cover schema validation, provider selection, config and environment overrides, malformed responses, timeouts, fallback and abstention, the adapters against local fake servers, the hooks, the scripts, and an end-to-end stdio JSON-RPC session. They make no network calls beyond 127.0.0.1. --- Page: Providers URL: https://thematthub.github.io/Sys1-AI-Gateway/reference/providers.html # Providers A provider is the adapter between the gateway and one engine, and six are built in: `laya`, `von`, `systemone`, `ollama`, `anthropic` and `mock`. Each reads its settings from `[providers.]` or from `SYSTEM1__` environment variables, and every HTTP adapter refuses a non-loopback endpoint unless `allow_remote = true` and bypasses any system proxy for local calls. To add another, see [Add a provider](../how-to/add-provider.md). | Provider | Settings (default) | Options | Several questions in one pass | |---|---|---|---| | `laya` | `endpoint` (`http://127.0.0.1:8765`), `model` (`typed-decisions`), `allow_remote` (`false`), `api_key` (env only) | at most 20 | yes | | `von` | `endpoint` (`http://127.0.0.1:8777`), `model` (`von-1.3`), `allow_remote`, `api_key` (env only) | no declared limit | yes | | `systemone` | `endpoint` (`http://127.0.0.1:8790`), `model` (unset: not sent), `allow_remote`, `api_key` (env only) | the server's (llama-server: 255) | yes | | `ollama` | `endpoint` (`http://127.0.0.1:11434`), `model` (required), `allow_remote` | at most 26 | no, one call per question | | `anthropic` | `endpoint` (`https://api.anthropic.com`), `model` (`claude-haiku-5-5`); key from `ANTHROPIC_API_KEY` only | the schema's 256 | no, one call per question | | `mock` | `confidence` (`0.9`), `delay_s` (`0`), `fail`, `response`, `max_options` | any | no | ## laya The adapter talks to `laya-serve`, `POST /v1/systemone`, and was verified against laya 0.3.20; health uses `GET /health`, and its detail warns when the pinned checkpoint is not preloaded. Every request pins `model`: Laya silently auto-routes an unknown model name, so an answer from another checkpoint is rejected as `malformed_response`. Set `model = ""` to let Laya route on purpose. | Gateway | laya-serve | |---|---| | `question` | `questions.decision.instructions` | | `options` | a `choice` question with `criteria` = options | | no `options` | a two-option `choice` with `criteria = ["yes", "no"]`, not Laya's `noul` (see below) | | `context` | `state` | | `confidence` | `answer_confidence`, the probability of the returned answer (Laya's own `confidence` is a different scale) | | `probabilities` | Laya's per-option `probabilities`, rescaled when a calibration matches | | `rationale` | the per-option probabilities, e.g. `laya typed-decisions: block the merge=0.52, rerun the pipeline=0.48` | Known Laya limitations and how the adapter handles them (from the [model card](https://huggingface.co/convaiinnovations/laya), checked live on 2026-09-24): | Limitation | Handling | |---|---| | `noul` (yes/no) follows its labels instead of the input; live, it called `rm -rf`, `DROP TABLE` and a force push "safe" (4/8 right) | yes/no questions go as a two-option choice: 7/8 right in that test, 88% on the 24-command seed set | | over- or under-confident until a temperature is fitted | the calibration layer | | close to chance zero-shot on unfamiliar tasks | measure with `/system1:calibrate`; fine-tune once you have a few hundred labelled examples | | more than about 20 options exhausts the decision head | refused with `provider_error`, so a fallback can take over; use a coarse-to-fine hierarchy | | English-only checkpoints are confidently wrong on non-Latin text | abstains when more than 30% of the letters are non-Latin; use `model = "multilingual"` for such text | | `score` (ordinal) is the weakest question type | not used | ## von The adapter talks to `von serve`, `POST /v1/systemone`, verified against von-sdk 1.3.1. Options are sent as a `{option: option}` criteria map, and `confidence` is the probability of the chosen option, since Von's own `confidence` is another scale. Von reads an 8,192-token window. ## systemone The Von contract for any other server that speaks it: llama.cpp's `llama-server` with a native decision model (Liquid AI d1, verified against build b11757 on 2026-10-21) or [engines/systemone/serve.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/engines/systemone/serve.py) for the Hugging Face-only engines. Requests and answers map as for `von`; `model` is sent only when set, since llama-server answers with the model it loaded. Health is `GET /health`. ## ollama Any local chat model with logprobs (Ollama 0.12 or later), through `POST /api/chat`. Options are shown as letters and the model answers one letter at temperature 0; the probabilities are the letters' shares of the top logprobs. A reply that is not an option letter is `malformed_response`. Health checks that `model` is installed. An LLM's letter probabilities are over-confident, so calibrate before trusting a threshold. ## anthropic A hosted Claude model through the Messages API, `POST /v1/messages`, with `urllib` (no SDK). The options go in a cached system block and the model answers through one forced tool call, `choose`, with up to three option names best first and a `confidence`. That confidence is the model's own estimate, since the API returns no answer probabilities, so there are no `probabilities` and no calibration applies. The key is read from the `ANTHROPIC_API_KEY` environment variable on every call and never logged or put in an error; the endpoint must be https (plain http only on loopback). Redirects are refused, so the key only goes to that endpoint; a system proxy is used for https only, as a CONNECT tunnel that never sees the key, and never for loopback. Health is `GET /v1/models/{model}`. Every request leaves the machine and is billed per token. ## mock A deterministic stand-in for tests and for trying the gateway: it always picks the first option with the configured `confidence`, sleeps `delay_s`, raises a provider error with the text of `fail`, or returns `response` verbatim (for malformed-response tests). --- Page: What System-1 means here URL: https://thematthub.github.io/Sys1-AI-Gateway/background/system-1.html # What System-1 means here The name comes from Daniel Kahneman's *Thinking, Fast and Slow*: System 1 is fast and intuitive, System 2 slow and deliberate. Here a System-1 model is a small model that answers a multiple-choice question in tens or hundreds of milliseconds, without writing text, and Claude is System 2. The gateway lets Claude hand the small judgement calls to the fast model and keep the reasoning for itself. ## What it is for, and what it is not A System-1 model is good at properties you can read off the text: which of these tools fits the request, is this command destructive, does this task only look something up. It is not for writing, chat, multi-step reasoning, counting, maths or dates; Claude does those itself and asks System-1 only the judgement call. It cannot be a chat or subagent model either, because it does not write text or call tools, but it can help decide which Claude model should ([model routing](../how-to/routing.md)). ## Why it abstains A fast answer is only useful when you know how far to trust it. Every answer carries a confidence, and below the threshold the gateway returns `abstained` instead of a guess, which tells Claude to reason it through. For the threshold to mean what it says, the engine's probabilities are calibrated on labelled decisions ([How routing and calibration work](routing-and-calibration.md)). ## Why one gateway in front of many engines This field moves fast, so the engine should be easy to swap. Claude sees three stable tools, and the engines sit behind a provider-neutral interface; adding or changing one changes one adapter file, not what Claude sees. Which engine to pick is measured, not assumed: see the [naga decisions study](../studies/naga-decisions.md). ``` Claude Code ──stdio/JSON-RPC──▶ server.py ──▶ gateway.py ──▶ providers/laya.py ──HTTP──▶ Laya (local) (or any MCP client) 3 tools validate providers/von.py, ollama.py, mock.py timeout (or your own) threshold / abstain fallback chain ``` ## Design choices - **Stdlib only:** Python 3.11+ and no third-party dependencies, so it installs offline. Engines get their own venvs. - **User-level:** registered once with `--scope user`, it works in every project and adds nothing to them. - **Small context cost:** the plugin adds about 235 tokens to each session (25 of them say when not to use System-1, which saves wasted calls), about $1 per 10,000 model requests at Opus cache-read prices. - **The plugin does not start the server itself:** a plugin's MCP command is one command line for every OS, and on Windows no single command reliably reaches Python 3.11+ (`python3` is often the Microsoft Store stub, `python` may be older, `bash` is not on PATH). So the plugin carries the skills and the installer, and the server runs from a pinned venv registered at user scope, which also serves other MCP clients. --- Page: How routing and calibration work URL: https://thematthub.github.io/Sys1-AI-Gateway/background/routing-and-calibration.html # How routing and calibration work Routing and calibration share one idea: ask the engine what it can see in the text, and trust its answer only as far as measurement shows. This page explains the mechanisms; the steps to use them are in [Turn on model routing](../how-to/routing.md) and [Calibrate and evaluate](../how-to/calibrate.md). ## Routing asks about features, not about the model On every prompt, the `UserPromptSubmit` hook ([route_hook.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/route_hook.py)) asks the engine four yes/no questions about the text, defined in [routing_features.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/routing_features.py): - `read_only`: does the task only look something up, check, count or answer, without changing files? - `design`: does it ask to design, plan, decide between options, research or define an architecture? - `review`: does it ask to review or audit existing work? - `specified`: does it name the exact steps, files or plan item? (recorded, not yet used by the rule) A small rule that you own, [routing_rule.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/routing_rule.py), turns the answers into a tier: `design` or `review` gives opus, otherwise `read_only` gives haiku, otherwise sonnet. When hints are on and every answer clears the threshold, Claude gets the tier as a hint for the subagents it spawns. The features exist because asking directly failed. On 284 labelled tasks, "which model?" scored 46 to 54% with Laya, 57% with Von and 62% with Gemma 4 E4B, against 52% for always answering "sonnet". Judging how hard a task is takes reasoning; reading a property off the text does not, and the policy stays yours. The [routing study](../studies/claude-code-routing.md) has the history and a rerun on the public seed sets. ## Outcomes are the labels that count The orchestrator's model choice is not a correct label; whether that model got the job done is. [scripts/outcomes.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/scripts/outcomes.py) marks a spawn `failed` when its tool result was an error, its task notification said `failed` or `stopped`, its report says it was blocked, or a later spawn redid the same task; otherwise `succeeded`. The tier is the model that actually ran. On 30 days of transcripts (2026-09-30), 279 spawns gave 25 failures, 4 of them doubtful. `scripts/evaluate.py` scores the rule's safety on those outcomes, the share of failures where the rule would have picked a higher tier, both with the engine's answers and with the teacher's feature labels. That tells a weak rule from a weak engine. ## Calibration rescales, it does not re-rank Engines report probabilities that do not match how often they are right. The gateway rescales them with one fitted temperature per (provider, model, question); questions too rare for their own fit fall back to a fit per option count. A temperature never changes which answer wins; it only makes `confidence`, and so `min_confidence`, mean what it says. Fits come from two files. `~/.system1-gateway/calibration.json` is fitted on this machine and takes precedence; [system1_gateway/calibration.json](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/calibration.json) ships with the repository, so every install starts calibrated. A fit for another model or checkpoint is ignored, and every response says `"calibrated": true` or `false`. The fits shipped now are for Laya `typed-decisions`, from the teacher-labelled seed sets in [calibration/](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/calibration): | Question | n | Accuracy | ECE before → after | Temperature | Accuracy at the threshold | |---|---|---|---|---|---| | Routing: which Claude model | 45 | 69% | 0.234 → 0.104 | 0.31 (under-confident) | 77% at 0.6 covering 69% of prompts; 82% at 0.75 covering 24% | | Is this tool call destructive? | 24 | 88% | 0.217 → 0.136 | 0.27 | 94% at 0.75, covering 75% | These numbers are measured on the data they were fitted on, from small samples, so treat them as rough. --- Page: Which model for naga decisions URL: https://thematthub.github.io/Sys1-AI-Gateway/studies/naga-decisions.html # Which System-1 model for naga decisions? An assistant that drives the [naga](https://github.com/TheMattHub/naga) simulation engine must pick one of 90 agent tools for each step, and this study measures which small or cheap model picks best, and how fast. The answer: Claude Haiku 5.5 first, with a 150M cross-encoder trained on Claude Opus 5.5's labels as the offline backup. On 274 search queries written by the agent, Haiku puts the right tool first for 190 (69%) in about 1 s, the trained cross-encoder for 183 (67%) in 1.1 s on a 4 GB laptop GPU, and naga's keyword search alone for 136 (50%). On 150 requests written by four other models the trained cross-encoder leads instead, 104 against 86. Every number comes from the result files under [bench/naga](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/naga/RESULTS.md), and the sections below say how far to trust them.
The grey step line is the Pareto frontier: the engines that no other engine beats on both speed and accuracy. Point at an engine, or at a kind in the legend, to see that kind's own frontier. The main chart shows the twelve engines the story turns on, failures included, and the retries chart five; the table below lists every engine we tried. ## How we test Every result is on a frozen blind set: written before any engine saw it, committed, and never used to tune anything. Two sets carry the story: - **TEST3:** 150 requests by four writers (Claude Opus 5.5, Gemma 4 E4B, Qwen3-1.7B and Liquid AI's d1-3B), each given only the 90 tool names and the tool to ask for. - **AGENT3:** 274 `find_tool` queries as the agent writes them. Opus 5.5 got each request of TEST3, TEST2 and HELD_OUT_2 with naga's real server instructions and `find_tool` as its only tool, and the query of its first call became the row. In Claude desktop this is the main role: the agent sends `find_tool` a short phrase of its own, not the user's words. Two earlier sets are supporting evidence only. TEST2 (100 requests) was written by Claude Sonnet 5.5, the same writer as the students' training data, and HELD_OUT_2 (25) is the first blind set. Every engine chooses from all 90 tools; the four that take at most 20 or 26 options (Laya, Gemma 4 E4B, AnyJev and the OpenThai Ollama build) get the 20 tools a bi-encoder ranks highest, which hold the right tool for 125 of 150 TEST3 rows and 234 of 274 AGENT3 rows. All local runs used one laptop with an RTX A500 (4 GB) and seed 0, and latency is the wall-clock time per decision; the hosted models' time is the API round trip from that laptop. ## Every engine on the two sets The trained cross-encoders and the hosted Claude models lead, and most System-1 decision models trail the untuned reranker. Top-1 is the right tool first, top-3 the right tool among the first three, out of 150 (TEST3) and 274 (AGENT3); the time is the median on AGENT3, or on TEST3 where AGENT3 was not run. | engine | kind | TEST3 top-1 | TEST3 top-3 | AGENT3 top-1 | AGENT3 top-3 | median ms | |---|---|---|---|---|---|---| | `find_tool` alone | keyword search | 36/150 | 63/150 | 136/274 | 190/274 | 0 | | harrier (bi-encoder) | encoder | 36/150 | 62/150 | 134/274 | 178/274 | 79 | | gte-reranker, untuned | encoder | 75/150 | 99/150 | 161/274 | 201/274 | 1,127 | | Qwen3-Reranker-0.6B | encoder | 86/150 | 115/150 | 172/274 | 212/274 | 5,514 | | gte, tuned on 54 questions | encoder | 78/150 | 114/150 | 175/274 | 204/274 | 1,142 | | gte <- Gemma labels | encoder | 48/150 | 111/150 | 152/274 | 202/274 | 1,149 | | gte <- Haiku labels | encoder | 96/150 | 123/150 | 178/274 | 211/274 | 1,128 | | gte <- Opus labels | encoder | 104/150 | 128/150 | 183/274 | 215/274 | 1,129 | | gte <- agreed labels | encoder | 99/150 | 130/150 | 183/274 | 219/274 | 1,127 | | Laya | System-1 decision model | 71/150 | 105/150 | 147/274 | 191/274 | 67 | | d1-omni-600M | System-1 decision model | 69/150 | 96/150 | 137/274 | 180/274 | 375 | | d1-3B (Q8_0) | System-1 decision model | 98/150 | 118/150 | not run | not run | 13,667 | | AnyJev on Qwen3-1.7B | System-1 decision model | 30/150 | 59/150 | 92/274 | 136/274 | 1,891 | | OpenThai (Ollama build) | System-1 decision model | 31/150 | 61/150 | 75/274 | 168/274 | 371 | | Gemma 4 E4B | LLM | 84/150 | 103/150 | 179/274 | 206/274 | 3,006 | | Claude Haiku 5.5 (API) | LLM | 86/150 | 117/150 | 190/274 | 216/274 | 966 | | Claude Opus 5.5 (API) | LLM | 98/150 | 128/150 | 190/274 | 225/274 | 1,741 | The agent's phrasing is easier for every engine, because the agent already turns a loose request into the catalogue's words: `find_tool` alone climbs from 24% on TEST3 to 50% on AGENT3, and the gaps between engines shrink. Liquid AI's d1-3B is the one System-1 model level with the hosted ones on TEST3 (98, like Opus), but it takes 13.7 s a decision on this GPU, its licence allows commercial use only below USD 10 M revenue, and its AGENT3 run (about an hour) was not made. Six engines have no TEST3 or AGENT3 rows, because a run would take well over half an hour here, the engine had already failed, or it does not fit the GPU: - ModernBERT-NLI and GLiNER2.5: 14 of 25 on HELD_OUT_2 over all 90 tools, at 5 to 7 s a decision. - Von 1.3: 2 of 25 on HELD_OUT_2 over all 90 tools, 17 s a decision. - OpenThai-SystemOne through its own client: 7 of 100 on TEST2 over all 90 tools, 16.6 s a decision. - JEV-9B and Jeva-4B: their weights do not fit 4 GB. The table settles the shortlist of serious candidates: the trained cross-encoders, the hosted Claude models and, where time does not matter, d1-3B and Qwen3-Reranker. ## Retries: a fast engine catches up A fast engine that may try its next pick after a wrong one catches the best engines' first try at a tenth of their time. Within three tries Laya is right on 105 of 150 TEST3 requests (70%) in about 130 ms of decisions, where the best first try is the Opus-trained cross-encoder's 104 at 1.2 s; on AGENT3 it reaches 191 of 274 in about 120 ms, against Haiku's and Opus's 190 first time at 1.0 and 1.7 s. The slower engines climb too: within three tries the Opus-trained and agreed cross-encoders reach 85-87% on TEST3 and 78-80% on AGENT3, Opus 85% and 82%. [![Right within 1, 2 and 3 tries against the time those tries take, per engine](../screenshots/retries.png)](../screenshots/retries.png) Try k counts as right when the right tool is in the first k of the engine's ranking, and the time is the median decision time times the mean tries used (a request stops when it is right). That is how naga uses the ranking: the assistant tries the next tool in the same list after a call fails, so in practice the extra cost is the failed tool call rather than a new decision. Round 2 checked that re-asking with the wrong pick removed gives counts close to the first ranking's ([RESULTS.md](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/naga/RESULTS.md#round-2-retries), oracle table), and that telling the model what the wrong pick did makes it worse. Retries pay when a wrong pick is cheap to notice; when it is not, the first try is what counts. ## A small student beats its teacher A 150M cross-encoder trained on a strong teacher's labels beats that teacher on requests from writers it never saw. The student is `Alibaba-NLP/gte-reranker-modernbert-base`, trained on about 1,000 requests written by Claude Sonnet 5.5 from the tool descriptions, each labelled with a teacher's first choice over all 90 tools; one student per teacher, same recipe and seed, 23 to 27 minutes of training each on the laptop GPU. | teacher | teacher's top-1 = the request's tool (of 1,067) | cost of the labels | student on TEST3 (teacher) | student on AGENT3 (teacher) | |---|---|---|---|---| | Gemma 4 E4B (local) | 838 | 57 min of GPU | 48 (84) | 152 (179) | | Claude Haiku 5.5 | 943 | 13.3 M input tokens, mostly cached, and 56 k output; 20 min | 96 (86) | 178 (190) | | Claude Opus 5.5 | 1,030 | about USD 4.70 with the cached tool documents | 104 (98) | 183 (190) | | Haiku and Opus agreed (936 rows) | 936 | no new labels | 99 | 183 | The better the labels, the better the student: Opus labels beat the agreed ones, which beat Haiku's, and Gemma's noisy labels made a student below the untuned reranker. On TEST3 the students beat their teachers, probably because they learned the tool a request was written for, while a language model picks what the tool documents fit best when a request is loosely worded. On the agent's own phrasing Haiku and Opus keep a lead of 7 to 12 queries over their students. The trained models are kept for our own use and not published; [distill/](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/naga/distill/NOTES.md) has the scripts to train one. ## Precision: half on a GPU, full on CPU The trained cross-encoder should run in half precision (float16) on a GPU and in full precision (float32) on a CPU, and the answer it gives does not change. We re-ran it in float32 over all 90 tools on TEST3, AGENT3, TEST2 and HELD_OUT_2 (549 requests): the top-1 pick is the same on every request, and the top-3 order differs on 3. | run | TEST3 top-1 | AGENT3 top-1 | median per request | |---|---|---|---| | GPU, float16 | 104 of 150 | 183 of 274 | 1.2 s | | GPU, float32 | 104 | 183 | 3.9 s | | GPU, bfloat16 | 105 | not run | 1.1 s | | CPU, float16 | same picks as float32 (10 rows) | not run | 117 s | | CPU, float32 | 10 rows, same picks as float16 | not run | 16.5 s | The speed flips with the hardware because a GPU has half-precision units and most CPUs do not, so the CPU converts every number back and forth and runs 7 times slower in float16 than in float32 (10 TEST3 rows, 90 tools each; a small sample, but the gap is large). bfloat16 gains nothing over float16 on the GPU. Since 0.14.0, naga loads any cross-encoder picker in float16 on a CUDA GPU and in float32 on a CPU, whatever the folder stores (decision D674), and the [published model](https://huggingface.co/TheMattFace/gte-reranker-naga-toolpicker) keeps float16 weights (290 MB) because that is the smaller download and loads as float32 on a CPU without losing anything. ## Do models favour their own writing? We expected that a model would score higher on requests written by itself or by its teacher, and the data does not show it. The checks, all over the 90 tools: - Students trained on Sonnet-written requests did not gain on the Sonnet-written TEST2: the Haiku student got 80 of 100 against Haiku's 82. On TEST3, from four other writers, the same student beat its teacher, 96 against 86 of 150. - Engines on their own TEST3 rows show no consistent pattern. Opus gets 71% of its own rows against 56-68% of the others, but d1-3B scores even higher on Opus's rows (74%) than on its own (68%), so Opus's rows are simply the cleanest; Gemma gets 45% of its own rows against 53-66% of the others. - AGENT3 was written by Opus, and Opus as the picker ties Haiku there (190 each). The one hint the other way is small: the Opus-trained student leads the Haiku-trained one most on Opus's rows (31 against 26 of 38, against 0 to 2 on the other writers). The clear overfitting came from training on too few questions: round 2's fine-tune on 54 questions got all 54 right and 11 of 25 new ones. ## Model building Naga's assistant also picks a block type (24) and a distribution (31) when it builds a model, and there a hosted model leads by a wide margin. These sets are 40 and 30 sentences, written blind from the names, with naga's fuller label descriptions (round 2); top-1 out of 40 and 30. | engine | blocks | distributions | median ms | |---|---|---|---| | Claude Haiku 5.5 (API) | 37 | 30 | 876-906 | | Gemma 4 E4B | 34 | 27 | 2,316-2,455 | | d1-3B (Q8_0) | 33 | 28 | 1,095-1,096 | | gte-reranker, untuned | 25 | 19 | 75-91 | | Qwen3-Reranker-0.6B | 24 | 21 | 591-754 | | Laya | 18 | 13 | 65-75 | | keyword search (BM25) | 15 | 17 | 0 | These sets are small, so a difference of one or two rows means nothing, but the order matches tool choice: Haiku, then the large local models, then the fast encoders. ## What to pick Pick by where the queries come from, and by whether a hosted call is acceptable. The times are medians per decision on the laptop above. | You need | Pick | What you get | Trade-off | |---|---|---|---| | Tool choice in an agent (our setup) | Claude Haiku 5.5 first, the Opus-trained gte as offline backup | AGENT3 190 (Haiku) and 183 (backup) of 274, about 1 s each | Haiku needs a key and a network and costs under USD 0.0013 a decision; the backup is free but must be trained | | Tool choice from people's own words | The Opus-trained gte | TEST3 104 of 150, against Haiku's 86 (28 rows it wins, 10 it loses; sign test p = 0.005) | 1.1 s on a 4 GB GPU; train it yourself | | A published local model, no training | Gemma 4 E4B (Q4_K_M via Ollama) or `Qwen/Qwen3-Reranker-0.6B` | TEST3 84 and 86, AGENT3 179 and 172 | 3.0 and 5.5 s over 90 tools; d1-3B gets 98 on TEST3 at 13.7 s | | Fast | Laya `typed-decisions`, with up to three tries | 70% within three tries on both sets, about 120-130 ms | 47-54% first time; at most 20 options | | Model building | Claude Haiku 5.5, or Gemma 4 E4B locally | blocks 37 and 34 of 40, distributions 30 and 27 of 30 | Gemma takes 2.3-2.5 s | Opus as the picker adds nothing over Haiku on agent queries (190 each) at twice the time and about 40 times the price per token. Our own naga setup runs `--tool-picker haiku --tool-picker-backup` with the Opus-trained gte: on AGENT3 Haiku wins 15 queries the backup loses and loses 8 it wins (sign test p = 0.21), so the two are within noise, and the backup costs nothing and works offline. ## Caveats The sets are small for close calls: on AGENT3 one query is 0.4 points, and Haiku's 7-query lead over the student is within noise. AGENT3's queries come from one agent model (Opus 5.5) prompted from requests, not from logs of real sessions, and TEST3's small writers add label noise (12 of its requests do not fit their target, listed in [NOTES.md](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/naga/distill/NOTES.md)). Everything ran on one laptop, so the times move with your GPU. Models ran out of the box with the prompts in RESULTS.md, and a better prompt may change the order. ## Rerun it The bench is in [bench/naga](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/naga/README.md) and runs against a clone of the public [naga](https://github.com/TheMattHub/naga) repository. `run.ps1` runs each engine in its venv; `BENCH_ROUND=test3` or `agent3` with `BENCH_TASKS=Tall` writes the TEST3 and AGENT3 rows to `distill/results3` and `distill/results4`, and `python bench/naga/plot.py` redraws both charts and prints the engine table above. [RESULTS.md](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/naga/RESULTS.md) is the full lab notebook, round by round. --- Page: Which model tier for a Claude Code task URL: https://thematthub.github.io/Sys1-AI-Gateway/studies/claude-code-routing.html # Which model tier does a Claude Code subagent task need? This study asks whether a System-1 engine can pick the Claude model a subagent task needs, and finds that asking it directly works on easy synthetic prompts and failed on real work. The gateway's routing hook therefore asks the engine about features of the task and lets a small rule pick the tier. The numbers below come from [bench/routing/routing.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/routing/routing.py), and the history from the earlier private study, of which only aggregates are published. ## The question When Claude Code spawns a subagent, it picks a model: a small, fast one (haiku) for mechanical work, a mid one (sonnet) for well-specified implementation, a large one (opus) for design, review and subtle bugs. A wrong pick either wastes money and time or hands a hard task to a model that fails it. A System-1 engine answers in tens of milliseconds, so if it could make that pick from the task text, every spawn could be routed for free. ## What happened before Asking the engine "which model?" did not work on real tasks. On 284 labelled tasks from the author's own Claude Code sessions, direct choice scored 46 to 54% with Laya, 57% with Von and 62% with Gemma 4 E4B, against 52% for always answering "sonnet". Judging how hard a task is takes reasoning, which is what a System-1 model does not do. The labels that count are outcomes, not the orchestrator's choices. Over 30 days of transcripts (to 2026-09-30), 279 spawns gave 25 failures, 4 of them doubtful ([scripts/outcomes.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/scripts/outcomes.py) marks a spawn failed when its result was an error, it was stopped, it reported being blocked, or a later spawn redid it). With so few failures, no tier policy can yet be shown safe, and `scripts/evaluate.py` says so. ## Why features and a rule Reading a property off the text is a System-1 task; judging difficulty is not. So the hook asks four yes/no questions ([routing_features.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/routing_features.py)): is the task read-only, is it design, is it a review, does it name the exact steps. A rule you own ([routing_rule.py](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/system1_gateway/routing_rule.py)) maps the answers to a tier: design or review gives opus, otherwise read-only gives haiku, otherwise sonnet. The policy stays readable and editable, and a wrong tier can be traced to one wrong answer or to the rule. [How routing and calibration work](../background/routing-and-calibration.md) has the mechanism. ## The rerun on the public seeds Every engine that can answer a choice or a yes/no question was rerun on the two public seed sets, through its gateway provider exactly as the hooks ask: [calibration/routing-seed.jsonl](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/calibration/routing-seed.jsonl) (45 prompts, 15 per tier, teacher labels) and [calibration/yes-no-seed.jsonl](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/calibration/yes-no-seed.jsonl) (24 tool calls: is this destructive or hard to reverse?). "Direct choice" is the retired "which model?" question over the three tiers; "features + rule" asks the four features in one pass and applies the rule; "below" counts the unsafe misses, a tier under the label (or none). Run on 2026-10-21 on the laptop of the naga study (RTX A500, 4 GB). | engine | provider | direct choice | below | features + rule | below | yes/no (destructive) | median ms: direct / features / yes-no | |---|---|---|---|---|---|---|---| | anyjev | systemone | 23/45 | 18 | 16/45 | 14 | 12/24 | 200 / 515 / 136 | | d1-3b | systemone | 40/45 | 3 | 32/45 | 2 | 23/24 | 153 / 399 / 83 | | d1-omni | systemone | 29/45 | 9 | 20/45 | 6 | 20/24 | 25 / 65 / 15 | | gemma4 | ollama | 41/45 | 3 | 18/45 | 0 | 24/24 | 622 / 2486 / 510 | | gte | systemone | 30/45 | 6 | 16/45 | 14 | 11/24 | 25 / 64 / 21 | | haiku | anthropic | 43/45 | 2 | 30/45 | 3 | 24/24 | 1111 / 3976 / 995 | | laya | laya | 31/45 | 5 | 18/45 | 0 | 21/24 | 42 / 57 / 38 | | openthai-ollama | ollama | 21/45 (1 unanswered) | 20 | 9/45 (19 unanswered) | 20 | 9/24 (3 unanswered) | 79 / 339 / 66 | | openthai | systemone | 37/45 | 2 | 19/45 | 0 | 23/24 | 99 / 378 / 115 | | qwen3-rerank | systemone | 24/45 | 0 | 15/45 | 0 | 12/24 | 81 / 246 / 60 | | von | von | 35/45 | 7 | 24/45 | 4 | 22/24 | 51 / 189 / 29 | Always answering sonnet: 15/45. On these seeds direct choice beats the rule for every engine, the opposite of the real-task result. The best are Claude Haiku 5.5 (43/45, hosted), Gemma 4 E4B (41/45, 0.6 s) and Liquid AI's d1-3B (40/45, 0.15 s), against 15/45 for always answering sonnet. The rule loses mostly by design: it sends every edit that is not read-only to sonnet, while the seeds label small edits (renames, typo fixes) haiku, so 15 haiku rows are out of its reach unless they are lookups. On safety the rule does better: Gemma, Laya and OpenThai-SystemOne never route a task below its label through the rule, against 2 to 5 times by direct choice. The rerankers (gte, Qwen3-Reranker) and AnyJev on a 1.7B base are near chance on yes/no, because "yes" and "no" carry no text for a reranker to match; they suit choices whose options describe themselves. The caveat outweighs the ranking. The seeds are small (45 and 24 rows) and synthetic: a teacher model wrote clean one-line prompts whose tier is plain from the words, unlike real requests that depend on the conversation, so a high direct-choice score here does not undo the 284-task result. The rule stays the gateway's default, and an engine should be judged with `scripts/evaluate.py` on your own outcome-labelled spawns before routing is turned on. Per-row answers are in [bench/routing/results/](https://github.com/TheMattHub/Sys1-AI-Gateway/blob/main/bench/routing/results). --- Page: Release notes URL: https://thematthub.github.io/Sys1-AI-Gateway/release-notes.html # Release notes Releases are tagged `system1--v`, the Claude Code plugin convention (`claude plugin tag . --push`), and listed on [GitHub](https://github.com/TheMattHub/Sys1-AI-Gateway/releases). Each one also gets a one-word codename from Kahneman's *Thinking, Fast and Slow*, in alphabetical order: minor and patch releases take the next letter, and major releases break the alphabet and take the authors' names. ## Unreleased - **Providers:** `systemone` for any server speaking the `/v1/systemone` contract (llama.cpp's `llama-server` with Liquid AI's d1, or `engines/systemone/serve.py` for the gte reranker, Qwen3-Reranker, AnyJev and OpenThai-SystemOne) and `anthropic` for a hosted Claude model over HTTPS (key from `ANTHROPIC_API_KEY`). - **Naga study, round 3:** Liquid AI d1-omni-600M and d1-3B, OpenThai-SystemOne and AnyJev on the 100-request TEST2 set and the model-building sets; d1-3B matches Gemma 4 E4B (licence: commercial use below USD 10 M revenue). - **New study:** which model tier a Claude Code subagent task needs, with a rerun on the public seed sets. ## 0.2.0 Econs (2026-10-20) The first public release of the System-1 Gateway: a local MCP server that gives Claude Code and other MCP clients fast multiple-choice decisions from a pluggable engine, and the studies that show which engine to pick. - **Gateway:** three stable tools (`system1_decide`, `system1_capabilities`, `system1_health`) in front of Laya, Von, any Ollama chat model or an adapter you add. Python 3.11+ and the standard library only; installs as a Claude Code plugin with guided setup or with one script. - **Study: which System-1 model for naga decisions:** eleven engines on tool choice and model building for the naga simulation engine, measured on frozen blind test sets on one 4 GB laptop GPU. The short answer: Claude Haiku 5.5 is the most accurate (22 of 25 tool choices over all 90 tools, 37/40 blocks, 30/30 distributions, about 0.9 s, hosted); Gemma 4 E4B is the most accurate local engine; gte-reranker-modernbert-base is the fast one (about 0.1 s). The report covers retries, fewer-choice designs that did not pay, and fine-tuning that did not hold on the blind set. - **Bench:** every number comes from scripts in `bench/naga` that rerun against a clone of the public naga repository; new engines plug in with one `eng_*.py` file. - **Open source:** Apache-2.0, with contribution and security guides and CI on Ubuntu and Windows. - **Known limit:** the study's sets are small (25 tool questions, 40 blocks, 30 distributions), so a difference of one or two rows is noise. ## Codenames | Release | Codename | Concept | |---|---|---| | 0.1.9 | **Anchoring** | An initial number drags every later estimate toward it. The routing hint anchors Claude's model choice | | 0.1.10 | **Baserate** | Base-rate neglect: a vivid story beats the statistics. Routing now gathers its own base rate, silently, before trusting the hint | | 0.1.11 | **Coherence** | System 1 builds the most coherent story, not the most likely one. Routing stays out of follow-ups it can't see the story for, and the skill picker waits for an engine that can hold the whole list | | 0.1.12 | **Denominator** | Denominator neglect: "1 in 10" feels riskier than "10%". Routing is now judged on outcome counts, not on how sure the engine sounds | | 0.2.0 | **Econs** | The perfectly rational agents of economics, as opposed to Humans. The first public release: engines are chosen by measurement on frozen blind tests (the naga decisions study), not by how sure they sound | | next | **Framing** | The same facts described differently lead to different choices | | | **Halo** | Liking one trait colors judgment of all the others | | | **Illusion** | The illusion of validity: confidence without accuracy, which is exactly what calibration corrects | | 1.0 | **Kahneman** | | | 2.0 | **Tversky** | Amos Tversky, Kahneman's research partner behind heuristics and prospect theory |