Devlog
Building Fleet Hive
Claude Fleet already coordinates multiple machines running Claude Code. Fleet Hive is the layer underneath that: one gateway for every model you have a key for — local Ollama boxes, Gemini, NVIDIA NIM, Groq, Cerebras — plus a small orchestrator that fans a prompt out to several of them at once and has models from different families adversarially judge each other's answers.
It was built the way it argues you should build things: an AI orchestrating research and implementation across a fleet of other models, catching a real hallucination from one of its own research sources along the way — not a hypothetical, an actual fabricated config syntax that made it one review pass deep before a different model family called it out.
01 · Origins
Routing answers “which model.” Hive answers “how many at once.”
Claude Fleet's existing model routing guide is about picking the cheapest model that can do a job correctly. That solves cost — it doesn't solve throughput. Even with a garage full of GPUs and a drawer of API keys, every one of those lanes spoke a different dialect: bespoke curl flags for Ollama, a different auth header for NIM, a different base URL for each cloud provider. Work happened one dispatch at a time no matter how much hardware was sitting idle.
The fix splits into two deliberately separable pieces. A gateway — LiteLLM Proxy, free and open-source — puts every lane behind one OpenAI-compatible endpoint with named aliases, load-balancing, cooldowns, and fallback chains. And a thin, stdlib-only Python CLI on top that does the part a router genuinely can't: fire the same prompt at several aliases in parallel, and convene a panel of judges — deliberately from different model families than the one being graded — to try to refute a generation rather than rubber-stamp it.
02 · Method
The build dogfooded the thing it was building
Research ran off-budget first: a round of web search plus a synthesis pass from a large cloud model produced a verdict (build the hybrid — gateway plus custom orchestrator) and a config sketch. Then a second model, from a different family, was asked to adversarially verify that sketch against the same evidence rather than take it on faith.
It caught real fabrications: three LiteLLM configuration constructs — a nonexistent model_group: key, an invented virtual_keys: section, a claim that budgets work over SQLite — that the first model had stated with total confidence and that were simply wrong, checked against the project's actual documentation. That near-miss is the whole argument for the judge-panel pattern baked into the CLI: a single fluent answer, even from a strong model, is not the same thing as a verified one.
The implementation that followed split the same way: separate files were generated in parallel by different providers — one by a NIM-hosted model, one by Gemini, one by a Groq-hosted model — reviewed by yet another set of models from different families, then integrated and live-tested end to end on real hardware. Building the thing whose whole point is “models checking each other” by actually having models check each other.
03 · Gotchas found the hard way
What live traffic revealed that no design doc predicted
Every one of these showed up only once real requests hit real machines — the kind of thing a spec review can't catch and a live gateway eventually will.
›A socket timeout doesn't bound total request timereliability
A client call passed timeout=300 to urllib and still hung for over twenty minutes. The gateway was dribbling keep-alive bytes while internally retrying a dead upstream, which resets what a plain socket read-timeout actually bounds. The fix wraps every gateway call in a hard total deadline enforced by a worker-thread future — the only way to guarantee a call actually returns by a fixed wall- clock time, independent of what the server on the other end is doing.
›Fallback chains follow exactly one hoplitellm
The config wired alias A to fall back to B, and separately wired B to fall back to C — reasonable to assume the router would walk A → B → C if both A and B failed. It doesn't. When B also fails, the router looks up B's own fallback list rather than continuing down A's original one. Every link in a chain needs its own explicit fallback entry; nothing walks a list further than one hop for you.
›Reasoning models can spend their whole budget on tokens you never seemodel behavior
A local reasoning model returned empty content no matter the prompt — turned out the gateway simply doesn't forward an Ollama-specific “disable thinking” parameter at all, so the model burned its entire token budget on hidden reasoning and had nothing left for the answer. The same failure shape reappeared later on two cloud models (a GLM reasoning tier and Gemini 2.5 Pro/Flash) under a small max_tokens: finishReason: MAX_TOKENS, zero visible text. The general lesson: a “thinking” model's hidden tokens and its visible answer draw from the same pool, so any caller-facing token budget has to assume that.
›A network fix that was really an old fix, forgottenops
One machine's local model server was reachable from its own terminal but invisible to every other machine on the network — bound to 127.0.0.1 instead of all interfaces. The exact same failure, on a different machine, had already been diagnosed and fixed weeks earlier and wasn't applied fleet-wide — the second occurrence cost real debugging time chasing gateway timeouts before anyone thought to check the simplest possible cause first.
›A bare Python User-Agent gets you blocked, not deniededge cases
One cloud provider's edge returned a 403 that looked like a bad API key — Cloudflare error 1010, “access denied by browser signature.” The key was fine; the block was on the default Python-urllib/3.x User-Agent string that Python's standard library sends by default. Every outbound call now sends a curl-style header instead.