Model fleet board
What the router does
Several of the sites in this collection use open-weights language models that we run ourselves, on rented servers and on our own workstations. Those models can only do a limited amount of work at the same moment, so a small program sits in front of them: the router.
Everything that wants a model, whether a background batch job that classifies or summarises thousands of pages or, where a feature offers it, a live request from a visitor, joins the same queue at the router. The router decides which backend takes each request, holds back when a backend says it is busy, and keeps a lane free for callers that must not wait behind a big batch.
The numbers below are counts the router keeps about itself. They start again from zero whenever the router restarts, so each total covers the period since the date printed on the page, not all time.
The router last restarted on 26 Oct 2026; every total counts from then. Measured 27 Oct 2026.
Capacity right now
- backends up
- 3 of 5
- Places the router can send work to that answered its latest health check, out of all it is set up to use.
- workstation nodes up
- 1 of 2
- Our own workstations that take some of the requests and answered the latest health check, out of all that are set up.
Work since the last restart
- callers
- 28
- Distinct jobs and services that have sent the router at least one request.
- requests received
- 54,733
- Every request to the model interface of the router that has been answered or has failed (chat, embeddings and the like): answered plus failed.
- requests answered
- 54,626
- Requests that came back with a successful response.
- requests failed
- 107
- Requests that came back with any other response.
- hand-offs to a backend
- 54,604
- Times the router passed a chat request to a backend. A retried request counts again and requests that are not chat (such as embeddings) are not counted, so this is a different count from the request counts above.
- busy signals
- 0
- Hand-offs where the backend replied "too many requests". Each is one of the hand-offs above, held back and not necessarily a failed request.
How the queue was shared
- hand-offs in a reserved lane
- 25,834
- Hand-offs in the main pool that ran in a lane kept free for one caller, so they never waited behind shared traffic.
- hand-offs in the shared pool
- 13,290
- Hand-offs in the main pool that ran in the shared part, where no single caller may take more than its share.
- hand-offs on a lent lane
- 45
- Hand-offs that borrowed a reserved lane while its owner was not using it.
- seconds spent waiting
- 59
- All hand-offs added together: the time they sat in the queue before a backend took them (whole seconds).