The LLM cache

Why it exists

A free model pool, always at its limit

All of our AI runs on a free, rate-capped model pool, and background jobs keep that pool at its limit around the clock.

Visitor questions used to queue behind those batch jobs for 10–25 s, or get dropped.

The rule is no paid AI APIs anywhere, not even as a fallback. Capacity cannot be bought, so it has to be saved.

Step by step

How a question flows

A website sends the service an AI request exactly as it would send it to the model. The service decides whether a model actually needs to run.

• Answer cache. A question asked before gets the saved answer back, and no model runs.
• Page routing. A question can be matched to the most relevant pages on the site instead, with no AI text generated.
• Model router. Only when a model is needed does the request go on to the model router, where 2 of the pool's 31 slots are reserved for visitors.
• Ask-later queue. When nothing matches, the question is saved so it can be answered properly and published later.
• Fail-open. If the cache is down, the site calls the model directly, so an outage never breaks a site.

What it does

Three jobs

• Answer cache. The same question asked before gets the saved answer in about 60 ms instead of seconds. One pilot site went from 3.6 s to 0.93 s on a repeat question. Exact matches are kept for 7 days. Near-identical questions can match only above a strict similarity threshold. When two visitors ask the same thing at the same moment, the model runs once.
• Page routing ("ask the register"). A visitor's question is matched to the most relevant pages on the site in about 0.5 s, and no AI text is generated. Nothing is invented, so it is safe for legal, health and finance. There are about 10,700 pages indexed across the sites. It powers the page finders on Rights and Limits, Police Census, Federal State Law and one more register that is not in the directory.
• Ask-later queue. When nothing matches, the question is saved so it can be answered properly and published later: visitor questions become content.

Design principles

The safety rules

• Fail-open. If the cache is down, sites call the model directly. An outage never breaks a site.
• A reserved visitor lane. 2 of the model pool's 31 slots are reserved for visitors and never lent to batch work.
• Rate limits. Per visitor, per site and overall, and cache hits never count. Switched on 28 Oct 2026.
• Strict matching. A near-identical question can match only above a strict similarity threshold, and is refused if the two differ in a number, a year, "not" or a currency: "within 14 days" never gets a "28 days" answer.
• Pre-generate, not per visitor. Reviewed FAQs, page routing and pre-recorded audio beat live AI on cost, speed and accuracy. For example, Court Judiciary shows FAQs that passed two independent reviews before publishing.
• Strictest on legal and health. Legal and health sites get page routing only, and no fuzzy reuse of answers.

As of 27 Oct 2026

The numbers so far

Page routing is the busy part: 285 successful lookups in the 6 days to 27 Oct 2026, typically about 0.5 s each.

successful page-routing lookups, in the 6 days to 27 Oct 2026

285

measured · 27 Oct 2026

typical time of one page-routing lookup, in the 6 days to 27 Oct 2026

~0.5 s

measured · 27 Oct 2026

pages indexed for page routing, across the sites

~10,700

measured · 27 Oct 2026

a repeat question answered from the answer cache

~60 ms

measured · 27 Oct 2026

one pilot site, on a repeat question

3.6 s → 0.93 s

measured · 27 Oct 2026

how long an exact match is kept

7 days

fact · 27 Oct 2026

model-pool slots reserved for visitors, never lent to batch work

2 of 31

fact · 27 Oct 2026

the day the service went live

18 Oct 2026

fact · 27 Oct 2026

The answer cache was built and proven, but in that same window its three pilot sites (The Fomite, CoverForge USA and MarketMoodIQ) sent it no traffic, so its hit rate then was effectively zero. No savings figure is claimed for it.

As of 28 Oct 2026

What's done

Six improvements, all complete.

1. Pilot sites reconnected. Found why the pilot sites went quiet, and reconnected them.
2. Cache pre-filled. Pre-filled the cache with answers to likely questions, instead of waiting for repeats.
3. Faster page routing. Sped up page routing with a faster embedding model, from about 500 ms to about 100 ms, and began logging "found nothing" questions as content gaps.
4. Rate limits on. Switched on the built-in rate limits, on 28 Oct 2026.
5. More registers. Added page routing to more registers: Charity Accounts Intel and the US court register, Court & Judiciary.
6. Near-duplicate matching. Turned on near-duplicate answer matching on more sites, only where questions genuinely repeat. Legal and health sites stay exact-match.

Every figure on this page

Measured means a count or a timing; fact means a setting or a date; reported means stated by the cache's maintainers and not measured here. Each carries its date.