Measured numbers
Counted by the collection, not written by hand. A figure we cannot support is left out rather than shown as zero.
- Pages
- 141
- 13th of 18 measured in Tech/AI, by sitemap URLs
- Page count over time
- 138 → 141 pages measured, 30 Sep to 25 Oct
- Languages
- English
- Deploys
- 7 in the last 30 days
- Last deploy 18 Oct 2026. From the public status feed
- llms.txt
- Published
- llms.txt 15.5 KB · llms-full.txt 25.1 KB
- Sitemap
- Listed in robots.txt
- Content last updated
- 24 Oct 2026
- As reported by the site itself
Tech and AI search
Read from the live site on 2026-10-23: its homepage, robots.txt and llms.txt. A line we could not read is left out.
- Built with
- Next.js
- Read from the homepage markup
- Structured data
- Dataset, Organization, WebSite
- schema.org types on the homepage
- AI crawler access
- Allows all 11 major AI crawlers
- From the site’s robots.txt
- llms.txt link depth
- 36 page links
- Markdown links in the site’s llms.txt
How it is built, by role
The same roles apply across the whole collection. Nothing in this block is specific to this site; what the site says about itself is quoted word for word below it.
- 1
Edge network
Visitors reach a site through an edge layer: a content delivery network with a web application firewall in front of it.
- 2
Application host
The edge talks to an application host that serves the site and renders its pages.
- 3
Data
What the pages show comes from a relational database or, for a site pre-rendered at build time, from data files built into its release.
- 4
Scheduled measurement
The collection measures its sites on a schedule: content dates and site health nightly, page counts read from each site’s sitemap weekly. Every reading is kept with its date. A site too large to re-walk is not re-measured, and shows no trend.
- 5
Public status feed
A monitoring panel sends a signed report on uptime, deploys and backups once a day, and the Operations page publishes it. A figure on this page that comes from that report says so.
In the site's own words
Each passage is copied unaltered from a page the site publishes, with the address it was taken from and the date we took it. We add nothing between them.
“We record how non-human clients behave on a website, and publish it.”
“The edge (Caddy) writes a structured access log for every request. A collector reads that log continuously and writes rows to Postgres. Nothing is sampled; nothing is inferred from a third-party analytics product; no JavaScript is required for a request to be counted.”
“The raw IP is used once, to look up an ASN and country, and is then stored only as a truncated salted hash.”
“Raw request rows are deleted after 90 days; only per-client daily totals survive.”
“Clients are matched to a catalogue by case-insensitive User-Agent substring. Anyone can send any User-Agent, so a match on its own is a claim, and the site labels it as one.”
“A client is marked verified only when the connecting address passes forward-confirmed reverse DNS: the address resolves to a hostname in a domain the operator publishes for its crawler, and that hostname resolves back to the same address. Unverified rows are still published, clearly marked, because “something claiming to be GPTBot arrived from an unrelated network” is itself worth knowing.”
“Nothing is done to non-compliant clients. They are not blocked, throttled, or served poisoned text. Feeding generated garbage to crawlers is a common tactic and this project does not use it: a measuring instrument that alters what it measures is not measuring.”
“Selection bias, by construction. This site advertises itself to agents. The population it observes is skewed toward clients that look for such advertisements. That is the point, and it is still a bias.”
Screenshots
Timeline
138 → 141 pages measured, 30 Sep to 25 Oct
| Date | Pages |
|---|---|
| 2026-09-30 | 138 |
| 2026-10-10 | 139 |
| 2026-10-18 | 139 |
| 2026-10-25 | 141 |
- 30 Sep 2026138 pages
- 10 Oct 2026139 pages
- 18 Oct 2026139 pages
- 25 Oct 2026141 pages