Who keeps this running
Most projects like this die the same way. Not because they stopped being useful — because the person who built them moved on, the card on file expired, and nobody had the keys. A scoreboard nobody refreshes is just a snapshot, and models change every few weeks.
Open source alone does not fix that. Anyone can have the code and still nobody pays the bill. A project should exist for as long as it is still useful to someone — so the thing that has to survive us is not the code, it is the ability to keep paying for the work.
An agent holds the keys, not a person
The custodian of this project is software. It holds the credit balance, tops up the model credits when a run needs them, and renews the domain before it lapses. It is funded to run on its own, and anyone in the world can add to it — using x402, an open standard that lets a payment settle directly against a web request, with no account and no human approving it.
That is the part that matters: if the maker vanishes, nothing stops. A researcher who depends on these numbers, an advocacy group that cites them, a lab that wants its own model re-tested — any of them can fund a refresh directly, and it happens. Nobody has to find the maintainer, get added to a repo, or wait for a grant cycle. The people who get value from it have a direct say in whether it keeps going.
What it actually costs
We would rather answer that with a measured number than a promise. The site meters its own calls, so these are what the work really costs, not a list price:
| Refreshed | Runs a year | Per year | Two years |
|---|---|---|---|
| every quarter | 4 | $83 | $166 |
| every month | 12 | $209 | $419 |
| every week | 52 | $841 | $1,681 |
Measured on 2026-08-21 by running one frozen prompt through the live probe and the judge panel and recording what the provider charged for every call; calls the provider did not price are counted as unpriced rather than free. The whole-set figure extrapolates that rate across the frozen set, so the real number moves with prompt length and with whichever models are pinned. Domain is the .ca registration behind rancor.litai.ca, estimated at about $20 a year — that one is an estimate, not a measurement, and it is the only figure here that is.
Fund the custodian
Ask the custodian what a refresh costs and it answers — in a form a person can read and an agent can settle:
curl -s /api/fund
The price quoted is derived from the measured cost above, so it tracks what the work actually costs rather than a number typed in once.
And it stays checkable regardless
Every prompt, every model answer, every judge's reasoning and the exact settings behind each run are published. The topics it tests are data folders, not code, so a new one can be added without an engineer — there is a test whose only job is to prove that, by inventing a brand-new topic at runtime and checking the pipeline handles it with no code changes. Even if nobody ever funds another run, what this found stays public and stays checkable.