Start with the part most companies would bury: the first time Helena took this test, she failed it.
The test is simple to describe and hard to pass. We generated a reference book — about 3.4 megabytes of it — describing Alder County, a place that does not exist, with invented people, mills, charters, harvests, and records, arranged the way real county records are arranged: the fact you need is never announced, it is filed. Because the county is invented, nothing in the book exists in any training data on Earth. No model can remember it. A model can only read it. Then we froze sixty questions — where is this value, join these two records, add these figures, what is not in the book at all — and froze the scorer that grades the answers. The questions and the scorer were fixed before Helena ever saw the book, and they have never been adjusted since.
On October 23, Cameron ran Moonshot's Kimi against it himself, on Kimi's own site, in its Instant mode on the free tier: the book attached whole, the sixty questions asked at once. Kimi answered in 2 minutes 47 seconds and scored 20 of 60. It attached evidence — the sentence its answer came from — to zero of its sixty answers.
Helena's first run, on the morning of October 25, scored 11 of 60. The anatomy of the failure matters more than the number, so here it is in plain words: she invented nothing. On question after question she answered honestly that she did not know — while the document she had been given contained the answer. On fifteen of the twenty-five questions that ask for a single buried fact, the passage containing the answer was in front of her and she declined to give it. A system that will not say what it is holding is safe, and it is also useless. We named the failure precisely, fixed it, and ran the identical frozen test again.
60 of 60. Every category perfect: the buried facts at every depth, including the five deepest (5 of 5), the joins, the sums, the structure questions, and the eight questions whose correct answer is that the book does not say — each of those answered as an absence, backed by a record that the whole document had been checked. Fifty-two answers carry the source sentence they came from. The other eight carry the check itself. Her answering time was 2 minutes 32 seconds — faster than Kimi's 2 minutes 47. Then, because one perfect run could be luck, the finished system ran the test twice more. 60 of 60. 60 of 60. Three consecutive perfect scores on a test that has never been changed.
The scorer deserves one honest paragraph, because it is strict with everyone. It demands exact values with their units, composed answers rather than bare numbers, and absence statements in recognizable form — and it grades our answers by the same rules it grades anyone's. Some of Kimi's answers lost points on form rather than fact: values that were right but arrived without their units, and honest "not recorded" answers whose wording the scorer did not recognize. Helena's first run lost its own honesty questions the same way, to the same strictness. The scorer was not adjusted after any run, for anyone. Its word is the word — including when its word was 11.
Where the test ran matters as much as the score. Helena took it entirely on a local machine: no account, no subscription, no per-question charge — the run cost the electricity a computer uses in a few minutes. The book was never uploaded to anyone's cloud, and everything she did with it stayed on the machine she runs on. She would score the same 60 with the network cable pulled out, because there is nothing on the other end of a cable that she needs.
We are not asking you to believe any of this. The whole kit — the book, the sixty questions, the answer key, and the scorer — is published for anyone to download. Run Kimi against it. Run any model you like against it. Run Helena against it. The numbers above are not a claim we are making about a test we control in private; they are the output of a test you can hold in your own hands, and the first, failing run is in the same record, because that is what a record is.
A two-point-eight-trillion-parameter system read the book and scored 20, with evidence for none of it. Ours read the same book, on an ordinary computer, and scored 60 — three times — with the receipts attached.
Download the test kit: https://github.com/Cameron-Harrison-Interactive/alder-county-showdown-kit
— Harrison Interactive