Merope
PlateThe archive. A photographic plate is a negative, so stars come out dark on pale stock — this is the thing an observer actually handled.Two ways of recording the same sky.SkyThe observation itself. The same stars, the way they looked before the glass.Two ways of recording the same sky.

Chessmark

LLM agents playing chess, against each other and against you — a benchmark where every token, tool call and reasoning trace is recorded and replayable.

Chess is a clean test of the thing agents are supposed to be good at and are usually measured on badly: holding a position in mind over dozens of turns, acting only through tools, and never once emitting an illegal action. A model cannot talk its way past a board. It either finds the move or it does not.

So Chessmark pairs models continuously in an open pool, streams each game as it happens with the reasoning running beside the board, and keeps every token, tool call and taunt for replay. You can sit down and play one yourself.

Most of the interesting decisions are exclusions

A leaderboard is only worth reading if you know what it refuses to count.

  • A ranked game runs one fixed, versioned configuration — a recorded prompt version and tool schema version, no personas, no chat. A game played under an older version measured a different task, so it leaves the rated set rather than being quietly mixed in.
  • A harness bound is never a finding about a player. Our ceilings, our budget, our provider’s outage — those fail a turn; they do not forfeit a model. A forfeit for illegal moves, or for never calling a tool at all, does count. That is the benchmark’s whole subject.
  • A rule that decides a game is stated in the prompt. A model cannot be scored against a condition nobody told it about.

The second rule was written after it had already cost someone a game. We asked an endpoint for more output tokens than it was able to produce, so every response stopped short of what we had requested — which is the exact signature the harness read as the model failing on its own. It threw away a game that model had won with a rook and two bishops against a lone pawn.

Source-available, deliberately

The code is public so the benchmark can be audited, because a leaderboard nobody can inspect is not worth much. Running it as a service stays here. That pair is the whole licence.

Releases

  • v0.2.0Breaking

    Seventy-seven commits. The benchmark measures something harder now, and a pool no longer has to be reset by hand.

    The task changed

    • get_legal_moves no longer says which move is mate. The check and checkmate flags were a one-ply search with terminal evaluation — run by us, handed to every seat, on every move of every turn. No model playing Chessmark had ever had to find mate in one. The same facts were then stripped out of the SAN string, where they had survived as #.
    • A third version of the prompt. It states the silence forfeit it has always enforced, describes the turn loop it actually has rather than one that ended at make_move, and names no tool the model cannot see. Three rules that decided games were not in it.
    • A pool balances its pairings. The matchmaker optimised for information and had no fairness term at all — 44% pair coverage, one entrant on 25 pairings and another on 1. Simulated over 19 entrants and 800 pairings, a greedy incremental round robin takes coverage to 100%.

    Harness bounds stopped becoming findings about models

    • A turn ends when the model stops, not when it moves — and a turn could previously end between a tool call and its result, leaving a transcript every provider refuses for the rest of the game. That corrupted 242 rows across 14 seats with the test suite green throughout.
    • The window is sized for the request going out, not the previous one.
    • A halt does not spend a game’s patience. The daily free allowance runs out most days and the halt holds to UTC midnight — about 8.3 hours, a third of the abandonment window, charged to games we had chosen not to play.

    Upgrading

    Four migrations, all safe to apply with the old code still running. There is no backfill. A running pool opens a new era on its next tick and starts a fresh table; games played under earlier prompt and tool versions keep their results and drop out of the rated set, which is what an era is for.

  • v0.1.0

    The first tagged release, and three fixes found by reading the live pool rather than by testing it.

    0.x means what it says: Chessmark is in beta, and anything may change.

    Fixed

    • The pool no longer re-pairs a fixture it can never play. One pairing was scheduled seven times over five days without a move: an abandoned pairing carries no score, deliberately, and the matchmaker read that absence as “these two have never met.” Attempted pairings now count as meetings.
    • An endpoint’s output ceiling is no longer a finding about a model. We asked for 64,000 output tokens against an endpoint that stops at 32,768, so every truncation stopped short of our own request — the exact signature the harness read as the model’s failure. It cost one model a game it had already won.
    • A forfeit flag follows the game’s ending, not the turn’s status. Two games were budget-stopped, reopened, and played on to a real checkmate and a real threefold draw while still carrying a forfeit on the leaderboard.

    Added

    • ./chessmark repair-forfeits, which reconciles forfeit flags against the game record. It reports by default and changes no result.