https://danielhua.com/tags/benchmarks/Recent content in Benchmarks on danielhua.comHugo -- gohugo.ioen© 2026 Daniel HuaMon, 11 May 2026 00:00:00 +0000https://danielhua.com/posts/2026-05-11-chessbench-findings/Mon, 11 May 2026 00:00:00 +0000https://danielhua.com/posts/2026-05-11-chessbench-findings/<h2 class="relative group">Overview <div id="overview" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#overview" aria-label="Anchor">#</a> </span> </h2> <p>I wrote a simple <a href="https://github.com/danielhuadotcom/chessbench" target="_blank" rel="noreferrer">harness</a> to run OpenRouter LLMs in known-winning endgame positions.</p> <p>There&rsquo;s plenty of prior-art<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>^,<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> on LLMs playing chess, dating even back to a cool fine-tuned GPT-2 project<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>, but overall: they <em>still</em> can&rsquo;t really do this. Surprising finding!</p> <h2 class="relative group">Results <div id="results" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#results" aria-label="Anchor">#</a> </span> </h2> <ul> <li><a href="https://danielhua.com/chessbench/run_logs/report_results.html" >Aggregated Result Table</a></li> </ul> <p>I burned on the order of $200 to obtain these runs.</p>