# How Do You Choose a Coding Agent Model? I Compared SelfBench Across 9 Real Repositories > Canonical: https://vxi.me/en/posts/coding-agent-model-selection-selfbench/ > Locale: en-us > Published: 2026-10-07 > Updated: 2026-10-07 > Markdown: https://vxi.me/en/posts/coding-agent-model-selection-selfbench.md Choosing a coding agent can look deceptively simple: check SWE-bench or a community leaderboard, pick the highest-scoring model, then wire Codex, Claude Code, or another harness into your project. I still look at those leaderboards. They are useful for narrowing the field. But once a model has to work inside a repository you actually maintain, the questions get much more specific. Is the codebase mostly TypeScript or Python? Do tests need a browser, Postgres, or Redis? Are your real changes usually UI work, caching fixes, SQL, or build-system changes? What does the difficulty distribution of your merged PRs actually look like? A public benchmark cannot answer all of that for you. I recently found [SelfBench](https://github.com/mupt-ai/self-bench) through a [Show HN post](https://news.ycombinator.com/item?id=49967408). Instead of building one more universal leaderboard, SelfBench turns **merged PRs from your own repository** into coding-agent evaluation tasks, then compares model, harness, reasoning effort, and cost on the same task set. The first thing I wanted to know was simple: does the repository really change the ranking? On October 7, 2026, I pulled every public SelfBench repository release available at the time—nine in total—and ran a separate first-party analysis over the published results. I also audited a sample of published tasks and traced the upstream guardrails in source. The result supports repository-specific evaluation, but it also killed a much stronger claim I could easily have written: **the winning setting does change by repository, yet with only a few dozen tasks per release, a difference of a few percentage points is nowhere near solid enough to call one model “clearly better.”** ## The useful idea in SelfBench is turning your PR history into your own task distribution SelfBench does not start from a hand-authored set of generic coding puzzles. It starts from work that was actually merged into a repository. The pipeline is roughly: \`\`\`text merged PR ↓ restore the pre-change base commit ↓ derive an engineering instruction ↓ prepare held-out tests + reference patch ↓ smoke / nop / oracle / determinism ↓ independent reviewer ↓ export a Harbor task ↓ run different model / harness / reasoning settings ↓ compare accuracy and cost \`\`\` The underlying [Harbor task format](https://www.harborframework.com/docs/tasks) fits this well. A task separates the instruction, container environment, tests, and solution. The agent works in its own environment, while a verifier grades the result with held-out material. SelfBench adds a layer on top: turning historical PRs into benchmark tasks. That difference matters for a team with a meaningful PR history. The model is no longer evaluated only on tasks somebody else designed. It is evaluated on the kinds of changes your team actually ships, with your repository layout, dependencies, test conventions, and accumulated engineering constraints. That is more useful than simply finding a few public benchmark repositories that happen to share your stack. I do not think this makes public benchmarks obsolete. It changes their job. I would use public leaderboards to create a shortlist, then use repository-specific tasks to decide what becomes the default inside my own project. ## Across 9 repositories, no single setting stayed on top At the time of the snapshot, SelfBench exposed nine public repositories. I pulled each release's task count, passes, accuracy, model, harness, reasoning level, and cost per task, then independently checked that the published accuracy was exactly \`passed / tasks\`. The top settings looked like this: | Repository | Tasks | Highest-accuracy setting | | --- | ---: | --- | | vercel/next.js | 38 | GPT-6.1 Sol / Codex / High — 84.2% | | PostHog/posthog | 36 | Claude Opus 5.5 / Claude Code / High — 69.4% | | earendil-works/pi | 50 | GPT-6 Sol / Codex / High — 84.0% | | getsentry/sentry | 81 | Claude Opus 5.5 and Claude Sonnet 5.5 / Claude Code / High — tied at 82.7% | | mupt-ai/self-bench | 38 | GPT-6.1 Sol / Codex / High — 76.3% | | vercel/eve | 38 | Claude Opus 5.5 / High and GPT-6.1 Sol / Low — tied at 73.7% | | vercel/vercel | 47 | Claude Opus 5.5 / Claude Code / High — 80.9% | | supabase/supabase | 36 | four settings tied at 77.8% | | pingdotgg/t3code | 39 | Claude Opus 5.5 / Claude Code / High — 76.9% | Counting ties, those nine repositories produced **six distinct exact top settings across five winning model families**. No complete setting won everywhere. I also compared the relative ranking of settings that appeared in multiple repositories using Spearman rank correlation. Pairwise correlations ranged from roughly 0.462 to 0.995, with a median of 0.807. That is a useful shape of result. It does not say every repository completely reshuffles the leaderboard. Strong models are often still strong. But the ranking is not stable enough to justify the assumption that “the public leaderboard winner will also be the winner in my repo.” That is why I now find “Which coding-agent model is best?” less useful than: > For this repository, this task distribution, this harness, and this cost budget, which setting is the better fit? The word *setting* matters. SelfBench correctly reports model, harness, and reasoning effort together. The same model can move on both accuracy and cost when you change the harness or effort level. Reducing the decision to “Claude vs GPT” throws away part of the experiment. One result was much more stable on the cost side: in all nine releases I inspected, the cheapest model family was **GPT-6 Luna**, although the winning low/high reasoning setting varied. That is still only a snapshot from October 7, 2026. Provider prices, reasoning configurations, and task lengths change. The reusable idea is to compare the accuracy × cost Pareto frontier, not to memorize today's dollar figure. You can inspect the current data directly in the [SelfBench public directory](https://selfbench.dev/api/public/directory). ## The result that changed my wording: 0 of 9 leaders separated cleanly from the runner-up If I stopped at the table above, it would be tempting to say something like “Next.js prefers GPT-6.1 Sol, while PostHog prefers Claude Opus.” The direction is fair. The confidence is not. These public releases contain only 36 to 81 tasks. Using each setting's published \`passed / tasks\`, I ran a simple Wilson 95% interval sensitivity check. I am not assuming benchmark tasks are strictly IID, and I am not pretending this is a formal significance test. I wanted a narrower answer: **when the leaderboard differs by only a few points, how robust does that gap look under a basic small-sample uncertainty check?** The answer was **0 / 9**. In none of the nine repositories did the top setting have an interval that sat entirely above the runner-up's interval. Next.js is an easy example. The top setting passed 32/38 tasks, or 84.2%. The next one passed 30/38, or 78.9%. A 5.3-point gap looks meaningful in a leaderboard, but the descriptive intervals still overlap heavily. Sentry is even tighter: 67/81 is 82.7%, while 66/81 is 81.5%. Writing “Claude Sonnet clearly beat GPT-6.1 Sol” from that result would be asking the leaderboard to support a claim it cannot support. This does **not** mean the models are equivalent. It does not mean SelfBench is useless. I think the opposite is true: repository-specific evaluation makes model selection much more realistic. But at this task count, the leaderboard is better used as **directional evidence**—who is clearly behind, who is on the Pareto frontier, and which settings are worth running again on your own codebase—rather than as a permanent verdict based on a 2% or 5% gap. If I were using repository-specific evals to set a company-wide default, drive model routing, or make a serious purchasing decision, I would want more tasks, repeated trials, and evidence that the ranking stays stable across batches. Running one private leaderboard once and declaring a long-term winner is not much better than doing the same thing with a public leaderboard. ## The biggest risk in an auto-generated benchmark is not agent cheating; it is a biased grader There is another part of SelfBench I was cautious about from the start: the task-authoring agent can inspect the original completed PR. That is convenient. It needs the finished change to derive an instruction, tests, and a reference solution. But it creates an obvious benchmark-design risk. If hidden tests are written too closely around the gold implementation, the task can drift from “satisfy the engineering request” toward “reproduce how the original author happened to implement it.” SelfBench clearly accounts for this risk. I pinned the upstream repository to commit \`67242b85569ea0c27af0c68bd2769ed2d02a05c4\` and checked the implementation rather than relying on README claims. The guardrails I verified include: - a static audit that blocks file overlap between the gold patch and held-out test patch; - verifier logic that restores agent changes to held-out paths and protects those paths before grading; - a nop gate that requires the unmodified base to fail the new tests while existing regression checks still pass; - an oracle gate that requires the reference patch to pass, with the fail-to-pass path repeated to check determinism; - an independent reviewer prompt that explicitly asks whether **a different but correct implementation should also pass the tests**. I also sampled two public tasks from Next.js and two from Sentry. I read the instruction, test patch, and gold patch without executing any third-party repository code. All four passed the structural checks I ran: zero file overlap between test and gold patches, no source PR number / GitHub URL / grader artifact name leaking into the instruction, digest-pinned environment images, held-out path protection, and benchmark canaries. The manual review was also reasonably healthy. For example, a Next.js stale-cache task asks for externally observable stale-while-revalidate behavior, same-key deduplication, and independent root-parameter keys. The held-out tests exercise those behaviors through app fixtures and cache APIs, while the gold patch changes the internal cache wrapper. A Sentry dropped-data UI task deliberately uses fairly flexible semantic regex checks for explanatory copy instead of pinning one exact sentence from the original implementation. That is the kind of grader design I want to see. But four tasks are still four tasks. This sample cannot prove the whole generation pipeline is always fair. The authoring agent has seen the completed implementation. Mechanical rules and a separate reviewer reduce coupling risk; they cannot logically erase that information advantage. Training contamination is another limitation. SelfBench adds benchmark canaries to published tasks and marks them so they should not be casually indexed. That can help with future filtering or contamination detection. It cannot prove that the original public PR, code, or discussion was never present in a model's training data before the benchmark existed. So “this benchmark comes from my repository” is not the same as “this benchmark is contamination-free.” Private repositories reduce that risk, but they introduce their own code, credential, and data-control concerns. ## How I would use this now: public leaderboards for screening, my repository for the default After this analysis, I am actually more willing to keep using public coding-agent benchmarks. I just do not want them to make the final decision. For a small personal project with little PR history, or for occasional agent use, building a repository-specific eval may not be worth the overhead. A public leaderboard plus two or three sensible candidate models, followed by some real usage and cost checks, is often enough. I would take repository-specific evals much more seriously when several conditions start to appear: the team uses multiple models or harnesses, API spend is material, you want one default agent instead of everyone choosing by feel, the repository already has dozens or hundreds of representative merged PRs, the test environment can be reconstructed reliably, or you are building model routing and need to know *which tasks should go to which setting*. I would not start with a huge benchmark. Start with a representative slice of PRs. Review the task instructions and hidden tests manually. Then look at accuracy, cost, failure patterns, and whether the relative ranking stays stable when you add more tasks or repeat the run. SelfBench already packages this workflow well enough that I do not see value in creating a duplicate companion repository for this article. If you want the implementation, start with [mupt-ai/self-bench](https://github.com/mupt-ai/self-bench). If you want to understand why its tasks can separate instruction, environment, solution, and verifier, the [Harbor Task Structure documentation](https://www.harborframework.com/docs/tasks) is the next useful read. If your problem is one layer lower—whether the tests written by an agent are trustworthy in the first place—I previously ran a [first-party Mutation Testing A/B experiment](/en/posts/ai-generated-tests-mutation-testing/). If the immediate issue is simply that a newly released Codex model does not appear in your client, see [synchronizing the Codex Model Catalog and preserving the upstream model name through Sub2API](/en/posts/codex-new-model-catalog-sub2api/). The most useful conclusion I got from SelfBench was not which model happened to rank first that day. It was that model selection can move one step beyond “average ability on somebody else's leaderboard” toward “how does this setting perform on the kind of code change I actually merge?” As long as you keep the second half of the lesson in view—**you have to evaluate the benchmark that evaluates the agent, too**—repository-specific evals are a much better decision tool than another more expensive form of leaderboard worship.