Two setups. One task. Your engineer decides.
Yardstick fits into the way your engineers already work. Every task they give it gets done twice, they pick the better result blind, and their picks add up to a verdict you can act on.
Give a task.
From the terminal, inside the repo: yardstick run "fix the cart
total bug". Or start it from a GitHub issue. It's the same request the engineer
would give any coding agent.
Two setups run it at once.
A setup is an agent and a model, plus whatever shapes them: settings, instruction files and skills. Claude Code with Sonnet against OpenCode with Kimi K3, say, or the same agent with and without your new instructions file.
Both start from the same code, with the same instructions and the same time limit, inside your own GitHub Actions and on your own API keys.
Pick blind.
Labelled A and B, nothing more
Which setup is A is decided at random for every task. No agent names, no model names, nothing to tell them apart except the work itself.
Everything needed to judge
The code changes from each side, what each agent did step by step, and whether CI passed. Time and cost stay hidden until after the pick, so price can't sway it.
Then keep the one you want
The pick is the verdict. Which change to keep is a separate choice, and it carries on as normal work: into the engineer's working copy, or as a pull request.
Get one clear answer.
Each decision is run as a campaign on real work.
beforeKnown up front
Before you start, Yardstick tells you how many comparisons the decision needs.
afterOne verdict
When the comparisons are done, you get one answer about which setup to use, such as "Kimi K3 is no worse than Sonnet on your work".
alongsideCost and CI
What each setup costs per task, and how often its changes pass CI, shown next to the verdict.
by taskWhere each setup is strong
Results broken down by kind of work, such as fixes, new features and maintenance.
What you can test.
Each new model, on your work
A new release comes out. Run it against what you use today and find out whether it's better for your code, not just on a leaderboard.
The cheaper alternative
Find out whether a lower-cost agent or open model is as good as the expensive one on your work.
Your own customizations
Check whether your instruction files, skills or agent settings actually improve results, or just feel like they do.
Run a pilot.
Pick the decision you're stuck on, whether that's two agents, two models, or the latest release, and let your engineers settle it on real work.