Repository retrieval, calibration, adversarial behavior, and scan time
The checked-in suites below contain the cases, baselines, misses, and confidence intervals.
Evidence
Measured honestly.
Misses included.
Every benchmark case is a real issue later closed by a merged pull request. FixMap sees the repository before the fix and tries to surface the file that actually changed.
Claim boundary
Three different questions need three different answers.
Retrieval measurements can show whether FixMap surfaced a known fixing file. They cannot, by themselves, show whether a coding agent used fewer tokens or completed the task better.
Focused context, test routes, impact candidates, and explicit uncertainty
These are product behaviors produced from repository evidence. They are not downstream outcome claims.
Agent tokens, cost, time, tool calls, and task success
The controlled protocol and evaluator are ready for real runs; no result is published until a complete run set exists.
Confidence calibration
Labels are evidence bands, not probabilities.
Across the held-out and external suites, high confidence is not more accurate than medium in this small sample. The counts are published so readers can see that limitation instead of treating a label as a calibrated chance of correctness.
Top labelCasesCorrectAccuracy
high6583%
medium191158%
low3133%
Adversarial gate
9/9 cases pass.
The checked-in adversarial record covers fabricated identifiers, wrong repositories, vague tasks, and generated output. Its measured false-confidence rate is 0%.
Held-out · tasks that did not name the file · 9 repositories
56%fixes surfaced in the top 3
Selected after the ranker was finished, and counting only the cases where FixMap had to locate the file rather than read it out of the task. This is the result to plan around.
- Top 1
- 33%
- Top 5
- 56%
Regression · 16 repositories
15/16fixes surfaced in the top 3
These cases guided development. They prove regressions are caught, not generalization.
- Top 1
- 11/16 · 69%
- Top 5
- 15/16 · 94%
Read this as “about half,” not as two significant figures. With 9 held-out cases in this cohort, one result changing moves Top-3 by roughly 11 points. The 95% interval is 27–81%.
Cohorts
Some tasks already contained their answer.
3 of the 12 held-out tasks name a fixing file outright — one as Location: lib/document.js:2339, two as a GitHub permalink to the exact lines. A ranker that reads explicit file mentions answers those by reading the task. Pooling them lets 3 cases carry the headline, so they are reported separately.
CohortCasesTop 1Top 3
Task did not name the file933%56%
Task named the file3100%100%
Pooled — what we published before1250%67%
This is a structural correction, not a measured effect size. The regression suite barely moves under the same split, and its named cases are 2 of 3 rather than 3 of 3 — being named does not guarantee a hit. With three cases per named cohort, how much a mention is worth is not established. What is established is that a generalization headline should not be computed over tasks that contain their own answer.
Baselines
Better than searching the repository?
A ranked list only earns its place if it beats what an agent already gets for free. Every arm below scores on the same repository scan — same files, same text samples. Each baseline is shown at its strongest candidate policy: pointed at every scanned file a keyword search just returns README.md, which would make this a strawman rather than a comparison.
ArmTop 1Top 3Top 5
Path extraction — read paths out of the task0%0%0%
Literal keyword search, code files only22%44%78%
BM25 retrieval, code files only44%56%100%
FixMap33%56%56%
FixMap does not beat BM25 over code files on repositories it was never tuned against. BM25 leads the Top 1 point estimate 4/9 to 3/9, while Top 3 ties at 5/9; both paired McNemar exact tests have p = 1.0. At Top 5 the baseline wins four cases FixMap misses and FixMap wins none: BM25 has the fixing file in its top five for 9 of 9 of these cases, FixMap for 5 of 9 (p = 0.125). FixMap does lead on the regression suite (69% vs 39% Top 1), but that is the suite whose cases shaped the ranker, and even there the lead is not significant against this baseline. We publish this because it is what the measurement says; closing the Top-5 recall gap is the next piece of work.
Your repository
Run the baseline comparison locally.
fixmap benchmark --repo . --last 50 backtests BM25-over-code, FixMap, and FixMap with Impact Graph on bounded recent Git history.
Parent snapshots
Every case runs before its target commit, so the target change and later co-change history are unavailable.
Identical corpus
All three arms see the same scanned files. The comparison never weakens a baseline with a noisier candidate set.
Separated cohorts
Tasks that name an expected path are reported separately from tasks that require retrieval.
Bounded and non-executing
Temporary worktrees, commit and file caps, and no repository code execution keep the run inspectable.
No universal performance claim is derived from one repository. Commit messages are imperfect task proxies, and historical changes reflect that project’s own maintenance patterns. The JSON output includes every eligible case, skip counts, cohort scores, and Wilson intervals so users can judge the evidence directly.
Controlled agent study
The protocol exists; the result does not yet.
FixMap publishes a frozen four-arm protocol for baseline, available, instructed, and Impact Graph-assisted runs. It requires the same agent model and version, pinned repository revision, complete transcripts, and one run per task and arm.
No fix-rate, turn-count, token, cost, or time-saved claim is made until real runs are completed and audited. The checked-in evaluator validates the protocol and rejects incomplete or mismatched run sets.
Held-out results
Every case, not just the wins.
The expected path is the file changed by the merged fix. “Top 3” means that path appeared in FixMap's first three results.
RepositoryExpected fileTop resultTop 3
react-hook-form/react-hook-form #13608
src/logic/createFormControl.tssrc/logic/createFormControl.tsHitsocketio/socket.io #5462
packages/engine.io-client/lib/socket.tspackages/engine.io-client/lib/transports/polling-xhr.tsHitsveltejs/svelte #18555
packages/svelte/src/internal/client/dom/blocks/boundary.jspackages/svelte/src/internal/client/dom/blocks/boundary.jsHitvitejs/vite #10136
packages/vite/src/node/server/bundledDev.tspackages/create-vite/template-vue-ts/src/App.vueMissMethodology
What the benchmark does—and does not—show.
Real issues
The task text comes from a public issue, not a prompt written to flatter the ranker.
Pre-fix repository
The repository is checked out before the merged change so the answer is not already embedded in the diff.
Mechanical scoring
A hit means a changed file appeared at a measured rank. Nothing is graded by taste.
No time-saved claim
FixMap has not run a controlled study of the same tasks with and without the tool, so this site does not invent one.
Trust the boundary
A starting map is useful even when it is imperfect.
FixMap narrows the search and exposes uncertainty. It does not replace reading, testing, or review.
Try the weakest sample