Evidence

Measured honestly.
Misses included.

Every benchmark case is a real issue later closed by a merged pull request. FixMap sees the repository before the fix and tries to surface the file that actually changed.

Open benchmark data

Held-out · 12 repositories

9/12

fixes surfaced in the top 3

Selected after the ranker was finished. This is the result to plan around.

Top 1
7/12 · 58%
Top 5
9/12 · 75%

Regression · 15 repositories

15/15

fixes surfaced in the top 3

These cases guided development. They prove regressions are caught, not generalization.

Top 1
10/15 · 67%
Top 5
15/15 · 100%

Read this as “about three quarters,” not as two significant figures. With twelve held-out cases, one result changing moves Top-3 by roughly eight points. The 95% interval is 47–91%.

Held-out results

Every case, not just the wins.

The expected path is the file changed by the merged fix. “Top 3” means that path appeared in FixMap's first three results.

RepositoryExpected fileTop resultTop 3
Automattic/mongoose #16379lib/document.jslib/document.jsHit
immerjs/immer #1045src/types/types-external.tssrc/types/types-external.tsHit
jestjs/jest #16174packages/jest-mock/src/index.tspackages/jest-config/src/readConfigFileAndSetRootDir.tsHit
knex/knex #5053lib/dialects/postgres/query/pg-querycompiler.jslib/dialects/postgres/query/pg-querycompiler.jsHit
mochajs/mocha #4526lib/reporters/xunit.js.github/CODE_OF_CONDUCT.mdHit
sindresorhus/got #2459source/core/index.tssource/core/index.tsHit
socketio/socket.io #5462packages/engine.io-client/lib/socket.tspackages/engine.io-client/lib/transports/polling-xhr.tsMiss
sveltejs/svelte #18555packages/svelte/src/internal/client/dom/blocks/boundary.jspackages/svelte/src/internal/client/dom/blocks/boundary.jsHit
vitejs/vite #10136packages/vite/src/node/server/bundledDev.tspackages/vite/src/types/commonjs.d.tsMiss
vuejs/core #11564packages/runtime-dom/src/index.tstsconfig.jsonMiss
winstonjs/winston #2610lib/winston/transports/file.jslib/winston/transports/file.jsHit
yargs/yargs #2497lib/utils/apply-extends.tslib/utils/apply-extends.tsHit

Methodology

What the benchmark does—and does not—show.

01

Real issues

The task text comes from a public issue, not a prompt written to flatter the ranker.

02

Pre-fix repository

The repository is checked out before the merged change so the answer is not already embedded in the diff.

03

Mechanical scoring

A hit means a changed file appeared at a measured rank. Nothing is graded by taste.

04

No time-saved claim

FixMap has not run a controlled study of the same tasks with and without the tool, so this site does not invent one.

Trust the boundary

A starting map is useful even when it is imperfect.

FixMap narrows the search and exposes uncertainty. It does not replace reading, testing, or review.

Try the weakest sample