Benchmarks

Finding the right file

A reproducible comparison on 300 real issues from SWE-bench Lite. The results are published as measured, including where Travsr does not lead.

In plain terms

Given a GitHub issue, which files does a tool put first? On queries written the way an agent writes them, BM25 leads Travsr 1.3.0 by 12 points of Acc@1 (68.7% vs 56.7%). Travsr leads the graph-based baseline (56.7% vs 41.7%). On raw, full-length issue text the gap is mostly abstention: Travsr declines to answer a large share of those queries rather than return a weak guess, which the score counts as a miss.

Rows marked counterfactual, recomputed from recorded output are not runs of changed code. They re-score the recorded output under a different rule, to show where the headroom is. They are never counted as the best result in a column.

What is measured

The task is file-level localization: rank the repository's files for an issue. The gold set is every file the reference patch touches. Acc@k means every gold file is in the top k. MRR is the mean of 1 / rank of the best gold file, 0 when it is not returned. Abstained means the tool returned no files. No language model is in the retrieval loop.

Dataset

princeton-nlp/SWE-bench_Lite, split test, revision 6ec7bb89b9342f664a54a6e0a6ea6501d3437cc2, all 300 instances (django 114, sympy 77, matplotlib 23, scikit-learn 23, pytest 17, sphinx 16, astropy 6, requests 6, pylint 6, xarray 5, seaborn 4, flask 3).

Agent-written queries (the target use)

1 to 3 short queries per issue written by an LLM from the issue text, each run separately and fused with reciprocal rank fusion (k = 60). All tools get the identical frozen query file.

Agent-written queries (the target use)
Tool Acc@1Acc@3Acc@5Acc@10MRR
BM25 68.7%84.7%89.0%92.7%0.779
Graph-based baseline 41.7%61.0%69.3%81.0%0.541
Travsr 1.3.0, shipped 56.7%77.0%83.7%88.0%0.686
Travsr, gate off counterfactual, recomputed from recorded output 54.7%75.7%82.7%87.3%0.671
Travsr, rows sorted by displayed score counterfactual, recomputed from recorded output 63.3%79.7%85.7%89.0%0.730
Travsr, gate off + score order counterfactual, recomputed from recorded output 59.7%77.7%83.7%88.3%0.703

Full issue text

The whole problem statement, the standard SWE-bench input.

Full issue text
Tool AbstainedAcc@1Acc@5Acc@10MRR
BM25 0.0%36.3%59.7%71.3%0.473
Graph-based baseline 0.0%20.0%49.3%58.3%0.324
Travsr 1.3.0, shipped 54.3%4.3%13.3%23.0%0.095
Travsr, gate off counterfactual, recomputed from recorded output 0.0%7.3%28.3%47.0%0.183
Travsr, gate off + score order counterfactual, recomputed from recorded output 0.0%9.7%32.3%48.7%0.227

Issue title

The first line of the problem statement.

Issue title
Tool AbstainedAcc@1Acc@5Acc@10MRR
BM25 0.0%31.3%62.0%73.7%0.449
Graph-based baseline 0.0%30.0%48.3%54.7%0.385
Travsr 1.3.0, shipped 39.0%12.7%33.7%38.0%0.220
Travsr, gate off counterfactual, recomputed from recorded output 0.0%18.0%45.7%55.7%0.314
Travsr, gate off + score order counterfactual, recomputed from recorded output 0.0%21.3%40.3%52.7%0.320

Per repository

Condition: Agent-written queries (the target use). Each cell is Acc@1 / Acc@5 / MRR.

Small samples: most repositories have 23 or fewer instances, so one instance moves Acc@1 by 4 to 33 points. Read these rows as direction, not as a ranking.

Per repository results, Acc@1 / Acc@5 / MRR
Repository n BM25Graph-based baselineTravsr 1.3.0, shippedTravsr, rows sorted by displayed scorecounterfactual
astropy/astropy 6 83% / 83% / 0.8433% / 67% / 0.5067% / 100% / 0.8183% / 100% / 0.92
django/django 114 67% / 89% / 0.7839% / 63% / 0.5158% / 85% / 0.7068% / 87% / 0.78
matplotlib/matplotlib 23 65% / 87% / 0.7439% / 61% / 0.4843% / 78% / 0.5952% / 74% / 0.62
mwaskom/seaborn 4 50% / 75% / 0.6175% / 100% / 0.8350% / 100% / 0.6925% / 100% / 0.63
pallets/flask 3 67% / 100% / 0.7833% / 67% / 0.47100% / 100% / 1.00100% / 100% / 1.00
psf/requests 6 50% / 100% / 0.6917% / 100% / 0.4333% / 83% / 0.5233% / 100% / 0.53
pydata/xarray 5 80% / 100% / 0.9060% / 80% / 0.6740% / 100% / 0.6740% / 100% / 0.62
pylint-dev/pylint 6 100% / 100% / 1.0033% / 67% / 0.4733% / 83% / 0.5650% / 83% / 0.62
pytest-dev/pytest 17 76% / 100% / 0.8647% / 88% / 0.6659% / 82% / 0.6665% / 88% / 0.73
scikit-learn/scikit-learn 23 70% / 96% / 0.7948% / 74% / 0.6265% / 87% / 0.7574% / 91% / 0.80
sphinx-doc/sphinx 16 88% / 100% / 0.9256% / 81% / 0.6769% / 94% / 0.8050% / 100% / 0.66
sympy/sympy 77 65% / 81% / 0.7340% / 69% / 0.5356% / 77% / 0.6662% / 78% / 0.70

Head to head

On how many instances each tool ranks the gold file higher. Ties include both tools missing it.

Instances where each tool of a pair ranks the gold file higher
Pair First better Second better Tie
Travsr 1.3.0, shipped vs BM25 Agent-written queries (the target use) 53 94 153
Travsr 1.3.0, shipped vs Graph-based baseline Agent-written queries (the target use) 126 69 105
Travsr, rows sorted by displayed score vs Travsr 1.3.0, shipped Agent-written queries (the target use) counterfactual, recomputed from recorded output 72 42 186
Travsr, rows sorted by displayed score vs BM25 Agent-written queries (the target use) counterfactual, recomputed from recorded output 54 76 170

Query leakage check

The query prompt forbids file names and paths and anything path-like is stripped, but bare module names still get through. In 101 of 300 instances (34%) a query contains a gold file module stem as a whole word. Some are real leaks, others are ordinary words that happen to be stems.

Acc@1 with and without a gold module name in a query
Tool Name in a query (101) No name (199)
BM25 78% 64%
Travsr 1.3.0, shipped 60% 55%
Graph-based baseline 44% 41%

Acc@1 with a gold stem in a query (101 instances) vs without (199 instances). Leakage helps BM25 most, but BM25 still leads by 9 points on the instances without it.

Caveats

Environment

Travsr
1.3.0, commit 5def41182c93a0947ec89fcec0d514e6c1dde8f5, binary sha256 bac5cbb61692c7458eb2b81642eae104090803dd3a3a68a5ad833a660e30a4b8
Reranker
ms-marco-MiniLM-L-6-v2, model_fp16.onnx, sha256 a59cbeb92524ab794e3bf3c7a93f5de73bd59b74241b65e81de46dc5bbf4c3ab, floors strong 0.8 / weak 0.5
Node.js
v22.22.3
Query writer
claude-sonnet-5-5 (no tools, no repo access), 887 queries, 2.96 per issue
Frozen agent queries
sha256 d972d2d764df3c365fed31b43b378a75edaa0fc24b158930e47544c4dfb1f3f0
Machine
Apple M3, 8 GB, macOS (arm64)

Source

Travsr tracking issue #998. Values transcribed from the Travsr tracking issue #998, rounded as published. Not produced by this harness. Counterfactual rows are recomputed from recorded output, not runs of changed code.