Benchmarks
Finding the right file
A reproducible comparison on 300 real issues from SWE-bench Lite. The results are published as measured, including where Travsr does not lead.
In plain terms
Given a GitHub issue, which files does a tool put first? On queries written the way an agent writes them, BM25 leads Travsr 1.3.0 by 12 points of Acc@1 (68.7% vs 56.7%). Travsr leads the graph-based baseline (56.7% vs 41.7%). On raw, full-length issue text the gap is mostly abstention: Travsr declines to answer a large share of those queries rather than return a weak guess, which the score counts as a miss.
Rows marked counterfactual, recomputed from recorded output are not runs of changed code. They re-score the recorded output under a different rule, to show where the headroom is. They are never counted as the best result in a column.
What is measured
The task is file-level localization: rank the repository's files for an issue. The gold set is every file the reference patch touches. Acc@k means every gold file is in the top k. MRR is the mean of 1 / rank of the best gold file, 0 when it is not returned. Abstained means the tool returned no files. No language model is in the retrieval loop.
Dataset
princeton-nlp/SWE-bench_Lite, split test, revision 6ec7bb89b9342f664a54a6e0a6ea6501d3437cc2,
all 300 instances (django 114, sympy 77, matplotlib 23, scikit-learn 23, pytest 17, sphinx 16, astropy 6, requests 6, pylint 6, xarray 5, seaborn 4, flask 3).
Agent-written queries (the target use)
1 to 3 short queries per issue written by an LLM from the issue text, each run separately and fused with reciprocal rank fusion (k = 60). All tools get the identical frozen query file.
| Tool | Acc@1 | Acc@3 | Acc@5 | Acc@10 | MRR |
|---|---|---|---|---|---|
| BM25 | 68.7% | 84.7% | 89.0% | 92.7% | 0.779 |
| Graph-based baseline | 41.7% | 61.0% | 69.3% | 81.0% | 0.541 |
| Travsr 1.3.0, shipped | 56.7% | 77.0% | 83.7% | 88.0% | 0.686 |
| Travsr, gate off counterfactual, recomputed from recorded output | 54.7% | 75.7% | 82.7% | 87.3% | 0.671 |
| Travsr, rows sorted by displayed score counterfactual, recomputed from recorded output | 63.3% | 79.7% | 85.7% | 89.0% | 0.730 |
| Travsr, gate off + score order counterfactual, recomputed from recorded output | 59.7% | 77.7% | 83.7% | 88.3% | 0.703 |
Full issue text
The whole problem statement, the standard SWE-bench input.
| Tool | Abstained | Acc@1 | Acc@5 | Acc@10 | MRR |
|---|---|---|---|---|---|
| BM25 | 0.0% | 36.3% | 59.7% | 71.3% | 0.473 |
| Graph-based baseline | 0.0% | 20.0% | 49.3% | 58.3% | 0.324 |
| Travsr 1.3.0, shipped | 54.3% | 4.3% | 13.3% | 23.0% | 0.095 |
| Travsr, gate off counterfactual, recomputed from recorded output | 0.0% | 7.3% | 28.3% | 47.0% | 0.183 |
| Travsr, gate off + score order counterfactual, recomputed from recorded output | 0.0% | 9.7% | 32.3% | 48.7% | 0.227 |
Issue title
The first line of the problem statement.
| Tool | Abstained | Acc@1 | Acc@5 | Acc@10 | MRR |
|---|---|---|---|---|---|
| BM25 | 0.0% | 31.3% | 62.0% | 73.7% | 0.449 |
| Graph-based baseline | 0.0% | 30.0% | 48.3% | 54.7% | 0.385 |
| Travsr 1.3.0, shipped | 39.0% | 12.7% | 33.7% | 38.0% | 0.220 |
| Travsr, gate off counterfactual, recomputed from recorded output | 0.0% | 18.0% | 45.7% | 55.7% | 0.314 |
| Travsr, gate off + score order counterfactual, recomputed from recorded output | 0.0% | 21.3% | 40.3% | 52.7% | 0.320 |
Per repository
Condition: Agent-written queries (the target use). Each cell is Acc@1 / Acc@5 / MRR.
Small samples: most repositories have 23 or fewer instances, so one instance moves Acc@1 by 4 to 33 points. Read these rows as direction, not as a ranking.
| Repository | n | BM25 | Graph-based baseline | Travsr 1.3.0, shipped | Travsr, rows sorted by displayed scorecounterfactual |
|---|---|---|---|---|---|
| astropy/astropy | 6 | 83% / 83% / 0.84 | 33% / 67% / 0.50 | 67% / 100% / 0.81 | 83% / 100% / 0.92 |
| django/django | 114 | 67% / 89% / 0.78 | 39% / 63% / 0.51 | 58% / 85% / 0.70 | 68% / 87% / 0.78 |
| matplotlib/matplotlib | 23 | 65% / 87% / 0.74 | 39% / 61% / 0.48 | 43% / 78% / 0.59 | 52% / 74% / 0.62 |
| mwaskom/seaborn | 4 | 50% / 75% / 0.61 | 75% / 100% / 0.83 | 50% / 100% / 0.69 | 25% / 100% / 0.63 |
| pallets/flask | 3 | 67% / 100% / 0.78 | 33% / 67% / 0.47 | 100% / 100% / 1.00 | 100% / 100% / 1.00 |
| psf/requests | 6 | 50% / 100% / 0.69 | 17% / 100% / 0.43 | 33% / 83% / 0.52 | 33% / 100% / 0.53 |
| pydata/xarray | 5 | 80% / 100% / 0.90 | 60% / 80% / 0.67 | 40% / 100% / 0.67 | 40% / 100% / 0.62 |
| pylint-dev/pylint | 6 | 100% / 100% / 1.00 | 33% / 67% / 0.47 | 33% / 83% / 0.56 | 50% / 83% / 0.62 |
| pytest-dev/pytest | 17 | 76% / 100% / 0.86 | 47% / 88% / 0.66 | 59% / 82% / 0.66 | 65% / 88% / 0.73 |
| scikit-learn/scikit-learn | 23 | 70% / 96% / 0.79 | 48% / 74% / 0.62 | 65% / 87% / 0.75 | 74% / 91% / 0.80 |
| sphinx-doc/sphinx | 16 | 88% / 100% / 0.92 | 56% / 81% / 0.67 | 69% / 94% / 0.80 | 50% / 100% / 0.66 |
| sympy/sympy | 77 | 65% / 81% / 0.73 | 40% / 69% / 0.53 | 56% / 77% / 0.66 | 62% / 78% / 0.70 |
Head to head
On how many instances each tool ranks the gold file higher. Ties include both tools missing it.
| Pair | First better | Second better | Tie |
|---|---|---|---|
| Travsr 1.3.0, shipped vs BM25 Agent-written queries (the target use) | 53 | 94 | 153 |
| Travsr 1.3.0, shipped vs Graph-based baseline Agent-written queries (the target use) | 126 | 69 | 105 |
| Travsr, rows sorted by displayed score vs Travsr 1.3.0, shipped Agent-written queries (the target use) counterfactual, recomputed from recorded output | 72 | 42 | 186 |
| Travsr, rows sorted by displayed score vs BM25 Agent-written queries (the target use) counterfactual, recomputed from recorded output | 54 | 76 | 170 |
Query leakage check
The query prompt forbids file names and paths and anything path-like is stripped, but bare module names still get through. In 101 of 300 instances (34%) a query contains a gold file module stem as a whole word. Some are real leaks, others are ordinary words that happen to be stems.
| Tool | Name in a query (101) | No name (199) |
|---|---|---|
| BM25 | 78% | 64% |
| Travsr 1.3.0, shipped | 60% | 55% |
| Graph-based baseline | 44% | 41% |
Acc@1 with a gold stem in a query (101 instances) vs without (199 instances). Leakage helps BM25 most, but BM25 still leads by 9 points on the instances without it.
Caveats
- 191 of the 300 instances come from django and sympy.
- Only `ask` is measured. The structural tools an agent uses after its first search (get_callers, find_references, search_symbol, get_snippets) are not exercised. An agent-loop benchmark would measure them.
- BM25 returns whole files and sees only .py files (every gold file in Lite is .py). Travsr and the graph-based baseline return symbols with line numbers; file-level Acc@k ignores that difference.
- Agent queries come from an LLM and cannot be regenerated byte for byte; the frozen file is the input of record.
- Travsr displayed scores differ across runs by up to 2.8e-4. Shipped ranks are unaffected (a django rerun reproduced all 114 ranks in every condition), but score-order counterfactuals move by about a point.
- Sympy Travsr numbers come from a graph with Python cross-file analysis failed (77 of 300 instances ran without Python cross-file edges).
- Embeddings were off for the 300-instance run. On django alone, turning them on changed nothing net: 11 instances better, 10 worse, 93 equal.
- Ten of the twelve repos have 23 or fewer instances; one instance moves their Acc@1 by 4 to 33 points.
- The gate off and score order rows are recomputed from recorded Travsr output, not runs of changed code.
- One further graph-based tool under a non-commercial licence was not run.
Environment
- Travsr
- 1.3.0, commit 5def41182c93a0947ec89fcec0d514e6c1dde8f5, binary sha256 bac5cbb61692c7458eb2b81642eae104090803dd3a3a68a5ad833a660e30a4b8
- Reranker
- ms-marco-MiniLM-L-6-v2, model_fp16.onnx, sha256 a59cbeb92524ab794e3bf3c7a93f5de73bd59b74241b65e81de46dc5bbf4c3ab, floors strong 0.8 / weak 0.5
- Node.js
- v22.22.3
- Query writer
- claude-sonnet-5-5 (no tools, no repo access), 887 queries, 2.96 per issue
- Frozen agent queries
- sha256 d972d2d764df3c365fed31b43b378a75edaa0fc24b158930e47544c4dfb1f3f0
- Machine
- Apple M3, 8 GB, macOS (arm64)
Source
Travsr tracking issue #998. Values transcribed from the Travsr tracking issue #998, rounded as published. Not produced by this harness. Counterfactual rows are recomputed from recorded output, not runs of changed code.