嵌入与Grep:重新思考代码搜索中的密集优势

媒体报道 Hugging Face Papers 原文语言:中文

代码搜索基准显示密集编码器显著领先于 BM25,但编码智能体使用 grep 表现良好。为此构建了现实的仓库级代码搜索基准 Repo-Premium,实验表明密集检索的优势源于有利于密集检索的评估选择以及缺乏代码稀有标识符的查询,并据此得出了三项改进指南。

全文

Code-search benchmarks show dense encoders well ahead of BM25, yet coding agents do well with grep. What does this dense advantage measure? Existing benchmarks cannot tell: they use curated queries, old code and a handicapped BM25. To find out, we build Repo-Premium, a realistic repository-level code-search benchmark. It ranks files in real repositories, with curated and raw queries in Python, JavaScript, TypeScript, Go, Rust and Java, pull requests from 2022 to 2026, and a preregistered test pool over 91 repositories. Our experiments trace the advantage to two sources: evaluation choices that favor dense retrieval, such as splitting files into chunks for BM25 and counting edited tests as relevant, and queries that lack the code's rare identifiers. Expanding the query with LLM-written identifiers (generate-then-grep) helps BM25 far more than dense retrieval, letting it match dense retrieval in every Python pool, even on pull requests opened after the LLM was released. No method we tried predicts which retriever will win on a given query. From these results we derive three guidelines: measure dense gains against a fair lexical baseline, fuse dense retrieval with BM25, and add the code's words to queries that lack them.

View arXiv page View PDF Add to collection

正文以来源语言显示。

参考来源

← 返回每日AI报告