Σχετικά με το έγγραφο αυτό
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Keshav Santhanam
- sl:arxiv_num : 2212.01340
- sl:arxiv_published : 2022-12-02T17:57:06Z
- sl:arxiv_summary : Neural information retrieval (IR) systems have progressed rapidly in recent
years, in large part due to the release of publicly available benchmarking
tasks. Unfortunately, some dimensions of this progress are illusory: the
majority of the popular IR benchmarks today focus exclusively on downstream
task accuracy and thus conceal the costs incurred by systems that trade away
efficiency for quality. Latency, hardware cost, and other efficiency
considerations are paramount to the deployment of IR systems in user-facing
settings. We propose that IR benchmarks structure their evaluation methodology
to include not only metrics of accuracy, but also efficiency considerations
such as a query latency and the corresponding cost budget for a reproducible
hardware setting. For the popular IR benchmarks MS MARCO and XOR-TyDi, we show
how the best choice of IR system varies according to how these efficiency
considerations are chosen and weighed. We hope that future benchmarks will
adopt these guidelines toward more holistic IR evaluation.@en
- sl:arxiv_title : Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking@en
- sl:arxiv_updated : 2022-12-02T17:57:06Z
- sl:bookmarkOf : https://arxiv.org/abs/2212.01340
- sl:creationDate : 2022-12-06
- sl:creationTime : 2022-12-06T19:27:25Z
Προτιμήσεις...