Learning-to-Rank Meets Evidence Set Selection: An Empirical Evaluation of RL Estimators for Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation systems typically retrieve a candidate pool of passages and then select a small evidence set for answer generation. Standard retrieval pipelines score passages individually, often ignoring interactions among selected items. Yet evidence quality is fundamentally a set-level property, since the value of one passage depends on which others are included. We study whether stochastic ranking methods from reinforcement learning and Learning-to-Rank can be used effectively in this setting. Focusing on Plackett--Luce ranking policies, we compare naive REINFORCE, PG-Rank, and PL-Rank for evidence selection. We evaluate selectors with LLM-based set utility and held-out downstream question-answering (QA) to test whether estimator advantages from decomposable ranking survive when reward is defined on selected evidence sets rather than rank-weighted relevance. We find that PG-Rank transfers well both to document ranking with set-level reward and to setwise selection over candidate sets, while PL-Rank's structural variance and efficiency advantages weaken when set utility cannot be factored across ranked positions. We also find that learned selectors improve over the BM25 baseline in QA, though the highest-utility methods do not always achieve the strongest QA scores.