arXiv preprint · 2026
Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG
* Equal contribution.
Summary
RINSE tests whether retrieved passages contain enough evidence to answer a question before any answer is generated. In retrieval-augmented generation (RAG), a language model answers from passages fetched from a corpus, and those passages often name the right entities while omitting the fact the answer needs. RINSE scores such sufficiency from the question and the passages alone, and it ranks sufficient evidence above insufficient evidence with a pairwise accuracy of 0.837 on six datasets, where chance is 0.5.
Key numbers
- 0.837Average pairwise accuracy across six datasets, against 0.746 for the strongest of 10 prior methods and 0.784 for a frontier model queried through an API
- 40.0–99.3%Share of insufficient-evidence questions that 12 generators still answer when told to abstain
- 36.5 msMedian latency per record on one NVIDIA A100 GPU, with no generated answer and no API call
Method
A passage is relevant when it is about the question and sufficient when it contains the facts the answer depends on. Testing sufficiency needs pairs of cases that differ only in whether the answer is supported. The paper builds such pairs by substitution (replace a required passage with a distractor, keeping the passage count), deletion (remove required support), and question swap (pair a passage with a related question it cannot answer). Naive versions of these constructions leak the label. On SQuAD 2.0 [1], counting how many question words appear in the passage separates answerable questions from unrelated ones with 0.993 accuracy, but only 0.573 against the unanswerable questions that annotators wrote about each passage. The benchmark therefore audits the cues it leaves, such as word overlap and passage position.
RINSE combines three signals, one per way evidence falls short. A question-coverage reader checks whether every part of the question appears in the evidence. An answer-span reader checks whether any single passage offers an answer. A small language model of 1.5 billion parameters reads the passages together and judges whether they suffice, which covers support spread across passages. Each signal is corrected for the size of the evidence set and standardized, and the three are summed with equal weights. The score depends only on the question and the evidence, so no generator runs.
Evaluation
Generators rarely notice a gap. Even with an explicit instruction to abstain, 12 generators answer 40.0 to 99.3% of insufficient-evidence questions. Pairwise accuracy (PA) measures how often a scorer ranks the sufficient side of a pair above the insufficient side, with ties counted as one half. Across six datasets built from HotpotQA [2], MuSiQue [3], NeoQA, SQuAD 2.0 [1], IIRC, and fresh English Wikipedia articles, RINSE scores 0.837 PA. The best of 10 prior methods, Self-RAG [4], scores 0.746. Sufficient Context [5], which asks a model directly whether the context suffices, scores 0.540 with a local 1.5B model and 0.784 with a frontier model.
RINSE is also the most consistent. Its weakest dataset scores 0.684, while every other local method falls to 0.644 or lower on at least one dataset. On 3,149 pairs formed from real retriever output, RINSE scores 0.813 against 0.674 for the strongest prior method. Methods that read a generated answer fail on NeoQA, whose events are invented: all four score between 0.503 and 0.520, because the generator stays confident in answers its evidence does not support.
| Method | Average pairwise accuracy | Lowest dataset score |
|---|---|---|
| RINSE | 0.837 | 0.684 |
| Self-RAG | 0.746 | 0.644 |
| Sufficient Context (API) | 0.784 | 0.676 |
Limitations
RINSE does not win everywhere. Self-RAG scores 0.752 on SQuAD 2.0 against 0.684 for RINSE. On NeoQA the answer-span reader scores 0.508, near chance, because the news articles keep plausible answer spans after the needed fact is deleted, and the question-coverage reader alone (0.799) beats the full sum (0.773).
The six datasets are in English, and the fresh Wikipedia set uses questions written by a language model, which can inherit that model's biases. PA measures how well a score orders evidence pairs. A deployment that abstains or retrieves again also needs a decision threshold, and the paper does not set one.
- retrieval-augmented generation
- evidence sufficiency
- answer abstention
- information retrieval
Citation
@article{chen2026relevance,
title={Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG},
author={Suting Chen and Peichun Hua and Yunming Xiao},
journal={CoRR},
volume={abs/2609.37469},
year={2026},
url={https://arxiv.org/abs/2609.37469},
doi={10.48550/arXiv.2609.37469},
eprinttype={arXiv},
eprint={2609.37469},
primaryClass={cs.CL}
}
References
- Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. ACL, 2018. arxiv.org/abs/1806.03822
- Zhilin Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018. arxiv.org/abs/1809.09600
- Harsh Trivedi et al. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the ACL, 2022. arxiv.org/abs/2108.00573
- Akari Asai et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR, 2024. arxiv.org/abs/2310.11511
- Hailey Joren et al. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR, 2025. arxiv.org/abs/2411.06037