Peichun Hua

arXiv preprint · 2026

Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG

Suting Chen*, Peichun Hua*, Yunming Xiao

* Equal contribution.

Summary

RINSE tests whether retrieved passages contain enough evidence to answer a question before any answer is generated. In retrieval-augmented generation (RAG), a language model answers from passages fetched from a corpus, and those passages often name the right entities while omitting the fact the answer needs. RINSE scores such sufficiency from the question and the passages alone, and it ranks sufficient evidence above insufficient evidence with a pairwise accuracy of 0.837 on six datasets, where chance is 0.5.

Key numbers

  • 0.837Average pairwise accuracy across six datasets, against 0.746 for the strongest of 10 prior methods and 0.784 for a frontier model queried through an API
  • 40.0–99.3%Share of insufficient-evidence questions that 12 generators still answer when told to abstain
  • 36.5 msMedian latency per record on one NVIDIA A100 GPU, with no generated answer and no API call

Method

A passage is relevant when it is about the question and sufficient when it contains the facts the answer depends on. Testing sufficiency needs pairs of cases that differ only in whether the answer is supported. The paper builds such pairs by substitution (replace a required passage with a distractor, keeping the passage count), deletion (remove required support), and question swap (pair a passage with a related question it cannot answer). Naive versions of these constructions leak the label. On SQuAD 2.0 [1], counting how many question words appear in the passage separates answerable questions from unrelated ones with 0.993 accuracy, but only 0.573 against the unanswerable questions that annotators wrote about each passage. The benchmark therefore audits the cues it leaves, such as word overlap and passage position.

RINSE combines three signals, one per way evidence falls short. A question-coverage reader checks whether every part of the question appears in the evidence. An answer-span reader checks whether any single passage offers an answer. A small language model of 1.5 billion parameters reads the passages together and judges whether they suffice, which covers support spread across passages. Each signal is corrected for the size of the evidence set and standardized, and the three are summed with equal weights. The score depends only on the question and the evidence, so no generator runs.

RINSE scores evidence with content coverage, answer-span evidence, and a cross-passage language model, then normalizes and combines the three signals.
RINSE combines three complementary evidence-sufficiency signals before answer generation. Figure 2 in the paper

Evaluation

Generators rarely notice a gap. Even with an explicit instruction to abstain, 12 generators answer 40.0 to 99.3% of insufficient-evidence questions. Pairwise accuracy (PA) measures how often a scorer ranks the sufficient side of a pair above the insufficient side, with ties counted as one half. Across six datasets built from HotpotQA [2], MuSiQue [3], NeoQA, SQuAD 2.0 [1], IIRC, and fresh English Wikipedia articles, RINSE scores 0.837 PA. The best of 10 prior methods, Self-RAG [4], scores 0.746. Sufficient Context [5], which asks a model directly whether the context suffices, scores 0.540 with a local 1.5B model and 0.784 with a frontier model.

RINSE is also the most consistent. Its weakest dataset scores 0.684, while every other local method falls to 0.644 or lower on at least one dataset. On 3,149 pairs formed from real retriever output, RINSE scores 0.813 against 0.674 for the strongest prior method. Methods that read a generated answer fail on NeoQA, whose events are invented: all four score between 0.503 and 0.520, because the generator stays confident in answers its evidence does not support.

Evidence sufficiency on six paired datasets
MethodAverage pairwise accuracyLowest dataset score
RINSE0.8370.684
Self-RAG0.7460.644
Sufficient Context (API)0.7840.676
Selected rows from Table 4. Chance pairwise accuracy is 0.5. The API row uses a frontier model; RINSE and Self-RAG run locally. Table 4 in the paper

Limitations

RINSE does not win everywhere. Self-RAG scores 0.752 on SQuAD 2.0 against 0.684 for RINSE. On NeoQA the answer-span reader scores 0.508, near chance, because the news articles keep plausible answer spans after the needed fact is deleted, and the question-coverage reader alone (0.799) beats the full sum (0.773).

The six datasets are in English, and the fresh Wikipedia set uses questions written by a language model, which can inherit that model's biases. PA measures how well a score orders evidence pairs. A deployment that abstains or retrieves again also needs a decision threshold, and the paper does not set one.

Citation

@article{chen2026relevance,
  title={Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG},
  author={Suting Chen and Peichun Hua and Yunming Xiao},
  journal={CoRR},
  volume={abs/2609.37469},
  year={2026},
  url={https://arxiv.org/abs/2609.37469},
  doi={10.48550/arXiv.2609.37469},
  eprinttype={arXiv},
  eprint={2609.37469},
  primaryClass={cs.CL}
}

References

  1. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. ACL, 2018. arxiv.org/abs/1806.03822
  2. Zhilin Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018. arxiv.org/abs/1809.09600
  3. Harsh Trivedi et al. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the ACL, 2022. arxiv.org/abs/2108.00573
  4. Akari Asai et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR, 2024. arxiv.org/abs/2310.11511
  5. Hailey Joren et al. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR, 2025. arxiv.org/abs/2411.06037