# How do you decide how many retrieval rounds to allow (depth of multi-step) before just answering with what’s gathered? What is the diminishing return, and how to measure it?

Source: https://zilliz.com/ai-faq/how-do-you-decide-how-many-retrieval-rounds-to-allow-depth-of-multistep-before-just-answering-with-whats-gathered-what-is-the-diminishing-return-and-how-to-measure-it
Author: zilliz

To determine the number of retrieval rounds in a multi-step system, you balance answer quality, computational cost, and user experience. A common approach is to set a **fixed maximum depth** (e.g., 3–5 steps) based on empirical testing, domain complexity, and acceptable latency. For example, a medical QA system might allow more steps than a trivia bot due to higher stakes. Beyond this threshold, additional steps often yield minimal improvements while increasing costs and delays. To measure diminishing returns, track metrics like the **percentage of new relevant information** added per step or the **stability of the answer** (e.g., how much the system’s output changes with each retrieval). If a step contributes less than a predefined threshold (e.g., <5% new relevant content), further retrievals are halted.

Diminishing returns can be quantified using **task-specific metrics**. For fact-based queries, measure precision/recall of retrieved documents at each step. For generative tasks, compute the similarity between outputs generated with *N* vs. *N+1* retrieval steps (e.g., using BERTScore or ROUGE). If similarity exceeds a threshold (e.g., 95%), additional steps are unlikely to help. Tools like **confidence scores** from LLMs (e.g., the model’s certainty in its answer) or **query expansion failure rates** (e.g., rephrased queries retrieving redundant content) can also signal when to stop. For instance, if a system retrieves the same documents across multiple rewritten queries, further steps add little value.

Practical implementation involves **adaptive termination**. Start with a baseline (e.g., 3 steps) and use A/B testing to compare answer quality, latency, and user satisfaction. For dynamic adjustment, employ rules like stopping when two consecutive retrievals return overlapping content (measured via Jaccard similarity or embedding-based clustering). In code, this might involve a loop that breaks early if `new_docs_similarity > 0.8` or `confidence_score > 0.9`. Open-source frameworks like LangChain or LlamaIndex provide built-in iterators (e.g., `AutoMergingRetriever`) that automate depth decisions by merging redundant context, offering a practical starting point for developers.