Dify Knowledge Base Retrieval Settings: How to Tune Them

Who this is for, and what you will be able to do

This guide is for anyone who has built a Dify knowledge base and wants its answers to come from the right passages. After reading it, you will know what each retrieval setting does, how to pick a retrieval depth, and how to measure whether a change helped.

Before you start

• A Dify knowledge base with at least one document that has been indexed. If a document's content is not being found, check its indexing result first^[2].
• A list of real questions your users are likely to ask. Test with complete questions, not bare keywords^[2].
• Access to the knowledge base's Settings > Retrieval page, where all the settings below are kept^[1].

What each setting does

The retrieval settings are in Settings > Retrieval of the knowledge base^[1]. Change only one setting at a time and re-test after each change, so you can tell which change had an effect^[1].

1. System reasoning model. During indexing, this model splits documents into chunks and sections by meaning, writes a summary for each section, and builds a knowledge graph that records the topics each chunk mentions and how those topics relate^[1]. It also plans the retrieval steps when the Deep or Research depth is used^[1]. Changing this model consumes no tokens, and documents already indexed keep what the old model built. To rebuild one document with the new model, click Re-index on that document^[1].
2. Embedding model. Converts each chunk into a vector, so that a question can be matched to chunks by meaning^[1]. Changing it regenerates the vectors for every document and consumes tokens^[1].
3. Rerank model. Orders the retrieved chunks by relevance and gives each chunk a score^[1].
4. Retrieval depth. Sets how thoroughly each retrieval searches. The options are covered in the next section^[1].
5. Top K. The maximum number of chunks one retrieval returns. A larger value gives more context but also more distracting chunks. A smaller value gives more focused results but can miss relevant chunks^[1].
6. Score threshold. Chunks scoring below the threshold are discarded. A high threshold keeps only close matches and may return nothing. A low threshold lets weaker matches through^[1].

Choosing a retrieval depth

There are three depths: Fast, Deep and Research^[1]. All three use the embedding model and the rerank model. Deep and Research also use the reasoning model^[1].

• Fast matches a question to chunks by meaning and by keyword. It is the fastest option and handles most direct questions, so it is the right place to start^[1].
• Deep searches more widely. It also follows the knowledge graph from the question to chunks on related topics, which makes it slower. Switch to it when the answer is spread across several related topics and Fast keeps missing it^[1].
• Research breaks a complex question into sub-questions, searches each one broadly, and uses section titles and summaries to decide which sections to read. It is the slowest option and uses the most tokens. It suits questions that need several rounds of retrieval, not direct ones^[1].

Test with real questions

Open the knowledge base's Retrieval test page and run a question. Run the same question at each depth, and set the depth that returns the right chunks most consistently as the knowledge base's default^[2]. Every run is saved under Records, so results from before and after a settings change can be compared^[2].

Save reference questions and run evaluations

A reference question is one that must return the correct chunks every time, such as the questions users ask most often. Save it together with the chunks it should return. Then choose the pass rule: all of those chunks must come back, or any one of them is enough^[2].

You can also add one directly on the Quality page. Click Add reference question, enter the question, click Find evidence and pick from the closest matches. A CSV file of up to 1 MB can be imported there^[2].

After any change to settings or models, go to Quality > Evaluation, click Run evaluation, choose the retrieval mode your app uses, and click Start evaluation. Every valid reference question is run again with the current settings. The evaluation passes only if each question returns all of its expected chunks^[2].

Evaluation skips three kinds of reference question^[2]:

• Draft: no expected chunks are set yet. Edit the question and select the chunks.
• Stale: an expected chunk no longer exists, for example after its document was re-uploaded. Edit the question and select the chunks again.
• Paused: the document holding an expected chunk is disabled. Re-enable the document and the question rejoins the evaluation.

Fix the question cases

When the right chunk is missing or ranked too low, click Save as question case under the result and choose a reason^[2].

• Retrieval miss: no chunk came back. Check the document's indexing result first^[2]. If indexing is fine, widen retrieval by raising the depth, raising Top K or lowering the score threshold. This can also bring in more unrelated chunks^[2].
• Low score: the chunk came back but ranked too far down. The score comes from the rerank model, so try a different rerank model and see whether the same chunk moves up^[2].

Question cases are kept under Quality > Question cases. After a fix, click Retest to run the question with the current settings. If the result is now correct, click Convert to reference question. Cases you do not plan to fix can be closed with Ignore^[2].

A workflow or agent that searches the knowledge base also creates question cases automatically when it fails to find content that is in the knowledge base. Keep an eye on this list after the app goes live, not only during testing^[2].

Common problems

• A question that used to pass now misses. The most recent change is the likely cause. Revert it, or find the reason and fix it, then run the evaluation again^[2].
• A wider search brings back noise. Raising Top K, lowering the threshold or moving to Research all bring in more chunks, and some of them will be irrelevant. Re-test and compare the results in Records before deciding^[2].

Next steps

The full Dify documentation covers the knowledge base retrieval settings and the testing and evaluation features in more detail^[1][2].

Adapted from Dify's official documentation pages *Retrieval settings* and *Test and tune retrieval*^[1][2], released under the CC BY 4.0 license.