Creating a knowledge base in Dify

Who this is for and what you will be able to do

This guide is for people already using Dify who want an AI app to answer questions from their own documents. After reading it, you should be able to create an Agent knowledge base in Dify, add content through a data source or uploaded files, set a sync strategy, and check the indexed documents and test retrieval before connecting the knowledge base to an app.

What a knowledge base is and why to build one first

A knowledge base is where you collect your documents so they can be searched. When an app needs to answer from your material, it retrieves the relevant passages from the knowledge base. This pattern is usually called retrieval-augmented generation, or RAG^[1]. This page covers Agent knowledge bases in Dify^[1].

Step 1: create the knowledge base

Go to the Knowledge page, switch to Agent, and click Create^[1].

Enter a name and a description. Set which members can view the knowledge base and which can connect it to their own apps. You can change these settings later in the knowledge base settings^[1].

The knowledge base uses the workspace's default model. If you want images inside your documents to be retrievable as well, switch to an embedding model that supports vision. The page links to a separate explanation of multimodal retrieval^[1].

Step 2: add content

Connect a data source when the content lives elsewhere and keeps changing. Upload files when you have standalone documents. One knowledge base can hold both kinds of content^[1].

The recommendation is to import a small set of representative documents first, tune retrieval until it works well, and then import the rest. Changing the embedding model later re-indexes every document. Doing that while the knowledge base is still small uses fewer tokens^[1].

Connecting a data source

Data source integrations such as Firecrawl, Notion, and Google Drive can import content from websites, document platforms, and cloud drives^[1]. The steps are as follows^[1]:

1. Choose a provider. Pick one of the data source integrations already installed in your workspace, or click More providers to browse and install a new one.
2. Enter the API key or authorized account under Integrations, then Data sources, to finish the connection.
3. Choose what to sync. For website crawling, name the data source, paste the URL, review the default settings under Crawl options, click Crawl and preview, and select the pages you want. For online documents and online cloud drives, pick the pages or files from the connected account, drive, or bucket.
4. Set the sync strategy. Choose a preset sync period, set a custom interval between 1 hour and 30 days, or switch to manual sync.

Pick the sync frequency to match how often the content changes. When a sync finds changes, the changed content is re-indexed, and re-indexing also uses tokens^[1].

If none of the existing integrations meets your needs, you can develop a data source plugin yourself, or request one from the community^[1].

Uploading files

A single file can be up to 15 MB^[1]. Batch upload is available on the Professional and Team plans^[1].

Step 3: check the index before connecting an app

When a document shows Ready on the Documents page, the work is not finished. The page says three more steps are needed before a knowledge base is suitable for real users^[1]:

• Review the indexed documents. Open each one and confirm that chapters, summaries, and segments were handled properly.
• Test retrieval and tune the settings. Ask the questions your app will actually receive, and adjust the settings until the results are stable and reliable.
• Connect the knowledge base to apps, Agents, and code, so that the app can answer from it^[1].

Limits and common questions

• Files are limited to 15 MB each. The page does not suggest a way to split larger files^[1].
• Batch upload is limited to the Professional and Team plans. The page does not list prices, so check the pricing page it links to if you need that information^[1].
• Both sync and changing the embedding model use tokens. Estimate how often your content changes and how large the knowledge base will become before you set a sync period or pick an embedding model^[1].

Next steps

To learn how to review indexed documents, test and tune retrieval, and connect the knowledge base to apps, Agents, and code, follow the three links on the page^[1].

Adapted from Dify's documentation page on creating an Agent knowledge base^[1], released under the CC BY 4.0 license.