FastGPT and Zilliz Cloud Integration

FastGPT is a knowledge-based question and answer system built on large language models (LLMs), offering ready-to-use capabilities for data processing and model invocation. It enables workflow orchestration through Flow visualization, facilitating complex question and answer scenarios. With its out-of-the-box features, developers can quickly create and deploy sophisticated QA systems across various knowledge domains.

By integrating with Zilliz Cloud (fully managed Milvus), FastGPT gains access to a fully managed, scalable vector database for storing and retrieving knowledge embeddings, enabling faster retrieval of relevant information, improved accuracy in answering queries, and the ability to handle large-scale knowledge bases effectively.

• Ready-to-use QA system with scalable storage: FastGPT provides out-of-the-box data processing and model invocation capabilities, while Zilliz Cloud ensures efficient vector storage and retrieval as your knowledge base grows.
• Visual workflow orchestration: FastGPT's Flow visualization lets you design complex question-answering workflows visually, backed by Zilliz Cloud's high-performance similarity search for the retrieval step.
• Docker-based rapid deployment: The integration supports both Milvus and Zilliz Cloud via Docker Compose, allowing teams to quickly deploy a complete QA system with minimal configuration.
• Improved retrieval accuracy: Zilliz Cloud's vector similarity search ensures that the most relevant knowledge is retrieved to augment LLM responses, reducing hallucinations and improving answer quality.

FastGPT serves as the application layer, providing the LLM-based question-answering platform with visual workflow orchestration, data processing, and model invocation capabilities. It handles user queries, orchestrates the RAG pipeline through its Flow interface, and generates responses using the retrieved context.

Zilliz Cloud serves as the vector database layer, storing and indexing knowledge base embeddings for fast similarity search. When a query comes in, Zilliz Cloud retrieves the most relevant documents to provide context for FastGPT's language model to generate accurate answers.

Together, FastGPT and Zilliz Cloud create a complete knowledge-based QA solution: documents are processed and embedded into Zilliz Cloud, and when users ask questions, FastGPT orchestrates the retrieval of relevant context through Zilliz Cloud's vector search, then generates informed responses using the LLM — all deployable via Docker Compose with minimal setup.

1. Download docker-compose.yml

Ensure that you have already installed Docker Compose. Execute the commands below to download the configuration files.

$ mkdir fastgpt
$ cd fastgpt
$ curl -O https://raw.githubusercontent.com/labring/FastGPT/main/projects/app/data/config.json

# milvus version
$ curl -o docker-compose.yml https://raw.githubusercontent.com/labring/FastGPT/main/files/docker/docker-compose-milvus.yml
# zilliz version
# curl -o docker-compose.yml https://raw.githubusercontent.com/labring/FastGPT/main/files/docker/docker-compose-zilliz.yml

If you're using the Zilliz version, adjust the MILVUS_ADDRESS and MILVUS_TOKEN link parameters in the docker-compose.yml file, which corresponds to the Public Endpoint and API Key in Zilliz Cloud.

2. Launch the Container

Execute in the same directory as docker-compose.yml. Ensure that the docker-compose version is ideally above 2.17, as some automation commands may not function otherwise.

# Launch the container
$ docker compose up -d
# Wait for 10s, OneAPI typically needs to restart a few times to initially connect to Mysql
$ sleep 10
# Restart oneapi
$ docker restart oneapi

3. Access OneAPI to Add Models

OneAPI can be accessed at ip:3001. The default username is root, and the password is 123456. You can alter the password after logging in. Using OpenAI's model as an example, click on the "Channel" tab, and select your chat model and embedding model under "Models". Input your OpenAI API Key in the "Secrets" section.

4. Setting Tokens

Click on the "Tokens" tab. By default, there is a token Initial Root Token. You can also create a new token and set a quota on your own. Click "Copy" on your token, ensuring that the value of this token matches the CHAT_API_KEY value set in the docker-compose.yml file.

5. Accessing FastGPT

FastGPT can be directly accessed at ip:3000 (please mind the firewall). The login username is root, with the password set to DEFAULT_ROOT_PSW within the docker-compose.yml environment variable. Should you require domain name access, you would need to install and configure Nginx on your own.

6. Stop the Container

Run the following command to stop the container.

$ docker compose down

• Deploying FastGPT with Milvus — Official Milvus tutorial for deploying FastGPT
• FastGPT GitHub Repository — FastGPT source code and community resources
• Build AI Apps with Milvus: Tutorials & Notebooks — Zilliz collection of Milvus tutorials and notebooks
• Enhancing ChatGPT with Milvus: Powering AI with Long-Term Memory — Zilliz tutorial on enhancing LLMs with vector storage
• FastGPT Documentation — Official FastGPT documentation