PrivateGPT and Zilliz Cloud Integration

PrivateGPT is a production-ready AI project that enables users to ask questions about their documents using Large Language Models without an internet connection while ensuring 100% privacy. It offers an API divided into high-level and low-level blocks, a Gradio UI client, and useful tools like bulk model download scripts and ingestion scripts. Conceptually, PrivateGPT wraps a RAG pipeline and exposes its primitives, being ready to use and providing a full implementation of the API and RAG pipeline.

By integrating with Zilliz Cloud (fully managed Milvus), PrivateGPT gains access to a fully managed, high-performance vector database designed for billion-scale vector storage and rapid similarity searches, providing organizations a secure and scalable framework for storing and retrieving embeddings within private environments for use cases like private chatbots, document analysis, and recommendation systems.

• 100% data privacy with scalable storage: PrivateGPT ensures complete data privacy by operating without internet connection, while Zilliz Cloud provides enterprise-grade vector storage that can be deployed in private, controlled environments.
• Production-ready RAG pipeline: PrivateGPT provides a complete RAG pipeline implementation out of the box, with Zilliz Cloud handling the high-performance vector storage and retrieval layer for scalable similarity search.
• Flexible model configuration: The integration supports customizable LLM and embedding models through Ollama, while Zilliz Cloud efficiently stores and indexes the generated embeddings regardless of model choice.
• Simple deployment: The setup requires minimal configuration — just install the Milvus module, update the vector store settings, and PrivateGPT is ready to use with Zilliz Cloud as the backend.

PrivateGPT serves as the application framework, providing the RAG pipeline, API, and Gradio UI for document ingestion and question-answering. It handles document processing, embedding generation through Ollama, query orchestration, and LLM-based response generation — all while maintaining complete data privacy.

Zilliz Cloud serves as the vector database backend, storing and indexing the document embeddings generated by PrivateGPT for fast similarity search. It enables efficient retrieval of relevant document context when users ask questions, supporting billion-scale vector storage with low-latency search.

Together, PrivateGPT and Zilliz Cloud create a secure, production-ready document QA solution: documents are ingested and embedded through PrivateGPT's pipeline and stored in Zilliz Cloud. When users ask questions, PrivateGPT retrieves relevant context from Zilliz Cloud's vector store and generates private, contextually informed responses using the local LLM — all without any data leaving the organization's environment.

1. Clone the PrivateGPT Repository

Clone the repository and navigate to it:

$ git clone https://github.com/zylon-ai/private-gpt
$ cd private-gpt

2. Install Poetry

Install Poetry for dependency management. Follow the instructions on the official Poetry website to install it.

3. Install Available Modules

Run the following command to use Poetry to install the required module dependencies:

$ poetry install --extras "llms-ollama embeddings-ollama vector-stores-milvus ui"

4. Start Ollama Service

Go to ollama.ai and follow the instructions to install Ollama on your machine. After the installation, make sure the Ollama desktop app is closed.

Start Ollama service (it will start a local inference server, serving both the LLM and the Embeddings):

$ ollama serve

Install the models to be used. The default settings-ollama.yaml is configured to use llama3.1 8b LLM (~4GB) and nomic-embed-text Embeddings (~275MB):

$ ollama pull llama3.1
$ ollama pull nomic-embed-text

5. Change Milvus Settings

In the file settings-ollama.yaml, set the vectorstore to Milvus:

vectorstore:
  database: milvus

You can also add custom Milvus configuration to specify your settings:

milvus:
  uri: http://localhost:19530
  collection_name: my_collection

The available configuration options are: uri — default is set to a local file; you can also set up a more performant Milvus server on Docker or Kubernetes, or use Zilliz Cloud by adjusting the uri and token to the Public Endpoint and API Key in Zilliz Cloud. token — pair with Milvus server on Docker/Kubernetes or Zilliz Cloud API key. collection_name — the name of the collection, default is "milvus_db". overwrite — overwrite the data in collection if it existed, default is True.

6. Start PrivateGPT

Once all settings are done, you can run PrivateGPT with a Gradio UI:

PGPT_PROFILES=ollama make run

The UI will be available at http://0.0.0.0:8001. You can play around with the UI and ask questions about your documents.

• Use Milvus in PrivateGPT — Official Milvus tutorial for using Milvus in PrivateGPT
• Securing AI: Advanced Privacy Strategies with PrivateGPT and Milvus — Zilliz blog on privacy strategies with PrivateGPT
• What are Private LLMs? Running Large Language Models Privately — Zilliz tutorial on private LLMs and PrivateGPT
• PrivateGPT GitHub Repository — PrivateGPT source code and community resources
• PrivateGPT Documentation — Official PrivateGPT documentation