Blog
Announcing Confluent's Kafka Connector for Milvus and Zilliz Cloud: Unlocking the Power of Real-Time AI

Announcing Confluent's Kafka Connector for Milvus and Zilliz Cloud: Unlocking the Power of Real-Time AI

Nov 03, 20234 min read

We are thrilled to announce the availability of the Confluent Kafka Sink Connector with open-source Milvus and Zilliz Cloud (the fully managed version of Milvus). This collaboration enables seamless, real-time vector data streaming from Confluent to Milvus or Zilliz vector databases, unleashing the full potential of unstructured data and significantly enhancing real-time Generative AI powered by large language models (LLMs) like OpenAI’s GPT-4.

The synergy of Zilliz and Confluent: enabling real-time vector streams

In the digital age, unstructured data is being generated at an unprecedented rate, constituting over 80% of the newly created information. Its sheer volume and constant updates present a significant challenge in swiftly digesting and extracting meaningful insights in real time. Vector embedding is a battle-tested approach for distilling insights from this unstructured data chaos. The crux of the issue now lies in finding efficient solutions to store, digest, and make the vast streams of real-time vector data easily searchable.

The synergy between Confluent and the Milvus/Zilliz vector database offers a perfect solution. This collaboration enables the continuous flow of real-time Confluent vector streams converted from unstructured data to be ported to Milvus/Zilliz, empowering developers to build applications for various use cases such as real-time semantic search, image/video/audio similarity search and retrieval augmented generation.

How you can benefit from the Zilliz and Confluent integration

The Confluent and Zilliz integration opens up possibilities for various sectors and applications. Here are some ways you can leverage this integration.

Enhancing Generative AI with a real-time knowledge base

By making incremental vector data searchable instantaneously, the Zilliz and Confluent integration brings up-to-date memory to GenAI applications for fresher and more accurate results. For instance, in the financial industry, investment firms can use Confluent to set up real-time data pipelines for streaming vector embeddings derived from SEC filings and the latest financial news from multiple sources. This data is then ingested into Milvus/Zilliz in real time via the Kafka sink connector for vector similarity searches. Milvus/Zilliz swiftly retrieves Top-K most relevant results, providing Large Language Models (LLMs) with the most current context to generate timely and actionable insights for traders, economists, and other decision-makers.

Optimizing personalized recommendations for e-commerce platforms

With this Kafka sink connector, e-commerce businesses can build recommender systems that adapt quickly to real-time inventory updates and customer behaviors. Vectors, generated from product attributes like images, names, and descriptions, are streamed in real time to Milvus/Zilliz through the Kafka sink connector. Milvus/Zilliz then uses these vector embeddings for immediate similarity searches, allowing the system to adjust recommendations based on up-to-the-minute data.

Getting started: simple steps to real-time innovation

Are you excited to harness the potential of real-time AI with Confluent and Zilliz? Here's how you can get started.

Step 1: Download the Kafka Sink Connector

Access the latest connector zip file from our GitHub repository or Confluent Hub.

Step 2: Configure your accounts

Sign up for free Confluent and Zilliz accounts. Create a topic in Confluent Cloud and a collection with a vector field in Zilliz Cloud. Make sure they have the same field name.

Step 3: Load the connector

Upload the zip file in the Connectors section of your Confluent Cloud cluster. Configure sensitive properties like tokens. Check out the custom-connector page for more details.

Step 4: Configure the connector

Enter your Confluent Cloud API Key and Secret. Provide Milvus connector configuration, including public endpoint, token, vector collection name, and Kafka topic.

Step 5: Launch the connector

Start the connector to stream data seamlessly from Kafka to Milvus. Verify data insertion into Zilliz Cloud, witnessing the magic of real-time data ingestion.

For detailed information and examples, check out our GitHub repository.

Embracing real-time AI with Zilliz and Confluent

Take the leap into the future of AI innovations by exploring our Confluent integration page for detailed resources.

New to Zilliz? Start for free with the new Starter Plan with no installation hassles and without requiring a credit card. You can also start your 30-day free trial of the Standard plan with $100 worth of credits upon registration and the opportunity to earn up to $200 in credits in total. You can contact us through our support portal if you encounter problems or questions when using Zilliz Cloud.

Not a Confluent customer yet? Begin your free trial of Confluent Cloud and receive $400 in credits for the first 30 days – no credit card required.

The era of real-time Generative AI has dawned. Join us in shaping the future of AI innovation. Let’s unlock the true potential of AI together.

Zilliz is the leader in vector database technology, offering an ultra-fast platform that enables a 10x faster vector retrieval. Built on cloud-native principles, Zilliz is designed to serve as the backbone for applications requiring complex similarity search and leveraging GenAI capabilities, such as retrieval-augmented generation. With Zilliz, developers can seamlessly integrate advanced search and retrieval functionalities into their applications, significantly enhancing user experience and backend performance. To explore further, visit www.zilliz.com.

Confluent is the data streaming platform pioneering a fundamentally new category of data infrastructure that sets data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion — designed to be the intelligent connective tissue enabling real-time data from multiple sources to stream across the organization constantly. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven back-end operations. To learn more, please visit www.confluent.io.

Fendy Feng
Fendy Feng is the Technical Marketing Writer at Zilliz. She has extensive experience developing and enhancing the impact of open-source projects in various global markets by producing high-quality, tailored content. Before joining Zilliz, Fendy worked as a Content Strategist at PingCAP, a fast-growing E-Series startup renowned for its open-source distributed SQL database.