Introducing Zilliz Cloud Pipelines: A One-Stop Service for Building AI-Powered Search
In recent years, the rapid advancement of AI and ML technologies has ushered in a new era of innovation, transforming how we interact with computers. Retrieval Augmented Generation (RAG) has emerged as a groundbreaking concept, enabling humans to access and generate information seamlessly. However, building an infrastructure for retrieval, the cornerstone of this technology, has posed many challenges.
Today, we’re excited to introduce Zilliz Cloud Pipelines, available in public preview, a game-changing solution that simplifies embedding creation and retrieval of unstructured data as vectors. Developed by ML experts with deep know-how in retrieval systems, it empowers developers to effortlessly convert unstructured data into vector embeddings to build high-quality semantic searches. Its scalability and quality optimization make it a reliable tool for handling large datasets and high-throughput queries without requiring extensive customization or infrastructure adjustments.
How Zilliz Cloud Pipelines helps
Milvus has been a game changer in the world of vector databases, offering several key advantages in performance and scalability through its innovation. To make this powerful technology even more accessible, we've introduced Zilliz Cloud, a fully managed platform for Milvus. This platform eliminates the operational challenges of scaling the system and ensures high availability.
However, we've consistently heard from our customers about challenges related to vector embeddings. Many have needed help choosing ML models to generate embeddings and vector indexing and search, needing more know-how to build an effective information retrieval system with vector search. These challenges involve complex data processing to transform unstructured data into a format suitable for semantic search. Technical expertise in AI and ML is often needed to develop accurate vector embeddings and search algorithms, which may be unfamiliar to app developers. Integrating semantic search into existing app infrastructures and ensuring scalability for growing data volumes and user queries while maintaining performance and relevance also pose challenges.
To address these issues, we've dedicated significant time and resources over the past few months to helping our customers overcome these hurdles by introducing a robust product suite that can transform unstructured data into a searchable vector collection. For the initial phase, it can transform text documents into vectors and retrieve the document chunks with a fully managed vector database service on Zilliz Cloud. Zilliz Cloud Pipelines enables easy and efficient transformations of unstructured data into high-quality vector embeddings, significantly lowering the barrier to developing AI-powered capabilities like RAG.
Zilliz Cloud Pipelines will enable you to:
Simplify the workflow for developers, from converting unstructured data into searchable vectors to retrieving them from vector databases
Empower developers to create high-quality vector embeddings effectively, regardless of their expertise
Ensure scalability for managing large datasets and high-throughput queries while maintaining high performance without extensive customization or infrastructure changes
How Zilliz Cloud Pipelines Works
Zilliz Cloud Pipelines consists of three specific pipelines: Ingestion, Search, and Deletion.
- Ingestion Pipeline is a tool designed to process unstructured data into searchable vector embeddings, then stored in a Zilliz Cloud Vector Database. It comprises various functions for transforming input data, such as creating vector embeddings from document chunks or preserving user-defined input values (metadata) as retrievable information during vector searches. Each Ingestion pipeline is linked to a single Vector Database collection, with its schema derived from the pipeline's functions. It supports two main function types: INDEX_DOC for splitting text documents into chunks and generating embeddings, and PRESERVE for storing additional metadata fields in the collection.
- Search pipeline enables semantic search by converting query text into vector embeddings with the SEARCH_DOC_CHUNK function and then retrieving the top-K most relevant document chunks, complete with their text and metadata.
- Deletion pipeline is designed to efficiently remove all chunks of a specified document from a collection, utilizing the PURGE_DOC_INDEX function to delete the entire document based on its doc_name.
In the demo below, we showcase the powerful capabilities of Zilliz Cloud Pipelines, highlighting its seamless process of converting, retrieving, and deleting vector chunks for text documents.
Zilliz Cloud Pipelines unlock a world of possibilities for developers, offering a versatile platform to address many use cases. Two standout Zilliz Cloud Pipelines applications are creating Knowledge Bases for RAG applications and revolutionizing real-time recommender systems.
Zilliz Cloud Pipelines improve the accuracy of Large Language Models (LLMs) for RAG applications by supplementing them with domain-specific or private knowledge. This addresses concerns about LLMs' reliance on potentially outdated data. By converting external knowledge and user queries into vector embeddings, the pipelines enable developers to fine-tune LLM responses, boosting accuracy and relevance in applications like chatbots and content generation systems.
Zilliz Cloud Pipelines can also transform text-based recommender systems by converting diverse textual sources into vector embeddings. This process, involving data transformation and filtering, tailors content to user preferences in real-time, enhancing personalization in applications like news recommenders or e-commerce platforms. The Pipelines' ability to deliver relevant and engaging content elevates user experiences and engagement.
These examples highlight how Zilliz Cloud Pipelines efficiently transform unstructured text into valuable insights and recommendations, enhancing user experiences across various applications with ease and simplicity.
Stay tuned for more pipelines
In the public preview phase, Zilliz Cloud Pipelines focuses on semantic search in text documents, crucial for RAG applications. Currently, it's available for free on the Serverless Tier. Future updates will introduce it to Standard and Enterprise tiers, improve text search capabilities, and add new features for image search, video copy detection, and multi-modal search, enhancing the flexibility and adaptability of Pipelines.
Ready to get going? If you still need to do so, sign up for Zilliz Cloud and start building with Pipelines for free within minutes. As you explore and use Pipelines, we would love to hear from you about your specific needs to help us improve.
Steffi LiSteffi is the Director of Product Marketing at Zilliz, with prior experience leading the GTM strategies for open-source data technologies such as Apache Kafka and Apache Airflow. Passionate about travel, yoga, and non-fiction reading, she brings a balanced and enriched worldview to her professional pursuits.
- How Zilliz Cloud Pipelines helps
- How Zilliz Cloud Pipelines Works
- Stay tuned for more pipelines
Content
Start Free, Scale Easily
Try the fully-managed vector database built for your GenAI applications.
Try Zilliz Cloud for Free