Purpose-built vector databases outperform dual-system setups by unifying Sparse-BM25 and semantic search in a single, efficient implementation.

large_jiang chen.jpeg

medium_jiang chen.jpeg

small_jiang chen.jpeg

thumbnail_jiang chen.jpeg

jiang chen.jpeg

Jiang Chen



_This post was originally published on [The New Stack](https://thenewstack.io/elasticsearch-was-great-but-vector-databases-are-the-future/) and is reposted here with permission._

For decades, keyword matching also known as full-text search, exemplified by Elasticsearch, has been the default choice for [information retrieval systems](https://zilliz.com/learn/what-is-information-retrieval) like enterprise search and recommendation engines. 

As AI-powered search technologies advance, there is a shift toward [semantic search](https://zilliz.com/glossary/semantic-search), enabling systems to understand both the meaning and intent behind user queries. Embedding models and [vector databases](https://zilliz.com/learn/what-is-vector-database) have become central to this shift. 

Semantic search surpasses keyword matching by representing data as vector embeddings, providing a more nuanced understanding of search intent and transforming applications ranging from [retrieval-augmented generation](https://zilliz.com/learn/Retrieval-Augmented-Generation) (RAG) to [multimodal search](https://milvus.io/docs/multimodal_rag_with_milvus.md#Multimodal-Search-with-Generative-Reranker).

In practice, effective information retrieval systems need both semantic understanding and exact keyword matching. For example, users expect search results to show concepts related to their search queries while also respecting the literal text used in the query, such as special terms and names and return the exact matching results. 

A semantic search powered by dense vectors helps to understand the meaning (like knowing that ‘car’ and “automobile” are the same) and [traditional full-text search](https://zilliz.com/learn/evolution-of-search-from-traditional-keyword-matching-to-vector-search-and-genai) provides the precise results users expect (like finding exact matches for “Python 3.9”). As a result, many organizations are adopting a hybrid search approach, combining the strengths of both methods to balance flexible semantic relevance with predictable exact keyword matching.


## The Hybrid Search Challenge

A common way to implement hybrid search is to use a purpose-built vector database like open source [Milvus](https://zilliz.com/what-is-milvus) for efficient and scalable semantic search, alongside traditional search engines like Elasticsearch or OpenSearch for full-text search. 

While this approach can yield good results, it also introduces a new layer of complexity. Managing two distinct search systems means dealing with separate infrastructures, configurations and maintenance tasks, creating a heavier operational burden and increasing the chance of potential integration issues.

![Elasticsearch vs Milvus on Hybrid search](https://assets.zilliz.com/Elasticsearch_vs_Milvus_on_Hybrid_search_7981aca132.jpg)

Figure: Elasticsearch vs Milvus on Hybrid search

A unified solution for hybrid search would provide many benefits:

- **Reduced infrastructure maintenance:** Managing one system rather than two, drastically reduces operational complexity, saving both time and resources. This also means less context switching and mental overhead to master two different sets of APIs.

- **Consolidated data management:** A unified table structure allows you to store both [dense](https://zilliz.com/learn/sparse-and-dense-embeddings) (vector-based) and [sparse](https://zilliz.com/learn/sparse-and-dense-embeddings#Sparse-versus-dense-embeddings-a-summary) (keyword-based) data alongside shared metadata labels. Using two separate systems would require storing metadata labels twice for both sides to be able to conduct metadata filtering.

- **Streamlined query**: A single request can carry out both semantic and full-text search tasks, eliminating the need to conduct two API calls to separate systems.

- **Enhanced security and access control**: A unified approach enables more straightforward and robust security management, as all access controls can be centrally administered within a vector database, enhancing security compliance and consistency.


## How a Unified Vector Approach Simplifies Hybrid Search

In semantic search, [machine learning models](https://zilliz.com/ai-models) “embed” text as points, known as [dense vectors](https://zilliz.com/learn/dense-vector-in-ai-maximize-data-potential-in-machine-learning), in a high-dimensional space based on its meaning. Texts with similar semantics are closer to each other in this space. For example, “apple” and “fruit” might be closer in this space than “apple” and “car.” This allows us to quickly find semantically related text by just calculating the distance between each point using [approximate nearest neighbor (ANN)](https://zilliz.com/glossary/anns) algorithms. 

This method can also be applied to full-text search by encoding documents and queries as sparse vectors. In sparse vectors, each dimension represents a term and the value indicates how important each term is in the document. 

Terms not present in the document have a value of zero. Since any given document typically uses only a small portion of all possible terms in the vocabulary, most terms will not appear in the document. This means the resulting vectors are sparse — most of their values are zero. For example, in the MS-MARCO data set commonly used for evaluating information retrieval tasks, while there are about 9 million documents and a million unique terms, a search system typically divides this large collection into smaller segments for easier management. 

Even at the segment level, with hundreds of thousands of terms in its vocabulary, each document usually contains fewer than 100 terms, which means over 99% of each vector's values are zero. This extreme sparsity has important implications for the way we store and process these vectors efficiently.

This sparsity pattern can be exploited to optimize search performance while maintaining accuracy. Vector databases originally designed for dense vectors can be adapted to handle these sparse vectors efficiently. For example, the open source vector database Milvus has just released native full-text search support using Sparse-BM25, a sparse vector implementation of the [BM25 algorithm](https://zilliz.com/learn/mastering-bm25-a-deep-dive-into-the-algorithm-and-application-in-milvus) used by Elasticsearch and other full-text search systems. Sparse-BM25 unlocks approximation-based optimization for full-text search with:

- **Efficient retrieval algorithm with data pruning:** By applying heuristic-based pruning to discard documents with the lowest sparse vector values in the segment index and ignoring low-value sparse vectors in the search query, a vector database can significantly reduce index size and optimize performance with minimal quality loss.

- **Unlocking further performance optimizations:** Representing term frequency as a sparse vector instead of a reversed index enables additional vector-based optimizations. These include:

  - Graph indexing is used for more efficient search than brute-force scans.

  - [Product quantization (PQ) / scalar quantization (SQ)](https://zilliz.com/learn/scalar-quantization-and-product-quantization) to further reduce memory footprint.

In addition to these optimizations, the Sparse-BM25 implementation also inherits several system-level advantages from the high-performance vector database Milvus:

- **Efficient low-level implementation and memory management:** The core vector indexing engine in Milvus is implemented in C++, offering more efficient memory management than a Java-based system like Elasticsearch. This alone reduces memory footprint by saving gigabytes compared to JVM-based approach.

- **Support for MMap:** Similar to Elasticsearch’s use of page-cache for index storage in both memory and disk, Milvus supports [memory mapping (MMap)](https://zilliz.com/blog/milvus-introduced-mmap-for-redefined-data-management-increased-storage-capability) to extend memory capacity when the index exceeds available memory.


## Why Traditional Search Stacks Fall Short on Vector Search

Elasticsearch was built for traditional inverted indexes, making optimizing the whole architecture for dense vector search fundamentally difficult. The impact is clear: Even with just 1 million vectors, Elasticsearch takes 200 milliseconds (as tested on fully-managed Elastic Cloud) to return the search result compared to 6ms on Milvus (as tested on fully-managed Zilliz Cloud) — that’s over 30x performance difference. The throughput measured by queries per second (QPS) also has a 3x difference, where the most performant instance on Zilliz Cloud runs at 6000 QPS while Elastic Cloud is 1900 QPS at most. Moreover, Zilliz Cloud is 15x faster in loading the vector data and building index than Elastic Cloud. This performance gap widens at scale, where Elasticsearch’s Java/JVM implementation struggles to match the scalability of C++/Go-based vector databases. Additionally, Elasticsearch lacks critical vector search features like disk-based indexes (DiskAnn, MMap), optimized metadata filtering, and range search.



![VectorDBBench Benchmarking Results.jpg](https://assets.zilliz.com/Vector_DB_Bench_Benchmarking_Results_d1d69875a3.jpg)


Figure: VectorDBBench Benchmarking Results (_[_source_](https://github.com/zilliztech/VectorDBBench#what-is-vectordbbench)_)


## Conclusion

Vector databases, exemplified by Milvus, are poised to surpass Elasticsearch as the unified solution for hybrid search. By integrating dense vector search with optimized sparse vector techniques, vector databases offer superior performance, scalability and efficiency. This unified approach simplifies infrastructure, reduces memory footprint and enhances search capabilities, making it the future of advanced search needs. As a result, vector databases provide a comprehensive solution that seamlessly combines semantic and full-text search, outperforming traditional search systems like Elasticsearch.


large_Elasticsearch Was Great, But Vector Databases Are the Future.png

medium_Elasticsearch Was Great, But Vector Databases Are the Future.png

small_Elasticsearch Was Great, But Vector Databases Are the Future.png

thumbnail_Elasticsearch Was Great, But Vector Databases Are the Future.png

Elasticsearch Was Great, But Vector Databases Are the Future.png

Elasticsearch Was Great, But Vector Databases Are the Future

RouteLLM is an open-source framework that enables developers to efficiently route tasks to the most suitable LLMs based on cost, latency, and accuracy.

large_Male-Author.jpeg

medium_Male-Author.jpeg

small_Male-Author.jpeg

thumbnail_Male-Author.jpeg

Male-Author.jpeg

Ruben Winastwan, Freelance Technical Writer

Ruben Winastwan

large_RouteLLM_ Learning to Route LLMs with Preference Data.png

medium_RouteLLM_ Learning to Route LLMs with Preference Data.png

small_RouteLLM_ Learning to Route LLMs with Preference Data.png

thumbnail_RouteLLM_ Learning to Route LLMs with Preference Data.png

RouteLLM_ Learning to Route LLMs with Preference Data.png

RouteLLM: Balancing Cost and Quality in LLM Deployments

RouteLLM: An Open-Source Framework for Navigating Cost-Quality Trade-Offs in LLM Deployment

Compare top vector database vendors, run benchmarks, and choose the right solution for your AI-driven applications. Download the guide now!


thumbnail_Emily-sq.png

Emily-sq.png

Emily Kurze

By 2025, an incredible 90% of new data will be [unstructured](https://zilliz.com/learn/introduction-to-unstructured-data)—text, images, audio, and videos that don’t neatly fit into rows, columns, or predefined models. This shift represents one of the biggest challenges in modern data management but also creates exciting opportunities to innovate in AI, recommendation systems, and [semantic search](https://zilliz.com/glossary/semantic-search).

Enter [vector databases](https://zilliz.com/learn/what-is-vector-database): cutting-edge systems built specifically to store and query high-dimensional [vector embeddings](https://zilliz.com/glossary/vector-embeddings). These embeddings transform unstructured data into actionable insights by capturing its meaning in numerical form to enable complex calculations and semantic comparisons that enable product recommenders, reverse image search, anomaly detection, and more. 

However, as the vector database landscape rapidly evolves, organizations face a crowded market filled with conflicting claims and confusing options. How can you choose the right solution for your use case?

Our comprehensive new resource, [_The Definitive Guide to Choosing a Vector Database_](https://zilliz.com/resources/guide/definitive-guide-choosing-vector-database), answers this question. Inside, you’ll discover:

- **Why purpose-built vector databases outperform traditional systems** like PostgreSQL and Elasticsearch for handling large-scale unstructured data.

- **How Approximate Nearest Neighbor (**[**ANN**](https://zilliz.com/glossary/anns)**) algorithms enable fast, scalable searches**, even with billions of high-dimensional vectors.

- **The features that matter most for AI-driven applications**, from hybrid search capabilities and GPU optimization to schema flexibility and multi-tenancy.

- **Comparisons of top players in the vector database market**, including [Milvus](https://zilliz.com/what-is-milvus), [Zilliz](https://zilliz.com/cloud), Pinecone, and Weaviate, to help you understand their strengths and trade-offs.

- **How to run benchmarks on your own data across vendors** using open-source tools to evaluate performance, scalability, and cost-effectiveness for your specific workloads.

Whether you’re building Retrieval Augmented Generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)) pipelines to reduce AI hallucinations, implementing recommendation systems, or enhancing semantic search, this guide equips you with the knowledge to make an informed choice.

Don’t let the complexity of the vector database market slow you down. Download [_The Definitive Guide to Choosing a Vector Database_](https://zilliz.com/resources/guide/definitive-guide-choosing-vector-database) today and take the first step toward making the most of your unstructured data.

[![Definitive Guide to Choosing a Vector Database ](https://assets.zilliz.com/Definitive_Guide_to_Choosing_a_Vector_Database_22ea755bb2.png)
](https://zilliz.com/resources/guide/definitive-guide-choosing-vector-database)

Making Sense of the Vector Database Landscape

large_Making Sense of the Vector Database Landscape-1.png

medium_Making Sense of the Vector Database Landscape-1.png

small_Making Sense of the Vector Database Landscape-1.png

thumbnail_Making Sense of the Vector Database Landscape-1.png

Making Sense of the Vector Database Landscape-1.png

Download | Definitive Guide to Choosing a Vector Database 

LaVA is a multimodal model that combines text-based LLMs with visual processing capabilities through visual instruction tuning. 


large_Visual Instruction Tuning-20241122-101617.png

medium_Visual Instruction Tuning-20241122-101617.png

small_Visual Instruction Tuning-20241122-101617.png

thumbnail_Visual Instruction Tuning-20241122-101617.png

Visual Instruction Tuning-20241122-101617.png

LLaVa and Visual Instruction Tuning Explained 

LLaVA: Advancing Vision-Language Models Through Visual Instruction Tuning

This blog explores how RAG works, RAG challenges, and advanced RAG techniques like Small to Slide RAG and ColPali. 


large_冯迪.jpg

medium_冯迪.jpg

small_冯迪.jpg

thumbnail_冯迪.jpg

Fendy Feng.jpg

Fendy Feng

Simon Mwaniki 

[Large language models](https://zilliz.com/glossary/large-language-models-(llms)) (LLMs) have demonstrated exceptional capabilities in understanding and generating various types of content, text or visual, making them widely adopted across various industries. Despite their versatility, LLMs often produce inaccurate or fabricated information—known as [hallucinations](https://zilliz.com/glossary/ai-hallucination)—due to limited domain knowledge and outdated data. 

[Retrieval-Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation) is a technique that addresses these challenges by combining LLMs' generative abilities with retrieval systems that fetch relevant information from external sources. By grounding LLM responses in real-world, up-to-date data, RAG improves the accuracy and contextual relevance of answers and enhances the system's overall efficiency. This combination of generation and retrieval drives innovation, transforms data processing, and makes LLMs more reliable in real-world applications.

At a recent [Unstructured Data Meetup](https://zilliz.com/learn/introduction-to-unstructured-data) hosted by Zilliz, [Yi Ding](https://www.linkedin.com/in/dingy/), formerly Head of Typescript and Partnerships at [LlamaIndex](https://www.llamaindex.ai/), shared insights on a new approach to RAG: Small to Slide. This method enhances the performance of [multimodal LLMs](https://zilliz.com/learn/top-10-best-multimodal-ai-models-you-should-know) when dealing with visual data like presentations or documents with images. In this blog, we’ll explore how RAG works, RAG challenges, and advanced RAG techniques like Small to Slide RAG. 

<iframe width="560" height="315" src="https://www.youtube.com/embed/i3Ou3OUo3T4?si=djSOIMHqdBGtU0RU&amp;start=916" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> 


## Understanding RAG

[RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) combines the power of LLMs with the ability to retrieve specific information from a knowledge base powered by [vector databases](https://zilliz.com/learn/what-is-vector-database). When a user asks a question, the RAG system searches the database for relevant information and uses this information to generate a more accurate response. Let’s take a look at how the RAG process works. 

![](https://assets.zilliz.com/RAG_architecture_a4710eb118.png)

*Figure: How RAG works*

A RAG system consists of three key components: an [embedding model](https://zilliz.com/blog/choosing-the-right-embedding-model-for-your-data), a [vector database](https://zilliz.com/learn/what-is-vector-database), and an LLM. 

- The embedding model converts documents into [vector embeddings](https://zilliz.com/glossary/vector-embeddings), which are stored in a vector database like [Milvus](https://zilliz.com/what-is-milvus). 

- When a user asks a question, the system transforms the query into a vector using the same embedding model. 

- The vector database then performs a [similarity search](https://zilliz.com/learn/vector-similarity-search) to retrieve the most relevant information. This retrieved information is combined with the original question to form a "question with context," which is then sent to the LLM. 

- The LLM processes this enriched input to generate a more accurate and contextually relevant answer.

This method is particularly valuable for handling specialized knowledge or dynamic, frequently updated information, ensuring that AI systems provide precise and reliable responses.

For example, a large e-commerce platform could use this approach to power its product recommendation system. The system could process millions of product descriptions, customer reviews, and usage data and store them as vectors. When a user searches for a product, the system quickly retrieves the most relevant items, considering not just keywords but the semantic meaning of the query.


## RAG Challenges and Advanced RAG Techniques 

While RAG offers powerful capabilities, it also faces challenges that can impact its effectiveness, especially as data becomes large and complex. 

- **Speed:** Searching larger databases takes longer. This can be mitigated by leveraging hardware-accelerated computing, such as optimizing for GPUs or other high-performance systems.

- **Accuracy:** Ensuring that we're retrieving the most relevant information is challenging. Developers need ways to balance performance and data accuracy based on their needs. 

- **Multimodal Data**: A basic RAG struggles with non-text data like images or charts. Handling different types of vector data becomes crucial in these cases.

In the following sections, we’ll explore several advanced RAG techniques—**Small to Big RAG, Small to Slide RAG, and ColPali**—that address these challenges, enhancing RAG systems' speed, accuracy, and versatility. 


## Small to Big RAG

**Small to Big RAG** is a method to enhance the accuracy of RAG systems. The core idea is to split documents into smaller, intent-focused chunks for precise searches but retrieve larger context sections rather than returning just the exact matching pieces to ensure LLMs can generate meaningful and complete responses. This is because while smaller chunks improve search precision by honing in on specific details, they can also miss critical context, leading to incomplete or inaccurate answers.

Let’s take a look at one example. 

Consider the query: _“_**_Where does Ms. Churilo work_**_?”_ 

When processed, the embedding model might prioritize the less common word _“Churilo”_ over more important terms like _“work.”_ As a result, the system could retrieve sentences mentioning _“Churilo”_ but fail to provide the key information: _“Chris is currently VP of Marketing at Zilliz.”_ 

Small to Big RAG addresses this limitation by retrieving the entire paragraph where her role is mentioned, giving the LLM the full context needed to deliver a correct and complete response.

![](https://assets.zilliz.com/Figure_Incorrect_retrieval_in_traditional_RAG_66123d0372.png)

*Figure: Incorrect retrieval in traditional RAG*

Small to Big RAG is particularly effective for handling complex or multi-layered text data. In a technical support system, for example, retrieving a single sentence with a relevant keyword might leave out surrounding details that are critical for understanding. Instead, Small to Big RAG retrieves all sections related to the query. For instance, when a user asks, _“How do I reset my password?”_ the system could retrieve the paragraph with the direct answer and include additional context, such as security recommendations, making the response more comprehensive and useful.

By combining the precision of smaller embeddings with the contextual depth of broader retrieval, Small to Big RAG ensures a balance between accuracy and completeness. This approach enables AI systems to provide answers that are not only precise but also rich in context, improving their reliability in real-world applications.


## Small to Slide RAG

While Small to Big RAG excels at handling text-based data, it struggles to capture information embedded in visuals such as slides, charts, or graphs. **Small to Slide RAG** is an improved RAG method that solves this problem by embedding text from visual elements like slides but retrieving the entire slide image. This retrieved result is then provided to and processed by a [multimodal LLM](https://zilliz.com/learn/top-10-best-multimodal-ai-models-you-should-know) capable of analyzing both text and images. 

Here's how it works:

- **Text Embedding:** The system extracts and embeds text from slides.

- **Visual Retrieval:** Instead of retrieving only text, it retrieves the entire slide image containing the relevant information.

- **Multimodal Processing:** The retrieved slide images are fed into a multimodal LLM (e.g., GPT-4 with vision capabilities) that can process both the visual and textual components simultaneously. 

This method is particularly useful for handling complex documents where crucial information is conveyed through visuals. For example, in quarterly financial reports, key insights are often found in graphs or charts that traditional text-based RAG systems might miss. By retrieving and analyzing the entire slide image, Small to Slide RAG ensures no critical detail is overlooked, providing a more accurate and comprehensive response.

To demonstrate the power of Small to Slide RAG, Yi Ding gave a demo of a practical example using an Nvidia slide deck. You can [check out the demo here](https://youtu.be/i3Ou3OUo3T4?list=PLPg7_faNDlT7SC3HxWShxKT-t-u7uKr--&t=2080). 

![](https://assets.zilliz.com/Figure_A_comparison_between_the_context_retrieved_by_a_traditional_RAG_and_a_Small_to_Slide_RAG_5f089a4ca3.png)

_Figure: A comparison between the context retrieved by a traditional RAG and a Small to Slide RAG_

In this demo, Yi Ding compared the context retrieved by a traditional RAG system with Small to Slide RAG when answering the same question: **_"Which SaaS platforms does NVIDIA AI Enterprise support?_**"

- **Traditional RAG:** The traditional text-based RAG system struggled to provide a complete answer. It relied solely on text extracted from the slides, missing critical information embedded in charts and diagrams. This limitation resulted in an incomplete and less informative response.

- **Small to Slide RAG:** In contrast, Small to Slide RAG delivered far better results. The system retrieved the actual slide images containing the relevant information and fed them into a multimodal LLM (e.g., GPT-4 with vision capabilities). This allowed the model to interpret both the text and visual elements, generating a much more comprehensive and accurate answer.

This example illustrates the unique strength of Small to Slide RAG: its ability to handle complex documents where key information is spread across text and visuals, making it an invaluable tool for processing visually rich content.

For these RAG methods to function smoothly, they require infrastructure to manage complex queries and retrieval operations. 


## ColPali: Pushing the Boundaries 

At the end of his talk, Yi introduced an emerging technique called [**ColPali**](https://zilliz.com/blog/colpali-enhanced-doc-retrieval-with-vision-language-models-and-colbert-strategy)**,** a novel document retrieval model that pushes the boundaries of Retrieval-Augmented Generation (RAG). Unlike traditional methods that convert documents with visual data (like slides and PDFs) into text, ColPali works directly with the **visual features of documents**, enabling it to index and retrieve information without the error-prone step of text extraction.


### How ColPali Works

While still in its early stages, ColPali changes the way we usually conduct document retrieval by focusing on visual data:

- Documents are treated as images, bypassing text conversion entirely.

- Multiple embeddings, similar to [visual n-grams](https://zilliz.com/learn/introduction-to-natural-language-processing-tokens-ngrams-bag-of-words-models), are generated per image to capture fine-grained visual features.

- Searches are conducted by comparing visual embeddings to the query, identifying the most relevant areas of the document.

![](https://assets.zilliz.com/Figure_French_document_about_oil_production_in_Kazakhstan_with_certain_areas_highlighted_1f0ecdf31f.png)

_Figure: French document about oil production in Kazakhstan, with certain areas highlighted_

Let’s look at the example shown in the image above: a French document about oil production in Kazakhstan. When asked about offshore oil production, ColPali highlights specific areas of the document—such as charts and diagrams—that are most relevant to the query. This ensures that both textual and visual elements are taken into account, offering a comprehensive understanding of the content.


### Why ColPali is Exciting

ColPali offers several key advantages over traditional RAG systems: 

- By working directly with images, it avoids inaccuracies introduced by converting complex documents (e.g., PDFs) to text.

- It captures visual information, such as layouts, diagrams, and formatting, that is often lost in text-based methods.

- ColPali enables retrieval and search for visual data, opening possibilities in fields like technical design, legal documentation, and more.

A practical application of ColPali could be in architectural design reviews. Imagine an AI system analyzing blueprints and technical drawings directly. When queried about specific design elements, the system could highlight relevant parts of the drawings, incorporating both textual annotations and visual features.


### ColPali Challenges and Limitations

While ColPali is promising, it is still in its early stages and comes with some challenges:

- **High Computational Demand:** Generating and comparing multiple embeddings per image is computationally intensive, making [GPU acceleration](https://zilliz.com/blog/milvus-on-gpu-with-nvidia-rapids-cuvs) essential for practical use.

- **Limited Ecosystem:** Compared to more established RAG techniques, there are fewer tools and resources available for implementing ColPali.

- **Language and Training-Specific Models:** ColPali models currently rely on specific training datasets, which may limit generalization across different domains.


## Pros and Cons of Each RAG Technique

Each of the techniques we have discussed above offers distinct advantages and limitations. 

![](https://assets.zilliz.com/Figure_Comparison_of_different_RAG_techniques_c75e332171.png)

Figure: Comparison of different RAG techniques

**Text-based RAG** is the simplest to implement, benefiting from extensive tooling support. However, it struggles with maintaining multimodal information and doesn't always handle chunking perfectly. On the other hand, **Small to Slide** improves the capture of multimodal data, though it demands a more intricate setup and works best with specific data types. Lastly, **ColPali** simplifies the process by avoiding the need for text conversion and integrating built-in attribution, but it requires GPUs and custom models, which may limit accessibility for some projects.


## Implementing Advanced RAG: Practical Considerations

If you're considering implementing these advanced RAG techniques in your own projects, here are some points to keep in mind. 


### Data Assessment

Look at the types of data you're working with. If you have many documents with charts, images, or complex layouts, approaches like Small to Slide RAG might be particularly beneficial. For example, if you're building a system to analyze financial reports, which often contain a mix of text, tables, and charts, Small to Slide RAG could provide more comprehensive insights than a text-only approach.


### Computational Resources

Methods like Small to Slide RAG and ColPali can be more computationally intensive than traditional RAG. Ensure you have the necessary resources, particularly when working with large datasets. 


### Model Selection

These advanced techniques often require specific types of models. For Small to Slide RAG, you'll need a multimodal LLM capable of processing text and images. Make sure you choose a model that fits your needs and that you have the resources to run it.


### Vector Database Capabilities

Choosing the right vector database is crucial for implementing a RAG system tailored to your needs. For example, if you're implementing a **Small to Big RAG**, you'll need a vector database capable of efficiently retrieving larger chunks of text using embeddings generated from smaller chunks. Scalability becomes key if your application deals with massive amounts of vector data, such as billion-scale datasets. Solutions like [**Milvus**](https://zilliz.com/what-is-milvus) and its managed service, [**Zilliz Cloud**](https://zilliz.com/cloud), are designed for these scenarios, offering highly scalable and efficient vector retrieval.


### Evaluation Metrics

Develop clear metrics to evaluate the performance of your RAG system, including the relevance of retrieved information, the accuracy of generated responses, and the speed of retrieval and generation.


### Iterative Development

RAG is evolving rapidly. Plan for an iterative development process that allows you to easily test and compare different approaches. Start with a basic RAG implementation, then gradually introduce more advanced techniques like Small to Big RAG or Small to Slide RAG. Continuously evaluate performance and user feedback to guide your development process.


## Conclusion

RAG techniques continue to evolve, offering different approaches like text-based RAG, Small to Slide RAG, and ColPali to address varying needs. While text-based RAG provides a solid starting point for general applications, more specialized cases, especially those handling visual data, can benefit from advanced methods like Small to Slide RAG or ColPali. 

Choosing the right method depends on your data and the resources at hand. Each approach provides a way to improve the efficiency and accuracy of AI systems, enabling a better balance between complexity, speed, and cost. By aligning your approach with your specific requirements, you can leverage RAG to build systems that offer users the right information at the right time in the right format. 


## Further Resources

- [What is RAG? ](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Multimodal RAG: Expanding Beyond Text for Smarter AI ](https://zilliz.com/blog/multimodal-rag-expanding-beyond-text-for-smarter-ai)

- [ColPali: Efficient Document Retrieval with Vision Language Models](https://arxiv.org/html/2407.01449v2)

- [What is Milvus | Milvus Documentation](https://milvus.io/docs/overview.md)

- [Deploying a Multimodal RAG System with vLLM and Milvus](https://zilliz.com/blog/deploy-multimodal-rag-using-vllm-and-milvus)

- [Evaluate Your Multimodal RAG Using Trulens ](https://zilliz.com/blog/evaluating-multimodal-rags-in-practice-trulens)

- [The Landscape of GenAI Ecosystem: Beyond LLMs and Vector Databases ](https://zilliz.com/blog/landscape-of-gen-ai-ecosystem-beyond-llms-and-vector-databases)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)

- [Build AI Apps with Milvus: Tutorials & Notebooks](https://zilliz.com/learn/milvus-notebooks)




large_Advanced RAG Techniques_ Bridging Text and Visuals for More Accurate Responses-20241125-114433.png

medium_Advanced RAG Techniques_ Bridging Text and Visuals for More Accurate Responses-20241125-114433.png

small_Advanced RAG Techniques_ Bridging Text and Visuals for More Accurate Responses-20241125-114433.png

thumbnail_Advanced RAG Techniques_ Bridging Text and Visuals for More Accurate Responses-20241125-114433.png

Advanced RAG Techniques_ Bridging Text and Visuals for More Accurate Responses-20241125-114433.png

Advanced RAG Techniques: Bridging Text and Visuals 

Advanced RAG Techniques: Bridging Text and Visuals for More Accurate Responses

We'll learn to build a Voice Assistant, a specialized Agentic RAG system designed for voice interactions, with Milvus, Llama 3.2, and other GenAI tools. 

large_Stephen Batifol.png

medium_Stephen Batifol.png

small_Stephen Batifol.png

thumbnail_Stephen Batifol.png

Stephen Batifol.png

Stephen Batifol is a Developer Advocate at Zilliz.

Stephen Batifol


_"Sorry, this feature isn't available in your region yet."_

If you're based in Europe, you've probably seen this message more times than you'd like to count. While the rest of the world celebrates each new AI breakthrough, many of us find ourselves pressing our noses against the metaphorical window, watching the party from outside. From [Llama 3.2](https://zilliz.com/blog/learn-llama-3-2-and-how-to-build-rag-with-llama-with-milvus) to OpenAI's Advanced Voice Mode, we're caught in an endless cycle of "Coming to your region... soon™️".

Well, I got tired of waiting. If we can't join the party, let's throw our own!

In this blog, we will guide you through building **a Voice Assistant**, a specialized Agentic RAG ([Retrieval-Augmented Generation](https://zilliz.com/learn/Retrieval-Augmented-Generation)) system designed for voice interactions. Using open-source projects such as [Milvus](https://zilliz.com/what-is-milvus) for the vector database, Llama 3.2, and various [GenAI](https://zilliz.com/learn/generative-ai) technologies, including Assemby AI, DuckDuckGo, and ElevenLabs, this voice assistant goes beyond mere processing—it intelligently understands and responds to voice queries.


## Key Technologies We'll Use

- **[Milvus](https://milvus.io/docs/overview.md)** is an open-source, high-performance, highly scalable [vector database](https://zilliz.com/learn/what-is-vector-database) that stores, indexes, and searches billion-scale unstructured data through high-dimensional [vector embeddings](https://zilliz.com/glossary/vector-embeddings). It is perfect for building modern AI applications such as Retrieval Augmented Generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)), semantic search, [multimodal search](https://zilliz.com/blog/multimodal-rag-expanding-beyond-text-for-smarter-ai), and recommendation systems. Milvus runs efficiently across various [environments](https://zilliz.com/blog/choose-the-right-milvus-deployment-mode-ai-applications), from laptops to large-scale distributed systems.
- **[Llama 3.2](https://zilliz.com/blog/learn-llama-3-2-and-how-to-build-rag-with-llama-with-milvus)** is the latest large language model ([LLM](https://zilliz.com/glossary/large-language-models-(llms))) provided by Meta. We'll use it to generate responses based on the retrieved information from Milvus.
- **Assembly AI** provides advanced speech-to-text APIs designed to convert spoken language into written text with a high degree of accuracy.
- **DuckDuckGo** is a search engine that emphasizes user privacy and provides the same search results for all users, unlike traditional search engines that tailor results based on user data.
- **ElevenLabs** specializes in advanced voice synthesis, allowing users to create realistic voice clones from brief audio samples.


## What We'll Build: An Agentic RAG System for Voice Interactions

There are several ways of building an Agentic RAG system. For those interested, I've written a blog walking through how to build [a local agentic RAG system](https://zilliz.com/blog/local-agentic-rag-with-langraph-and-llama3) using Llama 3.2, LangChain, and LangGraph.

However, this time we're taking a different route by building an agent from scratch, without relying on any existing frameworks. The architecture of this RAG system is broken down into multiple components (as shown below), each handling a specific part of this process.

![Figure- the architecture of this agentic RAG system.png](https://assets.zilliz.com/Figure_the_architecture_of_this_agentic_RAG_system_069e7f8b14.png)

_Figure: The architecture of this agentic RAG system_

Below is an overview of how this agentic RAG system works.

- The user sends their voice query to the system, starting the interaction.
- Assembly AI transcribes the user's voice to text in real-time.
- The transcribed text is processed by the Query Processor, which prepares it for vector analysis and keyword detection in the agentic RAG system.
- [Milvus](https://milvus.io/) vector database performs [semantic similarity searches](https://zilliz.com/glossary/semantic-search) and retrieves documents with high semantic similarity to the query.
- Concurrently, the Keyword Detector scans the text for specific keywords that might trigger specific actions, such as accessing the Google Calendar API if calendar-related keywords are identified.
- Documents retrieved from Milvus are checked for their relevance to the query by our agentic system. If relevant, they proceed to response generation using Llama 3.2; if not, a web search is initiated. If the documents from Milvus are not relevant, DuckDuckGo then conducts a web search.
- Llama 3.2 takes the information from either Milvus or DuckDuckGo or Keyword Detector, depending on the relevancy check, to generate a coherent and context-appropriate response.
- The response generated by Llama 3.2 is converted into speech using ElevenLabs’ Text-to-Speech technology.
- Finally, the synthesized voice response is delivered back to the user, completing the interaction loop with an intelligent, context-aware answer.

In the next section, we'll discuss how we build this agentic RAG system, particularly how this system handles various types of searches simultaneously.


## **Multi-Source Orchestration**

Unlike traditional RAG systems that follow a linear query → retrieve → generate pattern, our system thinks for itself. Let's explore the key components of our implementation:

💡 Note: For clarity and readability, we'll focus on the most important code snippets here. The complete implementation is available in [our GitHub repository](https://github.com/stephen37/voice-assistant).

**First, Milvus Vector Search**

```
milvus_results = self.milvus_wrapper.search_similar_text(text)
relevant_results = [result for result in milvus_results 
                   if result["distance"] > 0.6]

if relevant_results:
    context = "\n".join([result["text"] for result in milvus_results])
    augmented_query = f"""
    Context: {context}
    User Query: {text}
    Please answer based on the given context.
    """
```

When you ask a question, we first check our Milvus knowledge base for relevant results. If we find something relevant (similarity > 0.6), we use that to answer your query.

**Second, Calendar Integration**

```
elif any(keyword in text.lower() for keyword in 
        ["calendar", "schedule", "events", "appointment"]):
    events = await self.calendar_service.get_upcoming_events()
    augmented_query = f"""
    Calendar Events: {events}
    User Query: {text}
    Please answer based on their calendar events.
    """
```

If you're asking about your schedule, we skip the knowledge base entirely and go straight to your calendar. No need to search through vectors when we know exactly where the information is!

**Finally, Web Search Fallback**

```
else:
    web_results = self.web_searcher.search(text)
    if web_results:
        context = "\n".join(web_results[:3])
        augmented_query = f"""
        Web search results: {context}
        User Query: {text}
        Please answer based on the web search results.
        """
```

If we can't find anything relevant in Milvus and it's not a calendar query, we fall back to DuckDuckGo for a web search. We then get the top results, extract the most relevant paragraphs and feed that back to our LLM Llama 3.2.

Finally, Llama 3.2 takes the information as context to generate a coherent and accurate response.


## Why Milvus for RAG?

RAG systems are only as good as their retrieval speed and the quality of their retrieval. This is why Milvus is key here:

1. **Speed Matters**: When you're building a voice assistant, nobody wants to wait 5 seconds for an answer. Milvus can search through millions of vectors in milliseconds.
2. **Accuracy at Scale**: Milvus excels at managing high-dimensional vector searches through Approximate Nearest Neighbor ([ANN](https://zilliz.com/glossary/anns)) algorithms. With robust indexing options like [IVF_FLAT,](https://zilliz.com/learn/vector-index) [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW), and [DiskANN](https://zilliz.com/learn/DiskANN-and-the-Vamana-Algorithm), Milvus ensures high recall rates even as the vector database scales, maintaining the integrity of search results in extensive systems.


## The Results?

Our DIY voice assistant might not be the best one out there, but it's got some unique strengths:

- Modular design: Swap out any component (try Whisper instead of AssemblyAI, why not?)
- Full control: No black boxes, just transparent, hackable code
- Privacy-focused: Your data stays yours


## Conclusion

While this project is obviously far from what OpenAI Advanced Voice mode can do, besides, there's something satisfying about building your own AI assistant. It may not have Claude's charm or ChatGPT's polish, but it's got character. More importantly, it offers true ownership and control of your AI stack. By using open-source tools like the [Milvus](https://github.com/milvus-io/milvus) vector database, we've shown how we can build something that's:

- **Truly Yours**: Every component is transparent and modifiable
- **Geographically Unrestricted:** Deploy anywhere, serve anyone
- **Forever Available**: No sudden API changes or service discontinuations
- **Fully Customizable:** Add features, modify behavior, or optimize for specific needs
- **Privacy-First**: Your data stays on your infrastructure, under your control

And unlike some other AI tools I could mention, this one definitely knows where Europe is on a map!

The full code is available on our [Github](https://github.com/stephen37/voice-assistant).


## We'd Love to Hear What You Think!

If you like this blog post, please consider:

- ⭐ Giving us a star on [GitHub](https://github.com/milvus-io/milvus)
- 💬 Joining our [Milvus Discord community](https://discord.gg/FG6hMJStWu) to share your experiences
- 🔍 Exploring our [Bootcamp repository](https://github.com/milvus-io/bootcamp) for more examples of RAG applications with Milvus

***


large_Voice Assistant With Milvus and Llama 3.2.png

medium_Voice Assistant With Milvus and Llama 3.2.png

small_Voice Assistant With Milvus and Llama 3.2.png

thumbnail_Voice Assistant With Milvus and Llama 3.2.png

Voice Assistant With Milvus and Llama 3.2.png

Stop Waiting, Start Building: Voice Assistant With Milvus and Llama 3.2

Vector Quantization (VQ) is a data compression technique representing a large set of similar data points with a smaller set of representative vectors, known as centroids. 

large_Benito Headshot.jpg

medium_Benito Headshot.jpg

small_Benito Headshot.jpg

thumbnail_Benito Headshot.jpg

Benito Headshot.jpg

Benito Martin

large_Unlocking the Power of Vector Quantization_ Techniques for Efficient Data Compression and Retrieval.png

medium_Unlocking the Power of Vector Quantization_ Techniques for Efficient Data Compression and Retrieval.png

small_Unlocking the Power of Vector Quantization_ Techniques for Efficient Data Compression and Retrieval.png

thumbnail_Unlocking the Power of Vector Quantization_ Techniques for Efficient Data Compression and Retrieval.png

Unlocking the Power of Vector Quantization_ Techniques for Efficient Data Compression and Retrieval.png

What is Vector Quantization?

Unlocking the Power of Vector Quantization: Techniques for Efficient Data Compression and Retrieval

Explore Zilliz Cloud’s three CU options and learn how to choose the most suitable one for your business

thumbnail_RT.jpeg

RT.jpeg

Robert Guo



A Compute Unit (CU) in Zilliz Cloud refers to hardware resources catering to search requests and indexes. Zilliz Cloud provides three types of CUs: **Performance-optimized, Capacity-optimized, and Cost-optimized (coming soon)**. Each CU type comprises different CPU, memory, and storage resource combinations for different business needs. Therefore, selecting the appropriate CU options and sizes is crucial while configuring a Zilliz Cloud cluster.

## Performance-optimized CU

A performance-optimized CU is ideal for similarity retrieval tasks that require a **rapid response time** in milliseconds with a **high throughput** of at least 100 queries per second (QPS). Each CU can handle about 1.5 million 768-dim vectors.

This CU type is essential for (but not limited to) the following use cases:

- Generative AI applications
- Recommender systems
- Search engines
- Chatbots
- Content moderation
- Augmenting LLMs' knowledge base
- Anti-fraud systems

## Capacity-optimized CU

If your application handles tens of millions of vectors, consider using a capacity-optimized CU. Each CU can handle about 5 million 768-dim vectors. This type of CU can store much more data than a performance-optimized CU with a lower cost but also a lower performance. 

Capacity-optimized CUs are particularly useful for (but not limited to) the following scenarios:

- Searching through large-scale unstructured data, such as text, images, videos, and molecular structures
- Detecting copyright violations
- Verifying identities.

## Cost-optimized CU (coming soon)

The cost-optimized CU is ideal if you are not concerned with response time and have a very tight budget. Each CU can handle 20 million 768-dimensional vectors with a relatively reasonable price. While it has a higher search latency, it can hold up to 4 times more data as a capacity-optimized CU. 

This type of CU is perfect for offline tasks such as:

- Data labeling or clustering
- Deduplication
- Dataset outlier detection or class balancing.

## Evaluating three CU types

The table below provides an overview of the differences among Zilliz Cloud’s CU types.

|      **CU Type**      | **Latency** | **Throughput** | **Capacity** | **Cost per Million Vectors** (note: based on 768-dim vectors) |
| :-------------------: | :---------: | :------------: | :----------: | :-----------------------------------------------------------: |
| Performance-optimized |     Low     |      High      |      Low     |                    Starting from $65/month                    |
|   Capacity-optimized  |    Medium   |     Medium     |    Medium    |                    Starting from $20/month                    |
|     Cost-optimized    |     High    |       Low      |     High     |                          Coming soon                          |


### Performance comparison

To measure the performance of different CU options, we looked at two key indicators: search latency and throughput. We tested Zilliz Cloud's three types of CUs using two datasets with various `topk` values (10, 100, 250, 1000). The first dataset consists of 1,000,000 vectors with 768 dimensions, and the second has 5,000,000 vectors with the same dimension. 



|  top_k | /                        | /           | 10    | 100   | 250   | 1000     |
| :-----: | ------------------------ | ----------- | ----- | ----- | ----- | -------- |
| Latency | Performance-optimized CU | 1M 768dim   | <10ms | <10ms | <10ms | 10-20ms  |
|         | Capacity-optimized CU    | 5M 768dim   | <50ms | <50ms | <50ms | 50-100ms |
|         | Cost-optimized CU        | Coming soon |       |       |       |          |



The table above shows that the performance-optimized CU is the best choice for low latency, outperforming the capacity-optimized CU. It maintains a latency of under ten milliseconds for typical `topk` values of 10-250, five to ten times faster than the capacity-optimized. When dealing with `topk` values in the thousands, the latency for each CU type varies from 10-20 ms for the performance-optimized CU and 50-100 ms for the capacity-optimized CU. However, it's worth noting that even though the performance-optimized CU slows down in responses when conducting tasks with`topk` values in the thousands, its search latency is still suitable for many real-time applications. 

| top_k |                          |             | 10  | 100 | 250 | 1000 |
| ------ | ------------------------ | ----------- | --- | --- | --- | ---- |
| QPS    | Performance-optimized cu | 1M 768dim   | 520 | 440 | 270 | 150  |
|        | Capacity-optimized CU    | 5M 768dim   | 100 | 80  | 60  | 40   |
|        | Cost-optimized CU        | Coming soon |     |     |     |      |


When it comes to throughput, the performance-optimized CU is superior. It outperforms the capacity-optimized CU by four to five times.

### Capacity comparison

We tested Zilliz Cloud's three types of CUs using a standard set of vector dimensions: 128, 256, 512, 768, and 1024. 

| Vector dimensions | Number of vectors per CU (Millions) |       Number of vectors per CU (Millions)                |            Number of vectors per CU (Millions)       |
| ----------------- | ----------------------------------- | --------------------- | ----------------- |
| /                 | Performance-optimized CU            | Capacity-optimized CU | Cost-optimized CU |
| 128               | 5                                   | 25                    | Coming soon       |
| 256               | 2.96                                | 14.87                 | Coming soon       |
| 512               | 1.63                                | 8.22                  | Coming soon       |
| 768               | 1.5                                 | 5                     | 20                |
| 1024              | 0.86                                | 4.34                  | Coming soon       |

Based on the testing result in the table above, we find that:

- The cost-optimized CUs have the largest capacities in storing 768-dimensional vectors, 13 times and 4 times larger than the performance-optimized and capacity-optimized CUs respectively. 
- As the vector dimensions increase, more storage space is needed to hold the data. For instance, a CU can store roughly twice the number of 512-dimensional vectors compared to 1024-dimensional vectors.

**Note**: This experiment only focused on the primary key and vectors without adding scalar fields. However, if there are additional scalar fields like id, label, keywords, summary, URL, etc., the actual capacity of each CU type may differ from the table above. Therefore, it's essential to rely on empirical measurement for accuracy.

## Let’s look at some examples!

We have compared Zilliz Cloud's three CU options by looking through the lens of latency, throughput, capacity, and cost. But how do you choose the most suitable option for your business? Let's look at two examples to help you make the right choice.

### Example 1

Suppose you’re building an LLM-augmented chatbot that adopts Zilliz Cloud to store more than 10 million text chunks of private documents with a 768-dimension embedding vector. Your application requires Zilliz Cloud to support 1,000 QPS and retrieve the top 10 results with an end-to-end latency of less than 30 milliseconds. 

A performance-optimized CU is the only way to achieve a latency under 30ms. Since each performance-optimized CU can hold up to 1.5 million 768-dimensional vectors, you'll need at least seven CUs to handle all 10 million vectors. One CU can reach a peak QPS of 520 for throughput when the `topk` value is 10. To take 1,000 QPS, you'll need two replicas. 

Therefore, the best approach for this scenario is to use two replicas of a performance-optimized CU, each containing seven CUs.

### Example 2

Suppose your application detects copyright violations in images and needs to find similar ones from a pool of 100 million. Each image is embedded into a 768-dimensional vector. You don’t require real-time responses but expect the top 100 results with a throughput of 50 QPS.

Both capacity-optimized and performance-optimized CU can handle 50 requests per second when you retrieve the top 100 results. However, the capacity-optimized CU can store three times more vectors than the performance-optimized CU. Therefore, the capacity-optimized CU is the more suitable option for your needs.

Based on the test results, a single capacity-optimized CU can store up to 5.6 million 768-dimensional vectors. To accommodate your 100 million vectors, you will require a minimum of 20 CUs. When the `topk` value is 100, a single CU can reach a peak QPS of 80 for throughput. For 50 QPS, one replica is enough. Therefore, you'll need a cluster with 20 capacity-optimized CUs.

## Summary

Zilliz Cloud offers three types of CUs. If you need your application to be lightning-fast and responsive in real-time, the performance-optimized CU is the way to go. The capacity-optimized CU is the best choice for applications that require storing and retrieving tens of millions of vectors. If you're on a tight budget and okay with sacrificing speed and throughput, the cost-optimized CU is perfect for you.

## Getting started with Zilliz Cloud

Explore with our free tier (no credit card required), or try our [30-day enterprise trial](https://cloud.zilliz.com/signup) with up to \$200 in credits. Subscribe through any [cloud marketplace](https://zilliz.com/pricing) and receive an additional \$100 credit.

Dive deeper into the Zilliz Cloud [documentation](https://docs.zilliz.com/docs/quick-start).


large_June ？？——How to Select the Most Appropriate CU Type and Size for Your Zilliz Cloud_.png

medium_June ？？——How to Select the Most Appropriate CU Type and Size for Your Zilliz Cloud_.png

small_June ？？——How to Select the Most Appropriate CU Type and Size for Your Zilliz Cloud_.png

thumbnail_June ？？——How to Select the Most Appropriate CU Type and Size for Your Zilliz Cloud_.png

June ？？——How to Select the Most Appropriate CU Type and Size for Your Zilliz Cloud_.png

How to Select the Most Appropriate CU Type and Size for Your Business?

Comparing Zilliz Cloud’s Three CU Options

This new UI is cleaner, more intuitive, and specifically designed to streamline workflows, reduce cognitive load, and boost productivity

large_koko Lv.jpg

medium_koko Lv.jpg

small_koko Lv.jpg

thumbnail_koko Lv.jpg

koko Lv.jpg

Koko Lv

large_Introducing the Redesigned UI for Zilliz Cloud.png

medium_Introducing the Redesigned UI for Zilliz Cloud.png

small_Introducing the Redesigned UI for Zilliz Cloud.png

thumbnail_Introducing the Redesigned UI for Zilliz Cloud.png

Introducing the Redesigned UI for Zilliz Cloud.png

Zilliz Cloud’s Redesigned UI: A Streamlined and Intuitive User Experience

A 10x faster Performance with Cardinal vector search engine,  production-ready features including Multi-replica, Data Migration, Authentication, and more 


large_Steff-Li-Zilliz.JPG

medium_Steff-Li-Zilliz.JPG

small_Steff-Li-Zilliz.JPG

thumbnail_Steff-Li-Zilliz.JPG

Steff-Li-Zilliz.JPG

Steffi Li

[Vector databases](https://zilliz.com/learn/what-is-vector-database) are hitting their stride as more companies move AI applications into production, but this shift brings new challenges around performance, scale, and management. Today's release of [Zilliz Cloud](https://zilliz.com/cloud) tackles these challenges head-on with our new Cardinal vector search engine and a suite of enterprise-ready features.


## Breaking the Performance Barrier with Cardinal

[Cardinal](https://zilliz.com/blog/cardinal-most-performant-vector-search-engine), our new vector search engine that delivers a 10X performance boost in production environments, is at the heart of this release. This isn't just about raw speed - we've reimagined how [vector search](https://zilliz.com/learn/vector-similarity-search) should work at scale.

Cardinal introduces AutoIndex, which automatically tunes your index configuration based on data characteristics and hardware setup—no more manual tweaking or performance tradeoffs. The system continuously adapts to deliver optimal search performance, letting development teams focus on building features instead of managing infrastructure.

Under the hood, Cardinal employs a three-layer optimization approach. The algorithm layer combines advanced IVF-based and graph-based approaches, maintaining high accuracy even with filtered results. The engineering layer implements specialized memory allocation and multi-threading for real-world performance gains. At the lowest level, kernel optimizations for x86 and ARM platforms achieve up to 3X speedup over standard implementations.


## Production-Ready Features for Enterprise Scale

As vector databases become critical infrastructure for AI applications, we've enhanced Zilliz Cloud's enterprise capabilities across the board.


### Multi-replica Goes GA

Our multi-replica feature is now generally available, offering linear QPS improvements as you add replicas. For high-traffic applications, queries are automatically distributed across replicas, reducing latency and improving efficiency. The system also distributes replicas across availability zones, ensuring continuous operation even during zone outages.


### Infrastructure Improvements

We've increased the capacity of our performance-optimized compute units by 50%, now supporting up to 1.5 million vectors per CU (at 768 dimensions). For large-scale deployments, this translates to approximately 30% cost reduction.


### Enhanced Observability

Integration with Prometheus enables real-time monitoring of system metrics. We've implemented 41 alerts across 26 metrics covering everything from infrastructure health to data operations. This comprehensive monitoring ensures teams can proactively manage their vector search infrastructure.


### Simplified Data Migration

We've added migration pathways from Qdrant and Pinecone Serverless to support the growing ecosystem. The migration process preserves your data integrity while allowing you to use Zilliz Cloud's enhanced capabilities.


### Authentication and Global Presence

Security meets convenience with our new Auth0-based authentication system, supporting email, GitHub/Google authentication, and enterprise SSO through Okta. We've also expanded our global footprint with a new AWS Tokyo region, bringing our total to 12 regions worldwide.


## Developer Experience Improvements

Beyond the technical improvements, we've overhauled the developer experience. The new web console provides enhanced visualization tools and streamlined workflows. We've also revamped our payment system with extended billing cycles for enterprise workflows, automated notifications, and detailed cost visibility through itemized invoices.


## Getting Started

These features are available now across all Zilliz Cloud deployments. Explore with our free tier (no credit card required), or try our 30-day enterprise trial with up to \$200 in credits. Subscribe through any cloud marketplace and receive an additional \$100 credit.

For detailed documentation and migration guides, visit our[ documentation](https://docs.zilliz.com).


large_New for Zilliz Cloud_ 10X Performance Boost and Enhanced Enterprise Features (1).png

medium_New for Zilliz Cloud_ 10X Performance Boost and Enhanced Enterprise Features (1).png

small_New for Zilliz Cloud_ 10X Performance Boost and Enhanced Enterprise Features (1).png

thumbnail_New for Zilliz Cloud_ 10X Performance Boost and Enhanced Enterprise Features (1).png

New for Zilliz Cloud_ 10X Performance Boost and Enhanced Enterprise Features (1).png

New for Zilliz Cloud: 10X Performance Boost and Enhanced Enterprise Features

ScaNN is an open-source library developed by Google for fast, approximate nearest neighbor searches in large-scale datasets. 


large_What is ScaNN_.png

medium_What is ScaNN_.png

small_What is ScaNN_.png

thumbnail_What is ScaNN_.png

What is ScaNN_.png

What is ScaNN (Scalable Nearest Neighbors)?

Google’s ScaNN is a library for ANNS. This guide walks you you through implementing ScaNN and demonstrate how to integrate it with Milvus. 


Nearest neighbor search ([NNS](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google#The-Basics-Vector-Search-and-Nearest-Neighbor-Search)) is a fundamental technique for identifying similar items in large datasets, powering applications such as recommendation engines, image retrieval, and document search. For instance, in e-commerce, NNS algorithms can suggest products similar to what a customer is viewing, based on attributes like category, price, or reviews. However, as datasets grow in size and complexity, achieving both speed and accuracy becomes increasingly challenging.

To address these challenges, approximate nearest neighbor search ([ANNS](https://zilliz.com/glossary/anns)) offers a solution by trading a small degree of accuracy for significant improvements in speed and scalability. Google’s [**ScaNN (Scalable Nearest Neighbors)**](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google) is a library for ANNS, designed to handle large, high-dimensional datasets efficiently. By employing advanced techniques like clustering and compression, ScaNN delivers high-performance search while maintaining accuracy. 

In [my previous post](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google), I introduced the basics of vector search and ScaNN. In this post, we’ll guide you through implementing ScaNN and demonstrate how to integrate it with [**Milvus**](https://zilliz.com/what-is-milvus), an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) built for production-scale vector searches. Together, ScaNN and Milvus provide a robust and scalable solution for building next-generation applications.


## Getting Started with ScaNN

To use ScaNN, start by installing it with pip:

```
pip install scann
```


This command installs ScaNN and its dependencies. After the installations are done, the next step is to prepare the dataset on which we will conduct nearest-neighbor searches. 


### Preparing the Dataset

In this blog, we will use the [GloVe dataset](https://nlp.stanford.edu/projects/glove/), which provides pre-trained word embeddings. [Word embeddings](https://zilliz.com/glossary/glove) represent words as vectors in a high-dimensional space, where similar words are close together. For instance, king and queen have similar embeddings, making this dataset ideal for testing similarity searches.

Load the GloVe dataset as follows:

```
# Import necessary libraries
import numpy as np
import h5py
import os
import requests
import tempfile
import scann  # Install with: pip install scann

# Download and prepare the dataset
with tempfile.TemporaryDirectory() as temp_dir:
    # Fetching dataset from URL and saving it locally
    response = requests.get("http://ann-benchmarks.com/glove-100-angular.hdf5")
    file_path = os.path.join(temp_dir, "glove_data.hdf5")
    with open(file_path, 'wb') as file:
        file.write(response.content)
    
    # Loading data into memory with h5py
    glove_data = h5py.File(file_path, "r")

# Splitting data into vectors and queries
vectors = glove_data['train']
queries = glove_data['test']
print("Vectors shape:", vectors.shape)
print("Queries shape:", queries.shape)
```


This code downloads the GloVe dataset, stores it temporarily, and loads it into memory. The dataset is split into `train` (our main data, stored as `vectors`) and `test` (our sample queries, stored as `queries`). 


### Normalizing the Vectors

To ensure that [similarity calculations](https://zilliz.com/blog/similarity-metrics-for-vector-search) are meaningful, [normalize the vectors](https://zilliz.com/learn/layer-vs-batch-normalization-unlocking-efficiency-in-neural-networks). 

```
# Normalize vectors for efficient ScaNN indexing
normalized_vectors = vectors / np.linalg.norm(vectors, axis=1)[:, np.newaxis]
```


Each vector is divided by its magnitude and computed with `np.linalg.norm`, which brings each vector to a unit length. This is useful for dot-product similarity because it prevents longer vectors from dominating similarity calculations, leading to more accurate results.


### Configuring the ScaNN Index

The next step is to configure the ScaNN index. [An index](https://zilliz.com/learn/vector-index) is a data structure that organizes data to speed up search operations. By configuring the index, you control the balance between search speed and accuracy.

```
# Configure and build ScaNN indexer
# Setting up ScaNN for efficient search with hybrid scoring and reordering
index = scann.scann_ops_pybind.builder(normalized_vectors, 10, "dot_product").tree(
    num_leaves=2000, num_leaves_to_search=100, training_sample_size=250000).score_ah(
    2, anisotropic_quantization_threshold=0.2).reorder(100).build()
```


In the above code, we specify `normalized_vectors` as the dataset to index, with `10` as the number of nearest neighbors to retrieve for each query. We then set `dot_product` as the similarity metric which measures the angle between vectors. `num_leaves=2000` divides the data into 2000 clusters, which reduces the search space and speeds up searches by focusing on relevant clusters. We then set `num_leaves_to_search=100` to limit the search to 100 clusters, further balancing speed and accuracy. `training_sample_size=250000` specifies the number of samples to train the index, which speeds up indexing without using the entire dataset.

The `score_ah(2, anisotropic_quantization_threshold=0.2)` parameter enables asymmetric hashing, a data compression technique with a quantization threshold to compress data, reducing memory usage and improving retrieval speed. Finally, `reorder(100)` refines the top 100 results to ensure high accuracy.


### Running Searches

Once the index is configured, perform similarity searches to find the nearest neighbors of your queries. A search retrieves the closest matches based on the index’s configuration.

```
# Run initial search with default neighbors set to 10
neighbors, distances = index.search_batched(queries)

# Show search results for a sample query
sample_query_index = 0  # Change this index to see results for different queries
print(f"\nSample results for query {sample_query_index}:")
print("Nearest neighbor indices:", neighbors[sample_query_index])
print("Distances to nearest neighbors:", distances[sample_query_index])
```


In the above code, we run a batched search on the queries. The `neighbors` variable stores the indices of the closest vectors for each query, while `distances` contain the similarity scores for these neighbors. Lower distance values indicate closer similarity.

The expected output is as follows:

![](https://assets.zilliz.com/Nearest_neighbor_indices_and_distances_for_a_sample_query_in_Sca_NN_218a451a81.png)

_Figure: Nearest neighbor indices and distances for a sample query in ScaNN_

The output displays the nearest neighbor indices and their distances for the first query. The distances are similarity scores, where lower values indicate higher similarity.

Let’s now integrate ScaNN with Milvus and use a more human-friendly and simple dataset.


## Integrating ScaNN with Milvus

[Milvus ](https://zilliz.com/what-is-milvus)is a vector database optimized for storing and retrieving large collections (up to trillions) of vectors. Milvus natively integrates ScaNN within its architecture for fast search results. By combining ScaNN with Milvus, you gain the benefit of Milvus’s data management and scalability, making it possible to handle vast datasets efficiently.

In this section, we’ll walk through how to implement ScaNN with Milvus. 


### Step 1: Installing Milvus and Its Dependencies

If you haven’t installed Milvus, [follow this guide to install and run it](https://milvus.io/docs/quickstart.md) on your machine. 

Then, install the Milvus Python client, Pymilvus. This client lets you interact with Milvus and manage vector data collections using Python. PyMilvus also seamlessly integrates with various [embedding and reranking models](https://milvus.io/blog/introducing-pymilvus-integrations-with-embedding-models.md), making it easier to build modern AI applications with Milvus, particularly retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)). 

```
pip install "pymilvus[model]"
```



### Step 2: Initializing the Milvus Client and Creating a Collection

Initialize the Milvus client and define a schema for the collection. In Milvus, a collection is where you store and organize your vector data.

```
from pymilvus import MilvusClient, DataType
from pymilvus import model

# Initialize Milvus client with URI for the full server
client = MilvusClient(uri="http://localhost:19530")  # Replace with your Milvus server URI
# Define the collection schema with 768 dimensions
schema = client.create_schema(
    auto_id=False,
    enable_dynamic_field=False,
)
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=768)  # Set to 768
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=200)
```


The above code initializes the client and creates a connection to the Milvus server running locally on your computer. Then, it defines a schema for the collection, specifying that each entry will have an `id` (integer identifier), a `vector` field (100-dimensional float vector), and a `text` field (to store associated text data).


### Step 3: Configuring the ScaNN Index in Milvus

Configure ScaNN as the index type for this collection, organizing the data to make searches faster and more efficient.

```
# ScaNN index parameters with nlist
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="vector",
    index_type="SCANN",    # Use ScaNN as the index type
    metric_type="L2",
    nlist=128  
)

# Create collection in Milvus with ScaNN indexing
collection_name = "scann_collection"
if client.has_collection(collection_name=collection_name):
    client.drop_collection(collection_name=collection_name)

client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params
)
```


Setting `index_type="SCANN"` instructs Milvus to use ScaNN for efficient search. `metric_type="L2"` means Milvus will use Euclidean distance to measure similarity, and `nlist=128` divides the data into 128 clusters, which enhances search speed by focusing on relevant sections.


### Step 4: Inserting Data into Milvus

Generate [vector embeddings](https://zilliz.com/glossary/vector-embeddings) for sample documents and insert them into Milvus. This step stores each vector in Milvus with an associated `id` and text data.

```
# Define embedding function and prepare sample data
embedding_fn = model.DefaultEmbeddingFunction()
docs = [
    "Artificial intelligence was founded as an academic discipline in 1956.",
    "Alan Turing was the first person to conduct substantial research in AI.",
    "Born in Maida Vale, London, Turing was raised in southern England.",
]
vectors = embedding_fn.encode_documents(docs)  

# Insert sample data into Milvus
data = [{"id": i, "vector": vectors[i], "text": docs[i]} for i in range(len(vectors))]
client.insert(collection_name=collection_name, data=data)
```


In the above code, we encode the sample documents as vectors and insert them into Milvus. Each document is stored with a unique `id`, a 100-dimensional vector, and the original text for reference.


### Step 5: Performing Vector Search in Milvus

Next, perform a vector search in Milvus using ScaNN as the index. Run a search to find the nearest vector for a sample query.

```
# Perform vector search in Milvus using ScaNN index with nprobe
SAMPLE_QUESTION = "What's Alan Turing's achievement?"
query_vectors = embedding_fn.encode_queries([SAMPLE_QUESTION])

# Set search parameters, including nprobe for ScaNN
search_params = {"nprobe": 10}  # Adjust nprobe based on accuracy/speed trade-off

search_res = client.search(
    collection_name=collection_name,
    data=query_vectors,
    limit=1,
    output_fields=["text"],
    search_params=search_params  
)

# Retrieve and print the result
context = search_res[0][0]["entity"]["text"]
print("Search result:", context)
```


In the above code, we encode the query and use its embedding to search Milvus for a possible match. Setting `nprobe=10` instructs Milvus to search within 10 clusters, balancing accuracy and performance. The `limit=1` parameter restricts the results to the nearest match. Running this search retrieves the closest document to the query, allowing us to see the most relevant result.

Here is the expected output:

![](https://assets.zilliz.com/Closest_matching_document_for_a_sample_query_using_Sca_NN_indexing_in_Milvus_ab9e20578c.png)

Figure: Closest matching document for a sample query using ScaNN indexing in Milvus

The output shows the nearest match for the sample query, displaying the text of the most similar document. As you can see the most relevant result answers our question correctly. This means that Milvus and ScaNN were able to retrieve the correct result. 


## Conclusion

[**ScaNN**](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google) (Scalable Nearest Neighbors) is a library for implementing approximate nearest neighbor search, striking a balance between speed and accuracy for large, high-dimensional datasets. Its advanced techniques, such as clustering and compression, make it an ideal solution for modern applications like recommendation systems, image retrieval, and natural language processing.

[Milvus](https://zilliz.com/what-is-milvus), an open-source vector database, natively supports and integrates ScaNN within its architecture, enabling developers to build scalable, production-ready systems capable of handling billions of vector data efficiently. Together, these tools provide the flexibility and performance needed to tackle real-world challenges in vector search.

In addition to ScaNN, Milvus also supports and optimizes many other types of ANN algorithms such as [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW), [DiskANN](https://zilliz.com/learn/DiskANN-and-the-Vamana-Algorithm), and [IVF](https://zilliz.com/learn/vector-index), delivering optimal performance. For more details, see [Milvus documentation](https://milvus.io/docs/overview.md). 


## Further Resources

- [ScaNN GitHub repo](https://github.com/google-research/google-research/tree/master/scann)

- [What is ScaNN (Scalable Nearest Neighbors)? ](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google)

- [Announcing ScaNN: Efficient Vector Similarity Search](https://research.google/blog/announcing-scann-efficient-vector-similarity-search/)

- [Understanding ANN Benchmarks](https://zilliz.com/glossary/ann-benchmarks) 

- [What are Vector Databases and How Do They Work?](https://zilliz.com/learn/what-is-vector-database)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)

- [HNSWlib: A Graph-based Library for Fast ANN Search](https://zilliz.com/learn/learn-hnswlib-graph-based-library-for-fast-anns)




large_Getting Started with ScaNN-20241023-073632.png

medium_Getting Started with ScaNN-20241023-073632.png

small_Getting Started with ScaNN-20241023-073632.png

thumbnail_Getting Started with ScaNN-20241023-073632.png

Getting Started with ScaNN-20241023-073632.png

Getting Started with ScaNN 

Milvus offers row-level RBAC (Role-Based Access Control) which is a robust solution for managing data access with precision and efficiency. 


large_ken zhang.jpeg

medium_ken zhang.jpeg

small_ken zhang.jpeg

thumbnail_ken zhang.jpeg

ken zhang.jpeg

Ken Zhang is a Senior Product Manager at Zilliz

Ken Zhang

## Why Access Control Matters in Modern Data Systems

**Access control** is one of the most pressing challenges in modern data management, especially for enterprises. As organizations grow, particularly in environments with multiple departments and roles, both robust security and seamless access are critical. Let’s take a look at two examples where access control is especially important.


### Healthcare: Balancing Privacy and Collaboration

In the healthcare sector, organizations must protect patient privacy while fostering collaboration among medical professionals. Imagine a doctor who needs full access to a patient’s medical records to make an accurate diagnosis and treatment plan. However, that doctor shouldn’t be able to access records of patients they’re not treating. To meet this need, fine-grained access control is essential, ensuring that only authorized medical staff can view sensitive patient data. This level of control helps organizations comply with strict regulations like HIPAA while safeguarding privacy.


### Finance: Protecting Sensitive Data

The financial industry faces similar challenges when it comes to access control. Banks and financial institutions handle massive amounts of sensitive data—account details, transaction histories, credit scores, etc. This data is often converted into [vectors](https://zilliz.com/glossary/vector-embeddings) and stored in a [vector database](https://zilliz.com/learn/what-is-vector-database) like Milvus for fraud detection, risk analysis, and personalized customer experiences. Without proper access controls, sensitive data could be exposed to unauthorized parties, leading to significant financial and legal consequences. 

Fine-grained controls, such as row-level permissions, allow institutions to restrict access to specific users—for example, a client manager can access only their own clients' account data. Even broader access, required by risk management teams, can be carefully monitored and restricted to prevent misuse.


## Milvus Row-level RBAC: a Fine-Grained Access Control Solution 

**Role-Based Access Control (RBAC)** is a security model where access to resources is granted based on a user’s role within an organization. Roles define permissions, and users inherit those permissions, ensuring secure and efficient management of access rights. 

[**Milvus**](https://zilliz.com/what-is-milvus) is an open-source, high-performance vector database built for scale. It is perfect for building model AI applications such as [retrieval augmented generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation), [semantic search](https://zilliz.com/glossary/semantic-search) engines, recommendation systems, and chatbots. Milvus offers a fine-grained RBAC solution based on a permission model that uses bitmap indexing to enable row-level access control. This feature allows you to control access to specific Milvus resources and permissions based on user [roles and privileges](https://milvus.io/docs/users_and_roles.md#Users-and-Roles). Currently, Milvus RBAC is only available in Python and Java. 

Milvus RBAC offers several advantages:

- **Speed and Efficiency:** It allows for fast querying of permissions in large datasets.

- **Flexibility:** It adapts as roles and responsibilities evolve, ensuring permissions remain up-to-date.


### RBAC Fundamentals 

#### Roles and Permissions

- **Role**: Represents a user’s role in the system, with each role assigned a specific set of permissions.

- **Permission**: Specifies access rights to individual rows in a data connection (table) within Milvus, such as the ability to read, write, or delete specific data.


#### Bitmap Index building

Bitmap indexes are the foundation of this access control mechanism:

- Each role is associated with a bitmap that indicates the rows it can access.

- The bitmap's length matches the number of rows in the connection:

  - A **1** in a position means the role has access to that row.

  - A **0** means no access.


#### Bitmap Index Usage

- **Granting Permissions**: To give a role access to a row, set the corresponding bit in the role’s bitmap to **1**.

- **Checking Permissions**: To verify if a role can access a specific row, simply check if the corresponding bit in the bitmap is **1**.

Let’s say we have a connection (table), **Collection A**, that stores enterprise knowledge. Each row represents content identified by `doc_id` and its associated knowledge base, `kb_id`.

| Row ID | PK | Data   | doc_id | kb_id | Role  |
| ------ | -- | ------ | ------- | ------ | ----- |
| 1      | 0  | Data A | 1       | 1      | role1 |
| 2      | 1  | Data B | 1       | 1      | role1 |
| 3      | 2  | Data C | 2       | 1      | role1 |
| 4      | 3  | Data D | 2       | 1      | role1 |
| 5      | 4  | Data E | 3       | 2      | role2 |

Roles are defined as follows:

- **Role 1**: Can access rows 1, 2, 3, and 4 (`kb_id = 1`).

- **Role 2**: Can access row 5 (`kb_id = 2`).


#### Query Operations

When a user queries the data, their role’s bitmap is combined with the query conditions to filter rows they are authorized to access. For example, if a user with **Role 1** queries "all data," their bitmap **11110** will return rows 1–4 (`Data A`, `Data B`, `Data C`, and `Data D`).


#### Updating Permissions

To add or remove access for a role, simply update the corresponding bit in the role’s bitmap. For example, setting a bit to **1** grants access, while setting it to **0** revokes it.


#### Advantages and Considerations

- **Efficiency**: Bitmap operations are fast and well-suited for managing permissions in large datasets.

- **Low Storage Overhead**: Bitmaps consume minimal storage, even for datasets with millions of rows.

- **Flexibility**: Bitmap indexes support complex query conditions and combinations, making them highly adaptable to various use cases.


## Demonstrating Milvus Row-level RBAC

In this section, we’ll demonstrate how Milvus manages access to various knowledge bases of a large enterprise. 


### Use Case: An Enterprise RAG with Multiple Knowledge Bases

In a large enterprise, different departments often maintain separate knowledge bases—some public, others confidential—for their [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) application powered by Milvus. To manage permissions effectively across these knowledge bases, we implement role-based access controls (RBAC) based on entities or documents. For example, a super admin role (`admin`) might oversee access, with additional roles for specific business units like CEO, finance, sales, and developer, each having access to different sets of data.

![](https://assets.zilliz.com/Access_control_for_different_roles_in_a_large_enterprise_41cef36b66.png)

_Figure: Access control for different roles in a large enterprise_ 


### Defining Permission Columns

To manage access, we store permission data in an **array column** within Milvus. This column will define which roles have access to which rows of data. The `field_name` for this permission column is customizable, and the array’s size can be adjusted based on the user’s needs. A **BITMAP index** is then created for this column, making permission checks efficient. 

Below is how this setup is done in code:

```
# 1. Set up a Milvus client
client = MilvusClient(
    uri=CLUSTER_ENDPOINT
)

# 2. Create a collection
schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=False,
)

# 3. define schema 
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="data", datatype=DataType.VARCHAR, max_length=100)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=128)

# 4. add security column
schema.add_field(field_name="security_group", datatype=DataType.ARRAY, 
                 element_type=DataType.VARCHAR, max_capacity=10, max_length=100)

index_params = MilvusClient.prepare_index_params()
index_params.add_index(
    field_name="vector",
    index_type="IVF_FLAT",
    metric_type="L2",
    params={"nlist": 1024}
)

# 5. create bitmap index for security column
index_params.add_index(field_name="security_group", 
                       index_type="BITMAP")

# 6. create collection
client.create_collection(
    collection_name="test_collection",
    schema=schema,
    index_params=index_params
)
```



### Write Permissions

When inserting new data, you assign permissions by specifying which roles can access each row. This can be done by writing the corresponding role(s) into the `security_group` column. 

Here’s an example of how it works:

```
data =[]
data.append({
        "id": random.randint(0, 100000),
        "vector": [ random.uniform(-1, 1) for _ in range(128) ],
        "data": "data" + str(random.randint(0,100000)),
       # ceo role can read
      "security_group": ["ceo"]
})

data.append({
        "id": random.randint(0, 100000),
        "vector": [ random.uniform(-1, 1) for _ in range(128) ],
        "data": "data" + str(random.randint(0,100000)),
        # finance role can read
       "security_group": ["finance"]
})

data.append({
        "id": random.randint(0, 100000),
        "vector": [ random.uniform(-1, 1) for _ in range(128) ],
        "data": "data" + str(random.randint(0,100000)),
        # both sales and developer can read
      "security_group": ["sales", "finance"]
})

res = client.insert(collection_name="test_collection", data=data)
```



### Query Permission 

When performing search or query operations, it’s essential to restrict the results to only show the data that a user’s specific role has access to. Data outside of the user’s permitted roles will be hidden from the query results. Here’s how this can be done: 

**Querying Data Based on Role Permissions**

In the examples below, we use the `array_contains()` function to filter data based on role-specific permissions. Each query retrieves only the data that the given role is authorized to see.

```
res = client.query(
    collection_name="test_collection",
  # Query data visible only to the CEO role
   filter='array_contains(security_group, "ceo")',
    output_fields=["id", "data", "security_group"],
)
print("ceo role read:")
print(res)

res = client.query(
    collection_name="test_collection",
   # Query data visible only to the Sales role
   filter='array_contains(security_group, "sales")',
    output_fields=["id", "data", "security_group"],
)
print("sales role read:")
print(res)

res = client.query(
    collection_name="test_collection",
    # Query data visible only to the Developer role
    filter='array_contains(security_group, "develop")',
    output_fields=["id", "data", "security_group"],
)
print("developer role read:")
print(res)

res = client.query(
    collection_name="test_collection",
  # Query data visible to either the Developer or CEO roles
   filter='array_contains_any(security_group, ["develop", "ceo"])',
    output_fields=["id", "data", "security_group"],
)
print("developer or ceo role read:")
print(res)
```


Here’s an example of what the output would look like:

```
ceo role read:
data: [
"{'security_group': ['ceo'], 'id': 3443, 'data': 'data35077'}", 
"{'security_group': ['ceo'], 'id': 12181, 'data': 'data99090'}", 
"{'security_group': ['ceo'], 'id': 16551, 'data': 'data74619'}", 
"{'security_group': ['ceo'], 'id': 24466, 'data': 'data1373'}", ...
sales role read:
data: [
"{'data': 'data75305', 'security_group': ['sales'], 'id': 9122}", 
"{'data': 'data61054', 'security_group': ['sales'], 'id': 20087}", 
"{'data': 'data47948', 'security_group': ['sales', 'develop'], 'id': 21726}", 
"{'data': 'data8596', 'security_group': ['sales'], 'id': 40090}", ... 
developer role read:
data: [
"{'data': 'data1515', 'security_group': ['develop'], 'id': 6429}", 
"{'data': 'data47031', 'security_group': ['develop'], 'id': 10953}", 
"{'data': 'data47948', 'security_group': ['sales', 'develop'], 'id': 21726}", 
"{'data': 'data86894', 'security_group': ['develop'], 'id': 56980}"], ... 
developer or ceo role read:
data: [
"{'data': 'data35077', 'security_group': ['ceo'], 'id': 3443}",
 "{'data': 'data1515', 'security_group': ['develop'], 'id': 6429}", 
 "{'data': 'data47031', 'security_group': ['develop'], 'id': 10953}", 
 "{'data': 'data99090', 'security_group': ['ceo'], 'id': 12181}", ...
```


This approach ensures that each role sees exactly the data they're authorized to view while hiding any unauthorized data. You can also stack multiple roles in the `security_group` array, allowing for flexible and efficient permission management.

**Custom Filters with Role-Based Access**

In some cases, users may need to apply custom filters when querying data. These filters can be combined with role-based access to refine searches. By applying the role-based access control along with custom filter conditions, we can ensure that users only retrieve data they are permitted to see based on both the role and the specific query criteria.

For example: 

```
res = client.query(
    collection_name="test_collection",
   # Sales role queries data with the filter "pk in [1, 3, 5]"
   filter='pk in [1, 3, 5] && array_contains(security_group, "sales")',
    output_fields=["id", "data", "security_group"],
)

res = client.query(
    collection_name="test_collection",
   # Developer role queries data with the filter "pk > 10"
   filter='pk > 10 && array_contains(security_group, "develop")',
    output_fields=["id", "data", "security_group"],
)
```


In these examples, the queries not only check the role-based permissions but also apply additional filters (like `pk in [1, 3, 5]` or `pk > 10`) to narrow down the results based on specific criteria. This flexibility allows users to craft highly targeted queries while maintaining tight control over data access.


### Update Permissions

There are times when you need to change permissions—whether it's granting access to a specific role for a particular data row, or removing that access. Milvus makes this process easy with its `upsert` API, which allows you to update the permissions associated with a row of data.

Let’s walk through how you can use this API to modify permissions for a specific row.


#### Example: Updating Permissions for a Data Row

To update the permissions for a row, you simply adjust the `security_group` field. In this example, we'll add the "sales" role to a row that was previously only accessible by the "finance" role.

```
upsert_row_update = {
        "id": 101,
        "vector": upsert_vector,
        "data": upsert_data,
        # update role 
       "security_group": ["finance", "sales"]
}
res = client.upsert(
    collection_name="test_collection",
    data=upsert_row_update)
```



#### Result:

Before the upsert, the row with `pk = 101` looked like this:

```
pk = 101:
data: [" {'id': 101,
'data': 'data63309', 
'vector': [0.38069534, 0.15088418, -0.6266929, -0.6038463, 0.2516377...],
'security_group': ['finance'],"]  

after upsert
data: ["{'id': 101, 
'data': 'data63309', 
'vector': [0.38069534, 0.15088418, -0.6266929, -0.6038463, 0.2516377...], 
'security_group': ['finance', 'sales']}"] 
```


By using the `security_group` array column and bitmap index filtering, we've established a solid foundation for row-level read access control, allowing us to effectively manage permissions during queries. This method delivers strong performance and fine-grained control over access rights. However, it does require a more hands-on approach from administrators, who must carefully manage permissions when inserting or updating data and ensure a well-thought-out permission strategy when creating tables.


## Conclusion 

Implementing fine-grained access control is a critical component of modern data management, especially for industries dealing with sensitive information like healthcare and finance. [Milvus](https://milvus.io/) offers row-level RBAC (Role-Based Access Control) which is a robust solution for managing data access with precision and efficiency. 

This approach not only enhances security but also offers flexibility for evolving business needs, ensuring that access policies can adapt as roles and responsibilities change. With its powerful tools and flexible permissions model, Milvus empowers organizations to create highly secure, scalable data systems that meet regulatory requirements while offering seamless access to the right people. 

For even more dynamic permission management and inheritance, [Zilliz Cloud](https://zilliz.com/cloud), the fully managed service of Milvus, has even more fine-grained permission features. These enhancements will not only improve administrative efficiency but also offer greater flexibility, making it easier to meet a wider range of business requirements. For more information about Zilliz Cloud RBAC, check out its [documentation](https://docs.zilliz.com/docs/user-roles). 


## Further Reading

- [What are Vector Databases and How Do They Work? ](https://zilliz.com/learn/what-is-vector-database)

- [Build AI Apps with Milvus: Tutorials & Notebooks](https://zilliz.com/learn/milvus-notebooks)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

- [Safeguarding Data: Security and Privacy in Vector Database Systems](https://zilliz.com/learn/safeguarding-data-security-and-privacy-in-vector-database-systems)

- [Top 5 reasons to migrate from Open Source Milvus to Zilliz Cloud](https://zilliz.com/blog/top-5-reasons-to-migrate-milvus-to-zilliz-cloud)




large_Enabling Fine-Grained Access Control with Milvus Row-Level RBAC-20241127-121804 (1).png

medium_Enabling Fine-Grained Access Control with Milvus Row-Level RBAC-20241127-121804 (1).png

small_Enabling Fine-Grained Access Control with Milvus Row-Level RBAC-20241127-121804 (1).png

thumbnail_Enabling Fine-Grained Access Control with Milvus Row-Level RBAC-20241127-121804 (1).png

Enabling Fine-Grained Access Control with Milvus Row-Level RBAC-20241127-121804 (1).png

Enabling Fine-Grained Access Control with Milvus Row-Level RBAC

introduce Llama 3.1 and 3.2 and explore how to build a RAG app with Llama 3.2 and Milvus. 

large_Introduction to Llama 3.1!.png

medium_Introduction to Llama 3.1!.png

small_Introduction to Llama 3.1!.png

thumbnail_Introduction to Llama 3.1!.png

Introduction to Llama 3.1!.png

Learn Llama 3.2 and How to Build a RAG Pipeline with Llama and Milvus

LoRA (Low-Rank Adaptation) is a technique for efficiently fine-tuning LLMs by introducing low-rank trainable weight matrices into specific model layers.

large_LoRA_ Low-Rank Adaptation of Large Language Models-20241113-102322.png

medium_LoRA_ Low-Rank Adaptation of Large Language Models-20241113-102322.png

small_LoRA_ Low-Rank Adaptation of Large Language Models-20241113-102322.png

thumbnail_LoRA_ Low-Rank Adaptation of Large Language Models-20241113-102322.png

LoRA_ Low-Rank Adaptation of Large Language Models-20241113-102322.png

Efficient LLM Fine-tuning with LoRA (Low-Rank Adaptation)

LoRA Explained: Low-Rank Adaptation for Fine-Tuning LLMs

This blog will guide you through creating a Multimodal RAG with Milvus and vLLM. 

Imagine you've spent months fine-tuning your AI application around a specific [LLM](https://zilliz.com/glossary/large-language-models-(llms)) through an API provider. Then, out of the blue, you receive an email: "We're deprecating the model you're using in favor of our new version." Sound familiar? While cloud API providers offer the convenience of powerful, ready-to-use AI capabilities, relying solely on them also introduces several significant risks:


- **Lack of Control**: You have no control over model versions or updates.
- **Unpredictability**: You may face sudden changes in model behavior or capabilities.
- **Limited Insight**: There's often limited visibility into performance and usage patterns.
- **Privacy Concerns**: Data privacy can be a critical issue, especially when handling sensitive information.

So, what's the solution? How can you take back control? How can you mitigate these risks while enhancing your system's capabilities? The answer lies in building a more robust, independent system using open-source solutions.

This blog will guide you through creating a **Multimodal RAG** with [Milvus](https://zilliz.com/what-is-milvus) and [vLLM](https://zilliz.com/product/integrations/vllm). By leveraging the power of an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) combined open-source LLM inference, you can design a system capable of processing and understanding [multiple types of data](https://zilliz.com/learn/introduction-to-unstructured-data)—text, images, audio, and even videos. This approach not only puts you in complete control of the technology but also ensures a system that's both powerful and versatile, surpassing traditional text-based solutions.


## What We'll Build: a Multimodal RAG Fully Under Your Control

We're going to build a [Multimodal RAG](https://zilliz.com/blog/multimodal-rag-expanding-beyond-text-for-smarter-ai) system using Milvus and vLLM, illustrating how you can [self host your LLM](https://zilliz.com/blog/guide-to-self-hosting-compound-llm-systems) and gain full control over your AI applications. Our tutorial will guide you through creating a Streamlit application that demonstrates the power of integrating multiple data types. Here’s what we’ll cover:

1. Process video input by extracting frames and transcribing the audio

2. Store and efficiently index multimodal data using Milvus

   1. We use [OpenAI CLIP](https://zilliz.com/learn/exploring-openai-clip-the-future-of-multimodal-ai-learning) to encode the images into embeddings that can then be searched with Milvus
   2. We use the [Mistral Embedding](https://docs.mistral.ai/capabilities/embeddings/) model to encode the text into embeddings.

3. Retrieve relevant context based on user queries using Milvus

4. Generate responses using Pixtral running with vLLM, leveraging both visual and textual understanding

By the end of this tutorial, you'll have developed a flexible, scalable system entirely under your control-no more worrying about API deprecations or unexpected changes.


## What is Milvus?

**[Milvus](https://milvus.io/docs/overview.md)** is an open-source, high-performance, and highly scalable vector database that can store, index, and search billion-scale [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data) through high-dimensional [vector embeddings](https://zilliz.com/glossary/vector-embeddings). It is perfect for building modern AI applications such as retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)), semantic search, [multimodal search](https://zilliz.com/blog/multimodal-rag-expanding-beyond-text-for-smarter-ai), and recommendation systems. Milvus runs efficiently across various [environments](https://zilliz.com/blog/choose-the-right-milvus-deployment-mode-ai-applications), from laptops to large-scale distributed systems.


## What is vLLM?

The core idea of vLLM (Virtual Large Language Model) is to optimize the serving and execution of LLMs by utilizing efficient memory management techniques. Here are the key aspects:

- **Optimized Memory Management:** vLLM implements advanced memory allocation and management techniques to fully leverage available hardware resources. This optimization helps to run large language models efficiently, preventing memory bottlenecks that can hinder performance.
- **Dynamic Batching**: vLLM adapts batch sizes and sequences based on the memory and compute capabilities of the underlying hardware. This dynamic adjustment enhances processing throughput and minimizes latency during model inference.
- **Modular Design**: The architecture of vLLM is modular, facilitating straightforward integration with various hardware accelerators. This modularity also enables easy scaling across multiple devices or clusters, making it highly adaptable to different deployment scenarios.
- **Efficient Resource Utilization**: vLLM optimizes the use of critical resources such as CPUs, GPUs, and memory. This efficiency allows the system to support larger models and handle increased numbers of simultaneous requests, which is essential in production environments where both scalability and performance are key.
- **Seamless Integration**: Designed to integrate smoothly with existing machine learning frameworks and libraries, vLLM provides a user-friendly interface. This ensures that developers can easily deploy and manage large language models across a range of applications without extensive reconfiguration.


## Core Components of Our Multimodal RAG

The multimodal RAG app we are building comprises of the following key components:

- **[vLLM](https://github.com/vllm-project/vllm)** is the inference library we will use for the inference and serving of the Pixtral multimodal model.
- **[Koyeb](http://Koyeb.com)** provides the infrastructure layer for our deployment, offering a serverless platform specialized for AI workloads. With [native vLLM integration](https://www.koyeb.com/deploy/vllm) and automated GPU resource management, it makes it easy to deploy LLM while maintaining production-grade performance and scalability.
- **[Pixtral](https://mistral.ai/news/pixtral-12b/)** from **Mistral AI** acts as our multimodal brain, combining a 400M parameter vision encoder with a 12B parameter multimodal decoder. This architecture allows it to process both images and text within the same [context window](https://zilliz.com/glossary/context-window).
- **[Milvus](https://github.com/milvus-io/milvus)** provides the vector storage foundation, efficiently managing embeddings from different modalities. Its ability to handle multiple vector types and perform fast [similarity search](https://zilliz.com/learn/vector-similarity-search) makes it perfect for multimodal applications.

![Figure- The multimodal RAG architecture.png](https://assets.zilliz.com/Figure_The_multimodal_RAG_architecture_3fd1ccb67e.png)


Figure: The multimodal RAG architecture


## Getting Started

First, let's install our dependencies:

```Python
# Core LlamaIndex packages
pip install -U llama-index-vector-stores-milvus llama-index-multi-modal-llms-mistralai llama-index-embeddings-mistralai llama-index-multi-modal-llms-openai llama-index-embeddings-clip llama_index

# Video and audio processing
pip install moviepy pytube pydub SpeechRecognition openai-whisper ffmpeg-python soundfile

# Image processing and visualization
pip install torch torchvision matplotlib scikit-image git+https://github.com/openai/CLIP.git

# Utilities and infrastructure
pip install pymilvus streamlit ftfy regex tqdm
```


## Setting Up the Environment

We'll start by configuring our environment and importing necessary libraries:

```Python
import os
import base64
import json
from pathlib import Path
from dotenv import load_dotenv
from llama_index.core import Settings
from llama_index.embeddings.mistralai import MistralAIEmbedding

# Load environment variables
load_dotenv()

# Configure default embedding model
Settings.embed_model = MistralAIEmbedding(
    "mistral-embed", 
    api_key=os.getenv("MISTRAL_API_KEY")
)
```


## Video Processing Pipeline

The heart of our system is the video processing pipeline, which transforms raw video content into data that our RAG system can understand and process efficiently.

```Python
def process_video(video_path: str, output_folder: str, output_audio_path: str) -> dict:
    # Create output directory if it doesn't exist
    Path(output_folder).mkdir(parents=True, exist_ok=True)
    
    # Extract frames from video
    video_to_images(video_path, output_folder)
    
    # Extract and transcribe audio
    video_to_audio(video_path, output_audio_path)
    text_data = audio_to_text(output_audio_path)
    
    # Save transcription
    with open(os.path.join(output_folder, "output_text.txt"), "w") as file:
        file.write(text_data)
    
    os.remove(output_audio_path)
    return {"Author": "Example Author", "Title": "Example Title", "Views": "1000000"}
```

This pipeline breaks down videos into:

- Image frames (extracted at 0.2 FPS)
- Audio transcription using Whisper
- Metadata about the video


## Building the Vector Index

We use Milvus to store our multimodal embeddings. Here's how we create our index:

```Python
def create_index(output_folder: str):
    # Create different collections for text and images
    text_store = MilvusVectorStore(
        uri="milvus_local.db",
        collection_name="text_collection",
        overwrite=True,
        dim=1024
    )
    
    image_store = MilvusVectorStore(
        uri="milvus_local.db",
        collection_name="image_collection",
        overwrite=True,
        dim=512
    )
    
    storage_context = StorageContext.from_defaults(
        vector_store=text_store, 
        image_store=image_store
    )
    
    # Load and index documents
    documents = SimpleDirectoryReader(output_folder).load_data()
    return MultiModalVectorStoreIndex.from_documents(
        documents, 
        storage_context=storage_context
    )
```


## Query Processing with Pixtral

When a user asks a question, we need to:

1. Retrieve relevant context from our vector store
2. Process the query with Pixtral using both text and images

Here's our query processing function:

```Python
def process_query_with_image(query_str, context_str, metadata_str, image_document):
    client = OpenAI(
        base_url=os.getenv("KOYEB_ENDPOINT"),
        api_key=os.getenv("KOYEB_TOKEN")
    )
    
    with open(image_document.image_path, "rb") as image_file:
        image_base64 = base64.b64encode(image_file.read()).decode("utf-8")
    
    qa_tmpl_str = """
    Given the provided information, including relevant images and retrieved context
    from the video, accurately and precisely answer the query without any
    additional prior knowledge.
    ---------------------
    Context: {context_str}
    Metadata: {metadata_str}
    ---------------------
    Query: {query_str}
    Answer: """
    
    # Prepare messages for Pixtral
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": qa_tmpl_str.format(
                        context_str=context_str,
                        query_str=query_str,
                        metadata_str=metadata_str
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_base64}"
                    }
                },
            ],
        }
    ]
    
    completion = client.chat.completions.create(
        model="mistralai/Pixtral-12B-2409",
        messages=messages,
        max_tokens=300
    )
    return completion.choices[0].message.content
```


## Building the Streamlit Interface

Finally, we create a user-friendly interface with Streamlit:

```Python
def main():
    st.title("MultiModal RAG with Pixtral & Milvus")
    
    # Initialize session state
    if 'index' not in st.session_state:
        st.session_state.index = None
        st.session_state.retriever_engine = None
        st.session_state.metadata = None
    
    # Video input
    video_path = st.text_input("Enter video path:")
    
    if video_path and not st.session_state.index:
        with st.spinner("Processing video..."):
            # Process video and create index
            [... processing code ...]
    
    if st.session_state.index:
        st.subheader("Chat with the Video")
        query = st.text_input("Ask a question about the video:")
        
        if query:
            with st.spinner("Generating response..."):
                # Generate and display response
                [... query processing code ...]

if __name__ == "__main__":
    main()
```


### Running the Application

Before starting the application, ensure you have:

1. Set up your environment variables in `.env`
2. Installed all required dependencies

Then launch the application:

```Bash
streamlit run app.py
```

You'll see the homepage where you can:

- Upload videos for processing
- Ask questions about the content of the video
- Read the responses from Pixtral with relevant video frames

![Figure- The interface of your multimodal RAG app built with Milvus and Pixtral.png](https://assets.zilliz.com/Figure_The_interface_of_your_multimodal_RAG_app_built_with_Milvus_and_Pixtral_34db5892ff.png)


Figure: The interface of your multimodal RAG app built with Milvus and Pixtral

From now on, you can interact with the video and for example, learn more about the the Gaussian Distribution.

![Figure- Performing the multimodal search.png](https://assets.zilliz.com/Figure_Performing_the_multimodal_search_a416ae862d.png)


Figure: Performing the multimodal search


## Conclusion

In this blog post, we've demonstrated how to build a powerful multimodal RAG system using Milvus, Pixtral, and vLLM. Through the combination of Milvus's efficient vector storage capabilities and Pixtral's advanced multimodal understanding, we've created a system that can process, understand, and respond to queries about video content. And this system is fully under your control.


## We'd Love to Hear What You Think!

If you like this blog post, please consider:

- ⭐ Giving us a star on [GitHub](https://github.com/milvus-io/milvus)
- 💬 Joining our [Milvus Discord community](https://discord.gg/FG6hMJStWu) to share your experiences
- 🔍 Exploring our [Bootcamp repository](https://github.com/milvus-io/bootcamp) for more examples of multimodal applications with Milvus




large_Deploying a Multimodal RAG System Using vLLM and Milvus.png

medium_Deploying a Multimodal RAG System Using vLLM and Milvus.png

small_Deploying a Multimodal RAG System Using vLLM and Milvus.png

thumbnail_Deploying a Multimodal RAG System Using vLLM and Milvus.png

Deploying a Multimodal RAG System Using vLLM and Milvus.png

Multimodal RAG, Milvus vector database, vector database, retrieval augmented generation RAG 

Deploying a Multimodal RAG System with vLLM and Milvus

Deploying a Multimodal RAG System Using vLLM and Milvus

Transformers4Rec is a powerful and flexible library designed for creating sequential and session-based recommendation systems with PyTorch.

large_DALL·E 2024-04-01 09.34.43 - Create a colorful yet elegant avatar for a girl writer, presenting a faceless headshot silhouette with feminine features like long, styled hair. The b.webp

medium_DALL·E 2024-04-01 09.34.43 - Create a colorful yet elegant avatar for a girl writer, presenting a faceless headshot silhouette with feminine features like long, styled hair. The b.webp

small_DALL·E 2024-04-01 09.34.43 - Create a colorful yet elegant avatar for a girl writer, presenting a faceless headshot silhouette with feminine features like long, styled hair. The b.webp

thumbnail_DALL·E 2024-04-01 09.34.43 - Create a colorful yet elegant avatar for a girl writer, presenting a faceless headshot silhouette with feminine features like long, styled hair. The b.webp

DALL·E 2024-04-01 09.34.43 - Create a colorful yet elegant avatar for a girl writer, presenting a faceless headshot silhouette with feminine features like long, styled hair. The b.webp

ShriVarsheni R

## Introduction

As digital commerce and online content consumption continue to surge, the demand for effective, personalized recommendation engines is higher than ever. Shoppers and streamers alike expect tailored recommendations, whether they're searching for products, movies, or music. However, traditional recommendation systems, which rely on collaborative or content-based filtering, use static historical data and often struggle to adapt to changing user preferences. To address these limitations, recommendation systems that leverage dynamic AI models that capture evolving user behavior have become essential and popular. 

At a recent [Unstructured Data Meetup](https://zilliz.com/community/unstructured-data-meetup) hosted by Zilliz, [Kunal Sonalkar](https://www.linkedin.com/in/kunalsonalkar/), a Data Scientist at [Nordstrom](https://www.nordstrom.com/), shared insights on using **Transformers4Rec** to enable sequential and session-based recommendations to better serve users’ evolving needs. This blog post highlights key takeaways from Kunal’s presentation, with our perspective on the future of recommendation systems. You can also watch [Kunal’s full talk on YouTube](https://youtu.be/i3Ou3OUo3T4?list=PLPg7_faNDlT7SC3HxWShxKT-t-u7uKr--&t=2819) for more details.

<iframe width="560" height="315" src="https://www.youtube.com/embed/i3Ou3OUo3T4?si=al0Cjp0UkrjJJkIo&amp;start=2819" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>


## What is Transformers4Rec? 

[**Transformers4Rec**](https://github.com/NVIDIA-Merlin/Transformers4Rec/) is a powerful and flexible library designed for creating sequential and session-based recommendation systems with PyTorch. By integrating with Transformers, one of the most popular frameworks in natural language processing (NLP), Transformers4Rec bridges between NLP and recommender systems. 

The following figure shows the use of the transformers4Rec library in a recommender system. The input data for the library is a sequence of user interactions—such as items a user browses or adds to their cart during a session. Transformers4Rec processes these interactions to make intelligent predictions on what the user might want next. For example, imagine a customer is browsing an online bookstore. They first look at several science fiction novels, add one to their cart, and later view a few related author pages. Transformers4Rec analyzes this sequence of actions and predicts the customer might be interested in another popular sci-fi title or a related fantasy novel. 

![Figure- How Transformers4Rec works in a recommender system .png](https://assets.zilliz.com/Figure_How_Transformers4_Rec_works_in_a_recommender_system_92c2378bad.png)

_Figure: How Transformers4Rec works in a recommender system_ 

Kunal highlights that to build an effective recommendation system, it’s essential to understand customer intent—such as the type of product they’re interested in and whether they’re price-sensitive—to tailor recommendations accordingly.

For instance, two customers searching for "shoes" may receive different recommendations based on their previous interactions and preferences, like budget or brand preference.

![shoes.png](https://assets.zilliz.com/shoes_0ee056719f.png)



## Transformers4Rec Architecture 

As we mentioned previously, Transformers4Rec’s architecture is designed to adapt transformer models for sequential and session-based recommendations by processing user interaction data in a structured way. Its architecture includes four main components that work together to make predictions: **Feature Aggregation, Sequence Masking, Sequence Processing, and Prediction Head.** 

![Figure- Transformers4Rec Architecture .png](https://assets.zilliz.com/Figure_Transformers4_Rec_Architecture_2b1b62f15f.png)


Figure: Transformers4Rec Architecture 


### Feature Aggregation

The **Feature Aggregation** component gathers data about user interactions—both continuous and categorical variables—to build a comprehensive profile for recommendation. For example, in an e-commerce recommendation system, categorical features might include attributes like brand and product type, while continuous features might capture price, discount rates, or recent interactions (e.g., the number of clicks on a product in the last 24 hours).

![Figure- Feature Aggregation in Transformers4Rec.png](https://assets.zilliz.com/Figure_Feature_Aggregation_in_Transformers4_Rec_a37529db6c.png)


Figure: _Feature Aggregation in Transformers4Rec_

Before feeding these sequences into a transformer block, the interaction data needs to be pre-processed into a standardized format called [vector embeddings](https://zilliz.com/glossary/vector-embeddings) (numerical representations of interaction data in a high-dimensional space). Inputs can be organized by user ID or session ID, with the sequence of interactions aggregated into a single vector representation, known as an "interaction embedding." This embedding contains contextual information about user interactions, such as the type of user actions, items involved, and interaction order, and can be stored in a [vector database](https://zilliz.com/learn/what-is-vector-database) for efficient retrieval and [semantic search](https://zilliz.com/glossary/semantic-search). A robust vector database like [Milvus](https://milvus.io/docs/overview.md) or its managed service [Zilliz Cloud](https://zilliz.com/cloud), which integrates well with [PyTorch](https://zilliz.com/product/integrations/pytorch), can streamline this process. Additionally, Transformers4Rec includes an NVTabular module to easily convert raw data into the required format, making data preparation more manageable. 


## Sequence Masking

The **Sequence Masking** component in Transformers4Rec applies masks to user interactions to prevent data leakage and maintain sequence order, ensuring the model respects the timeline of events. For instance, when predicting a user’s next item, this module hides any future actions, allowing the model to learn in a causal, step-by-step manner.

Different types of sequence masking in Transformers4Rec control what the model "sees" during training or inference, which helps it capture meaningful patterns in user behavior and improves its ability to predict the next interaction. Transformers4Rec provides multiple options in sequence masking:

- **Causal Language Modeling (CLM)**: This technique masks future items in a sequence, training the model to predict each item based only on prior items. CLM is ideal for sequential recommendations where the order of interactions is crucial.

- **Masked Language Modeling (MLM)**: In MLM, certain positions in the sequence are randomly masked, and the model learns to predict the hidden items using the surrounding context. This technique helps the model understand item relationships without depending on strict order.

- **Random Token Detection (RTD)**: RTD replaces a random item in the sequence with an unrelated one, and the model learns to spot this incorrect item. This method teaches the model to distinguish genuine patterns from random noise or anomalies.

- **Permutation Language Modeling (PLM)**: Here, items in the sequence are shuffled, and the model is trained to predict items in the new, permuted order. PLM encourages flexibility by helping the model recognize patterns in any order.

![Figure-  Difference in Casual LM and Masked LM during Sequence Masking.png](https://assets.zilliz.com/Figure_Difference_in_Casual_LM_and_Masked_LM_during_Sequence_Masking_ee437de1d7.png)


_Figure:  Difference in Casual LM and Masked LM during Sequence Masking_

Usually, 20-30% of the positional embeddings are masked before training. The masking probability also plays a role in the robustness of the recommendation system. 

These masking techniques allow the model to learn from user interactions in various ways, enhancing its ability to make accurate, context-aware recommendations.


### Sequence Processing

The **Sequence Processing** component feeds input features to transformer models to predict masked items or positions in a user’s interaction sequence. Transformers4Rec supports various architectures for sequence processing, including XLNet, GPT-2, and LSTM, allowing users to choose the most suitable model for their recommendation system.

The interaction vector embeddings are passed into the transformer block, as illustrated in the code snippet below. Users can configure the transformer’s architecture by specifying parameters such as the number of layers and the sequence length. A sequential block in PyTorch can be created by passing the aggregated inputs, applying masking, and enabling the desired model configuration, enabling flexible setup for tailored recommendation tasks.

![ Sequence processing in Pytorch .png](https://assets.zilliz.com/Fig_Sequence_processing_in_Pytorch_e03f5bec60.png)


    Fig) _Sequence processing in Pytorch_ 


### Prediction Head

The Prediction Head is the final module in the Transformers4Rec framework, where outputs from the transformer model are used to generate actionable recommendations. 

![Figure- Outputs from the transformer model are used to generate actionable recommendations. .png](https://assets.zilliz.com/Figure_Outputs_from_the_transformer_model_are_used_to_generate_actionable_recommendations_225f6c279c.png)


_Figure: Outputs from the transformer model are used to generate actionable recommendations._ 

This module supports various use cases tailored to business needs, including:

- **Next Item Prediction**: This option predicts the next item a user is likely to interact with based on their previous actions, helping deliver timely and relevant recommendations.

- **Binary Classification**: Here, the model calculates the probability that a user will click on a specific item. This is valuable for click-through rate (CTR) prediction, as it helps determine which items are most likely to engage users, aiding in layout and content optimization.

- **Regression**: Regression tasks enable the prediction of continuous values, such as the time a user might spend on an item, the number of interactions expected, or even an estimated purchase amount.

The diagram below illustrates the end-to-end production pipeline, including all four components—Feature Aggregation, Sequence Masking, Sequence Processing, and the Prediction Head—working together to deliver accurate and context-aware recommendations.  

![Figure-  End-to-End pipeline of a recommendation system built using Transformers4Rec.png](https://assets.zilliz.com/Figure_End_to_End_pipeline_of_a_recommendation_system_built_using_Transformers4_Rec_f5d1d58b98.png)


_Figure:  End-to-End pipeline of a recommendation system built using Transformers4Rec_


## Evaluation and Challenges for Using Transformers4Rec

Building a recommendation system with Transformers4Rec requires both effective evaluation and strategic planning to overcome scalability and maintenance challenges.


### System Evaluation 

The effectiveness of a recommendation system depends largely on how accurately it meets user needs. For Transformers4Rec, common evaluation metrics include precision and recall, which measure how well the system’s top ‘N’ recommendations align with user preferences. Additionally, ranking metrics like Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) are also used to assess the relevance and order of recommended items.


### Challenges

Scaling a recommendation system powered by Transformers4Rec comes with significant computational costs, particularly when deploying at large scale with GPU resources. Building and maintaining such a system also demands extensive infrastructure, covering everything from data collection and storage to model deployment and real-time inference.

Choosing the right infrastructure is crucial to creating a system that can quickly retrieve relevant recommendations. For example, using a [vector database](https://zilliz.com/learn/what-is-vector-database) like [Milvus](https://milvus.io/) allows for fast [similarity searches](https://zilliz.com/learn/vector-similarity-search) and efficient storage of vector embeddings, which integrate well into the recommendation pipeline.

Another key challenge is handling new or frequently changing product catalogs. Transformers4Rec needs to provide accurate recommendations for these new items, even with limited historical data. This requires adaptability in both the model and the data infrastructure to update recommendations based on fresh context.


## Summary 

In this blog, we discussed how Transformers4Rec uses NLP-inspired techniques to create dynamic, personalized recommendation systems. With components like Feature Aggregation, Sequence Masking, Sequence Processing, and the Prediction Head, the framework captures complex user patterns to deliver real-time, relevant recommendations. Key metrics, including precision, recall, MAP, and NDCG, help evaluate system effectiveness, ensuring recommendations meet user needs.

We also discussed the challenges of scaling Transformers4Rec, particularly around infrastructure costs and storage needs, where tools like the [Milvus](https://milvus.io/) vector database support efficient vector storage and retrieval. Fine-tuning parameters, such as vector dimensions and sequence length, is a key technique that optimizes model performance, making Transformers4Rec a flexible and powerful solution for modern recommendation systems.


## Further Reading

- [Transformers4Rec GitHub repo](https://github.com/NVIDIA-Merlin/Transformers4Rec)

- [Mamba: A Potential Transformer Replacement ](https://zilliz.com/learn/mamba-architecture-potential-transformer-replacement)

- [Understanding Transformer Models Architecture and Core Concepts](https://zilliz.com/learn/decoding-transformer-models-a-study-of-their-architecture-and-underlying-principles)

- [Efficient Vector Search in RecSys with Milvus and NVIDIA Merlin](https://zilliz.com/blog/efficient-vector-similarity-search-recommender-workflows-using-milvus-nvidia-merlin)

- [Transforming Ad Recommendations: SmartNews's Journey with Milvus](https://zilliz.com/blog/transforming-ad-recommendations-smartnews-journey-with-milvus)

- [Unlock Advanced Recommendation Engines with Milvus' New Range Search](https://zilliz.com/blog/unlock-advanced-recommendation-engines-with-milvus-new-range-search)

- [Create a Movie Recommendation Engine with Milvus and Python ](https://zilliz.com/blog/create-a-movie-recommendation-engine-with-milvus-and-python)


large_Transformers4rec_ Harnessing NLP Advancements  for Cutting-Edge Recommender Systems.png

medium_Transformers4rec_ Harnessing NLP Advancements  for Cutting-Edge Recommender Systems.png

small_Transformers4rec_ Harnessing NLP Advancements  for Cutting-Edge Recommender Systems.png

thumbnail_Transformers4rec_ Harnessing NLP Advancements  for Cutting-Edge Recommender Systems.png

Transformers4rec_ Harnessing NLP Advancements  for Cutting-Edge Recommender Systems.png

Transformers4Rec: Bringing NLP Power to Modern Recommendation Systems

Vector Database Stories