Blog
Elasticsearch vs Vald Selecting the Right Database for GenAI Applications

Elasticsearch vs Vald Selecting the Right Database for GenAI Applications

Nov 23, 20249 min read

As AI-driven applications evolve, the importance of vector search capabilities in supporting these advancements cannot be overstated. This blog post will discuss two prominent databases with vector search capabilities: Elasticsearch and Vald. Each provides robust capabilities for handling vector search, an essential feature for applications such as recommendation engines, image retrieval, and semantic search. Our goal is to provide developers and engineers with a clear comparison, aiding in the decision of which database best aligns with their specific requirements.

What is a Vector Database?

Before we compare Elasticsearch vs Vald let's first explore the concept of vector databases.

A vector database is specifically designed to store and query high-dimensional vectors, which are numerical representations of unstructured data. These vectors encode complex information, such as the semantic meaning of text, the visual features of images, or product attributes. By enabling efficient similarity searches, vector databases play a pivotal role in AI applications, allowing for more advanced data analysis and retrieval.

Common use cases for vector databases include e-commerce product recommendations, content discovery platforms, anomaly detection in cybersecurity, medical image analysis, and natural language processing (NLP) tasks. They also play a crucial role in Retrieval Augmented Generation (RAG), a technique that enhances the performance of large language models (LLMs) by providing external knowledge to reduce issues like AI hallucinations.

There are many types of vector databases available in the market, including:

Purpose-built vector databases such as Milvus, Zilliz Cloud (fully managed Milvus)
Vector search libraries such as Faiss and Annoy.
Lightweight vector databases such as Chroma and Milvus Lite.
Traditional databases with vector search add-ons capable of performing small-scale vector searches.

Elasticsearch is a search engine based on Apache Lucene with vector search as an add-on. Vald is a purpose-built vector database. This post compares their vector search capabilities.

Elasticsearch: Overview and Core Technology

Elasticsearch is an open source search engine built on top of the Apache Lucene library. It’s known for real time indexing and full text search so it’s a go to search for heavy applications and log analytics. Elasticsearch lets you search and analyse large amounts of data fast and efficiently.

Elasticsearch was built for search and analytics, with features like fuzzy searching, phrase matching and relevance ranking. It’s great for scenarios where complex search queries and real time data retrieval is required. With the rise of AI applications, Elasticsearch has added vector search capabilities so it can do similarity search and semantic search, which is required for AI use cases like image recognition, document retrieval and Generative AI.

Vector Search

Vector search is integrated in Elasticsearch through Apache Lucene. Lucene organises data into immutable segments that are merged periodically, vectors are added to the segments the same way as other data structures. The process involves buffering vectors in memory at index time, then serializing these buffers as part of segments when needed. Segments are merged periodically for optimization, and searches combine vector hits across all segments.

For vector indexing, Elasticsearch uses the HNSW (Hierarchical Navigable Small World) algorithm which creates a graph where similar vectors are connected to each other. This is chosen for its simplicity, strong benchmark performance and ability to handle incremental updates without requiring complete retraining of the index. The system performs vector searches typically in tens or hundreds of milliseconds, much faster than brute force approaches.

Elasticsearch’s technical architecture is one of its biggest strengths. The system supports lock free searching even during concurrent indexing and maintains strict consistency across different fields when updating documents. So if you update both vector and keyword fields, searches will see either all old values or all new values, data consistency is guaranteed. While the system can scale beyond available RAM, performance optimizes when vector data fits in memory.

Beyond the core vector search capabilities, Elasticsearch provides practical integration features that makes it super valuable. Vector searches can be combined with traditional Elasticsearch filters, so you can do hybrid search that mixes vector similarity with full text search results. The vector search is fully compatible with Elasticsearch’s security features, aggregations and index sorting, so it’s a complete solution for modern search use cases.

Vald: Overview and Core Technology

Vald is a powerful tool for searching through huge amounts of vector data really fast. It's built to handle billions of vectors and can easily grow as your needs get bigger. The cool thing about Vald is that it uses a super quick algorithm called NGT to find similar vectors.

One of Vald's best features is how it handles indexing. Usually, when you're building an index, everything has to stop. But Vald is smart - it spreads the index across different machines, so searches can keep happening even while the index is being updated. Plus, Vald automatically backs up your index data, so you don't have to worry about losing everything if something goes wrong.

Vald is great at fitting into different setups. You can customize how data goes in and out, making it work well with gRPC. It's also built to run smoothly in the cloud, so you can easily add more computing power or memory when you need it. Vald spreads your data across multiple machines, which helps it handle huge amounts of information.

Another neat trick Vald has is index replication. It stores copies of each index on different machines. This means if one machine has a problem, your searches can still work fine. Vald automatically balances these copies, so you don't have to worry about it. All of this makes Vald a solid choice for developers who need to search through tons of vector data quickly and reliably.

Key Differences

When choosing between Elasticsearch and Vald for vector search, your choice will depend on your use case. Let’s compare these across the areas that matter most to engineering teams.

Core Search Technology

Elasticsearch uses the HNSW (Hierarchical Navigable Small World) algorithm for vector search. HNSW creates a graph connecting similar vectors, makes searches faster and more efficient. Searches are in milliseconds, with strong consistency across field updates.

Vald takes a different approach, using the NGT algorithm for vector similarity search. It’s designed for billions of vectors, with the architecture focused on vector operations.

Data Management and Indexing

Elasticsearch is good at handling data. It combines full-text search with vector capabilities, so you can mix vector similarity searches with keyword filters. The system uses immutable segments for data organisation, merges them periodically for optimisation.

Vald is focused on distributed indexing. Its main feature is the ability to update indices across multiple machines without stopping searches. This means you can search while you update your data, which is very useful for applications that need to be always up.

Integration and Use Cases

Elasticsearch is good if you need both traditional search and vector search. Its vector search integrates well with existing Elasticsearch features like security, aggregations and index sorting. If you’re already using Elasticsearch or need a full search solution then Elasticsearch is a good choice.

Vald is for vector search integration through gRPC. It’s built for cloud and focused on vector operations. If your main need is pure vector search at scale then Vald’s specialisation might be more suitable.

Scaling and Reliability

Elasticsearch is best when vector data fits in memory, but can scale beyond RAM. It’s consistent during updates and supports concurrent operations without locking.

Vald uses index replication across machines for reliability. It manages data distribution and balancing automatically, so it’s resilient to machine failures. This architecture is good for large scale vector operations.

When to Choose Elasticsearch vs Vald

Elasticsearch: For Combined Search Needs

Elasticsearch is best when you need a full search solution that handles both traditional and vector search. It’s the right choice if your app needs text search, structured data queries and vector similarity search all together. The platform is good for companies that already use the ELK Stack for logging or search, need strong data consistency or want to combine vector search with text filtering. Examples are e-commerce platforms using image similarity with text filters, content recommendation systems mixing semantic and keyword search or document retrieval systems that need both vector embeddings and full-text search.

Vald: For Pure Vector Search at Scale

Vald is the better choice when you need to handle huge amounts of vector data efficiently. It’s built for applications that need pure vector similarity search at scale, especially when continuous indexing is critical. Vald is good for scenarios where you have billions of vectors, need to update indices without downtime or want a system that handles distributed operations and failovers automatically. This is good for large-scale image similarity search, real-time recommendation engines or any application where vector operations are the main requirement.

Conclusion

The choice between Elasticsearch and Vald is yours. Elasticsearch is a full platform that combines traditional search with vector capabilities so it’s good for applications that need both. Vald is a specialized solution for pure vector search at scale with strong distributed capabilities. Your choice should be based on whether you need a search platform with vector capabilities (Elasticsearch) or a vector search solution (Vald). Consider your existing infrastructure, the scale of vector operations you need and whether you need more search features beyond vector similarity.

Read this to get an overview of Elasticsearch and Vald but to evaluate these you need to evaluate based on your use case. One tool that can help with that is VectorDBBench, an open-source benchmarking tool for vector database comparison. In the end, thorough benchmarking with your own datasets and query patterns will be key to making a decision between these two powerful but different approaches to vector search in distributed database systems.

Using Open-source VectorDBBench to Evaluate and Compare Vector Databases on Your Own

VectorDBBench is an open-source benchmarking tool for users who need high-performance data storage and retrieval systems, especially vector databases. This tool allows users to test and compare different vector database systems like Milvus and Zilliz Cloud (the managed Milvus) using their own datasets and find the one that fits their use cases. With VectorDBBench, users can make decisions based on actual vector database performance rather than marketing claims or hearsay.

VectorDBBench is written in Python and licensed under the MIT open-source license, meaning anyone can freely use, modify, and distribute it. The tool is actively maintained by a community of developers committed to improving its features and performance.

Download VectorDBBench from its GitHub repository to reproduce our benchmark results or obtain performance results on your own datasets.
Take a quick look at the performance of mainstream vector databases on the VectorDBBench Leaderboard.
Read the following blogs to learn more about vector database evaluation.

Further Resources about VectorDB, GenAI, and ML

Chloe Williams
Chloe Williams is a technical writer at Zilliz.

Content

Start Free, Scale Easily

Try the fully-managed vector database built for your GenAI applications.

Try Zilliz Cloud for Free