Learn
How to Pick the Right Vector Database for Your Use Case

Benchmarking Vector Database Performance: Techniques and Insights

Mar 15, 20246 min read

This article digs explicitly into the key evaluation metrics and benchmarking tools for vector databases. Additionally, it offers insights to aid in evaluating vector databases for informed decision-making.

By Min Tian

Read the entire series

Introduction

Today, the growth of unstructured data and the rise of AI and LLMs have highlighted vector databases as a crucial infrastructure. As the focus shifts to these tools, how do you assess and select the right one for your business? This article digs explicitly into the key evaluation metrics and benchmarking tools for vector databases. Additionally, it offers insights to aid in evaluating vector databases for informed decision-making.

Understanding Vector Databases

Vector databases are designed for managing unstructured data like images, videos, texts, and audio using high-dimensional numerical representations called vector embeddings. The key differences between vector and traditional relational databases include:

Traditional databases handle structured or semi-structured data with fixed formats. In contrast, vector databases manage unstructured data using vector embeddings.
Traditional databases conduct precise searches, whereas vector databases specialize in semantic similarity searches using a machine-learning technique called Approximate Nearest Neighbor (ANN).

Vector databases are the vector store for retrieval augmented generation (RAG) applications that address LLMs' hallucinations. They are also widely used in various modern applications, such as recommender systems, chatbots, anomaly detection, semantic search, and video deduplication. Each of these applications will have specific vector database needs that you should consider when reviewing the evaluation metrics. This will help you choose the best tool for your project. Here is a list of some use cases and their requirements:

For product recommendation, high cost-effectiveness is crucial for helping to keep prices down when serving millions of customers, and high performance (fast querying) is necessary for a good user experience.
Research scientists use molecular search to aid in applications like new drug discovery; high performance is not necessary, but querying the database should always return the most accurate results with as little hardware costs as possible.
For applications such as real-time fraud detection, a single true negative or false positive can have dire consequences, so it makes economic sense to maximize performance and accuracy using more expensive hardware (i.e., GPUs/ASICs).

Vector Database Evaluation Metrics

When assessing a vector database, performance, scalability, and functionality are the top three most crucial metrics.

Performance

Like other database systems, performance is critical when evaluating vector databases. Key metrics include insertion capacity and speed, query latency, and maximum throughput (QPS). However, since vector databases conduct approximate rather than precise searches, you need to consider two additional metrics: indexing construction time and recall rate.

Index construction time: the duration needed to build vector indexes
Recall rate: a metric denoting retrieval accuracy.

Building indexes requires significant computational resources, leading to a trade-off between query accuracy and efficiency. Prioritizing accuracy may affect query speed and vice versa. Therefore, balancing both aspects is vital rather than focusing solely on latency and query speed.

Scalability and functionality

Scalability assesses the database's ability to effectively handle rapidly growing data volumes. Functionality evaluates support for enterprise-level features like multi-tenancy, disaster recovery, and multi-index support.

Vector Database Performance Evaluation Tools

In terms of vector database evaluation, two prominent benchmarking tools stand out: ANN Benchmark and VectorDBBench.

ANN Benchmark

ANN-Benchmark is an external benchmark tool for evaluating various vector index algorithms across real datasets. Vector indexing, a pivotal and resource-intensive component of vector databases, directly influences the overall database performance.

The graph below is an example of benchmarking results generated by ANN Benchmark. It demonstrates the results of testing recall/queries per second of various algorithms based on the GIST1M dataset (1M vectors with 960 dimensions). It plots the recall rate on the x-axis against QPS on the y-axis, illustrating each algorithm's performance at different levels of retrieval accuracy.

Recall_Queries_Per_Second_3bde2f0da5.png

VectorDBBench

VectorDBBench is an open-source benchmarking tool tailored for open-source vector databases such as Milvus and Weaviate and fully managed services like Zilliz Cloud and Pinecone. Notably, it provides separate insights into QPS and recall rates, a feature particularly valuable for fully managed vector search services.

The charts below are examples of benchmarking results generated by VectorDBBench. They demonstrate the testing results for QPS and the recall rate of various mainstream vector databases when processing 500,000 vectors with 1,536 dimensions.

Benchmarking_Mainstream_Vector_Databeses_QPS_500_K_cde7b82807.png

Benchmarking_Mainstream_Vector_Databases_Recall_Rate_500_K_ae21e6abbb.png

Note: Many fully managed vector search services do not expose their parameters for user tuning, so VectorDBBench displays QPS and recall rates separately.

ANN Benchmark vs. VectorDBBench

ANN Benchmark excels at evaluating vector index algorithms, aiding in selecting and comparing different vector searching libraries. However, it is unsuitable for assessing complex and mature vector databases and overlooks situations like filtered vector searching.

The engineers at Zilliz created VectorDB Bench to be tailored for comprehensive vector database evaluation. It considers essential factors such as resource consumption, data loading capacity, and system stability. By segregating the test client and vector database and ensuring independent deployment, VectorDB Bench allows for testing that closely mirrors real-world production environments.

Performance Evaluation Tips and Insights

Understanding the nuances of performance evaluation is crucial for effectively assessing vector databases' capabilities. When making your database selection, it is important to consider the methodologies for evaluating both insertion and query performance.

How to Accurately Assess Insertion Performance

Accurately evaluating insertion performance involves an examination of both maximum insertion capacity and insertion time.

To gauge the maximum insertion capacity, use a single process for serial insertion of small data batches until insertion requests are declined. This approach allows the testing client to read original data in manageable batches, alleviating memory constraints and mitigating excessive pressure on the database from multiple writing processes, which could prematurely constrain throughput and distort maximum capacity testing.

Insertion time should encompass the duration from inserting the initial dataset until efficient querying becomes feasible. Notably, constructing vector indexes consumes substantial computational resources, resulting in a time gap between data insertion completion and the database's ability to facilitate efficient querying. Merely tracking the time taken for write requests might lead to misleadingly high write speeds if the database employs aggressive insertion strategies, delaying index construction.

How to Accurately Assess Query Performance

Assessing the query performance of vector databases typically involves three key metrics: latency, queries per second (QPS), and recall rate.

Latency testing measures the time taken for a single query under serial testing conditions. P99 latency is a commonly used metric representing the duration within which 99% of queries are completed. It offers a more nuanced perspective than average latency and aligns closely with user experience.

Important to Note: While latency testing is straightforward, it's heavily influenced by network conditions, especially for cloud products accessed through public networks.

QPS refers to a database's query capability under high concurrency. It is achieved by simultaneously sending multiple requests from the test client to maximize database CPU utilization and observe throughput. Unlike latency, QPS is less susceptible to network fluctuations, providing a comprehensive evaluation of a vector database's real-world performance.

Assessing the recall rate of a vector database is generally straightforward; however, it's important to note that relying solely on recall metrics is insufficient for evaluating query performance.

Dataset Impact on Performance

In real-world tests, performance results differ significantly among various vector databases when exposed to diverse datasets. Larger datasets present more formidable challenges to vector databases' distributed architecture, leading to decreased performance. The dimensionality and distribution of testing datasets also profoundly influence testing results.

Therefore, evaluating a vector database using testing datasets featuring varying data sizes, dimensions, and data distributions can lead to more precise and comprehensive testing outcomes.

Conclusion

This blog explored vector databases and performance evaluation techniques, focusing on critical metrics like insertion capacity and query latency. We discussed ANN Benchmark and VectorDBBench, highlighting their roles in assessing vector indexing algorithms and purpose-built databases. Additionally, we shared insights on accurately evaluating database performance. Armed with these insights, you can confidently navigate the complexities of vector database assessment and ensure optimal performance in today's data-driven landscape.

Min Tian
Software Engineer at Zilliz

Next: Developing Custom Applications with Vector Databases

Content

Start Free, Scale Easily

Try the fully-managed vector database built for your GenAI applications.

Try Zilliz Cloud for Free