Learn
How to Pick the Right Vector Database for Your Use Case

Introducing an Open Source Vector Database Benchmark Tool for Choosing the Ideal Vector Database for Your Project

Jun 16, 20236 min read

Introducing VectorDBBench, an open-source vector database benchmark tool for choosing the ideal vector database for your project.

By Li Liu

Read the entire series

Selecting a suitable vector database for your project is crucial. With various options available, making an informed decision requires a comprehensive performance evaluation. That's where our new open source vector database benchmark tool comes into play. This tool was designed specifically for vector databases and enables you to unlock the true potential of your preferred database by measuring its performance across various critical metrics. This blog will explore this vector database benchmarking tool's exciting features and benefits, empowering you to make confident decisions for your data-driven projects.

A little background on vector database benchmark tools

When I thought about building a vector benchmarking tool, I needed to consider the purpose of conducting a benchmark. Benchmarking aims to measure and compare the performance of a system, application, or component under different circumstances. Developers can use it to evaluate the effectiveness and efficiency of different approaches and identify improvement areas.

Defining precise requirements based on your use case is crucial to conducting a meaningful benchmark. For instance, if you are working with large datasets, you might be interested in how many vectors can be ingested or search performance (how fast you can retrieve the relevant data). Additionally, you should test the filtering performance to see how well the system handles complex queries with large datasets.

Finally, it is crucial to consider the shape of your data. The shape of the data refers to the number of vectors and the dimensions of the data that you plan to store and use in a vector database. When you are designing your vector database benchmark, the shape of your data can influence the vector database’s performance.

With these things in mind, we developed a tool to help developers quickly benchmark available vector stores and tailor the benchmark to their specific needs.

Design goals for our open-source vector database benchmarking

Here are some of the design goals that we considered when building an open source vector database benchmarking tool.

Flexible and Extensible: The vector database benchmarking tool should be flexible and extensible. It should support multiple vector database systems, allowing you to benchmark and compare different options effortlessly. Moreover, the tool should have a modular architecture to enable the addition of more vector databases, metrics, and custom test scenarios, empowering you to tailor the evaluation according to your specific requirements.

Realistic Workload Simulation: A vector database benchmarking tool should leverage your own workloads which can serve as a realistic workload simulation to ensure accurate performance evaluation. Simulating your real-world use cases and query patterns provides insights into the database's behavior under various scenarios. This simulation helps you gauge how well your vector database performs in practical situations and determine its suitability for your project.

Interactive Reports and Visualization: This tool should have an intuitive system that generates reports and visualizations that allow for easy identification of performance bottlenecks, comparison of databases, and uncovering optimization opportunities. These reports will be valuable decision-making resources and facilitate effective team communication.

Open Source Community Collaboration: This tool should be open-source to foster collaboration among the community of vector database users and developers. By sharing insights, best practices, and performance results, the community collectively contributes to improving and refining this tool and, ultimately, helping the developer to choose the right tool for the job.

Introducing VectorDBBench — the open-source vector database benchmarking tool

VectorDBBench is an open-source benchmarking tool designed for users who require high-performance data storage and retrieval systems. This tool allows users to test and compare different vector database systems' performance to determine their specific use case's most suitable database system. Using VectorDBBench, users can make informed decisions based on the actual vector database performance of the systems they are evaluating rather than relying on marketing claims or anecdotal evidence.

Vector DB Bench is written in Python and licensed under the MIT open-source license, meaning anyone can freely use, modify, and distribute it. The tool is actively maintained by a community of developers committed to improving its features and performance.

How to get started

Download VectorDBBench from GitHub and install using the following:

pip install vectordb-bench

then Run it

init_bench

The first screen you will see is the Vector Database Benchmark page. This page shows the results of tests already conducted for the current month. From this page, you can link to the QPS with Pricing page to see the results sorted by the retail pricing for the cloud services. These tests already have a comprehensive test across different size datasets.

To perform your own tests, you can go to the Run Your Test page to set that up.

How to set it up for your own tests

To use the tool for your own tests, go to the Run Your Test page and select the vector databases that you want to test, and add the configurations of those vector databases. Depending on the database, this will include a uri, username, password, and db label. The tool currently supports six vector databases: Milvus, Zilliz, Pinecone, Weaviate, Qdrant, and Elasticsearch. You then will set up the type of test you want to run (Capacity or Search Performance), the index type, the Use Case (Search, Low or High Filtering), and the Dataset size (Small, Medium, and Large).

Configuring VBBench | Zilliz.png

After selecting the desired configurations, you can run the test and wait for the results.

Search Results.png

On the results page, you can view the outcome of the test. If you selected more than one database for testing, you will see the results presented for vector database comparison. Since the test runs on your local instance, sharing the results will be up to your discretion. Learn more about open source vector databases and the benefits of vector search.

Community

I'd love to hear your thoughts on this tool and you can join us in GitHub or on the Vector DB Bench slack channel.

Conclusion on open source vector database benchmarks

Choosing the suitable vector database for your project is a critical decision that can significantly impact your data management and analysis. With our open-source vector database benchmarking tool, you can now evaluate the performance of your favorite vector databases comprehensively, objectively, and efficiently. With accurate insights and metrics, you can confidently select the vector database that best aligns with your project's requirements and scale your data-driven initiatives to new heights. Embrace the power of performance evaluation and unlock the full potential of your vector database performance with our benchmarking tool.

Li Liu
Li Liu is the Principal Engineer at Zilliz, leading the vector searching research and development. Before joining Zilliz, he was a Senior Engineer at Meta, designing and shaping numerous advertising stream data frameworks. With a Master's degree from Carnegie Mellon University, he boasts extensive experience in databases and big data. Li Liu's expertise in technology and innovation continues to drive advancements in vector searching, leaving a lasting impact on the field.

Next: How to Choose A Vector Database: Elastic Cloud vs. Zilliz Cloud

Content

Start Free, Scale Easily

Try the fully-managed vector database built for your GenAI applications.

Try Zilliz Cloud for Free