Blog
The Cost of Open Source Vector Databases: An Engineer’s Guide to DYI Pricing

The Cost of Open Source Vector Databases: An Engineer’s Guide to DYI Pricing

Apr 08, 20248 min read

As engineers, we often start our projects by tapping into open-source software. For instance, when setting up a Retrieval Augmented Generation (RAG) system, we lean on open-source vector databases such as Milvus, which we can get up and running with a simple pip install. This method is simple and free, making it a no-brainer for us.

Then, there's the allure of cloud services like AWS. For smaller projects, the cost can be surprisingly low, sometimes just a few dollars a month. However, as we scale our projects and our needs become more intricate, our expenses can skyrocket. This is the crux of usage-based billing, which can turn into a significant financial burden as usage intensifies.

For large-scale projects, the discussion often revolves around the decision to manage resources in-house, such as running MinIO, versus relying on services like Amazon S3. These pivotal decisions demand careful consideration. However, we've noticed that not all software engineers and engineering managers invest the necessary time to thoroughly evaluate these options.

Even when a managed service offers an affordable solution, some engineers still prefer to manage their open-source vector database setups. When asked why, the answers vary from the satisfaction derived from hands-on management and the career growth opportunities it presents to a more resigned attitude of "my manager would never sign off on this expense."

Engineering managers' responses are mixed. Some believe more in their team's ability to deliver than in external providers. Often, they need help weighing the pros and cons or justifying the investment required for managed services. For many managers, the familiar routine involves requesting more headcounts for maintenance, not necessarily a budget for managed services.

This habit highlights a broader issue within our field. Despite over a decade of cloud services' widespread use, we're still navigating how best to leverage managed services.

So, How Much Does An Open Source Vector Database Really Cost?

Start With Some Fairly Obvious and Easy-To-Quantify Expenses

When we dive into the world of running an open-source vector database like Milvus in some production format, the initial thrill of "free software" quickly gets a reality check by hardware costs. Let's break it down into two main hardware areas you must consider.

First up is the backbone of a database. Running a distributed database like Milvus isn't just about having the database up and running; it's also about setting up a dependency that supports its running. Before setting up Milvus, you need to figure out WAL deployment (with options like Kafka or Pulsar), secure metadata storage (hello, etcd), and orchestrate the whole shebang with Kubernetes. Remember the load balancer to manage traffic, plus monitoring and logging tools to keep everything in check. If your project is smaller, these components can significantly affect your hardware budget. It's like setting up a mini data center, and even though we love tinkering, the costs can add an extra layer of challenge.

Then, there's the core of the operation. The vector database costs themselves. Setting up EC2 instances (or their equivalents) for worker nodes is essential and tailored to your specific performance and capacity needs, no matter the scale of your usage. You’ll also need storage solutions like S3 or Azure Blob. Additionally, don’t forget about networking costs because transferring all that data in and out is a given expense.

Some Aspects of Running an Open Source Vector Databases Are More Challenging To Quantify

But that doesn’t mean you should avoid considering them or that you will end up paying these costs later, whether you want to or not.

It starts with capacity planning. Everyone begins with guesses about capacity—the number of vectors, their dimensions, the metadata volume, and query per second (QPS). But let's be honest: these guesses often miss the mark. Overprovisioning seems like playing it safe, but it locks up resources you might never use. Underprovisioning? That's a fast track to downtime and emergency troubleshooting sessions nobody wants.

Additionally, getting the capacity right involves more than choosing the correct number of instances. At Zilliz, it's about deeply understanding the requirements of diverse use cases and continuously aligning the infrastructure to meet those needs efficiently.

It's not just about choosing the hardware; there's a whole setup phase to consider. Tasks like configuring Kubernetes, scripting with Terraform, developing a GUI, and finalizing your backup and replication strategies are not simple tasks. They consume time and demand a high level of expertise.

Then, there is routine maintenance. Routine maintenance might not be flashy, but skipping it is a gamble. Staying on top of updates, mainly bug fixes and security patches, is non-negotiable. It's not just about keeping your system functional; it's about safeguarding it against known vulnerabilities and ensuring it can effectively support new features.

Another critical operational task is to watch for workload imbalances and be ready to adjust. Proactively managing your resources can prevent performance bottlenecks and save costs in the long run. And when it's time to expand, doing so strategically can keep you from scrambling to scale an already maxed-out system.

Planning when things go wrong is as crucial as the setup itself. You'll need to become very familiar with your choice of open-source vector databases, which helps troubleshoot. Another pro tip is to build a solid disaster recovery plan to ensure you can bounce back with minimal impact.

“Why is my vector database slow?” tax. Even with careful capacity planning and tuning, someone will eventually ask, “Why is my Milvus so slow?” Latency issues—expected at 100 ms but hitting 200 ms, or occasional spikes to 5,000 ms—can be a puzzle. Resolving these isn’t straightforward and leans heavily on having specialized knowledge. Finding and fixing slowdowns becomes even more challenging if your team is stretched thin across Milvus, Kafka, and Elasticsearch. It boils down to a choice: invest in hiring and training experts focused on specific vector databases or brace for the impact of performance issues.

Some Costs Are Nearly Impossible to Quantify

We've covered the straightforward costs, which we can calculate if we know what an engineer's time is worth. However, a whole category of costs is more challenging to pin down. These aren't minor; they can be make-or-break for your project, especially when it comes to something as complicated as a vector database for mission-critical workloads.

Time to Market. Before your app hits production, there's a bunch of prep work—like getting your vector database tuned just right. Delays here can range from a minor annoyance to giving competitors the lead. It's not just about being first but not being left behind.

Engineering Morale and Retention. Here's the straight talk—engineers want to solve problems, not babysit systems. Sure, we expect some on-call duties and maintenance, but that's assuming these tasks are balanced, and we're moving towards automating the tedious stuff. If we're stuck with endless maintenance and no end in sight, that's a fast track to a demotivated and potentially shrinking team. Plus, unhappy engineers aren't just looking for the exit; they're not putting their best into their work.

Risk and Its Ripple Effects. Do you have a team of vector database wizards? Great, your risk is lower but not gone. You could hit near-perfect uptime. But if your team's learning as they go, expect bumps. We're talking about more than just downtime—data loss, security slip-ups, and fines. And downtime isn't just about the immediate hit; it's the recovery slog, the 4 a.m. crisis shifts, and how often you're firefighting instead of improving.

How to Assess Costs in Vector Database Management

After we calculate the straightforward costs and those related to the time engineers spend setting up vector databases like Milvus, we face a bigger question: Should we manage everything on our own, or is it better to use managed services?

It would be best to work on some performance tests to gather data first. A vector database's most critical performance test comes from seeing how it handles real-life workloads. That means setting up test environments that mimic actual operations, pushing them to see how they perform. This step is crucial because it shows us how fast the database can run and how it behaves under stress—information we need to decide if a setup is worth the investment.

After collecting this performance data, we turn it into a straightforward comparison: How much does handling a specific volume of data or a set number of queries per second cost? This method of comparing costs is well-approved in database benchmarking, helping us see clearly which option offers the best value.

Optimizing for Cost

Reducing the cost per query is possible, both from your side and your cloud provider's. One straightforward strategy is to adopt dynamic scaling, which avoids paying for resources you don't use. However, it's worth remembering the challenges, like the potential for under-provisioning, which we've already discussed.

Adjusting the balance between recall accuracy, latency, and throughput according to your project needs can also help manage costs. This involves choosing the right index type for your situation. For instance, DiskANN might be your pick for moderate recall with acceptable latency and throughput, whereas IVF_Flat could be better for high-accuracy scenarios despite its higher latency and lower throughput.

Another approach is to use MMap to store less data in memory, which can save costs but may reduce performance. This choice should align with the demands of your use cases.

At Zilliz, we focus on cost optimizations that fit different use cases. We continuously enhance Zilliz Cloud (the fully managed version of Milvus) with new features released monthly to ensure the best price-performance ratio for your vector database needs.

Making a Smart Economic Choice

Deciding on how to manage our vector database ultimately comes down to looking at the numbers and making an intelligent call based on what's most cost-effective. This means considering everything from the direct costs of running the servers to whether we might need more advanced hardware or if we can achieve our goals more economically through smart engineering.

The key here is to present the options and their costs in a way that's easy to grasp, ensuring that when we discuss these choices with others in our team or with decision-makers, we're talking in clear, plain terms. It's not about avoiding hard work; it's about making sure we're investing our efforts and resources where they'll have the most impact.

Steffi Li
Steffi is the Director of Product Marketing at Zilliz, with prior experience leading the GTM strategies for open-source data technologies such as Apache Kafka and Apache Airflow. Passionate about travel, yoga, and non-fiction reading, she brings a balanced and enriched worldview to her professional pursuits.