# Milvus Reference Architectures Source: https://zilliz.com/blog/milvus-reference-architectures Author: Christy Bergman Published: 2024-05-10T03:00:00.000Z This blog addresses some commonly asked questions regarding [Milvus](https://zilliz.com/what-is-milvus) resource allocation based on specific use cases. Those questions include:  - How much CPU and memory resources are needed for Milvus, based on a specific number of users or requests per second (RPS)? - How much CPU and memory resources are needed for Milvus, based on different mixes of READ and WRITE? ## Understanding Your Workload Characteristics The first step in allocating resources to Milvus is to understand your workload characteristics. These factors play a crucial role in determining Milvus' computational power and memory requirements. Below is an [example list](https://docs.gitlab.com/ee/administration/reference_architectures/) of Linux package-based reference architectures, where RPS means Requests Per Second: - [Up to 20 RPS or 1,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/1k_users.html) API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS - [Up to 40 RPS or 2,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html) API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS - [Up to 60 RPS or 3,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/3k_users.html) API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS - [Up to 100 RPS or 5,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/5k_users.html) API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS - [Up to 200 RPS or 10,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/10k_users.html) API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS - [Up to 500 RPS or 25,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/25k_users.html) API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS - [Up to 1000 RPS or 50,000 users](https://docs.gitlab.com/ee/administration/reference_architectures/50k_users.html) API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS ## Estimating Resource Requirements To estimate the resource requirements for Milvus, we need to make a few assumptions: 1. Reads: Each web request and Git pull is a READ operation. 2. Writes: Each Git push is considered a WRITE operation. 3. Volume and ratio of reads/writes: Milvus are assumed to be the same as the read/write ratio of API calls per number of users. 4. Queries per second (QPS): must match the API RPS (requests per second) requirement per number of users. We also need to estimate the data size per read/write request. We’ll assume a common GenAI use case: - Vector dimension: 1024 floating point numbers - Size in bytes per floating point number: 4 KB - Top\_k (number of returned vectors): 10 vectors per search request - Size of a collection (database table): 1 million vectors per write request - Database index type: [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) Based on these assumptions, we can do back-of-napkin math to estimate the data size per read or write. Assume the vector dimension is 1024, and each vector takes 1024 * 4 bytes = 4 KB.  Assume a typical top_k = 10 vectors per read. With these assumptions: - **Each Milvus read operation processes around 40 KB of data.**  - **Each Milvus write operation is estimated to involve 40 MB of data.** Milvus offers both insert (create an entirely new collection) and upsert (modify a few rows) functionalities (See the blog [Milvus insert, upsert, delete](https://milvus.io/docs/insert-update-delete.md) for more information).  We’ll over-estimate each WRITE operation to be a whole collection insert rather than just a few rows upserts. Databases need to consider not only data size but search and insert speed.  We will assume the collection is indexed using the popular [HNSW index](https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md), which has Big-O notation, O(log n), search time. With these assumptions, here is our conversion of Web-based users/RPS/reads/writes to Vector Database QPS/data size architecture tiers: - Up to 1,000 users = 20 QPS with 1 million vectors - Up to 2,000 users = 40 QPS with 1 million vectors - Up to 3,000 users = 60 QPS with 1 million vectors - Up to 5,000 users = 100 QPS with 2 million vectors - Up to 10,000 users = 200 QPS with 4 million vectors - Up to 25,000 users = 500 QPS with 10 million vectors - Up to 50,000 users = 1000 QPS with 20 million vectors ## Load Testing and Benchmarking To ensure the accuracy of our resource estimations, we load-tested and benchmarked the architecture tiers on [VectorDBBench](https://zilliz.com/vector-database-benchmark-tool?database=ZillizCloud%2CMilvus%2CElasticCloud%2CPgVector%2CPinecone%2CQdrantCloud%2CWeaviateCloud\&dataset=medium\&filter=none%2Clow%2Chigh\&tab=1). We assumed the default Segment, Partition, Shard, Data node, Query node, and Index node sizes for the [Milvus architecture](https://milvus.io/docs/architecture_overview.md) itself. Due to Milvus's autoscaling capabilities, performance is linear with the data size and cluster resources! Below is a table showing the recommended Milvus and [Zilliz Cloud](https://zilliz.com/cloud) (the fully managed Milvus) resource sizes for different data capacities and QPS requirements.   The table below shows data capacity in millions of 1024_dimension vectors. Milvus resources are given in several CPUs and GB of memory.  For cost comparison, we show [Zilliz Cloud resource sizes](https://docs.zilliz.com/docs/cu-types-explained#evaluate-performance), given in Compute Units (cu), in either performance or capacity types. **Table of recommended Milvus and Zilliz resource sizes per User/RPS tiers** | | | | | | | | ------- | ------------------- | --------------- | ------------ | ---------------- | --------------- | | Users | Data capacity | QPS benchmarked | RPS required | Milvus resource | Zilliz resource | | 3,000 | 1m\_1024d vectors | 1200 | 60 | 8CPU, 32G | 1cu-perf | | 3,000 | 1m\_1024d vectors | 2400 | 60 | 16CPU, 64G | 2cu-perf | | 3,000 | 1m\_1024d vectors | 3600 | 60 | 24CPU, 96G | 4cu-perf | | 10,000 | 3.7m\_1024d vectors | 360 | 200 | 16CPU, 64G | 2cu-cap | | 10,000 | 3.7m\_1024d vectors | 700 | 200 | 64CPU, 256G | 4cu-cap | | 25,000 | 10m\_1024d vectors | 600 | 500 | 196CPU, 768G | 12cu- cap | | 250,000 | 100m\_1024d vectors | 6000 | 5000 | 19200CPU, 76800G | 1200cu- cap | _Table of recommended Milvus and Zilliz resource sizes per number of Users/RPS tiers. Milvus scaling is linear with respect to data size and required QPS._ From the table above, when data size and QPS need to reach a certain threshold, it could be more cost-effective to run Milvus from Zilliz Cloud instead of on-premises. ## Conclusion By understanding your workload characteristics, estimating resource requirements based on assumptions, and leveraging load testing and benchmarking tools such as [VectorDBBench](https://zilliz.com/vector-database-benchmark-tool?database=ZillizCloud%2CMilvus%2CElasticCloud%2CPgVector%2CPinecone%2CQdrantCloud%2CWeaviateCloud\&dataset=medium\&filter=none%2Clow%2Chigh\&tab=1), you can confidently provision the necessary resources for your Milvus deployment.  Refer to our cluster [sizing guide](https://medium.com/@zilliz_learn/demystifying-the-milvus-sizing-tool-2c0afe7fe963) for a deeper dive.  Remember, as your workload evolves, it's essential to regularly review and adjust your resource allocation to maintain peak performance. ## References - ​​HNSW:  - Milvus architecture:  - Blog Milvus Packaging Dependencies: - Blog Milvus Sizing Tool:   - Shards, Partitions, Segments: - Zilliz Cloud CU types:  - [VectorDBBench Tool](https://zilliz.com/vector-database-benchmark-tool?database=ZillizCloud%2CMilvus%2CElasticCloud%2CPgVector%2CPinecone%2CQdrantCloud%2CWeaviateCloud\&dataset=medium\&filter=none%2Clow%2Chigh\&tab=1) for Benchmarking Milvus, Zilliz Cloud, and many other maintream vector databases