VectorDBBench - A Vector Database Benchmark Tool
VectorDBBench provides unbiased vector database benchmark results for mainstream vector databases and cloud services, and it's your go-to tool for the ultimate performance and cost-effectiveness of vector database comparison. Designed with ease-of-use in mind, VectorDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of vector database cloud services and open-source vector databases a breeze.
Prepare to delve into the world of VectorDBBench, and let it guide you in uncovering your perfect vector database match.
System
- PgVector
- ElasticCloud
- Pinecone
- ZillizCloud
- QdrantCloud
- WeaviateCloud
- Milvus
Host/Instance Type
- 2c8g
- upTo2.5c8g
- p2.x1
- p1.x1
- s1.x1
- s1.x1-2node
- p2.x1-8node
- p1.x1-8node
- 1cu-perf
- 8cu-perf
- 1cu-cap
- 2cu-cap
- 2c8g-1node
- 4c16g-1node
- 4c16g-5node
- standard
- bus_crit
- 2c8g-disk
- 4c16g-disk
- 2c8g-hnsw
- 16c64g-hnsw
Metrics
- QPS
- QP$
- Latency
System & MachineQPS ScoreThis is a 1-100 score based on each system's performance in different cases according to a specific rule. A higher score denotes better performance.
- ZillizCloud-8cu-perf99.6039
- Milvus-16c64g-hnsw46.8312
- QdrantCloud-4c16g-5node31.1555
- Pinecone-p2.x1-8node29.2506
- ZillizCloud-2cu-cap17.418
- Milvus-4c16g-disk14.1567
- ZillizCloud-1cu-perf5.284
- Milvus-2c8g-hnsw4.3963
- QdrantCloud-4c16g-1node4.1808
- ZillizCloud-1cu-cap3.7789
- Pinecone-p2.x13.7139
- Pinecone-p1.x1-8node2.5564
- Pinecone-p1.x12.3723
- Milvus-2c8g-disk2.1384
- Pinecone-s1.x1-2node1.7839
- Pinecone-s1.x11.1502
- ElasticCloud-upTo2.5c8g1.1436
- QdrantCloud-2c8g-1node1.1303
- WeaviateCloud-standard0.9862
- WeaviateCloud-bus_crit0.9726
- PgVector-2c8g0.5739
Detailed Comparison_qps
These are the specific QPS performance data with recall for each system across various cases.
ZillizCloud(1cu-perf) | ZillizCloud(8cu-perf) | ZillizCloud(1cu-cap) | ZillizCloud(2cu-cap) | WeaviateCloud(standard) | WeaviateCloud(bus_crit) | QdrantCloud(2c8g-1node) | QdrantCloud(4c16g-1node) | QdrantCloud(4c16g-5node) | PgVector(2c8g) | Pinecone(p2.x1) | Pinecone(p1.x1) | Pinecone(s1.x1) | Pinecone(s1.x1-2node) | Pinecone(p2.x1-8node) | Pinecone(p1.x1-8node) | Milvus(2c8g-disk) | Milvus(4c16g-disk) | Milvus(2c8g-hnsw) | Milvus(16c64g-hnsw) | ElasticCloud(upTo2.5c8g) | |
Search Performance Test (500K Dataset, 1536 Dim) | 297.5 @0.974 | 1871 @0.9602 | 143 @0.9818 | 379.9721 @0.982 | 46.8622 @0.9957 | 43.5017 @0.9957 | 78.7196 @0.9203 | 188.6436 @0.9175 | 633.6033 @0.919 | 0.8836 @0.8528 | 228.4 @0.9348 | 67.63 @0.8064 | 15.33 @0.8064 | 16.34 @0.879 | 322.7 @0.9478 | Insufficient Capacity | 37.432 @0.9975 | 321.6048 @0.989 | 180.2757 @0.9942 | 626.5243 @0.9954 | 11.2945 @0.996 |
Search Performance Test (5M Dataset, 1536 Dim) | Insufficient Capacity | 556.7 @0.9723 | Insufficient Capacity | 71.74 @0.9883 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 95.5682 @0.9463 | Timeout | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 10.45 @0.8208 | 265.5 @0.9332 | Insufficient Capacity | Insufficient Capacity | 22.1467 @0.9972 | Insufficient Capacity | 78.4227 @0.9982 | Timeout |
Filtering Search Performance Test (500K Dataset, 1536 Dim, Filter 1%) | 228.3 @0.994 | 1583 @0.9836 | 106 @0.9887 | 287 @0.9865 | 1.8388 @0.9957 | 1.5668 @0.9957 | 68.3111 @0.9202 | 155.6991 @0.9171 | 434.4062 @0.9181 | 0.8937 @0.8525 | 181.5 @0.9345 | 63.35 @0.8065 | 15.13 @0.8065 | 16.18 @0.8793 | 303.8 @0.9478 | 147.7 @0.9707 | 37.0696 @0.9976 | 303.2551 @0.9876 | 179.0033 @0.9943 | 599.4213 @0.9955 | 17.3271 @0.9961 |
Filtering Search Performance Test (5M Dataset, 1536 Dim, Filter 1%) | Insufficient Capacity | 294.3 @0.9939 | Insufficient Capacity | 34.6654 @0.9961 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 80.3192 @0.9462 | Timeout | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 9.8 @0.8212 | 180.2 @0.9335 | Insufficient Capacity | Insufficient Capacity | 21.5388 @0.997 | Insufficient Capacity | 78.5351 @0.9982 | Timeout |
Filtering Search Performance Test (500K Dataset, 1536 Dim, Filter 99%) | 584 @1 | 2345 @1 | 189 @1 | 412 @1 | 45.0666 @1 | 41.5443 @1 | 210.2147 @0.9996 | 445.3289 @0.9999 | 1509.3293 @0.9995 | 1.2145 @0.7487 | 205.7 @0.9586 | 176.7 @1 | 17.41 @1 | 36.11 @1 | 730.7 @0.9586 | 782.5 @1 | 81.1915 @1 | 394.5418 @1 | 526.8846 @1 | 2098.2113 @1 | 26.26 @0.9999 |
Filtering Search Performance Test (5M Dataset, 1536 Dim, Filter 99%) | Insufficient Capacity | 295.6 @1 | Insufficient Capacity | 42.169 @1 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 166.7252 @0.9988 | Timeout | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 14.84 @0.96 | 104.3 @0.9563 | Insufficient Capacity | Insufficient Capacity | 37.878 @1 | Insufficient Capacity | 275.6292 @1 | Timeout |
Search Performance Test (1M Dataset, 768 Dim) | 516.27 @0.9463 | 2884.689 @0.8801 | 330.0144 @0.9507 | 579.9416 @0.9213 | 63.1365 @0.991 | 67.9121 @0.9909 | Insufficient Capacity | 240.7209 @0.8887 | 537.4975 @0.8903 | 10.6271 @0.8898 | 261.798 @0.9262 | 46.6189 @0.8737 | 18.7634 @0.8737 | 20.7437 @0.9291 | 365.0835 @0.945 | 131.2549 @0.9867 | 100.6667 @0.9909 | 536.0726 @0.9728 | 274.5407 @0.9807 | 1258.7043 @0.9799 | 15.2269 @0.9888 |
Filtering Search Performance Test (1M Dataset, 768 Dim, Filter 1%) | 354.8416 @0.9802 | 1689.5799 @0.9493 | 271.6585 @0.9678 | 425.2529 @0.9686 | 0.7512 @0.9908 | 0.7636 @0.9908 | Insufficient Capacity | 189.4399 @0.8889 | 372.0466 @0.8904 | 10.8507 @0.8897 | 166.1851 @0.9264 | 42.4856 @0.8741 | 18.3619 @0.8741 | 20.2993 @0.9293 | 325.5271 @0.9452 | 127.9337 @0.9869 | 101.1399 @0.9907 | 467.179 @0.9697 | 236.5672 @0.981 | 1075.8776 @0.98 | 15.1749 @0.989 |
Filtering Search Performance Test (1M Dataset, 768 Dim, Filter 99%) | 427.5229 @1 | 1517.6792 @1 | 216.5226 @1 | 397.0539 @1 | 30.1358 @1 | 32 @1 | Insufficient Capacity | 313.5116 @0.9999 | 1156.2898 @0.9989 | 75.7055 @0.9999 | 121.7169 @0.9693 | 138.9479 @0.9979 | 25.2744 @0.9979 | 26.4719 @1 | 596.7942 @0.9693 | 595.8462 @1 | 52.2606 @1 | 431.7512 @1 | 309.4833 @1 | 1494.8493 @1 | 27.6181 @0.9999 |
Search Performance Test (10M Dataset, 768 Dim) | Insufficient Capacity | 822.5318 @0.9294 | Insufficient Capacity | 123.9553 @0.971 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 110.248 @0.898 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 8.6675 @0.8369 | 303.2044 @0.9246 | Insufficient Capacity | Insufficient Capacity | 61.0661 @0.9911 | Insufficient Capacity | 178.6585 @0.9843 | Timeout |
Filtering Search Performance Test (10M Dataset, 768 Dim, Filter 1%) | Insufficient Capacity | 378.9146 @0.9758 | Insufficient Capacity | 59.1479 @0.9906 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 87.2601 @0.898 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 7.8121 @0.8369 | 136.0345 @0.9244 | Insufficient Capacity | Insufficient Capacity | 58.9326 @0.9911 | Insufficient Capacity | 178.3732 @0.9844 | Timeout |
Filtering Search Performance Test (10M Dataset, 768 Dim, Filter 99%) | Insufficient Capacity | 218.6854 @1 | Insufficient Capacity | 40.999 @1 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 125.7846 @0.975 | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | Insufficient Capacity | 16.869 @0.9814 | 66.7221 @0.963 | Insufficient Capacity | Insufficient Capacity | 42.5977 @1 | Insufficient Capacity | 229.3526 @1 | Timeout |