For "Single tenant LAION 100M" below, we refer to the "Laion 100m" (Click to copy S3 URI) Dataset (100M * 768 float dense vectors).
For "Multitenant Cohere 10M" Dataset below, we refer to the "Cohere Large" (Click to copy S3 URI) Dataset (10M * 768 float dense vectors). All data are randomly splitted across 1000 tenants.
Pareto cost lines show how much sustained QPS each cloud vector database can deliver for a given operating spend. They combine measured search results with public pricing models.
Workload
Single tenant LAION 100M
Cost basis
Query only
Cost period
Price order
Cost vs. QPS ParetoQuery only · lower is better · USD / hour
This case measures the write-to-serve lifecycle: how long bulk data takes to finish inserting, when it can be searched reliably, and when background indexing has fully caught up.
Notes:For exactly how we defined "inserted", "searchable", "fully indexed", please check VectorDBBench source code for each client. For basic intuition:
This case measures query behavior after cloud data is already loaded and searchable. It compares peak concurrent QPS, P99 latency, and recall under different response payloads, scalar-filter selectivity, and tenancy modes so the chart shows both throughput and result quality instead of rewarding speed alone.
Mode
Single tenant LAION 100M
Filter
unfiltered
Payload
ids only
Latency
Max concurrency P99 latency
Display cost
topK = 100
Vector Search Latency and QPSunfiltered · ids only · Max concurrency P99
ProductMax Concurrency P99 LatencyMax Concurrency QPSrecall@10Query cost @ max QPS
This case measures the first query after an idle cold period against the warmed steady-state query path. It isolates cold-start behavior from normal search throughput so the chart shows whether a product has a material warm-up penalty after inactivity.
Mode
unfiltered
Cold / Warm Latencyunfiltered
Zilliz Cloud Capacity 12CU
55 / 54 ms
Turbopuffer Pinned
64 / 45 ms
Zilliz Cloud Tiered 4CU
122 / 57 ms
Pinecone Serverless
271 / 60 ms
Turbopuffer
2048 / 322 ms
Cold / Warm Ratiolower is better
Zilliz Cloud Capacity 12CU
1.01×
Turbopuffer Pinned
1.42×
Zilliz Cloud Tiered 4CU
2.16×
Pinecone Serverless
4.52×
Turbopuffer
6.36×
Notes:
We note that while certain products may have a more dramatic cold/warm ratio at p99 percentile, this usually indicates a network shaking problem in later queries and cannot be fully reproduced. Thus we stick with the more faithful definition of cold/warm latency, i.e. the first query for each round.
The timing for when a product's collection becomes cold is rather ambiguous since most products don't offer public APIs to provide such info. In order to simulate real world production settings, for cold latency benchmarking, we ensure to wait at least 24 hours since the last operations on the products for the collections to become as cold as possible.
Run VDBBench on your own data
These results use public datasets. Your vectors, filters and traffic decide what you actually get—run the same open-source benchmark on them.
Test Zilliz Cloud
Benchmark Zilliz Cloud on your data
Sign up for Zilliz Cloud—$200 in free credits with a work email.
Create a cluster and copy its endpoint and API key.
Install VDBBench with pip install vectordb-bench and start it with init_bench.
Load your vectors as Parquet files on the Custom Dataset page and run the case against your cluster.
Evaluating a migration from Elasticsearch, OpenSearch or pgvector, or running more than 100M vectors? A Zilliz engineer runs the benchmark on your data with you.