# What factors should be controlled to make fair performance comparisons between two vector database systems (e.g., ensuring the same hardware, similar index build configurations, and using the same dataset)?

Source: https://zilliz.com/ai-faq/what-factors-should-be-controlled-to-make-fair-performance-comparisons-between-two-vector-database-systems-eg-ensuring-the-same-hardware-similar-index-build-configurations-and-using-the-same-dataset
Author: zilliz

To ensure fair performance comparisons between two vector database systems, you must control variables that directly impact speed, accuracy, and resource usage. Here are the key factors to standardize:

### 1. **Hardware and Software Environment** 
Use identical hardware specifications for both systems, including CPU (model, core count), RAM size, storage type (SSD vs. HDD), and GPU acceleration if applicable. For example, test both databases on the same AWS EC2 `c5.4xlarge` instance to eliminate hardware-driven performance differences. Software dependencies like OS version, driver versions (e.g., CUDA for GPU-accelerated systems), and runtime libraries (Python, Java) must also match. Disable unrelated background processes to minimize interference.

### 2. **Dataset and Queries** 
Use the same dataset with identical characteristics: 
- **Size**: Test with the same number of vectors (e.g., 1 million 768-dimensional vectors). 
- **Distribution**: Ensure data distribution (e.g., clustered vs. uniform) is consistent. 
- **Queries**: Use the same set of query vectors and search parameters (e.g., top-k=10). For reproducibility, use standard benchmarks like the SIFT-1M dataset or ANN Benchmarks’ glove-100-angular. 

Preprocessing steps (normalization, quantization) must match. For example, if one system requires L2-normalized vectors, apply the same to the other.

### 3. **Index Configuration and Workload** 
Configure indexes with equivalent parameters for a fair comparison: 
- **Index Type**: Compare HNSW vs. HNSW, not HNSW vs. IVF. 
- **Build Parameters**: Set comparable values (e.g., HNSW’s `efConstruction=200` and `M=16`). 
- **Query Parameters**: Use the same `efSearch` value during queries. 

Simulate identical workloads: 
- **Query Throughput**: Test with the same QPS (queries per second). 
- **Concurrency**: Match client thread counts or connection pools. 
- **Mixed Workloads**: If testing writes and reads, use the same insert/query ratio. 

### 4. **Benchmarking Methodology** 
- **Metrics**: Measure the same criteria (e.g., 95th percentile latency, recall@10, throughput). 
- **Warm-Up**: Run preliminary queries to preload caches and stabilize performance. 
- **Averaging**: Execute multiple test runs (e.g., 5 iterations) and average results. 
- **Resource Monitoring**: Track CPU/RAM/disk usage during tests to identify bottlenecks. 

### Example Scenario 
To compare Milvus and Qdrant: 
1. Use AWS `c6i.8xlarge` instances (32 vCPUs, 64GB RAM). 
2. Load the LAION-5B dataset subset (10M vectors, 768D) with identical normalization. 
3. Configure HNSW with `efConstruction=128`, `M=24`, and query with `efSearch=64`. 
4. Measure throughput (QPS) and recall@10 under 50 concurrent client threads. 

By standardizing these factors, differences in results will reflect the databases’ inherent performance, not external variables. Document all configurations publicly (e.g., in a GitHub repo) for reproducibility.