Vector databases like Milvus and Weaviate handle vector storage and indexing differently, relying on a mix of memory-mapped files, custom storage engines, and optimized data structures. Here’s a breakdown of their underlying mechanisms:
Milvus uses a distributed architecture that separates storage and compute. For vector storage, it relies on cloud-native object storage (e.g., S3, MinIO) or local disk for persisting raw vectors and metadata. Vectors are stored in a columnar format (similar to Parquet) to optimize bulk reads. Indexes like IVF (Inverted File Index), HNSW (Hierarchical Navigable Small World), or ANNOY (Approximate Nearest Neighbors Oh Yeah) are built on top of these vectors and stored as separate files. During query execution, indexes are loaded into memory for fast access. Milvus also uses memory-mapped files to manage large datasets efficiently, allowing the OS to handle paging between disk and memory. This reduces the overhead of manual memory management. For metadata (e.g., schema, index configurations), Milvus relies on external databases like etcd or MySQL to track state across distributed nodes.
Weaviate, in contrast, uses a custom storage engine designed for vector-first workloads. It stores vectors, inverted indexes (for filtering), and HNSW graphs in a proprietary format on disk, using a log-structured merge (LSM) tree approach for writes. Data is organized into segments that are periodically merged in the background. Weaviate heavily leverages memory-mapped files to keep frequently accessed index structures (like HNSW layers) in memory while allowing the OS to manage eviction to disk. This balances performance with memory constraints. Unlike Milvus, Weaviate bundles its storage engine (based on RocksDB-like LSM trees) and doesn’t require external systems for metadata, making it a self-contained binary. For hybrid search (vector + keyword), it combines vector indices with inverted indexes using techniques like Roaring Bitmaps for fast filtering.
Key Differences: Milvus emphasizes scalability via separation of storage and compute, while Weaviate prioritizes simplicity with an all-in-one design. Both use memory-mapped files to optimize memory usage but differ in storage backends—Milvus relies on external object storage, whereas Weaviate uses an embedded LSM engine. Their indexing strategies (e.g., HNSW in Weaviate vs. multiple algorithm options in Milvus) reflect trade-offs between flexibility and ease of deployment.