Blog
pgvector vs Aerospike: Choosing the Right Vector Database for Your AI Apps

pgvector vs Aerospike: Choosing the Right Vector Database for Your AI Apps

Oct 06, 20249 min read

What is a Vector Database?

Before we compare pgvector and Aerospike, let's first explore the concept of vector databases.

A vector database is specifically designed to store and query high-dimensional vectors, which are numerical representations of unstructured data. These vectors encode complex information, such as the semantic meaning of text, the visual features of images, or product attributes. By enabling efficient similarity searches, vector databases play a pivotal role in AI applications, allowing for more advanced data analysis and retrieval.

Common use cases for vector databases include e-commerce product recommendations, content discovery platforms, anomaly detection in cybersecurity, medical image analysis, and natural language processing (NLP) tasks. They also play a crucial role in Retrieval Augmented Generation (RAG), a technique that enhances the performance of large language models (LLMs) by providing external knowledge to reduce issues like AI hallucinations.

There are many types of vector databases available in the market, including:

Purpose-built vector databases such as Milvus, Zilliz Cloud (fully managed Milvus)
Vector search libraries such as Faiss and Annoy.
Lightweight vector databases such as Chroma and Milvus Lite.
Traditional databases with vector search add-ons capable of performing small-scale vector searches.

pgvector is a traditional database and Aerospike is a distributed, scalable NoSQL database. Both have vector search capabilities as an add-on. This post compares their vector search capabilities.

pgvector: Overview and Core Technology

pgvector is an extension for PostgreSQL that adds support for vector operations. It allows users to store and query vector embeddings directly within their PostgreSQL database, providing vector similarity search capabilities without the need for a separate vector database.

Key features of pgvector include:

Support for exact and approximate nearest neighbor search
Integration with PostgreSQL's indexing mechanisms
Ability to perform vector operations like addition and subtraction
Support for various distance metrics (Euclidean, cosine, inner product)

pgvector, by default, employs exact nearest neighbor search, which guarantees perfect recall but can be slower for large datasets. To optimize performance, pgvector offers the option to create indexes for approximate nearest neighbor search. This approach trades some accuracy for significantly improved speed, which is often a worthwhile tradeoff in many real-world applications.

It's important to note that adding an approximate index can change the results of your queries. This is different from typical database indexes, which don't affect the actual results returned. The two types of approximate indexes supported by pgvector are:

HNSW (Hierarchical Navigable Small World): Introduced in pgvector version 0.5.0, HNSW is known for its high performance and quality of results. It builds a multi-layer graph structure that allows for fast traversal during searches.
IVFFlat (Inverted File Flat): This method divides the vector space into clusters. During a search, it first identifies the most relevant clusters and then performs an exact search within those clusters. This can significantly speed up searches in large datasets.

The choice between these index types depends on your specific use case, considering factors like dataset size, required query speed, and acceptable trade-off in accuracy. HNSW generally offers better performance but may use more memory, while IVFFlat can be more memory-efficient but might be slightly slower or less accurate in some cases.

When implementing pgvector in your project, try to experiment with both index types and their parameters to find the optimal configuration for your specific needs. This process of fine-tuning can impact the performance and accuracy of your vector search operations.

Wanna learn how to get started using pgvector? Check out this tutorial!

What is Aerospike? An Overview

Aerospike is a NoSQL database for high-performance real-time applications. It has added support for vector indexing and searching so it’s suitable for vector database use cases. The vector capability is called Aerospike Vector Search (AVS) and is in Preview. You can request early access from Aerospike.

AVS only supports Hierarchical Navigable Small World (HNSW) indexes for vector search. When updates or inserts are made in AVS, record data including the vector is written to the Aerospike Database (ASDB) and is immediately visible. For indexing, each record must have at least one vector in the specified vector field of an index. You can have multiple vectors and indexes for a single record so you can search on the same data in different ways. Aerospike recommends assigning upserted records to a specific set so you can monitor and operate on them.

AVS has a unique way of building the index, it’s concurrent across all AVS nodes. While vector record updates are written directly to ASDB, index records are processed asynchronously from an indexing queue. This is done in batches and distributed across all AVS nodes, so it uses all the CPU cores in the AVS cluster and is scalable. Ingestion performance is highly dependent on host memory and storage layer configuration.

For each item in the indexing queue, AVS processes the vector for indexing, builds the clusters for each vector and commits those to ASDB. An index record contains a copy of the vector itself and the clusters for that vector at a given layer of the HNSW graph. Indexing uses vector extensions (AVX) for single instruction, multiple data parallel processing.

AVS queries during ingestion to “pre-hydrate” the index cache because records in the clusters are interconnected. These queries are not counted as query requests but show up as reads against the storage layer. This way, the cache is populated with relevant data and can improve query performance. This shows how AVS handles vector data and builds indexes for similarity search so it can scale for high-dimensional vector searches.

Key Differences

When deciding between pgvector and Aerospike for vector search, here are the key factors to consider.

Search Methodology:

pgvector supports exact and approximate nearest neighbor search. It has two types of approximate indexes: HNSW (Hierarchical Navigable Small World) and IVFFlat (Inverted File Flat). HNSW builds a multi-layer graph for fast traversal, IVFFlat divides the vector space into clusters. Aerospike Vector Search (AVS) only supports HNSW indexes for vector search.

Data Handling:

pgvector integrates with PostgreSQL so you can store and query vector embeddings along with your traditional relational data. If you need to combine vector search with structured data operations this can be useful. Aerospike being a NoSQL database is designed for high performance real-time applications and may be more suitable for semi-structured or unstructured data at scale.

Scalability and Performance:

pgvector uses PostgreSQL’s indexing mechanisms which can be good for many use cases. But for very large datasets you may need to tune your indexes and queries carefully. Aerospike is designed for high scalability and has a unique concurrent indexing process across all nodes in the cluster. This distributed approach can be better for large scale vector search operations.

Flexibility and Customization:

pgvector allows you to perform various vector operations like addition and subtraction and supports multiple distance metrics (Euclidean, cosine, inner product). It integrates seamlessly with PostgreSQL’s rich set of features and extensions. Aerospike may have less flexibility in terms of SQL-like operations but more options for fine tuning performance at scale.

Integration and Ecosystem:

pgvector has the benefit of PostgreSQL’s large ecosystem of tools and integrations. If your existing stack is heavily invested in PostgreSQL then pgvector could be a natural fit. Aerospike while less common may have specific integrations that are valuable for high performance real-time applications.

Ease of Use:

pgvector can be easy to set up and use if you’re already familiar with PostgreSQL. The learning curve may be steeper for Aerospike if you’re new to NoSQL databases. However both require careful consideration of index types and parameters to optimize performance.

Cost:

pgvector is an open-source extension for PostgreSQL so could be lower cost. Aerospike offers both open-source and enterprise editions, AVS is currently in preview. The total cost will depend on your specific deployment and scale.

Security:

Both have security features but the details are different. PostgreSQL has a robust set of authentication and access control mechanisms that pgvector can use. Aerospike has security features but you’d need to check their documentation for the most up-to-date info on encryption, authentication and access control for their vector search.

When to Choose Each Technology

Use pgvector:

pgvector is a good choice when you already have PostgreSQL and want to add vector search to your existing relational database. It’s good for projects that need to combine vector operations with SQL queries or when you have structured data with a vector component. pgvector is good for exact nearest neighbor search or small to medium sized datasets where query performance is not a bottleneck.

Use Aerospike:

Aerospike with Vector Search (AVS) is better suited for high performance, real-time applications that need to handle large scale vector search. It’s a good option when you’re building systems that require low latency vector similarity search across huge datasets. Aerospike’s distributed indexing is particularly useful for applications in areas like recommendation systems, real-time fraud detection or large scale image or text similarity search where speed and scalability is key.

Conclusion:

pgvector stands out for its PostgreSQL integration, a familiar environment for developers who work with relational databases and flexibility to combine vector search with structured data operations. Aerospike is high performance, scalable vector search for large datasets, with distributed indexing potentially better for massive scale. Your choice between these two should be based on your use case, existing infrastructure, data volume and performance requirements. Consider your team’s expertise, the nature of your data (structured vs semi-structured), the scale of your vector search needs and the real-time performance of your application when making your decision.

While this article provides an overview of pgvector and Aerospike, it's key to evaluate these databases based on your specific use case. One tool that can assist in this process is VectorDBBench, an open-source benchmarking tool designed for comparing vector database performance. Ultimately, thorough benchmarking with specific datasets and query patterns will be essential in making an informed decision between these two powerful, yet distinct, approaches to vector search in distributed database systems.

Using Open-source VectorDBBench to Evaluate and Compare Vector Databases on Your Own

VectorDBBench is an open-source benchmarking tool designed for users who require high-performance data storage and retrieval systems, particularly vector databases. This tool allows users to test and compare the performance of different vector database systems such as Milvus and Zilliz Cloud (the managed Milvus) using their own datasets and determine the most suitable one for their use cases. Using VectorDBBench, users can make informed decisions based on the actual vector database performance rather than relying on marketing claims or anecdotal evidence.

VectorDBBench is written in Python and licensed under the MIT open-source license, meaning anyone can freely use, modify, and distribute it. The tool is actively maintained by a community of developers committed to improving its features and performance.

Download VectorDBBench from its GitHub repository to reproduce our benchmark results or obtain performance results on your own datasets.
Take a quick look at the performance of mainstream vector databases on the VectorDBBench Leaderboard.
Read the following blogs to learn more about vector database evaluation.

Further Resources about VectorDB, GenAI, and ML

Chloe Williams
Chloe Williams is a technical writer at Zilliz.

Content

Start Free, Scale Easily

Try the fully-managed vector database built for your GenAI applications.

Try Zilliz Cloud for Free