Lucidworks Fusion Data Discovery Platform
Lucidworks is an enterprise search technology company that offers an application development platform, commercial support, and value-add software for open-source Apache Lucene and Apache Solr. The company built Lucidworks Fusion in a modern, containerized, cloud-native architecture that simplifies the process of deploying AI-powered data discovery and search applications. To ensure users find relevant data fast, Lucidworks implemented semantic search using the vector similarity approach. Milvus, an open-source vector database built by Zilliz, is used to power the core semantic search module inside Lucidworks Fusion.
Objective
Build a data discovery platform that delivers relevant semantic search results at speed.
Challenges
- Return semantic search results on large-scale datasets as quickly as possible. - Retrieve roughly 20 relevant matches per query from a dataset that contains millions of vectors. - Maintain efficiency amid thousands of simultaneous semantic search requests.
Why verctor database
- Milvus can store hundreds of billions of vectors across multiple collections to ensure that search is fast and results are relevant. - Support for a wide range of indexes makes semantic search blazing fast without sacrificing recall. - When more speed is needed, Milvus can run even faster on GPU. - Optimization for batch requests means Milvus can maintain efficiency even under a heavy load of requests per second (RPS).
Results
- 1,500 RPS against a collection of over one million documents, with an average response time of around 40 milliseconds.
"Milvus can handle vectors at any scale efficiently and at lightning speed. Without a vector search engine like Milvus, similarity searches would not be feasible across the entire vector space."
Read the full storyshare this