July 7, 2021 by Zilliz
On June 28, 2021, LF AI & Data, an umbrella foundation of the Linux Foundation supporting open source innovation in artificial intelligence, machine learning, deep learning, and data, released version 2.0 of its open-source vector database Milvus. Milvus 2.0 is a cloud-native, full redesign of Milvus 1.0, and a leap forward in the iterative process of optimizing vector database architectures for AI and MLOps.
Xiaofan Luan, Director of Engineering at Zilliz, summarized the redesign as an effort to build a future-proof version of vector database: cloud-native, flexible, and forever easy to use.
"Only architectures supporting storage and computing disaggregation can take full advantage of elasticity of public clouds," Luan argued, "As a cloud-ready vector database, Milvus 2.0 is resilient, manageable, and observable. It employs the microservice design and relies on Kubernetes and cloud services to achieve component orchestration."
He then moved on to the flexibility feature of this new release: "Milvus 2.0 supports both scalar and vector data and enables hybrid search between them to leverage synergy between structured and unstructured data. Unlike Milvus 1.0, which has eventual consistency only, Milvus 2.0 supports tunable consistency at the request level to support more scenarios. Moreover, Milvus 2.0 has introduced the unified Lambda architecture to integrate the processing of incremental and historical data."
"Making our vector database easy to use is always on the top of our list," claimed Xiaofan. Of all the initiatives designed specifically to facilitate user journey with Milvus, he cited the object relational mapping (ORM) encapsulation of the Python client as an attempt to close the gap between the proof of concept (PoC) for AI algorithms and the real production. He also pointed out that, unlike Milvus 1.0, which requires a cluster sharding middleware solution Mishards to scale, Milvus 2.0 is born for massive-scale datasets and guarantees linear scalability.
According to Luan, optimizing vector retrieval costs on massive datasets is the biggest challenge facing the space. The Milvus team is planning on introducing heteogeneous hardware acceleration at the physical layer to reduce CPU overhead and the overall costs. "Milvus has roots in building databases designed specifically for AI, and this remains a primary goal of the project," the ambitious Director continued, "We will follow the motto 'AI for DB' and look to replacing knob tuning of parameters with AI-enabled system-parameter tuning to further reduce learning costs for our users."
Milvus 2.0 is the world's most advanced open-source vector database built on a cloud-native architecture that completely disaggregates storage from computation, making it easy to scale in any AI or MLOps scenario. Milvus 2.0 is available now on GitHub. To learn more about the design, key features, and future plans for Milvus check out the Zilliz blog.
Please contact Jingyu Zhang by email at email@example.com.