Learn how to build a Milvus cluster based on JuiceFS, a shared file system designed for cloud-native environments.

Collaborations between open-source communities are a magical thing. Not only do passionate, intelligent, and creative volunteers keep open-source solutions innovative, they also work to bring different tools together in interesting and useful ways. [Milvus](https://milvus.io/), the world's most popular vector database, and [JuiceFS](https://github.com/juicedata/juicefs), a shared file system designed for cloud-native environments, were united in this spirit by their respective open-source communities. This article explains what JuiceFS is, how to build a Milvus cluster based on JuiceFS shared file storage, and the performance users can expect using this solution.

## **What is JuiceFS?**

JuiceFS is a high-performance, open-source distributed POSIX file system, which can be built on top of Redis and S3. It was designed for cloud-native environments and supports managing, analyzing, archiving, and backing up data of any type. JuiceFS is commonly used for solving big data challenges, building artificial intelligence (AI) applications, and log collection. The system also supports data sharing across multiple clients and can be used directly as shared storage in Milvus.

After data, and its corresponding metadata, are persisted to object storage and [Redis](https://redis.io/) respectively, JuiceFS serves as a stateless middleware. Data sharing is realized by enabling different applications to dock with each other seamlessly through a standard file system interface. JuiceFS relies on Redis, an open-source in-memory data store, for metadata storage. Redis is used because it guarantees atomicity and provides high performance metadata operations. All data is stored in object storage through the JuiceFS client. The architecture diagram is as follows:

![juicefs-architecture.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/juicefs_architecture_2023b37a4e.png)
###### *Overall architecture of JuiceFS.*

## **Build a Milvus cluster based on JuiceFS**

A Milvus cluster built with JuiceFS (see architecture diagram below) works by splitting upstream requests using Mishards, a cluster sharding middleware, to cascade the requests down to its sub-modules. When inserting data, Mishards allocates upstream requests to the Milvus write node, which stores newly inserted data in JuiceFS. When reading data, Mishards loads the data from JuiceFS through a Milvus read node to memory for processing, then collects and returns results from sub-services upstream.

![milvus-cluster-built-with-juicefs.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/milvus_cluster_built_with_juicefs_3a43cd262c.png)
###### *Architecture of Milvus cluster built with JuiceFS.*

### **Step 1: Launch MySQL service**

Launch the MySQL service on **any** node in the cluster. For details, see [Manage Metadata with MySQL](https://milvus.io/docs/v1.1.0/data_manage.md).

### **Step 2: Create a JuiceFS file system**

For demonstration purposes, the pre-compiled binary JuiceFS program is used. Download the correct [installation package](https://github.com/juicedata/juicefs/releases) for your system and follow the JuiceFS [Quick Start Guide](https://github.com/juicedata/juicefs/blob/main/docs/en/quick_start_guide.md) for detailed install instructions. To create a JuiceFS file system, first set up a Redis database for metadata storage. It is recommended that for public cloud deployments you host the Redis service on the same cloud as the application. Additionally, set up object storage for JuiceFS. In this example, Azure Blob Storage is used; however, JuiceFS supports almost all object services. Select the object storage service that best suits the demands of your scenario.

After configuring the Redis service and object storage, format a new file system and mount JuiceFS to the local directory:

```
1 $ export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=XXX;AccountKey=XXX;EndpointSuffix=core.windows.net"
2 $ ./juicefs format \
3 --storage wasb \
4 --bucket https://<container> \
5 ... \
6 localhost test #format
7 $ ./juicefs mount -d localhost ~/jfs #mount
8
```

> If the Redis server is not running locally, replace the localhost with the following address: `redis://<user:password>@host:6379/1`.

When the installation succeeds, JuiceFS returns the shared storage page **/root/jfs**.

![installation-success.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/installation_success_9d05279ecd.png)

### **Step 3: Start Milvus**

All the nodes in the cluster should have Milvus installed, and each Milvus node should be configured with read or write permission. Only one Milvus node can be configured as write node, and the rest must be read nodes. First, set the parameters of sections `cluster` and `general` in the Milvus system configuration file **server_config.yaml**:

**Section** `cluster` 

| **Parameter** | **Description** | **Configuration** |
| :------------ | :----------------------------- | :---------------- |
| `enable` | Whether to enable cluster mode | `true` |
| `role` | Milvus deployment role | `rw`/`ro` |

**Section** `general` 

```
# meta_uri is the URI for metadata storage, using MySQL (for Milvus Cluster). Format: mysql://<username:password>@host:port/database
general:
 timezone: UTC+8
 meta_uri: mysql://root:milvusroot@host:3306/milvus
```


During installation, the configured JuiceFS shared storage path is set as **/root/jfs/milvus/db**.

```
1 sudo docker run -d --name milvus_gpu_1.0.0 --gpus all \ 
2 -p 19530:19530 \ 
3 -p 19121:19121 \ 
4 -v /root/jfs/milvus/db:/var/lib/milvus/db \ #/root/jfs/milvus/db is the shared storage path 
5 -v /home/$USER/milvus/conf:/var/lib/milvus/conf \ 
6 -v /home/$USER/milvus/logs:/var/lib/milvus/logs \ 
7 -v /home/$USER/milvus/wal:/var/lib/milvus/wal \ 
8 milvusdb/milvus:1.0.0-gpu-d030521-1ea92e 
9
```

After the installation completes, start Milvus and confirm that it is launched properly.
Finally, start the Mishards service on **any** of the nodes in the cluster. The image below shows a successful launch of Mishards. For more information, refer to the GitHub [tutorial](https://github.com/milvus-io/bootcamp/tree/new-bootcamp/deployments/juicefs).

![mishards-launch-success.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/mishards_launch_success_921695d3a8.png)

## **Performance benchmarks**

Shared storage solutions are usually implemented by network-attached storage (NAS) systems. Commonly used NAS systems types include Network File System (NFS) and Server Message Block (SMB). Public cloud platforms generally provide managed storage services compatible with these protocols, such as Amazon Elastic File System (EFS).

Unlike traditional NAS systems, JuiceFS is implemented based on Filesystem in Userspace (FUSE), where all data reading and writing takes place directly on the application side, further reducing access latency. There are also features unique to JuiceFS that cannot be found in other NAS systems, such as data compression and caching.

Benchmark testing reveals that JuiceFS offers major advantages over EFS. In the metadata benchmark (Figure 1), JuiceFS sees I/O operations per second (IOPS) up to ten times higher than EFS. Additionally, the I/O throughput benchmark (Figure 2) shows JuiceFS outperforms EFS in both single- and multi-job scenarios.

![performance-benchmark-1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/performance_benchmark_1_b7fcbb4439.png) 
###### *Figure 1. Metadata benchmark*
![performance-benchmark-2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/performance_benchmark_2_e311098123.png)
###### *Figure 2. Sequential read/write benchmark.*

Additionally, benchmark testing shows first query retrieval time, or time to load newly inserted data from disk to memory, for the JuiceFS-based Milvus cluster is just 0.032 seconds on average, indicating that data is loaded from disk to memory almost instantaneously. For this test, first query retrieval time is measured using one million rows of 128-dimensional vector data inserted in batches of 100k at intervals of 1 to 8 seconds.

JuiceFS is a stable and reliable shared file storage system, and the Milvus cluster built on JuiceFS offers both high performance and flexible storage capacity.

## **Learn more about Milvus**

Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

![writer bio-changjian gao.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/writer_bio_changjian_gao_68018f7716.png)
![writer bio-jingjing jia.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/writer_bio_jingjing_jia_a85d1c2e3b.png)

large_blog_juiceFS.jpg

medium_blog_juiceFS.jpg

small_blog_juiceFS.jpg

thumbnail_blog_juiceFS.jpg

blog_juiceFS.jpg

Building a Milvus Cluster Based on JuiceFS

Building a news content recommendation system with semantic vectors is easy with Milvus, an open-source vector database. 

With [71% of Americans](https://www.socialmediatoday.com/news/new-research-shows-that-71-of-americans-now-get-news-content-via-social-pl/593255/) getting their news recommendations from social platforms, personalized content has quickly become how new media is discovered. Whether people are searching for specific topics, or interacting with recommended content, everything users see is optimized by algorithms to improve click-through rates (CTR), engagement, and relevance. Sohu is a NASDAQ-listed Chinese online media, video, search and gaming group. It leveraged [Milvus](https://milvus.io/), an open-source vector database built by [Zilliz](https://zilliz.com/), to build a semantic vector search engine inside its news app. This article explains how the company used user profiles to fine-tune personalized content recommendations over time, improving user experience and engagement.

## Recommending content using semantic vector search

Sohu News user profiles are built from browsing history and adjusted as users search for, and interact with, news content. Sohu’s recommender system uses semantic vector search to find relevant news articles. The system works by identifying a set of tags that are expected to be of interest to each user based on browsing history. It then quickly searches for relevant articles and sorts the results by popularity (measured by average CTR), before serving them to users. 

The New York Times alone publishes [230 pieces of content](https://www.theatlantic.com/technology/archive/2016/05/how-many-stories-do-newspapers-publish-per-day/483845/) a day, which offers a glimpse at the magnitude of new content an effective recommendation system must be capable of processing. Ingesting large volumes of news demands millisecond similarity search and hourly matching of tags to new content. Sohu chose Milvus because it processes massive datasets efficiently and accurately, reduces memory usage during search, and supports high-performance deployments.

## Understanding a news recommendation system workflow

Sohu’s semantic vector search-based content recommendation relies on the Deep Structured Semantic Model (DSSM), which uses two neural networks to represent user queries and news articles as vectors. The model calculates the cosine similarity of the two semantic vectors, then the most similar batch of news is sent to the recommendation candidates pool. Next, news articles are ranked based on their estimated CTR, and those with the highest predicted click-through rate are displayed to users.

### Encoding news articles into semantic vectors with BERT-as-service

To encode news articles into semantic vectors, the system uses the [BERT-as-service](https://github.com/hanxiao/bert-as-service.git) tool. If the word count of any piece of content exceeds 512 while using this model, information loss occurs during the embedding process. To help overcome this, the system first extracts a summary and encodes it into a 768-dimensional semantic vector. Then the two most relevant topics from each news article are extracted, and the corresponding pre-trained topic vectors (200-dimensions) are identified based on topic ID. Next the topic vectors are spliced into the 768-dimensional semantic vector extracted from the article summary, forming a 968-dimensional semantic vector. 

New content continuously comes in through Kafta, and is converted into semantic vectors before being inserted into the Milvus database.

### Extracting semantically similar tags from user profiles with BERT-as-service

The other neural network of the model is user semantic vector. Semantically similar tags (e.g., coronavirus, covid, COVID-19, pandemic, novel strain, pneumonia) are extracted from user profiles based on interests, search queries, and browsing history. The list of acquired tags is sorted by weight, and the top 200 are divided into different semantic groups. Permutations of the tags within each semantic group are used to generate new tag phrases, which are then encoded into semantic vectors through BERT-as-service

For each user profile, sets of tag phrases have a [corresponding set of topics](https://github.com/baidu/Familia) that are marked by a weight indicating a user's interest level. The top two topics out of all relevant topics are selected and encoded by the machine learning (ML) model to be spliced into the corresponding tag semantic vector, forming a 968-dimensional user semantic vector. Even if the system generates the same tags for different users, different weights for tags and their corresponding topics, as well as explicit variance between each user's topic vectors, ensures recommendations are unique

The system is able to make personalized news recommendations by calculating the cosine similarity of the semantic vectors extracted from both user profiles and news articles.

![Sohu01.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Sohu01_1e466fe0c3.jpg)
###### *Sohu News DSSM structure.*

### Computing new semantic user profile vectors and inserting them to Milvus

Semantic user profile vectors are computed daily, with data from the previous 24-hour period processed the following evening. Vectors are inserted into Milvus individually and run through the query process to serve relevant news results to users. News content is inherently topical, requiring computation to be run hourly to generate a current newsfeed that contains content that has a high predicted click-through rate and is relevant to users. News content is also sorted into partitions by date, and old news is purged daily.

### Decreasing semantic vector extraction time from hours to days

Retrieving content using semantic vectors requires converting tens of millions of tag phrases into semantic vectors every day. This is a time-consuming process that would require days to complete even when running on graphics processing units (GPU), which accelerate this type of computation. To overcome this technical issue, semantic vectors from the previous embedding must be optimized so that when similar tag phrases surface corresponding semantic vectors are directly retrieved.

The semantic vector of the existing set of tag phrases is stored, and a new set of tag phrases that is generated daily gets encoded into MinHash vectors. [Jaccard distance](https://milvus.io/docs/metric.md) is used to compute similarity between the MinHash vector of the new tag phrase and the saved tag phrase vector. If the Jaccard distance exceeds a pre-defined threshold, the two sets are considered similar. If the similarity threshold is met, new phrases can leverage the semantic information from previous embeddings. Tests suggest a distance above 0.8 should guarantee enough accuracy for most situations.

Through this process, the daily conversion of the tens of millions of vectors mentioned above is reduced from days to around two hours. Although other methods of storing semantic vectors might be more appropriate depending on specific project requirements, computing similarity between two tag phrases using Jaccard distance in a Milvus database remains an efficient and accurate method in a wide variety of scenarios.

![Sohu02.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Sohu02_d50fccc538.jpg)
###### *Using Jaccard distance to compute similarity between two sets.*

## Overcoming “bad cases” of short text classification

When classifying news text, short news articles have fewer features for extraction than longer ones. Because of this, classification algorithms fail when content of varying lengths is run through the same classifier. Milvus helps solve this issue by searching for multiple pieces of long text classification information with similar semantics and reliable scores, then using a voting mechanism to modify short text classification. 

### Identifying and resolving misclassified short text 

Precise classification of each news article is crucial to providing useful content recommendations. Since short news articles have fewer features, applying the same classifier for news with different lengths results in a higher error rate for short text classification. Human labeling is too slow and inaccurate for this task, so BERT-as-service and Milvus are used to quickly identify misclassified short text in batches, correctly reclassify them, then use batches of data as a corpus for training against this problem.

BERT-as-service is used to encode a total number of five million long news articles with a classifier score greater than 0.9 into semantic vectors. After inserting the long text articles into Milvus, short text news is encoded into semantic vectors. Each short news semantic vector is used to query the Milvus database and obtain the top 20 long news articles with the highest cosine similarity to the target short news. If 18 of the top 20 semantically similar long news appear to be in the same classification and it differs from that of the query short news, then the short news classification is considered incorrect and must be adjusted to align with the 18 long news articles.

This process quickly identifies and corrects inaccurate short text classifications. Random sampling statistics show that after short text classifications are corrected, the overall accuracy of text classification exceeds 95%. By leveraging the classification of high-confidence long text to correct the classification of short text, the majority of bad classification cases are corrected in a short amount of time. This also offers a good corpus for training a short text classifier.

![Sohu03.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Sohu03_a43074cf5f.jpg)
###### *Flowchart of discovery of "bad cases" of short text classification.*

## Milvus can power real-time news content recommendation and more

Milvus greatly improved the real-time performance of Sohu’s news recommendation system, and also bolstered the efficiency of identifying misclassified short text. If you’re interested in learning more about Milvus and its various applications:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

large_blog_SohuNews.jpg

medium_blog_SohuNews.jpg

small_blog_SohuNews.jpg

thumbnail_blog_SohuNews.jpg

blog_SohuNews.jpg

Learn how Milvus was used to build an intelligent news recommendation system inside an app.

Building an Intelligent News Recommendation System Inside Sohu News App

Milvus v1.1.0 is available now. Discover new features, improvements, and bug fixes made to the world' most popular vector database.

[Milvus](https://github.com/milvus-io) is an ongoing open-source software (OSS) project focused on building the world's fastest and most reliable vector database. New features inside Milvus v1.1.0 are the first of many updates to come, thanks to long-term support from the open-source community and sponsorship from Zilliz. This blog article covers the new features, improvements, and bug fixes included with Milvus v1.1.0.

**Jump to:**

- [New features](#new-features)
- [Improvements](#improvements)
- [Bug fixes](#bug-fixes)

 

## New features

Like any OSS project, Milvus is a perpetual work in progress. We strive to listen to our users and the open-source community to prioritize the features that matter most. The latest update, Milvus v1.1.0, offers the following new features:

### Specify partitions with `get_entity_by_id()` method calls

To further accelerate vector similarity search, Milvus 1.1.0 now supports retrieving vectors from a specified partition. Generally, Milvus supports querying vectors through specified vector IDs. In Milvus 1.0, calling the method `get_entity_by_id()` searches the entire collection, which can be time consuming for large datasets. As we can see from the code below, `GetVectorsByIdHelper` uses a `FileHolder` structure to loop through and find a specific vector. 

```
std::vector<meta::CollectionSchema> collection_array; 
 auto status = meta_ptr_->ShowPartitions(collection.collection_id_, collection_array); 
 
 collection_array.push_back(collection); 
 status = meta_ptr_->FilesByTypeEx(collection_array, file_types, files_holder); 
 if (!status.ok()) { 
 std::string err_msg = "Failed to get files for GetVectorByID: " + status.message(); 
 LOG_ENGINE_ERROR_ << err_msg; 
 return status; 
 } 
 
 if (files_holder.HoldFiles().empty()) { 
 LOG_ENGINE_DEBUG_ << "No files to get vector by id from"; 
 return Status(DB_NOT_FOUND, "Collection is empty"); 
 } 
 
 cache::CpuCacheMgr::GetInstance()->PrintInfo(); 
 status = GetVectorsByIdHelper(id_array, vectors, files_holder); 
DBImpl::GetVectorsByIdHelper(const IDNumbers& id_array, std::vector<engine::VectorsData>& vectors, 
 meta::FilesHolder& files_holder) { 
 // attention: this is a copy, not a reference, since the files_holder.UnMarkFile will change the array internal 
 milvus::engine::meta::SegmentsSchema files = files_holder.HoldFiles(); 
 LOG_ENGINE_DEBUG_ << "Getting vector by id in " << files.size() << " files, id count = " << id_array.size(); 
 
 // sometimes not all of id_array can be found, we need to return empty vector for id not found 
 // for example: 
 // id_array = [1, -1, 2, -1, 3] 
 // vectors should return [valid_vector, empty_vector, valid_vector, empty_vector, valid_vector] 
 // the ID2RAW is to ensure returned vector sequence is consist with id_array 
 using ID2VECTOR = std::map<int64_t, VectorsData>; 
 ID2VECTOR map_id2vector; 
 
 vectors.clear(); 
 
 IDNumbers temp_ids = id_array; 
 for (auto& file : files) { 
```

However, this structure is not filtered by any partitions in `FilesByTypeEx()`. In Milvus v1.1.0, it is possible for the system to pass partition names to the `GetVectorsIdHelper` loop so that the `FileHolder` only contains segments from specified partitions. Put differently, if you know exactly which partition the vector for a search belongs to, you can specify the partition name in a `get_entity_by_id()` method call to accelerate the search process.

We not only made modifications to code controlling system queries at the Milvus server level, but also updated all our SDKs(Python, Go, C++, Java, and RESTful) by adding a parameter for specifying partition names. For example, in pymilvus, the definition of `get_entity_by_id` `def get_entity_by_id(self, collection_name, ids, timeout=None)` is changed to `def get_entity_by_id(self, collection_name, partition_tags=None, ids, timeout=None)`.

 

### Specify partitions with `delete_entity_by_id()` method calls

To make vector management more efficient, Milvus v1.1.0 now supports specifying partition names when deleting a vector in a collection. In Milvus 1.0, vectors in a collection can only be deleted by ID. When calling the delete method, Milvus will scan all vectors in the collection. However, it is far more efficient to scan only relevant partitions when working with massive million, billion, or even trillion vector datasets. Similar to the new feature for specifying partitions with `get_entity_by_id()` method calls, modifications were made to the Milvus code using the same logic.

 

### New method `release_collection()`

To free up memory Milvus used to load collections at runtime, a new method `release_collection()` has been added in Milvus v1.1.0 to manually unload specific collections from cache.

 

## Improvements

Although new features are usually all the rage, it's also important to improve what we already have. What follows are upgrades and other general improvements over Milvus v1.0.

 

### Improved performance of `get_entity_by_id()` method call

The chart below is a comparison of vector search performance between Milvus v1.0 and Milvus v1.1.0:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Segment file size = 1024 MB 
> Row count = 1,000,000 
> Dim = 128 

| Query ID Num | v 1.0.0 | v1.1.0 |
| :-----------: | :-----------: | :-----------: |
| 10 | 9 ms | 2 ms |
| 100 | 149 ms | 19 ms |

 

### Hnswlib upgraded to v0.5.0

Milvus adopts multiple widely used index libraries, including Faiss, NMSLIB, Hnswlib, and Annoy to simplify the process of choosing the right index type for a given scenario.

Hnswlib has been upgraded from v0.3.0 to v0.5.0 in Milvus 1.1.0 due to a bug detected in the earlier version. Additionally, upgrading Hnswlib improves `addPoint()` performance in index building.

A Zilliz developer created a pull request (PR) to improve Hnswlib performance while building indexes in Milvus. See [PR #298](https://github.com/nmslib/hnswlib/pull/298) for details.

The chart below is a comparison of `addPoint()` performance between Hnswlib 0.5.0 and the proposed PR:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Dataset: sift_1M (row count = 1000000, dim = 128, space = L2)

| | 0.5.0 | PR-298 |
| :-----------: | :-----------: | :-----------: |
| M = 16, ef_construction = 100 | 274406 ms | 265631 ms |
| M = 16, ef_construction = 200 | 522411 ms | 499639 ms |

 

### Improved IVF index training performance

Creating an index includes training, inserting and writing data to disk. Milvus 1.1.0 improves the training component of index building. The chart below is a comparison of IVF index training performance between Milvus 1.0 and Milvus 1.1.0:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Dataset: sift_1m (row_count = 1000000, dim = 128, metric_type = L2)

| | v1.0.0 (ms) | v1.1.0 (ms) |
| :-----------: | :-----------: | :-----------: |
| ivf_flat (nlist = 2048) | 90079 | 81544 |
| ivf_pq (nlist = 2048, m=16) | 103535 | 97115 |
| ivf_pq (nlist = 2048, m=32) | 108638 | 104558 |
| ivf_flat (nlist = 4096) | 340643 | 310685 |
| ivf_pq (nlist = 4096, m=16) | 351982 | 323758 |
| ivf_pq (nlist = 4096, m=32) | 357359 | 330887 |


 

## Bug fixes

We also fixed some bugs to make Milvus more stable and efficient when managing vector datasets. See [Fixed Issues](https://milvus.io/docs/v1.1.0/release_notes.md#Fixed-issues) for more details.

large_v1.1 cover.jpeg

medium_v1.1 cover.jpeg

small_v1.1 cover.jpeg

thumbnail_v1.1 cover.jpeg

v1.1 cover.jpeg

Milvus v1.1.0 has arrived! New features, improvements, and bug fixes are available now.

Inside Milvus 1.1.0

The finance industry has long leveraged open-source software to extract insights from large, unstructured datasets. Discover how Milvus, an open-source vector database, can power AI applications including banking chatbots, recommender systems, and semantic text mining. 

Banks and other financial institutions have long been early adopters of open-source software for big data processing and analytics. In 2010, Morgan Stanley [began using](https://www.forbes.com/sites/tomgroenfeldt/2012/05/30/morgan-stanley-takes-on-big-data-with-hadoop/?sh=19f4f8cd16db) the open-source Apache Hadoop framework as part of a small experiment. The company was struggling to successfully scale traditional databases to the massive volumes of data its scientists wanted to leverage, so it decided to explore alternative solutions. Hadoop is now a staple at Morgan Stanley, helping with everything from managing CRM data to portfolio analysis. Other open-source relational database software such as MySQL, MongoDB, and PostgreSQL have been indispensable tools for making sense of big data in the finance industry.

Technology is what gives the financial services industry a competitive edge, and artificial intelligence (AI) is rapidly becoming the standard approach to extracting valuable insights from big data and analyzing activity in real-time across the banking, asset management, and insurance sectors. By using AI algorithms to convert unstructured data such as images, audio, or video to vectors, a machine-readable numeric data format, it is possible to run similarity searches on massive million, billion, or even trillion vector datasets. Vector data is stored in high-dimensional space, and similar vectors are found using similarity search, which requires a dedicated infrastructure called a vector database.

![01 (1).jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/01_1_cb99f15886.jpg)
###### *Development of open source big data technologies.*

[Milvus](https://github.com/milvus-io/milvus) is an open-source vector database built specifically for managing vector data, which means engineers and data scientists can focus on building AI applications or conducting analysis—instead of the underlying data infrastructure. The platform was built around AI application development workflows and is optimized to streamline machine learning operations (MLOps). For more information about Milvus and its underlying technology, check out our [blog](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View).

Common applications of AI in the financial services industry include algorithmic trading, portfolio composition and optimization, model validation, backtesting, Robo-advising, virtual customer assistants, market impact analysis, regulatory compliance, and stress testing. This article covers three specific areas where vector data is leveraged as one of the most valuable assets for banking and financial companies:

1. Enhancing customer experience with banking chatbots
2. Boosting financial services sales and more with recommender systems
3. Analyzing earnings reports and other unstructured financial data with semantic text mining

 

### Enhancing customer experience with banking chatbots

Banking chatbots can improve customer experiences by helping consumers select investments, banking products, and insurance policies. Digital services are rising rapidly in popularity in part due to trends accelerated by the coronavirus pandemic. Chatbots work by using natural language processing (NLP) to convert user-submitted questions into semantic vectors to search for matching answers. Modern banking chatbots offer a personalized natural experience for users and speak in a conversational tone. Milvus provides a data fabric well suited for creating chatbots using real-time vector similarity search.

Learn more in our demo that covers building [chatbots with Milvus](https://zilliz.com/blog/building-intelligent-chatbot-with-nlp-and-milvus).

![02 (1).jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/02_1_8c298c45e5.jpg)

 

#### Boosting financial services sales and more with recommender systems:

The private banking sector uses recommender systems to increase sales of financial products through personalized recommendations based on customer profiles. Recommender systems can also be leveraged in financial research, business news, stock selection, and trading support systems. Thanks to deep learning models, every user and item is described as an embedding vector. A vector database offers an embedding space where similarities between users and items can be calculated.

Learn more from our [demo](https://zilliz.com/blog/graph-based-recommendation-system-with-milvus) covering graph-based recommendation systems with Milvus.

 

#### Analyzing earnings reports and other unstructured financial data with semantic text mining:

Text mining techniques had a substantial impact on the financial industry. As financial data grows exponentially, text mining has emerged as an important field of research in the domain of finance.

Deep learning models are currently applied to represent financial reports through word vectors capable of capturing numerous semantic aspects. A vector database like Milvus is able to store massive semantic word vectors from millions of reports, then conduct similarity searches on them in milliseconds.

Learn more about how to [use deepset's Haystack with Milvus](https://medium.com/deepset-ai/semantic-search-with-milvus-knowledge-graph-qa-web-crawlers-and-more-837451eae9fa).

 

### Don’t be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_03 (1).jpg

medium_03 (1).jpg

small_03 (1).jpg

thumbnail_03 (1).jpg

03 (1).jpg

Milvus can be used to build AI applications for the finance industry including chatbots, recommender systems, and more.

Accelerating AI in Finance with Milvus, an Open-Source Vector Database

VOVA, an emerging e-commerce platform, prioritizes convenient and positive user experiences. Learn how the company used Milvus to build an image search tool to help shoppers find items they encounter in real life.

Jump to:
 
- [How does image search work?](#how-does-image-search-work)
- [Target detection using the YOLO model](#target-detection-using-the-yolo-model)
- [Image feature vector extraction with ResNet](#image-feature-vector-extraction-with-resnet)
- [Vector similarity search powered by Milvus](#vector-similarity-search-powered-by-milvus)
- [VOVA's shop by image tool](#vovas-shop-by-image-tool)

Online shopping surged in 2020, [up 44%](https://www.digitalcommerce360.com/2021/02/15/ecommerce-during-coronavirus-pandemic-in-charts/) in large part due to the coronavirus pandemic. As people sought to socially distance and avoid contact with strangers, no-contact delivery became an incredibly desirable option for many consumers. This popularity has also led to people buying a greater variety of goods online, including niche items that can be hard to describe using a traditional keyword search.

To help users overcome the limitations of keyword-based queries, companies can build image search engines that allow users to use images instead of words for search. Not only does this allow users to find items that are difficult to describe, but it also helps them shop for things they encounter in real life. This functionality helps build a unique user experience and offers general convenience that customers appreciate.

VOVA is an emerging e-commerce platform that focuses on affordability and offering a positive shopping experience to its users, with listings covering millions of products and support for 20 languages and 35 major currencies. To enhance the shopping experience for its users, the company used Milvus to build image search functionality into its e-commerce platform. The article explores how VOVA successfully built an image search engine with Milvus. 

 

### How does image search work?

VOVA's shop by image system searches the company's inventory for product images that are similar to user uploads. The following chart shows the two stages of the system process, the data import stage (blue) and the query stage (orange):

1. Use the YOLO model to detect targets from uploaded photos;
2. Use ResNet to extract feature vectors from the detected targets; 
3. Use Milvus for vector similarity search.

![Vova-1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Vova_1_47ee6f2da9.png)
###### *System process of VOVA's search by image functionality.*

 

### Target detection using the YOLO model

VOVA's mobile apps on Android and iOS currently support image search. The company uses a state-of-the-art, real-time object detection system called YOLO (You only look once) to detect objects in user uploaded images. The YOLO model is currently in its fifth iteration. 

YOLO is a one-stage model, using only one convolutional neural network (CNN) to predict categories and positions of different targets. It is small, compact, and well suited for mobile use. 

YOLO uses convolutional layers to extract features and fully-connected layers to obtain predicted values. Drawing inspiration from the GooLeNet model, YOLO’s CNN includes 24 convolutional layers and two fully-connected layers.

As the following illustration shows, a 448 &times; 448 input image is converted by a number of convolutional layers and pooling layers to a 7 &times; 7 &times; 1024-dimensional tensor (depicted in the third to last cube below), and then converted by two fully-connected layers to a 7 &times; 7 &times; 30-dimensional tensor output.

The predicted output of YOLO P is a two-dimensional tensor, whose shape is [batch,7 &times;7 &times;30]. Using slicing, P[:,0:7&times;7&times;20] is the category probability, P[:,7&times;7&times;20:7&times;7&times;(20+2)] is the confidence, and P[:,7&times;7&times;(20+2)]:] is the predicted result of the bounding box.

![vova-2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/vova_2_1ccf38f721.png)
###### *YOLO network architecture.*

 

### Image feature vector extraction with ResNet

VOVA adopted the residual neural network (ResNet) model to extract feature vectors from an extensive product image library and user uploaded photos. ResNet is limited because as the depth of a learning network increases, the accuracy of the network decreases. The image below depicts ResNet running the VGG19 model (a variant of the VGG model) modified to include a residual unit through the short circuit mechanism. VGG was proposed in 2014 and includes just 14 layers, while ResNet came out a year later and can have up to 152.

The ResNet structure is easy to modify and scale. By changing the number of channels in the block and the number of stacked blocks, the width and depth of the network can be easily adjusted to obtain networks with different expressive capabilities. This effectively solves the network degeneration effect, where accuracy declines as the depth of learning increases. With sufficient training data, a model with improving expressive performance can be obtained while gradually deepening the network. Through model training, features are extracted for each picture and converted to 256-dimensional floating point vectors.

![vova-3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/vova_3_df4b810281.png)
###### *ResNet structure.*

 

### Vector similarity search powered by Milvus
 
VOVA's product image database includes 30 million pictures and is growing rapidly. To quickly retrieve the most similar product images from this massive dataset, Milvus is used to conduct vector similarity search. Thanks to a number of optimizations, Milvus offers a fast and streamlined approach to managing vector data and building machine learning applications. Milvus offers integration with popular index libraries (e.g., Faiss, Annoy), supports multiple index types and distance metrics, has SDKs in multiple languages, and provides rich APIs for managing vector data.

Milvus can conduct similarity search on trillion-vector datasets in milliseconds, with a query time under 1.5 seconds when nq=1 and an average batch query time under 0.08 seconds. To build its image search engine, VOVA referred to the design of Mishards, Milvus' sharding middleware solution (see the chart below for its system design), to implement a highly available server cluster. By leveraging the horizontal scalability of a Milvus cluster, the project requirement for high query performance on massive datasets was met.

![vova-4.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/vova_4_e305f1955c.png)
###### *Mishards architecture in Milvus.*


### VOVA's shop by image tool
 
The screenshots below show the VOVA search by image shopping tool on the company's Android app.

![vova-5.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/vova_5_c4c25a3bae.png)
###### *Screenshots of VOVA's search by image shopping tool.*
 
As more users search for products and upload photos, VOVA will continue to optimize the models that power the system. Additionally, the company will incorporate new Milvus functionality that can further enhance the online shopping experience of its users.

### Reference

**YOLO:**

https://arxiv.org/pdf/1506.02640.pdf

https://arxiv.org/pdf/1612.08242.pdf

**ResNet:**

https://arxiv.org/abs/1512.03385

**Milvus:**

https://milvus.io/docs/overview.md

large_vova-thumbnail.jpg

medium_vova-thumbnail.jpg

small_vova-thumbnail.jpg

thumbnail_vova-thumbnail.jpg

vova-thumbnail.jpg

Discover how Milvus, an open-source vector database, was used by e-commerce platform VOVA to power shopping by image. 

Building a Search by Image Shopping Experience with VOVA and Milvus

We are proud to announce Milvus v1.0, a stable, long-term support version of  the world’s most popular open-source vector database. Discover the highlight features that Milvus v1.0 has to offer.

![Milvus.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Milvus_510cf50aee.jpeg)

Milvus is an open-source vector database designed to manage massive million, billion, or even trillion vector datasets. Milvus has broad applications spanning new drug discovery, computer vision, autonomous driving, recommendation engines, chatbots, and much more. 

In March, 2021 Zilliz, the company behind Milvus, released the platform's first long-term support version—Milvus v1.0. After months of extensive testing, a stable, production ready version of the world's most popular vector database is ready for prime time. This blog article covers some Milvus fundamentals as well as key features of v1.0.

 

### Milvus distributions

Milvus is available in CPU-only and GPU-enabled distributions. The former relies exclusively on CPU for index building and search; the latter enables CPU and GPU hybrid search and index building that further accelerates Milvus. For example, using the hybrid distribution, CPU can be used for search and GPU for index building, further improving query efficiency.

Both Milvus distributions are available in Docker. You can either compile Milvus from Docker (if your operating system supports it) or compile Milvus from source code on Linux (other operating systems are not supported). 

 

### Embedding vectors

Vectors are stored in Milvus as entities. Each entity has one vector ID field and one vector field. Milvus v1.0 supports integer vector IDs only. When creating a collection within Milvus, vector IDs can be automatically generated or manually defined. Milvus ensures auto-generated vector IDs are unique however, manually defined IDs can be duplicated within Milvus. If manually defining IDs, users are responsible for making sure all IDs are unique.

 

### Partitions

Milvus supports creating partitions in a collection. In situations where data is inserted regularly and historical data isn't significant (e.g., streaming data), partitions can be used to accelerate vector similarity search. One collection can have up to 4,096 partitions. Specifying a vector search within a specific partition narrows the search and may significantly reduce query time, particularly for collections that contain more than a trillion vectors.

 

### Index algorithm optimizations
Milvus is built on top of multiple widely-adopted index libraries, including Faiss, NMSLIB, and Annoy. Milvus is far more than a basic wrapper for these index libraries. Here are some of the major enhancements that have been made to the underlying libraries:

- IVF index performance optimizations using the Elkan k-means algorithm.
- FLAT search optimizations.
- IVF_SQ8H hybrid index support, which can reduce index file sizes by up to 75% without sacrificing data accuracy. IVF_SQ8H is built upon IVF_SQ8, with identical recall but much faster query speed. It was designed specifically for Milvus to harnesses the parallel processing capacity of GPUs, and the potential for synergy between CPU/GPU co-processing.
- Dynamic instruction set compatibility.

 

### Search, index building, and other Milvus optimizations

The following optimizations have been made to Milvus to improve search and index building performance. 

- Search performance is optimized in situations when the number of queries (nq) is less than the number of CPU threads. 
- Milvus combines search requests from a client that take the same topK and search parameters. 
- Index building is suspended when search requests come in. 
- Milvus automatically preloads collections to memory at start.
- Multiple GPU devices can be assigned to accelerate vector similarity search.

 

### Distance metrics

Milvus is a vector database built to power vector similarity search. The platform was built with MLOps and production level AI applications in mind. Milvus supports a wide range of distance metrics for calculating similarity, such as Euclidean distance (L2), inner product (IP), Jaccard distance, Tanimoto, Hamming distance, superstructure, and substructure. The last two metrics are commonly used in molecular search and AI-powered new drug discovery.

 

### Logging

Milvus supports log rotation. In the system configuration file, milvus.yaml, you can set the size of a single log file, the number of log files, and log output to stdout.

 

### Distributed solution

Mishards, a Milvus sharding middleware, is the distributed solution for Milvus With one write node and an unlimited number of read nodes, Mishards unleashes the computational potential of server cluster. Its features include request forwarding, read/write splitting, dynamic/horizontal scaling, and more.

 

### Monitoring

Milvus is compatible with Prometheus, an open-source system monitoring and alerts toolkit. Milvus adds support for Pushgateway in Prometheus, making it possible for Prometheus to acquire short-lived batch metrics. The monitoring and alerts system works as follows:

- The Milvus server pushes customized metrics data to Pushgateway.
- Pushgateway ensures ephemeral metric data is safely sent to Prometheus.
- Prometheus continues pulling data from Pushgateway.
- Alertmanager is used to set the alert threshold for different indicators and send alerts via email or message.

 

### Metadata management

Milvus uses SQLite for metadata management by default. SQLite is implemented in Milvus and does not require configuration. In a production environment, it is recommended that you use MySQL for metadata management. 

 

### Engage with our open-source community:

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_Milvus.jpeg

medium_Milvus.jpeg

small_Milvus.jpeg

thumbnail_Milvus.jpeg

Milvus.jpeg

	 Milvus v1.0 is available now. This stable, long-term support version powers image/video search, chatbots, and many more AI applications.

What's Inside Milvus 1.0?

Learn how Milvus is used to mitigate threats to critical data and strengthen cybersecurity with real-time virus detection.

Cybersecurity remains a persistent threat to both individuals and businesses, with data privacy concerns increasing for [86% of companies](https://www.getapp.com/resources/annual-data-security-report/) in 2020 and just [23% of consumers](https://www.firstdata.com/downloads/pdf/digital-commerce-cybersecurity-ebook.pdf) believing their personal data is very secure. As malware becomes steadily more omnipresent and sophisticated, a proactive approach to threat detection has become essential. [Trend Micro](https://www.trendmicro.com/en_us/business.html) is a global leader in hybrid cloud security, network defense, small business security, and endpoint security. To protect Android devices from viruses, the company built Trend Micro Mobile Security—a mobile app that compares APKs (Android Application Package) from the Google Play Store to a database of known malware. The virus detection system works as follows:

- External APKs (Android application package) from the Google Play Store are crawled.
- Known malware is converted into vectors and stored in [Milvus](https://www.milvus.io/docs/v1.0.0/overview.md).
- New APKs are also converted into vectors, then compared to the malware database using similarity search.
- If an APK vector is similar to any of the malware vectors, the app provides users with detailed information about the virus and its threat level.

To work, the system has to perform highly efficient similarity search on massive vector datasets in real time. Initially, Trend Micro used [MySQL](https://www.mysql.com/). However, as its business expanded so did the number of APKs with nefarious code stored in its database. The company’s algorithm team began searching for alternative vector similarity search solutions after quickly outgrowing MySQL.

 

### Comparing vector similarity search solutions

There are a number of vector similarity search solutions available, many of which are open source. Although the circumstances vary from project to project, most users benefit from leveraging a vector database built for unstructured data processing and analytics rather than a simple library that requires extensive configuration. Below we compare some popular vector similarity search solutions and explain why Trend Micro chose Milvus.

#### Faiss
[Faiss](https://ai.facebook.com/tools/faiss/) is a library developed by Facebook AI Research that enables efficient similarity search and clustering of dense vectors. The algorithms it contains search vectors of any size in sets. Faiss is written in C++ with wrappers for Python/numpy, and supports a number of indexes including IndexFlatL2, IndexFlatIP, HNSW, and IVF.

Although Faiss is an incredibly useful tool, it has limitations. It only works as a basic algorithm library, not a database for managing vector datasets. Additionally, it does not offer a distributed version, monitoring services, SDKs, or high availability, which are the key features of most cloud-based services. 

#### Plug-ins based on Faiss & other ANN search libraries
There are several plug-ins built on top of Faiss, NMSLIB, and other ANN search libraries that are designed to enhance the basic functionality of the underlying tool that powers them. Elasticsearch (ES) is a search engine based on the Lucene library with a number of such plugins. Below is an architecture diagram of an ES plug-in:

![1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/1_3ce4e516c3.png)

###### *Architecture diagram of an Elasticsearch plug-in.*

Built in support for distributed systems is a major advantage of an ES solution. This saves developers time and companies money thanks to code that doesn’t have to be written. ES plug-ins are technically advanced and prevalent. Elasticsearch provides a QueryDSL (domain-specific language), which defines queries based on JSON and is easy to grasp. A full set of ES services makes it possible to conduct vector/text search and filter scalar data simultaneously.

Amazon, Alibaba, and Netease are a few large tech companies that currently rely on Elasticsearch plug-ins for vector similarity search. The primary downsides with this solution are high memory consumption and no support for performance tuning. In contrast, [JD.com](http://jd.com/) has developed its own distributed solution based on Faiss called [Vearch](https://github.com/vearch/vearch). However, Vearch is still an incubation-stage project and its open-source community is relatively inactive.

#### Milvus 
[Milvus](https://www.milvus.io/) is an open-source vector database created by [Zilliz](https://zilliz.com). It is highly flexible, reliable, and blazing fast. By encapsulating multiple widely adopted index libraries, such as Faiss, NMSLIB, and Annoy, Milvus provides a comprehensive set of intuitive APIs, allowing developers to choose the ideal index type for their scenario. It also provides distributed solutions and monitoring services. Milvus has a highly active open-source community and over 5.5K stars on [Github](https://github.com/milvus-io/milvus).

#### Milvus bests the competition
We compiled a number of different test results from the various vector similarity search solutions mentioned above. As we can see in the following comparison table, Milvus was significantly faster than the competition despite being tested on a dataset of 1 billion 128-dimensional vectors.

| **Engine** | **Performance (ms)** | **Dataset Size (million)** |
| :---- | :---- | :---- |
| ES | 600 | 1 |
| ES + Alibaba Cloud | 900 | 20 |
| Milvus | 27 | 1000+ |
| SPTAG | Not good | |
| ES + nmslib, faiss | 90 | 150 |

###### *A comparison of vector similarity search solutions.*

After weighing the pros and cons of each solution, Trend Micro settled on Milvus for its vector retrieval model. With exceptional performance on massive, billion-scale datasets, it's obvious why the company chose Milvus for a mobile security service that requires real-time vector similarity search.

 

### Designing a system for real-time virus detection
Trend Micro has more than 10 million malicious APKs stored in its MySQL database, with 100k new APKs added each day. The system works by extracting and calculating Thash values of different components of an APK file, then uses the Sha256 algorithm to transform it into binary files and generate 256-bit Sha256 values that differentiate the APK from others. Since Sha256 values vary with APK files, one APK can have one combined Thash value and one unique Sha256 value.

Sha256 values are only used to differentiate APKs, and Thash values are used for vector similarity retrieval. Similar APKs may have the same Thash values but different Sha256 values.

To detect APKs with nefarious code, Trend Micro developed its own system for retrieving similar Thash values and corresponding Sha256 values. Trend Micro chose Milvus to conduct instantaneous vector similarity search on massive vector datasets converted from Thash values. After similarity search is run, the corresponding Sha256 values are queried in MySQL. A Redis caching layer is also added to the architecture to map Thash values to Sha256 values, significantly reducing query time.

Below is the architecture diagram of Trend Micro’s mobile security system.

![image-20210118-022039.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/image_20210118_022039_ae824b663c.png)

###### *Architecture diagram for Trend Micro Mobile Security.*

 

Choosing an appropriate distance metric helps improve vector classification and clustering performance. The following table shows the [distance metrics](https://www.milvus.io/docs/v1.0.0/metric.md#binary) and the corresponding indexes that work with binary vectors. 

| **Distance Metrics** | **Index Types** |
| ---- | ---- |
| - Jaccard - Tanimoto - Hamming | - FLAT - IVF_FLAT |
| - Superstructure - Substructure | FLAT |

###### *Distance metrics and indexes for binary vectors.*

 

Trend Micro converts Thash values into binary vectors and stores them in Milvus. For this scenario, Trend Micro is using Hamming distance to compare vectors.

Milvus will soon support string vector ID, and integer IDs won’t have to be mapped to the corresponding name in string format. This makes the Redis caching layer unnecessary and the system architecture less bulky.

Trend Micro adopts a cloud-based solution and deploys many tasks on [Kubernetes](https://kubernetes.io/). To achieve high availability, Trend Micro uses [Mishards](https://www.milvus.io/docs/v1.0.0/mishards.md), a Milvus cluster sharding middleware developed in Python. 

![image-20210118-022104.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/image_20210118_022104_3001950ee8.png)

###### *Mishards architecture in Milvus.*

 

Trend Micro separates storage and distance calculation by storing all vectors in the [EFS](https://aws.amazon.com/efs/) (Elastic File System) provided by [AWS](https://aws.amazon.com/). This practice is a popular trend in the industry. Kubernetes is used to start multiple reading nodes, and develops LoadBalancer services on these reading nodes to ensure high availability.

To maintain data consistency Mishards supports just one writing node. However, a distributed version of Milvus with support for multiple writing nodes will be available in the coming months. 

 

### Monitoring and Alert Functions
Milvus is compatible with monitoring systems built on [Prometheus](https://prometheus.io/), and uses [Grafana](https://grafana.com/), an open-source platform for time-series analytics, to visualize various performance metrics.

Prometheus monitors and stores the following metrics:

- Milvus performance metrics including insertion speed, query speed, and Milvus uptime. 
- System performance metrics including CPU/GPU usage, network traffic, and disk access speed.
- Hardware storage metrics including data size and total file number.

The monitoring and alert system works as follows: 

- A Milvus client pushes customized metrics data to Pushgateway.
- The Pushgateway ensures short-lived, ephemeral metric data is safely sent to Prometheus.
- Prometheus keeps pulling data from Pushgateway. 
- Alertmanager sets the alert threshold for different metrics and raises alarms through emails or messages.

 

### System Performance
A couple months have passed since the ThashSearch service built on Milvus was first launched. The graph below shows that end-to-end query latency is less than 95 milliseconds.

![image-20210118-022116.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/image_20210118_022116_a0c735ce20.png)

###### *Query latency for Thash search service built on Milvus.*

 

Insertion is also fast. It takes around 10 seconds to insert 3 million 192-dimensional vectors. With help from Milvus, the system performance was able to meet the performance criteria set by Trend Micro. 

 

### Don’t be a stranger
- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_TrendMicro.jpg

medium_blog_TrendMicro.jpg

small_blog_TrendMicro.jpg

thumbnail_blog_TrendMicro.jpg

blog_TrendMicro.jpg

Making with Milvus: Detecting Android Viruses in Real Time for Trend Micro

Learn more about using semantic machine learning methodologies to power more relevant search results across your organization.

[Semantic search](https://lucidworks.com/post/what-is-semantic-search/) is a great tool to help your customers—or your employees—find the right products or information. It can even surface difficult-to-index information for better results. That said, if your semantic methodologies aren’t being deployed to work fast, they won’t do you any good. The customer or employee isn’t just going to sit around while the system takes its time responding to their query—and a thousand others are likely being ingested at the same time.

How can you make semantic search fast? Slow semantic search isn’t going to cut it.

Fortunately, this is the kind of problem Lucidworks loves to solve. We recently tested a modest-sized cluster—read on for more details—that resulted in 1500 RPS (requests per second) against a collection of over one million documents, with an average response time of roughly 40 milliseconds. Now that’s some serious speed.

 

### Implementing Semantic Search
To make lightning-fast, machine learning magic happen, Lucidworks has implemented semantic search using the semantic vector search approach. There are two critical parts.

 

#### Part One: The Machine Learning Model

First, you need a way to encode text into a numerical vector. The text could be a product description, a user search query, a question, or even an answer to a question. A semantic search model is trained to encode text such that text that is semantically similar to other text is encoded into vectors that are numerically “close” to one another. This encoding step needs to be fast in order to support the thousand or more possible customer searches or user queries coming in every second.

 

#### Part Two: The Vector Search Engine
Second, you need a way to quickly find the best matches to the customer search or user query. The model will have encoded that text into a numerical vector. From there, you need to compare that to all the numerical vectors in your catalog or lists of questions and answers to find those best matches—the vectors that are “closest” to the query vector. For that, you will need a vector engine that can handle all of that information effectively and at lightning speed. The engine could contain millions of vectors and you really just want the best twenty or so matches to your query. And of course, it needs to handle a thousand or so such queries every second.

To tackle these challenges, we added the vector search engine [Milvus](https://doc.lucidworks.com/fusion/5.3/8821/milvus) in our [Fusion 5.3 release](https://lucidworks.com/post/enhance-personalization-efforts-with-new-features-in-fusion/). Milvus is open-source software and it is fast. Milvus uses FAISS ([Facebook AI Similarity Search](https://ai.facebook.com/tools/faiss/)), the same technology Facebook uses in production for its own machine learning initiatives. When needed, it can run even faster on [GPU](https://en.wikipedia.org/wiki/Graphics_processing_unit). When Fusion 5.3 (or higher) is installed with the machine learning component, Milvus is automatically installed as part of that component so you can turn on all of these capabilities with ease.

The size of the vectors in a given collection, specified when the collection is created, depends on the model that produces those vectors. For example, a given collection could store the vectors created from encoding (via a model) all of the product descriptions in a product catalog. Without a vector search engine like Milvus, similarity searches would not be feasible across the entire vector space. So, similarity searches would have to be limited to pre-selected candidates from the vector space (for example, 500) and would have both slower performance and lower quality results. Milvus can store hundreds of billions of vectors across multiple collections of vectors to ensure that search is fast and results are relevant.

 

### Using Semantic Search

Let’s get back to the semantic search workflow, now that we’ve learned a little about why Milvus might be so important. Semantic search has three stages. During the first stage, the machine learning model is loaded and/or trained. Afterwards, data is indexed into Milvus and Solr. The final stage is the query stage, when the actual search occurs. We’ll focus on those last two stages below.

 

### Indexing into Milvus
![Lucidworks-1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Lucidworks_1_47a9221723.png)
###### *Architectural diagram for indexing into Milvus.*

As shown in the above diagram, the query stage begins similarly to the indexing stage, just with queries coming in instead of documents. For each query:

1. The query is sent to the [Smart Answers](https://lucidworks.com/products/smart-answers/) index pipeline.
2. The query is then sent to the ML model.
3. The ML model returns a numeric vector (encrypted from the query). Again, the type of model determines the size of the vector.
4. The vector is sent to Milvus, which then determines which vectors, in the specified Milvus collection, best match the provided vector.
5. Milvus returns a list of unique IDs and distances corresponding to the vectors determined in step four.
6. A query containing those IDs and distances is sent to Solr.
7. Solr then returns an ordered list of the documents associated with those IDs.

 

### Scale Testing
In order to prove that our semantic search flows are running at the efficiency we require for our customers, we run scale tests using Gatling scripts on the Google Cloud Platform using a Fusion cluster with eight replicas of the ML model, eight replicas of the query service, and a single instance of Milvus. Tests were run using the Milvus FLAT and HNSW indexes. The FLAT index has 100% recall, but is less efficient – except when the datasets are small. The HNSW (Hierarchical Small World Graph) index still has high quality results and it has improved performance on larger datasets.

Let’s jump into some numbers from a recent example we ran:

![Lucidworks-2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Lucidworks_2_3162113560.png)
###### *Performance of Milvus FLAT and HNSW indexes on a small dataset.*

![Lucidworks-3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Lucidworks_3_3dc17f0ed8.png)
###### *Performance of Milvus FLAT and HNSW indexes on a medium dataset.*

![Lucidworks-4.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Lucidworks_4_8a6edd2f59.png)
###### *Performance of Milvus FLAT and HNSW indexes on a large dataset.*

 

### Getting Started

The [Smart Answers](https://lucidworks.com/products/smart-answers/) pipelines are designed to be easy-to-use. Lucidworks has [pre-trained models that are easy-to-deploy](https://doc.lucidworks.com/how-to/734/set-up-a-pre-trained-cold-start-model-for-smart-answers) and generally have good results—though training your own models, in tandem with pre-trained models, will offer the best results. Contact us today to learn how you can implement these initiatives into your search tools to power more effective and delightful results.


> This blog is reposted from: https://lucidworks.com/post/how-to-build-fast-semantic-search/?utm_campaign=Oktopost-Blog+Posts&utm_medium=organic_social&utm_source=linkedin

medium_lucidworks.png

small_lucidworks.png

thumbnail_lucidworks.png

lucidworks.png

Build Semantic Search at Speed

Milvus accelerates machine learning application development and machine learning operations (MLOps). With Milvus, you can rapidly develop a minimum viable product (MVP) while keeping costs at lower limits.

![blog cover.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/blog_cover_4a9807b9e0.png)

[Milvus](https://milvus.io/) is an open-source vector database. It supports adding, deleting, updating, and near real-time search of massive vector datasets created by extracting feature vectors from unstructured data using AI models. With a comprehensive set of intuitive APIs, and support for multiple widely adopted index libraries (e.g., Faiss, NMSLIB, and Annoy), Milvus accelerates machine learning application development and machine learning operations (MLOps). With Milvus, you can rapidly develop a minimum viable product (MVP) while keeping costs at lower limits.

"What resources are available for developing an AI application with Milvus?” is commonly asked in the Milvus community. Zilliz, the [company](https://zilliz.com/) behind Milvus, developed a number of demos that leverage Milvus to conduct lightening-fast similarity search that powers intelligent applications. Source code of Milvus solutions can be found at [zilliz-bootcamp](https://github.com/zilliz-bootcamp). The following interactive scenarios demonstrate natural language processing (NLP), reverse image search, audio search, and computer vision.


Feel free to try out the solutions to gain some hands-on experience with specific scenarios! Share your own application scenarios via: 
- [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ)
- [GitHub](https://github.com/milvus-io/milvus/discussions)

 

**Jump to:**

- [Natural language processing (chatbots)](#natural-language-processing-chatbots)
- [Reverse image search](#reverse-image-search-systems)
- [Audio search](#audio-search-systems)
- [Video object detection (computer vision)](#video-object-detection-computer-vision)

 

### Natural language processing (chatbots)
Milvus can be used to build chatbots that use natural language processing to simulate a live operator, answer questions, route users to relevant information, and reduce labor costs. To demonstrate this application scenario, Zilliz built an AI-powered chatbot that understands semantic language by combining Milvus with [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)), a machine learning (ML) model developed for NLP pre-training.

👉Source code：[zilliz-bootcamp/intelligent_question_answering_v2](https://github.com/zilliz-bootcamp/intelligent_question_answering_v2)

![1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/1_c301a9e4bd.png)
###### *AI-powered chatbot built with Milvus and BERT.*

#### How to use

1. Upload a dataset that includes question-answer pairs. Format questions and answers in two separate columns. Alternatively, a [sample dataset](https://zilliz.com/solutions/qa) is available for download.

2. After typing in your question, a list of similar questions will be retrieved from the uploaded dataset.

3. Reveal the answer by selecting the question most similar to your own.

👉Video：[[Demo] QA System Powered by Milvus](https://www.youtube.com/watch?v=ANgoyvgAxgU)

#### How it works

Questions are converted into feature vectors using Google’s BERT model, then Milvus is used to manage and query the dataset.

**Data processing:**

1. BERT is used to convert the uploaded question-answer pairs into 768-dimensional feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Question, and corresponding answer, vector IDs are stored in PostgreSQL.

**Searching for similar questions:**

1. BERT is used to extract feature vectors from a user's input question.
2. Milvus retrieves vector IDs for questions that are most similar to the input question.
3. The system looks up the corresponding answers in PostgreSQL.

 

### Reverse image search systems
Reverse image search is transforming e-commerce through personalized product recommendations and similar product lookup tools that can boost sales. In this application scenario, Zilliz built a reverse image search system by combining Milvus with [VGG](https://towardsdatascience.com/how-to-use-a-pre-trained-model-vgg-for-image-classification-8dd7c4a4a517), an ML model that can extract image features.

👉Source code：[zilliz-bootcamp/image_search](https://github.com/zilliz-bootcamp/image_search)

![2.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/2_09000e2e2e.jpeg)
###### *Reverse image search system built with Milvus and VGG.*

#### How to use
1. Upload a zipped image dataset comprised of .jpg images only (other image file types are not accepted). Alternatively, a [sample dataset](https://zilliz.com/solutions/image-search) is available for download.
2. Upload an image to use as the search input for finding similar images.

👉Video: [[Demo] Image Search Powered by Milvus](https://www.youtube.com/watch?v=mTO8YdQObKY)

#### How it works
Images are converted into 512-dimensional feature vectors using the VGG model, then Milvus is used to manage and query the dataset.

**Data processing:**

1. The VGG model is used to convert the uploaded image dataset to feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Image feature vectors, and corresponding image file paths, are stored in CacheDB.

**Searching for similar images:**

1. VGG is used to convert a user’s uploaded image into feature vectors.
2. Vector IDs of images most similar to the input image are retrieved from Milvus.
3. The system looks up the corresponding image file paths in CacheDB.

 

### Audio search systems
Speech, music, sound effects, and other types of audio search makes it possible to quickly query massive volumes of audio data and surface similar sounds. Applications include identifying similar sound effects, minimizing IP infringement, and more. To demonstrate this application scenario, Zilliz built a highly efficient audio similarity search system by combining Milvus with [PANNs](https://arxiv.org/abs/1912.10211)—a large-scale pretrained audio neural networks built for audio pattern recognition.

👉Source code：[zilliz-bootcamp/audio_search](https://github.com/zilliz-bootcamp/audio_search)
![3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/3_419bac3dd2.png)
###### *Audio search system built with Milvus and PANNs.*

#### How to use
1. Upload a zipped audio dataset comprised of .wav files only (other audio file types are not accepted). Alternatively, a [sample dataset](https://zilliz.com/solutions/audio-search) is available for download.
2. Upload a .wav file to use as the search input for finding similar audio.

👉Video: [[Demo] Audio Search Powered by Milvus](https://www.youtube.com/watch?v=0eQHeqriCXw)
#### How it works
Audio is converted into feature vectors using PANNs, large-scale pre-trained audio neural networks built for audio pattern recognition. Then Milvus is used to manage and query the dataset.

**Data processing:**

1. PANNs converts audio from the uploaded dataset to feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Audio feature vector IDs and their corresponding .wav file paths are stored in PostgreSQL.

**Searching for similar audio:**

1. PANNs is used to convert a user’s uploaded audio file into feature vectors.
2. Vector IDs of audio most similar to the uploaded file are retrieved from Milvus by calculating inner product (IP) distance.
3. The system looks up the corresponding audio file paths in MySQL.

 

### Video object detection (computer vision)
Video object detection has applications in computer vision, image retrieval, autonomous driving, and more. To demonstrate this application scenario, Zilliz built a video object detection system by combining Milvus with technologies and algorithms including [OpenCV](https://en.wikipedia.org/wiki/OpenCV), [YOLOv3](https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b), and [ResNet50](https://www.mathworks.com/help/deeplearning/ref/resnet50.html).

👉Source code: [zilliz-bootcamp/video_analysis](https://github.com/zilliz-bootcamp/video_analysis)

![4.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/4_54b4ceb2ad.png)
###### *Video object detection system with Milvus.*

#### How to use
1. Upload a zipped image dataset comprised of .jpg files only (other image file types are not accepted). Ensure that each image file is named by the object it depicts. Alternatively, a [sample dataset](https://zilliz.com/solutions/video-obj-analysis) is available for download. 
2. Upload a video to use for analysis.
3. Click the play button to view the uploaded video with object detection results shown in real time.

👉Video: [[Demo] Video Object Detection System Powered by Milvus](https://www.youtube.com/watch?v=m9rosLClByc)

#### How it works
Object images are converted into 2048-dimensional feature vectors using ResNet50. Then Milvus is used to manage and query the dataset.

**Data processing:**

1. ResNet50 converts object images to 2048-dimensional feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Audio feature vector IDs and their corresponding image file paths are stored in MySQL.

**Detecting objects in video:**

1. OpenCV is used to trim the video.
2. YOLOv3 is used to detect objects in the video.
3. ResNet50 converts detected object images into 2048-dimensional feature vectors.

Milvus searches for the most similar object images in the uploaded dataset. Corresponding object names and image file paths are retrieved from MySQL.

small_blog cover.png

thumbnail_blog cover.png

blog cover.png

How to Make 4 Popular AI Applications with Milvus

Ensuring investments in artificial intelligence are successful requires effectively managing the life cycle of AI models and the applications they are used in. This article explains the concept of machine learning operations, or MLOps, and how the open-source vector database Milvus can be used to advance MLOps programs of varying complexity.

Building machine learning (ML) applications is a complex and iterative process. As more companies realize the untapped potential of unstructured data, the demand for [AI-powered data processing and analytics](https://zilliz.com/blog/Thanks-to-Milvus-Anyone-Can-Build-a-Vector-Database-for-1-Billion-Images) will continue to rise. Without effective machine learning operations, or MLOps, most ML application investments will wither on the vine. Research has found that [as little as 5%](https://www.forbes.com/sites/cognitiveworld/2020/03/31/modelops-is-the-key-to-enterprise-ai/?sh=44c0f5066f5a) of the AI adoptions companies plan to deploy actually reach deployment. Many organizations incur "model debt," where changes in market conditions, and failure to adapt to them, result in unrealized investments in models that linger unrefreshed (or worse, never get deployed at all).

This article explains MLOps, a systemic approach to AI model life cycle management, and how the open-source vector data management platform [Milvus](https://milvus.io/) can be used to operationalize AI at scale.

 

### What is MLOps?

Machine learning operations (MLOps), also known as model operations (ModelOps) or AI model operationalization, is necessary to build, maintain, and deploy AI applications at scale. As companies seek to apply the AI models they develop to hundreds of different scenarios, it is mission critical that models in use, and those under development, are operationalized across the entire organization. MLOps involves monitoring a machine learning model throughout its lifecycle, and governing everything from underlying data to the effectiveness of a production system that relies on a particular model.

![01.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/01_362a07d156.jpg)

###### *MLOps governs the life cycle of a machine learning model.*

Gartner [defines](https://www.gartner.com/en/information-technology/glossary/modelops) ModelOps as the governance and life cycle management of a wide range of operationalized artificial intelligence and decision models. The core functionality of MLOps can be broken down as follows:

- **Continuous integration/continuous delivery (CI/CD):** A set of best practices borrowed from developer operations (DevOps), CI/CD is a method for delivering code changes more frequently and more reliably. [Continuous integration](https://www.gartner.com/en/information-technology/glossary/continuous-integration-ci) promotes implementing code changes in small batches while monitoring them with strict version control. [Continuous delivery](https://www.gartner.com/smarterwithgartner/5-steps-to-master-continuous-delivery/) automates the delivery of applications to various environments (e.g., testing and development environments). 

- **Model development environments (MDE):** A complex process for building, reviewing, documenting, and examining models, MDE help ensure models are created iteratively, documented as they are developed, trusted, and reproducible. Effective MDE ensures models can be explored, researched, and experimented on in a controlled manner.

- **Champion-challenger testing:** Similar to A/B testing methodology used by marketers, [champion-challenger testing](https://medium.com/decision-automation/what-is-champion-challenger-and-how-does-it-enable-choosing-the-right-decision-f57b8b653149) involves experimenting with different solutions to aid the decision-making process that proceeds committing to a single approach. This technique involves monitoring and measuring performance in real time to identify which deviation works best.

- **Model versioning:** As with any complex system, machine learning models are developed in steps by many different people—resulting in data management questions around versions of data and ML models. Model versioning helps manage and govern the iterative process of ML development where data, models, and code may evolve at different rates.

- **Model store and rollback:** When a model is deployed, its corresponding image file must be stored. Rollback and recovery abilities allow MLOps teams to revert to a previous model version if needed.

Using just one model in a production application presents a number of difficult challenges. MLOps is a structured, repeatable method that relies on tools, technology, and best practices to overcome technical or business problems that arise during a machine learning model’s life cycle. Successful MLOps maintains efficiency across teams working to build, deploy, monitor, retrain, and govern AI models and their use in production systems. 

 

### Why is MLOps necessary?

As depicted in the ML model life cycle above, building a machine learning model is an iterative process that involves incorporating new data, retraining models, and dealing with general model decay over time. These are all issues that traditional developer operations, or DevOps, does not address or provide solutions for. MLOps has become necessary as a way to manage investment in AI models and ensure a productive model life cycle. Because machine learning models will be leveraged by a variety of different production systems, MLOps becomes integral to making sure requirements can be met across different environments and amid varying scenarios.

![02.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/02_403e7f2fe2.jpg)

###### *A machine learning model being deployed in a cloud environment that feeds into an application.*

 

The simple illustration above depicts a machine learning model being deployed in a cloud environment that feeds into an application. In this basic scenario, a number of problems could arise that MLOps helps overcome. Because the production application relies on a specific cloud environment, there are latency requirements that the data scientists who developed the ML model don’t have access to. Operationalizing the model life cycle would make it possible for data scientists or engineers with deep knowledge of the model to identify and troubleshoot problems that arise in specific production environments. 

Not only are machine learning models trained in different environments from the production applications they are used in, but they also often rely on historical datasets that differ from the data used in production applications. With MLOps the entire data science team, from those developing the model to people working at the application level, have a means of sharing and requesting information and assistance. The rate at which data and markets change make it imperative that there is as little friction as possible between all key stakeholders and contributors that will come to depend on a given machine learning model. 

### Supporting the transition to Software 2.0

[Software 2.0](https://karpathy.medium.com/software-2-0-a64152b37c35) is the idea that software development will experience a paradigm shift as artificial intelligence increasingly plays a central role in writing AI models that power software applications. Under Software 1.0, development involves programmers writing explicit instructions using a specific programming language (e.g., Python, C++). Software 2.0 is far more abstract. Although people provide input data and set parameters, neural networks are difficult for humans to understand due to their sheer complexity— with typical networks containing millions of weights that impact outcomes (and sometimes billions or trillions). 

DevOps was built around Software 1.0’s reliance on specific instructions dictated by programmers using languages, but never considered the life cycle of a machine learning model that powers a variety of different applications. MLOps addresses the need for the process of managing software development to change alongside the software under development. As Software 2.0 becomes the new standard for computer-based problem solving, having the right tools and processes for managing model life cycles will make or break investments in new technology. Milvus is an open-source vector similarity search engine built to support the transition to Software 2.0 and manage model life cycles with MLOps.

![03.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/03_c63c501995.jpg)

###### *How Milvus supports the transition to Software 2.0.*

 

### Operationalizing AI at scale with Milvus

Milvus is a vector data management platform that was made specifically for storing, querying, updating, and maintaining massive, trillion-scale vector datasets. The platform powers vector similarity search and can integrate with widely adopted index libraries, including Faiss, NMSLIB, and Annoy. By pairing AI models that convert unstructured data to vectors with Milvus, applications spanning new drug development, biometric analysis, recommendation systems and much more can be created. 

[Vector similarity search](https://blog.milvus.io/vector-similarity-search-hides-in-plain-view-654f8152f8ab) is the go-to solution for unstructured data data processing and analytics, and vector data is quickly emerging as a core data type. A comprehensive data management system like Milvus facilitates operationalizing AI in many ways, including:

- Providing an environment for model training that ensures more aspects of development are done in one place, facilitating cross team collaboration, model governance, and more. 

- Offering a comprehensive set of APIs that support popular frameworks such as Python, Java, and Go, making it easy to integrate a common set of ML models. 

- Compatibility with Google Colaboratory, a Jupyter notebook environment that runs in a browser, simplifies the process of compiling Milvus from source code and running basic Python operations.

- Automated machine learning (AutoML) functionality makes it possible to automate the tasks associated with applying machine learning to real world problems. Not only does AutoML lead to efficiency improvements, but it makes it possible for non-experts to take advantage of machine learning models and techniques.


Regardless of the machine learning applications you’re building today, or the plans you have for applications in the future, Milvus is a flexible data management platform created with Software 2.0 and MLOps in mind. To learn more about Milvus or make contributions, find the project on [Github](https://github.com/milvus-io). To get involved with the community or ask questions, join our [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ) channel. Hungry for more content? Check out the following resources:

- [Milvus Is an Open-Source Scalable Vector Database](https://zilliz.com/blog/Milvus-Is-an-Open-Source-Scalable-Vector-Database)
- [Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search](https://zilliz.com/blog/Milvus-Was-Built-for-Massive-Scale-Think-Trillion-Vector-Similarity-Search)
- [Set Up Milvus in Google Colaboratory for Easy ML Application Building](https://zilliz.com/blog/Set-Up-Milvus-in-Google-Colaboratory-for-Easy-ML-Application-Building)

large_milvus.jpg

medium_milvus.jpg

small_milvus.jpg

thumbnail_milvus.jpg

milvus.jpg

MLOps is superseding DevOps as we make the transition to Software 2.0. Find  what model operations is, and how the open-source vector database Milvus supports it. 

Operationalize AI at Scale with Software 2.0, MLOps, and Milvus

Beike wanted to create a personalized home search experience. Discover how they leveraged Milvus to build an intelligent real estate search platform capable of providing tailored recommendations in near real-time. 

Artificial intelligence (AI) has [powerful applications](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#d62b) in real estate that are transforming the home search process. Tech savvy real estate professionals have been taking advantage of AI for years, recognizing its ability to help clients find the right home faster and simplify the process of purchasing property. The coronavirus pandemic has [accelerated](https://www.pwc.com/ca/en/industries/real-estate/emerging-trends-in-real-estate-2021/shifting-landscape-proptech.html) interest, adoption, and investement in property technology (or proptech) worldwide, suggesting it will play an increasingly greater role in the real estate industry moving forward. 

This article explores how Beike used vector similarity search to build a house hunting platform that provides personalized results and recommends listings in near real-time.

**Jump to:**

- [What is vector similarity search?](#what-is-vector-similarity-search)

- [How does Milvus manage vector data?](#how-does-milvus-manage-vector-data)

- [How does Beike use Milvus to make house hunting smarter?](#how-does-beike-use-milvus-to-make-house-hunting-smarter)

 

### What is vector similarity search?

[Vector similarity search](https://medium.com/unstructured-data-service/vector-similarity-search-hides-in-plain-view-654f8152f8ab) has applications spanning a wide variety of artificial intelligence, deep learning, and traditional vector calculation scenarios. The proliferation of AI technology is in part attributed to vector search and its ability to make sense of unstructured data, which includes things like images, video, audio, behavior data, documents, and much more.

Unstructured data makes up an estimated 80-90% of all data, and extracting insights from it is quickly becoming a requrement for businesses that want to remain competitive in an ever-changing world. Increasing demand for unstructured data analytics, rising compute power, and declining compute costs have made AI-enabled vector search more accessible than ever.

![beike-blog-img1.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/beike_blog_img1_2dc95cac08.jpg)
###### *Understanding the difference between structured and unstructured data.*

Traditionally, unstructured data has been a challenge to process and analyze at scale because it doesn't follow a predefined model or organizational structure. Neural networks (e.g., CNN, RNN, and BERT) make it possible to convert unstructured data into feature vectors, a numerical data format that can be easily interpreted by computers. Algorithms are then used to calculate similarity between vectors using metrics like cosine similarity or Euclidean distance.

Ultimately, vector similarity search is a broad term that desribes techniques for identifying similar things in massive datasets. Beike uses this technology to power an intelligent home search engine that automatically recommends listings based on individual user preferences, search history, and property criteria—accelerating the real estate search and purchase process. Milvus is an open-source vector database that connects information with algorithms, enabling Beike to develop and manage its AI real estate platform.

 

### How does Milvus manage vector data?

Milvus was built specifically for large-scale vector data management, and has applications spanning image and video search, chemical similarity analysis, personalized recommendation systems, conversational AI, and much more. Vector datasets stored in Milvus can be efficiently queried, with most implementations following this general process:

![beike-blog-img2.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/beike_blog_img2_d5abb58f95.jpg)
###### *How does Milvus manage vector data?*

 

### How does Beike use Milvus to make house hunting smarter?

Commonly described as China's answer to Zillow, [Beike](https://bj.ke.com/) is an online platform that allows real estate agents to list properties for rent or sale. To help improve the home search experience for house hunters, and to help agents close deals faster, the company built an AI-powered search engine for its listing database. Beike’s real estate listing database was converted into feature vectors then fed into Milvus for indexing and storage. Milvus is then used to conduct similarity search based on an input listing, search criteria, user profile, or other criteria.

For example, when searching for more homes similar to a given listing, features such as floor plan, size, orientation, interior finishings, paint colors, and more are extracted. Since the original database of property listings listing data has been [indexed](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212), searches can be conducted in mere milliseconds. Beike’s final product had an average query time of 113 milliseconds on a dataset containing over 3 million vectors. However, Milvus is capable of maintaining efficient speeds on trillion-scale datasets—making light work of this relatively small real estate database. In general, the system follows the following process:

1. Deep learning models (e.g., CNN, RNN, or BERT) convert unstructured data to feature vectors, which are then imported to Milvus.

2. Milvus stores and indexes the feature vectors.

3. Milvus returns similarity search results based on user queries. 

![milvus-overview-diagram.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/milvus_overview_diagram_d17cda0e47.png)

###### *An overview of Milvus.*

 

Beike’s intelligent real estate search platform is powered by a recommendation algorithm that calculates vector similarity using cosine distance. The system finds similar homes based on favorite listings and search criteria. At a high level, it works as follows:

1. Based on an input listing, characteristics such as floor plan, size, and orientation are used to extract 4 collections of feature vectors.

2. The extracted feature collections are used to perform similarity search in Milvus. Results of the query for each collection of vectors are a measure of similarity between the input listing and other similar listings. 

3. The search results from each of the 4 vector collections are compared then used to recommend similar homes. 

![beike-intelligent-house-platform-diagram.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/beike_intelligent_house_platform_diagram_6e278da118.jpg)

###### *An overview of Beike’s intelligent house hunting platform.*

 

As the figure above shows, the system implements an A/B table switching mechanism for updating data. Milvus stores the data for the first T days in table A, on day T+1 it starts to store data in table B, on day 2T+1, it starts to rewrite table A, and so forth. 

 

### To learn more about making things with Milvus, check out the following resources:

- [Building an AI-Powered Writing Assistant for WPS Office](https://zilliz.com/blog/Building-an-AI-Powered-Writing-Assistant-with-WPS-Office)

- [Making with Milvus: AI-Powered News Recommendation Inside Xiaomi's Mobile Browser](https://zilliz.com/blog/Making-with-Milvus-AI-Powered-News-Recommendation-Inside-Xiaomi-Mobile-Browser)

large_blog_realistate search.jpg

medium_blog_realistate search.jpg

small_blog_realistate search.jpg

thumbnail_blog_realistate search.jpg

blog_realistate search.jpg

AI is transforming the real estate industry, discover how intelligent proptech accelerates the home search and purchase process. 

Making With Milvus: AI-Infused Proptech for Personalized Real Estate Search

Learn how to use Milvus data migration tool to greatly improve efficiency of data management and reduce DevOps costs.

### Overview
[MilvusDM](https://github.com/milvus-io/milvus-tools) (Milvus Data Migration) is an open-source tool designed specifically for importing and exporting data files with Milvus. MilvusDM can greatly improve data mangement efficiency and reduce DevOps costs in the following ways:

- [Faiss to Milvus](#faiss-to-milvus): Import unzipped data from Faiss to Milvus.

- [HDF5 to Milvus](#hdf5-to-milvus): Import HDF5 files to Milvus.

- [Milvus to Milvus](#milvus-to-milvus): Migrate data from a source Milvus to a different target Milvus.

- [Milvus to HDF5](#milvus-to-hdf5): Save data in Milvus as HDF5 files.

![milvusdm blog 1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/milvusdm_blog_1_199cbdebe7.png)
###### *Importing and exporting data files with Milvus.*

MilvusDM is hosted on [Github](https://github.com/milvus-io/milvus-tools) and can be easily installed by running the command line ```pip3 install pymilvusdm```. MilvusDM allows you to migrate data in a specific collection or partition. In the following sections, we will explain how to use each data migration type. 

 


### Faiss to Milvus

#### Steps
1.Download **F2M.yaml**: 

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/F2
```

2.Set the following parameters:

- ``` data_path ```: Data path (vectors and their corresponding IDs) in Faiss.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: Delete data before insertion if the collection or partition already exists.

- ```dest_collection_name```: Name of receiving collection for data import.

- ```dest_partition_name```: Name of receiving partition for data import. 

- ```collection_parameter```: Collection-specific information such as vector dimension, index file size, and distance metric.

```
F2M:
 milvus_version: 1.0.0
 data_path: '/home/data/faiss.index'
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'append' # 'skip/append/overwrite'
 dest_collection_name: 'test'
 dest_partition_name: ''
 collection_parameter:
 dimension: 256
 index_file_size: 1024
 metric_type: 'L2'
```
3.Run **F2M.yaml:**

```
$ milvusdm --yaml F2M.yaml
```

#### Sample Code

1.Read Faiss files to retrieve vectors and their corresponding IDs.

```
ids, vectors = faiss_data.read_faiss_data()
```

2.Insert the retrieved data into Milvus: 

```
insert_milvus.insert_data(vectors, self.dest_collection_name, self.collection_parameter, self.mode, ids, self.dest_partition_name)
```

 

### HDF5 to Milvus
#### Steps
1.Download **H2M.yaml**.

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/H2M.yaml
```

2.Set the following parameters:

- ```data_path```: Path to the HDF5 files. 

- ```data_dir```: Directory holding the HDF5 files.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: Delete data before insertion if the collection or partition already exists.

- ```dest_collection_name```: Name of receiving collection for data import.

- ```dest_partition_name```: Name of receiving partition for data import.

- ```collection_parameter```: Collection-specific information such as vector dimension, index file size, and distance metric. 

> Set either ```data_path``` or ```data_dir```. Do **not** set both. Use ```data_path``` to specify multiple file paths, or ```data_dir``` to specify the directory holding your data file.

```
H2M:
 milvus-version: 1.0.0
 data_path:
 - /Users/zilliz/float_1.h5
 - /Users/zilliz/float_2.h5
 data_dir:
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'overwrite' # 'skip/append/overwrite'
 dest_collection_name: 'test_float'
 dest_partition_name: 'partition_1'
 collection_parameter:
 dimension: 128
 index_file_size: 1024
 metric_type: 'L2'
```
3.Run **H2M.yaml:**

```
$ milvusdm --yaml H2M.yaml
```

#### Sample Code

1.Read the HDF5 files to retrieve vectors and their corresponding IDs:

```
vectors, ids = self.file.read_hdf5_data()
```

2.Insert the retrieved data into Milvus:

```
ids = insert_milvus.insert_data(vectors, self.c_name, self.c_param, self.mode, ids,self.p_name)
```

 

### Milvus to Milvus
#### Steps
1.Download **M2M.yaml**.

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2M.yaml
```

2.Set the following parameters:

- ```source_milvus_path```: Source Milvus work path.

- ```mysql_parameter```: Source Milvus MySQL settings. If MySQL is not used, set mysql_parameter as ''.

- ```source_collection```: Names of the collection and its partitions in the source Milvus.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: If the collection or partition already exists, delete the data before inserting it.Delete data before insertion if the collection or partition already exists.

```
M2M:
 milvus_version: 1.0.0
 source_milvus_path: '/home/user/milvus'
 mysql_parameter:
 host: '127.0.0.1'
 user: 'root'
 port: 3306
 password: '123456'
 database: 'milvus'
 source_collection:
 test:
 - 'partition_1'
 - 'partition_2'
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'skip' # 'skip/append/overwrite'
```

3.Run **M2M.yaml.**

```
$ milvusdm --yaml M2M.yaml
```

#### Sample Code

1.According to a specified collection or partition's metadata, read the files under **milvus/db** on your local drive to retrieve vectors and their corresponding IDs from the source Milvus. 

```
collection_parameter, _ = milvus_meta.get_collection_info(collection_name)
r_vectors, r_ids, r_rows = milvusdb.read_milvus_file(self.milvus_meta, collection_name, partition_tag) 
```

2.Insert the retrieved data into the target Milvus.


```
milvus_insert.insert_data(r_vectors, collection_name, collection_parameter, self.mode, r_ids, partition_tag)
```

 

### Milvus to HDF5
#### Steps
1.Download **M2H.yaml**:

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2H.yaml
```


2.Set the following parameters:

- ```source_milvus_path```: Source Milvus work path.

- ```mysql_parameter```: Source Milvus MySQL settings. If MySQL is not used, set mysql_parameter as ''.

- ```source_collection```: Names of the collection and its partitions in the source Milvus.

- ```data_dir```: Directory for holding the saved HDF5 files.

```
M2H:
 milvus_version: 1.0.0
 source_milvus_path: '/home/user/milvus'
 mysql_parameter:
 host: '127.0.0.1'
 user: 'root'
 port: 3306
 password: '123456'
 database: 'milvus'
 source_collection: # specify the 'partition_1' and 'partition_2' partitions of the 'test' collection.
 test:
 - 'partition_1'
 - 'partition_2'
 data_dir: '/home/user/data'
```

3.Run **M2H.yaml**:

```
$ milvusdm --yaml M2H.yaml
```

#### Sample Code
1.According to a specified collection or partition's metadata, read the files under **milvus/db** on your local drive to retrieve vectors and their corresponding IDs.

```
collection_parameter, version = milvus_meta.get_collection_info(collection_name)
r_vectors, r_ids, r_rows = milvusdb.read_milvus_file(self.milvus_meta, collection_name, partition_tag)
```

2.Save the retrieved data as HDF5 files.

```
data_save.save_yaml(collection_name, partition_tag, collection_parameter, version, save_hdf5_name)
```

### MilvusDM File Structure

The flow chart below shows how MilvusDM performs different tasks according to the YAML file it receives:

![milvusdm blog 2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/milvusdm_blog_2_7824b16e5e.png)
###### *How does MilvusDM work?*

MilvusDM file structure:

- pymilvusdm

 - core

 - **milvus_client.py**: Performs client operations in Milvus.

 - **read_data.py**: Reads the HDF5 data files on your local drive. (Add your code here to support reading data files in other formats.)

 - **read_faiss_data.py**: Reads the data files in Faiss.

 - **read_milvus_data.py**: Reads the data files in Milvus.

 - **read_milvus_meta.py**: Reads the metadata in Milvus.

 - **data_to_milvus.py**: Creates collections or partitions based on parameters in YAML files and imports the vectors and the corresponding vector IDs into Milvus.

 - **save_data.py**: Saves the data as HDF5 files.

 - **write_logs.py**: Writes logs during runtime.

 - **faiss_to_milvus.py**: Imports data from Faiss into Milvus.

 - **hdf5_to_milvus.py**: Imports data in HDF5 files into Milvus.

 - **milvus_to_milvus.py**: Migrates data from a source Milvus to the target Milvus.

 - **milvus_to_hdf5.p**y: Exports data in Milvus and saves them as HDF5 files.

 - **main.py**: Performs corresponding tasks according to the received YAML file.

 - **setting.py**: Configurations relating to running the MilvusDM code.

- **setup.py**: Creates **pymilvusdm** file packages and uploads them to PyPI (Python Package Index).

 

### Recap

MilvusDM primarily handles migrating data in and out of Milvus, which includes Faiss to Milvus, HDF5 to Milvus, Milvus to Milvus, and Milvus to HDF5. 

The following features are planned for upcoming releases:

- Import binary data from Faiss to Milvus.

- Blocklist and allowlist for data migration between source Milvus and target Milvus.

- Merge and import data from multiple colletions or partitions in source Milvus into a new collection in target Milvus.

- Backup and recovery of the Milvus data.

The MilvusDM project is open sourced on [Github](https://github.com/milvus-io/milvus-tools). Any and all contributions to the project are welcome. Give it a star 🌟, and feel free to file an [issue](https://github.com/milvus-io/milvus-tools/issues) or submit your own code!

large_Generic Tool Announcement.jpg

medium_Generic Tool Announcement.jpg

small_Generic Tool Announcement.jpg

thumbnail_Generic Tool Announcement.jpg

Generic Tool Announcement.jpg

Introducing Milvus Data Migration Tool

iYUNDONG continually seeks to support its users with real-time image retrieval services. When it decided to develop its App, it chose Milvus to build its core image search system. Discover how Milvus, an open-source vector database, can help build an intelligent image retrieval system that can extract sport event highlights.

iYUNDONG is an Internet company aiming to engage more sport lovers and participants of events such as marathon races. It builds [artificial intelligence (AI)](https://en.wikipedia.org/wiki/Artificial_intelligence) tools that can analyze media captured during sporting events to automatically generate highlights. For example, by uploading a selfie, a user of the iYUNDONG sports App who took part in a sport event can instaneously retrieve his or her own photos or video clips from a massive media dataset of the event.

 

One of the key features of iYUNDONG App is called “Find me in motion“. Photographers usually take massive volumes of photos or videos during a sporting event such as a marathon race, and would upload the photos and videos in real time to the iYUNDONG media database. Marathon runners who want to see their highlighted moments can retrieve pictures including themselves simply by uploading one of their selfies. This saves them a lot of time because an image retrieval system in the iYUNDONG App does all the image matching. [Milvus](http://milvus.io/) is adopted by iYUNDONG to power this system as Milvus can greatly accelerate the retrieval process and return highly accurate results.

 

**Jump to:**
- [Difficulties and solutions](#difficulties-and-solutions)
- [What is Milvus](#what-is-milvus)
- [Why Milvus](#why-milvus)
- [System and Workflow](#system-and-workflow)
- [iYUNDONG App Interface](#iyundong-app-interface)
- [Conclusion](#conclusion)

 

### Difficulties and solutions

iYUNDONG faced the following issues and successfully found corresponding solutions when building its image retrieval system. 

- Event photos must be immediately available for search.

iYUNDONG developed a function called InstantUpload to ensure that event photos are available for search immediately after they are uploaded.

- Storage of massive datasets

Massive data such as photos and videos are uploaded to the iYUNDONG backend every millisecond. So iYUNDONG decided to migrate onto cloud storage systems including [AWS](https://aws.amazon.com/), [S3](https://aws.amazon.com/s3/?nc1=h_ls), and [Alibaba Cloud Object Storage Service (OSS)](https://www.alibabacloud.com/product/oss) for handling gargantuan volumes of unstructured data in a secure, fast and reliable way.

- Instant reading

In order to achieve instant reading, iYUNDONG developed its own sharding middleware to achieve horizontal scalability easily and mitigate the impact on the system from disk reading. In addition, [Redis](https://redis.io/) is used to serve as a caching layer to ensure consistent performance in situation of high concurrency.

- Instant extraction of facial features

In order to accurately and efficiently extract facial features from user-uploaded photos, iYUNDONG developed a proprietary image conversion algorithm which converts images into 128-dimensional feature vectors. Another issue encountered was that, oftentimes, many users and photographers uploaded images or videos simultaneously. So system engineers needed to take dynamic scalability into consideration when deploying the system. More specifically, iYUNDONG fully leveraged its elastic compute service (ECS) on the cloud to achieve dynamic scaling.

- Quick and large-scale vector search

iYUNDONG needed a vector database to store its large number of feature vectors extracted by AI models. According to its own unique business application scenario, iYUNDONG expected the vector database to be able to: 
1. Perform blazing fast vector retrieval on ultra-large datasets. 
2. Achieve mass storage at lower costs.

Initially, an average of 1 million images were processed annually, so iYUNDONG stored all its data for search in RAM. However, in the past two years, its business boomed and saw an exponential growth of unstructured data – the number of images in iYUNDONG’s database exceeded 60 milllion in 2019, meaning that there were more than 1 billion feature vectors that needed to be stored. A tremendous amount of data inevitably made the iYUNDONG system heavily-built and resource-consuming. So it had to continuously invest in hardware facilities to ensure high performance. Specifically, iYUNDONG deployed more search servers, larger RAM, and a better-performing CPU to achieve greater efficiency and horizontal scalability. However, one of the defects of this solution was that it drove the operating costs prohibitively high. Therefore, iYUNDONG started to explore a better solution to this issue and pondered on leveraging vector index libraries like Faiss to save costs and better steer its business. Finally iYUNDONG chose open-source vector database Milvus.

 

### What is Milvus

Milvus is an open-source vector database that is easy to use, highly flexible, reliable, and blazing fast. Combined with various deep learning models such as photo and voice recognition, video processing, natural language processing, Milvus can process and analyze unstructured data that are converted into vectors by using various AI algorithms. Below is the workflow of how Milvus processes all unstructured data:

● Unstructured data are converted into embedding vectors by deep learning models or other AI algorithms.

● Then embedding vectors are inserted into Milvus for storage. Milvus also builds indexes for those vectors.

● Milvus performs similarity search and returns accurate search results based on various business needs.

![iYUNDONG Blog 1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/i_YUNDONG_Blog_1_d8abe065ae.png)
###### *An overview of Milvus.*

 

### Why Milvus
Since the end of 2019, iYUNDONG has carried out a series of testings on using Milvus to power its image retrieval system. The testing results turned out that Milvus outperforms other mainstream vector databases as it supports multiple indexes and can efficiently reduce RAM usage, significantly compressing the timeline for vector similarity search. 

Moreover, new versions of Milvus are released regularly. Over the testing period, Milvus has went through multiple version updates from v0.6.0 to v0.10.1.

Additionally, with its active open-source community and powerful out-of-the-box features, Milvus allows iYUNDONG to operate on a tight development budget.

 

### System and Workflow
iYUNDONG’s system extracts facial features by detecting faces in event photos uploaded by photographers first. Then those facial features are converted into 128-dimensional vectors and stored in the Milvus library. Milvus creates indexes for those vectors and can instantaneously return highly accurate results. 

Other additional information such as photo IDs and coordinates indicating the position of a face in a photo are stored in a third-party database.

Each feature vector has its unique ID in the Milvus library. iYUNDONG adopted the [Leaf algorithm](https://github.com/Meituan-Dianping/Leaf), a distributed ID generation service developed by [Meituan](https://about.meituan.com/en) basic R&D platform, to associate the vector ID in Milvus with its corresponing additional information stored in another database. By combining the feature vector and the additional information, the iYUNDONG system can return similar results upon user search.

 

### iYUNDONG App Interface
A series of latest sports events are listed on the homepage. By tapping one of the events, users can see the full details.

After tapping the button on the top of the photo gallery page, users can then upload a photo of their own to retrieve images of their highlights. 

![iyundong-interface.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/iyundong_interface_3da684d206.jpg)
###### *iYUNDONG app interface.*

 

### Conclusion
This article introduces how iYUNDONG App builds an intelligent image retrieval system that can return accurate search results based on user uploaded photos varying in resolution, size, clarity, angle, and other ways that complicate similarity search. With the help of Milvus, iYUNDONG App can successfully run millisecond-level queries on a database of 60+ million images. And accuracy rate of photo retrieval is constantly above 92%. Milvus makes it easier for iYUNDONG to create a powerful, enterprise-grade image retrieval system in a short time with limited resources.


Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

large_blog_iyundong.jpg

medium_blog_iyundong.jpg

small_blog_iyundong.jpg

thumbnail_blog_iyundong.jpg

blog_iyundong.jpg

Making with Milvus: Intelligent image retrieval system for sports App iYUNDONG

Extracting Event Highlights Using iYUNDONG Sports App

We are proud to announce Milvus v1.0, a stable, long-term support version of the world’s most popular open-source vector database. Discover what Milvus v1.0 has to offer, and plans for the platform’s future.

Zilliz is proud to announce the release of Milvus v1.0. After months of extensive testing Milvus v1.0, which is based on a stable version of Milvus v0.10.6, is available for use.

Milvus v1.0 offers the following key features:

- Support for mainstream similarity metrics, including Euclidean distance, inner product, Hamming distance, Jaccard coefficient, and more.
- Integration with, and improvements to, SOTA ANNs algorithms, including Faiss, Hnswlib, Annoy, NSG, and more. 
- Scale-out capability through the Mishards sharding proxy.
- Support for processors commonly used in AI scenarios, including X86, Nvidia GPU, Xilinx FPGA, and more.

See the [Release Notes](https://www.milvus.io/docs/v1.0.0/release_notes.md) for additional Milvus v1.0 features. 

Milvus is an ongoing open-source software (OSS) project. Its first major release has the following implications for users:

- Milvus v1.0 will receive long-term support (3+ years).
- The most stable Milvus release to date is well structured and ready for integration with existing AI ecosystems.

### The first version of Milvus with long-term support

Thanks in part to sponsorship from Zilliz, the Milvus community will provide bug fix support for Milvus v1.0 until December 31st, 2024. New features will be available only in releases following v1.0.

See [The Milvus release guideline](https://milvus.io/docs/v1.0.0/milvus_release_guideline.md) for information about release cadences and more.

### Toolchain enhancements and seamless AI ecosystem integration

Beginning with v1.0, Milvus' toolchain will be a primary development focus. We plan to create the necessary tooling and utilities to meet the needs of the Milvus user community. 

Stability makes integrating Milvus with AI ecosystems a breeze. We are seeking further collaboration between the Milvus community and other AI-focused OSS communities. We encourage contributions to the new AI ASICs (application-specific integrated circuits) in Milvus. 

### The future of Milvus

We believe Milvus has a bright future thanks to the following factors:

- Regular contributions from developers in the Milvus community.
- Support for integration with any cloud-native environment.

We have drafted [community charters](https://milvus.io/docs/v1.0.0/milvus_community_charters.md) to help guide, nurture, and advance the Milvus community as our technology and user base grows. The charters include several technical decisions made to attract more participants to the community.

- Golang will now be used to develop the Milvus engine however, the ANNS algorithm component will still be developed in C++.
- The forthcoming distributed version of Milvus will use existing cloud components as much as possible.

We are thrilled to partner with the open-source software community to build the next-generation cloud data fabric made for AI. Let's get to work!

### Don’t be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/)
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

Milvus v1.0 is available now. This stable, long-term support version powers image/video search, chatbots, and many more AI applications.

Milvus 1.0: The World's Most Popular Open-Source Vector Database Just Got Better

Discover how Xiaomi leveraged AI and Milvus to build an intelligent news recommendation system capable of finding the most relevant content for users of its mobile web browser. 

From social media feeds to playlist recommendations on Spotify, [artificial intelligence](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View) already plays a major role in the content we see and interact with each day. In an effort to differentiate their mobile web browser, multinational electronics manufacturer Xiaomi built an AI-powered news recommendation engine. [Milvus](https://milvus.io/), an open-source vector database built specifically for similarity search and artificial intelligence, was used as the application’s core data management platform. This article explains how Xiaomi built its AI-powered news recommendation engine, and how Milvus and other AI algorithms were used. 

 


### Using AI to suggest personalized content and cut through news noise

With the New York Times alone publishing over [230 pieces](https://www.theatlantic.com/technology/archive/2016/05/how-many-stories-do-newspapers-publish-per-day/483845/) of content each day, the sheer volume of articles produced makes it impossible for individuals to get a comprehensive view of all the news. To help sift through large volumes of content, and recommend the most relevant or interesting pieces, we increasingly turn to AI. Although recommendations remain far from perfect, machine learning is increasingly necessary to cut through the constant stream of new information pouring out of our increasingly complex and interconnected world.

Xiaomi makes and invests in smartphones, mobile apps, laptops, home appliances, and many more products. In an effort to differentiate a mobile browser that comes preinstalled on many of 40+ million smartphones the company sells each quarter, Xiaomi built a news recommendation system into it. When users launch Xiaomi’s mobile browser, artificial intelligence is used to recommend similar content based on user search history, interests, and more. Milvus is an open-source vector similarity search database used to accelerate retrieval of related articles.

 

### How does AI-powered content recommendation work?

At its core, news recommendation (or any other type of content recommendation system) involves comparing input data to a massive database to find similar information. Successful content recommendation involves balancing relevance with timeliness, and efficiently incorporating huge volumes of new data—often in real time. 

To accommodate massive datasets, recommendation systems are typically divided into two stages:

1. **Retrieval**: During retrieval, content is narrowed down from the broader library based on user interests and behavior. In Xiaomi’s mobile browser, thousands of pieces of content are selected from a massive dataset that contains millions of news articles.
2. **Sorting**: Next, content selected during retrieval is sorted according to certain indicators before being pushed to the user. As users engage with recommended content, the system adapts in real time to provide more relevant suggestions. 

News content recommendations need to be made in real-time based on user behavior and recently published content. Additionally, suggested content must match user interests and search intent as much as possible.

 

### Milvus + BERT = intelligent content suggestions

Milvus is an open-source vector similarity search database that can be integrated with deep learning models to power applications spanning natural language processing, identity verification, and much more. Milvus indexes large vector datasets to make search more efficient, and supports a variety of popular AI frameworks to simplify the process of developing machine learning applications. These characteristics make the platform the ideal for storing and querying vector data, a critical component of many machine learning applications. 

Xiaomi selected Milvus to manage vector data for its intelligent news recommendation system because it is fast, reliable, and requires minimal configuration and maintenance. However, Milvus must be paired with an AI algorithm to build deployable applications. Xiaomi selected BERT, short for Bidirectional Encoder Representation Transformers, as the language representation model in its recommendation engine. BERT can be used as a general NLU (natural language understanding) model that can drive a number of different NLP (natural language processing) tasks. Its key features include:

- BERT’s transformer is used as the main framework of the algorithm and is capable of capturing explicit and implicit relationships within and between sentences.
- Multi-task learning goals, masked language modeling (MLM), and next sentence prediction (NSP).
- BERT performs better with greater amounts of data, and can enhance other natural language processing techniques such as Word2Vec by acting as a conversion matrix.

![Blog_Xiaomi_1.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Xiaomi_1_6301344312.jpeg)
###### *Milvus working with BERT.*

 

BERT’s network architecture uses a multi-layer transformer structure that abandons the traditional RNN and CNN neural networks. It works by converting the distance between two words at any position into one through its attention mechanism, and solves the dependency issue that has persisted in NLP for some time.

![Blog-Xiaomi-2.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Xiaomi_2_fe5cf2e401.jpeg)

###### *A transformer’s network architecture in BERT.*

 

![Blog-Xiaomi-3.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Xiaomi_3_5d10b51440.jpeg)

###### *BERT’s network structure. ‘Trm’ represents the transformer network architecture depicted above.*

 

BERT provides a simple and a complex model. The corresponding hyperparameters are as follows: BERT BASE: L = 12, H = 768, A = 12, total parameter 110M; BERT LARGE: L = 24, H = 1024, A = 16, the total number of parameters is 340M.

In the above hyperparameters, L represents the number of layers in the network (i.e. the number of Transformer blocks), A represents the number of self-Attention in Multi-Head Attention, and the filter size is 4H.

 


### Xiaomi’s content recommendation system 

Xiaomi browser-based news recommender system relies on three key components: vectorization, ID mapping, and approximate nearest neighbor (ANN) service. 

Vectorization is process where article titles are converted into general sentence vectors. The SimBert model, based on BERT, is used in Xiaomi’s recommendation system. SimBert is a 12-layer model with a hidden size of 768. Simbert uses the training model Chinese L-12_H-768_A-12 for continuous training (training task being “metric learning +UniLM”, and has trained 1.17 million steps on a signle TITAN RTX with the Adam optimizer (learning rate 2e-6, batch size 128). Simply put, this is an optimized BERT model.

ANN algorithms compare vectorized article titles to the entire news library stored in Milvus, then return similar content for users. ID mapping is used to obtain relevant information such as page views and clicks for corresponding articles.

![Blog-Xiaomi-N1.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Xiaomi_N1_f4749b3131.jpeg)

###### *How content recommendation works.*

 

The data stored in Milvus that powers Xiaomi’s news recommendation engine is constantly being updated, including additional articles and activity information. As the system incorporates new data, old data must be purged. In this system, full data updates are done for the first T-1 days and incremental updates are done in the subsequent T days.

At defined intervals, old data is deleted and processed data of the T-1 days is inserted into the collection. Here newly generated data is incorporated in real time. Once new data is inserted, similarity search is conducted in Milvus. Retrieved articles are again sorted by click rate and other factors, and the top content is shown to users. In a scenario like this where data is frequently updated and results must be delivered in real time, Milvus' ability to rapidly incorporate and search new data makes it possible to drastically accelerate news content recommendation in Xiaomi’s mobile browser. 

 

### Milvus makes vector similarity search better

Vectorizing data and then calculating similarity between vectors is the most commonly used retrieval technology. The rise of ANN-based vector similarity search engines has greatly improved the efficiency of vector similarity calculations. Compared with similar solutions, Milvus offers optimized data storage, abundant SDKs, and a distributed version that greatly reduces the workload of building a retrieval layer. Additionally, Milvus' active open-source community is a powerful resource that can help answer questions and troubleshoot problems as they arise.

If you would like to learn more about vector similarity search and Milvus, check out the following resources:

- Check out [Milvus](https://github.com/milvus-io/milvus) on Github. 
- [Vector Similarity Search Hides in Plain View](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View)
- [Accelerating Similarity Search on Really Big Data with Vector Indexing](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing)

Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

 Discover how Xiaomi leveraged AI and Milvus to build an intelligent news recommendation system capable of finding the most relevant content for users of its mobile web browser. 

Making with Milvus: AI-Powered News Recommendation Inside Xiaomi's Mobile Browser

How to build a deep learning powered recommender system

## Background Introduction

With the continuous development of network technology and the ever-expanding scale of e-commerce, the number and variety of goods grow rapidly and users need to spend a lot of time to find the goods they want to buy. This is information overload. In order to solve this problem, recommendation system came into being.

The recommendation system is a subset of the Information Filtering System, which can be used in a range of areas such as movies, music, e-commerce, and Feed stream recommendations. The recommendation system discovers the user’s personalized needs and interests by analyzing and mining user behaviors, and recommends information or products that may be of interest to the user. Unlike search engines, recommendation system do not require users to accurately describe their needs, but model their historical behavior to proactively provide information that meets user interests and needs.

In this article we use PaddlePaddle, a deep learning platform from Baidu, to build a model and combine Milvus, a vector similarity search engine, to build a personalized recommendation system that can quickly and accurately provide users with interesting information.

## Data Preparation

We take MovieLens Million Dataset (ml-1m) [1] as an example. The ml-1m dataset contains 1,000,000 reviews of 4,000 movies by 6,000 users, collected by the GroupLens Research lab. The original data includes feature data of the movie, user feature, and user rating of the movie, you can refer to ml-1m-README [2] .

ml-1m dataset includes 3 .dat articles: movies.dat、users.dat and ratings.dat.movies.dat includes movie’s features, see example below:

 MovieID::Title::Genres
 1::ToyStory(1995)::Animation|Children's|Comedy

This means that the movie id is 1, and the title is 《Toy Story》, which is divided into three categories. These three categories are animation, children, and comedy.

users.dat includes user’s features, see example below:

 UserID::Gender::Age::Occupation::Zip-code
 1::F::1::10::48067

This means that the user ID is 1, female, and younger than 18 years old. The occupation ID is 10.

ratings.dat includes the feature of movie rating, see example below:

UserID::MovieID::Rating::Timestamp
1::1193::5::978300760

That is, the user 1 evaluates the movie 1193 as 5 points.

## Fusion Recommendation Model

In the film personalized recommendation system we used the Fusion Recommendation Model [3] which PaddlePaddle has implemented. This model is created from its industrial practice.

First, take user features and movie features as input to the neural network, where:

- The user features incorporate four attribute information: user ID, gender, occupation, and age.
- The movie feature incorporate three attribute information: movie ID, movie type ID, and movie name.

For the user feature, map the user ID to a vector representation with a dimension size of 256, enter the fully connected layer, and do similar processing for the other three attributes. Then the feature representations of the four attributes are fully connected and added separately.

For movie features, the movie ID is processed in a manner similar to the user ID. The movie type ID is directly input into the fully connected layer in the form of a vector, and the movie name is represented by a fixed-length vector using a text convolutional neural network. The feature representations of the three attributes are then fully connected and added separately.

After obtaining the vector representation of the user and the movie, calculate the cosine similarity of them as the score of the personalized recommendation system. Finally, the square of the difference between the similarity score and the user’s true score is used as the loss function of the regression model.

![1-user-film-personalized-recommender-Milvus.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/1_user_film_personalized_recommender_Milvus_9ec39f501d.jpg)
###### *PaddlePaddle's fusion recommendation model.*

## System Overview

Combined with PaddlePaddle’s fusion recommendation model, the movie feature vector generated by the model is stored in the Milvus vector similarity search engine, and the user feature is used as the target vector to be searched. Similarity search is performed in Milvus to obtain the query result as the recommended movies for the user.

![2-system-overview.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/2_system_overview_5652afdca7.jpg)
###### *Fusion recommendation model combined with Milvus.*

> The inner product (IP) method is provided in Milvus to calculate the vector distance. After normalizing the data, the inner product similarity is consistent with the cosine similarity result in the fusion recommendation model.

## Application of Personal Recommender System

There are three steps in building a personalized recommendation system with Milvus, details on how to operate please refer to Mivus Bootcamp [4].

### Step 1：Model Training

 # run train.py
 $ python train.py

Running this command will generate a model recommender_system.inference.model in the directory, which can convert movie data and user data into feature vectors, and generate application data for Milvus to store and retrieve.

### Step 2：Data Preprocessing

 # Data preprocessing, -f followed by the parameter raw movie data file name
 $ python get_movies_data.py -f movies_origin.txt

Running this command will generate test data movies_data.txt in the directory to achieve pre-processing of movie data.

### Step 3：Implementing Personal Recommender System with Milvus

 # Implementing personal recommender system based on user conditions
 $ python infer_milvus.py -a <age>-g <gender>-j <job>[-i]

Running this command will implement personalized recommendations for specified users.

The main process is:

- Through the load_inference_model, the movie data is processed by the model to generate a movie feature vector.
- Load the movie feature vector into Milvus via milvus.insert.
- According to the user’s age / gender / occupation specified by the parameters, it is converted into a user feature vector, milvus.search_vectors is used for similarity retrieval, and the result with the highest similarity between the user and the movie is returned.

Prediction of the top five movies that the user is interested in:

 TopIdsTitleScore
 03030Yojimbo2.9444923996925354
 13871Shane2.8583481907844543
 23467Hud2.849525213241577
 31809Hana-bi2.826111316680908
 43184Montana2.8119677305221558 

## Summary

By inputing user information and movie information to the fusion recommendation model we can get matching scores, and then sort the scores of all movies based on the user to recommend movies that may be of interest to the user. **This article combines Milvus and PaddlePaddle to build a personalized recommendation system. Milvus, a vector search engine, is used to store all movie feature data, and then similarity retrieval is performed on user features in Milvus.** The search result is the movie ranking recommended by the system to the user.

Milvus [5] vector similarity search engine is compatible with various deep learning platforms, searching billions of vectors with only millisecond response. You can explore more possibilities of AI applications with Milvus with ease!

## Reference
1. MovieLens Million Dataset (ml-1m): http://files.grouplens.org/datasets/movielens/ml-1m.zip
2. ml-1m-README: http://files.grouplens.org/datasets/movielens/ml-1m-README.txt
3. Fusion Recommendation Model by PaddlePaddle: https://www.paddlepaddle.org.cn/documentation/docs/zh/beginners_guide/basics/recommender_system/index.html#id7
4. Bootcamp: https://github.com/milvus-io/bootcamp/blob/master/demo/recommender_system
5. Milvus: https://milvus.io/en/

small_header.jpg

thumbnail_header.jpg

header.jpg

Building Personalized Recommender Systems with Milvus and PaddlePaddle

Learn how Tokopedia used semantic search to make their search 10x smarter

At Tokopedia, we understand that the value in our product corpus is only unlocked when our buyers can find products that are relevant to them, so we strive to improve the relevance of search results.

To further that effort, we are introducing **similarity search** on Tokopedia. If you go to the search result page on mobile devices, you will find a “…” button that exposes a menu that gives you the option to search for products similar to the product.

## Keyword-based search
Tokopedia Search uses **Elasticsearch** for the search and ranking of products. For each search request, we first query Elasticsearch, which ranks products according to the search query. ElasticSearch stores each word as a sequence of numbers representing [ASCII](https://en.wikipedia.org/wiki/ASCII) (or UTF) codes for each letter. It builds an [inverted-index](https://en.wikipedia.org/wiki/Inverted_index) to quickly find out, which documents contain words from the user query, and then finds the best match among them using various scoring algorithms. These scoring algorithms pay little attention to what the words mean, but rather to how frequently they occur in the document, how close they are to each other, etc. ASCII representation obviously contains enough information to convey the semantics (after all we, humans, can understand it). Unfortunately, there’s no good algorithm for the computer to compare ASCII-encoded words by their meaning.

## Vector representation
One solution to this would be to come up with an alternative representation, which tells us not only about the letters contained in the word but also something about its meaning. For example, we could encode *which other words our word is frequently used together with* (represent by the probable context). We’d then assume that similar contexts represent similar things, and try to compare them using mathematical methods. We could even find a way to encode whole sentences by their meaning.

![Blog_How we used semantic search to make our search 10x smarter_2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_2_776af567a8.png)
###### *Vector representation also reveals the meaning of words compared with ASCII representation.*

## Select an embedding similarity search engine
Now that we have feature vectors, the remaining issue is how to retrieve from the large volume of vectors the ones that are similar to the target vector. When it comes to the embeddings search engine, we tried POC on several engines available on Github some of them are FAISS, Vearch, Milvus.

We prefer Milvus to other engines based on load test results. On the one hand, we have used FAISS before on other teams and hence would like to try something new. Compared to Milvus, FAISS is more of an underlying library, therefore not quite convenient to use. As we learned more about Milvus, we finally decided to adopt Milvus for its two main features:

- Milvus is very easy to use. All you need to do is to pull its Docker image and update the parameters based on your own scenario.

- It supports more indexes and has detailed supporting documentation.

In a nutshell, Milvus is very friendly to users and the documentation is quite detailed. If you come across any problem, you can usually find solutions in the documentation; otherwise, you can always get support from the Milvus community.

## Milvus cluster service
After deciding to use Milvus as the feature vector search engine, we decided to use Milvus for one of our Ads service use-case where we wanted to match [low fill rate](https://www.tradegecko.com/blog/wholesale-management/what-is-fill-rate-and-why-does-it-matter-for-wholesalers) keywords with high fill rate keywords. We configured a standalone node in a development (DEV) environment and started serving, it had been running well for a few days, and giving us improved CTR/CVR metrics. If a standalone node crashed in production, the entire service would become unavailable. Thus, we need to deploy a highly available search service.

Milvus provides both Mishards, a cluster sharding middleware, and Milvus-Helm for configuration. In Tokopedia we use Ansible playbooks for infrastructure setup so we created a playbook for infra orchestration. The diagram below from Milvus’ documentation shows how Mishards works:

![Blog_How we used semantic search to make our search 10x smarter_3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_3_4fa0c8a1a1.png)
###### *How Mishards works.*

Mishards cascade a request from upstream down to its sub-modules splitting the upstream request, and then collects and returns the results of the sub-services to upstream. The overall architecture of the Mishards-based cluster solution is shown below:
![Blog_How we used semantic search to make our search 10x smarter_4.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_4_724618be4e.jpeg)
###### *Overall architecture of Mishards.*

The official documentation provides a clear introduction of Mishards. You can refer to [Mishards](https://milvus.io/cn/docs/v0.10.2/mishards.md) if you are interested.

In our keyword-to-keyword service, we deployed one writable node, two read-only nodes, and one Mishards middleware instance in GCP, using Milvus ansible. It has been stable so far. A huge component of what makes it possible to efficiently query the million-, billion-, or even trillion-vector datasets that similarity search engines rely on is [indexing](https://milvus.io/docs/v0.10.5/index.md), a process of organizing data that drastically accelerates big data search.

## How does vector indexing accelerate similarity search?
Similarity search engines work by comparing input to a database to find objects that are most similar to the input. Indexing is the process of efficiently organizing data, and it plays a major role in making similarity search useful by dramatically accelerating time-consuming queries on large datasets. After a massive vector dataset is indexed, queries can be routed to clusters, or subsets of data, that are most likely to contain vectors similar to an input query. In practice, this means a certain degree of accuracy is sacrificed to speed up queries on really big vector data.

An analogy can be drawn to a dictionary, where words are sorted alphabetically. When looking up a word, it is possible to quickly navigate to a section that only contains words with the same initial — drastically accelerating the search for the input word’s definition.

## What next, you ask?
![Blog_How we used semantic search to make our search 10x smarter_5.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_5_035480c8af.jpeg)

As shown above, there is no solution that fits all, we always want to improve the model’s performance used for getting the embeddings.

Also, from a technical point of view, we want to run multiple learning models at the same time and compare the results from the various experiments. Watch this space for more information on our experiments like image search, video search.

 

## References:
- Mishards Docs：https://milvus.io/docs/v0.10.2/mishards.md
- Mishards: https://github.com/milvus-io/milvus/tree/master/shards
- Milvus-Helm: https://github.com/milvus-io/milvus-helm/tree/master/charts/milvus

 

*This blog article is reposted from: https://medium.com/tokopedia-engineering/how-we-used-semantic-search-to-make-our-search-10x-smarter-bd9c7f601821*

Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

large_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

medium_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

small_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

thumbnail_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

Blog_How we used semantic search to make our search 10x smarter_1.jpeg

How we used semantic search to make our search 10x smarter

Milvus was built from the ground up to power vector similarity search for huge volumes of virtually any data type. Discover how this open-source platform is optimized for fast and efficient massive-scale vector data management. 

Every day, an incalculable number of business-critical insights are squandered because companies can’t make sense of their own data. Unstructured data, such as text, image, video, and audio, is estimated to account for 80% of all data — but just 1% of it is ever analyzed. Fortunately, [artificial intelligence (AI)](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f), open-source software, and Moore’s law are making machine-scale analytics more accessible than ever before. Using vector similarity search, it is possible to extract value from massive unstructured datasets. This technique involves converting unstructured data into feature vectors, a machine-friendly numerical data format that can be processed and analyzed in real time.

Vector similarity search has applications spanning e-commerce, security, new drug development, and more. These solutions rely on dynamic datasets containing millions, billions, or even trillions of vectors, and their usefulness often depends on returning near instantaneous results. [Milvus](https://milvus.io/) is an open-source vector data management solution built from the ground up for efficiently managing and searching large vector datasets. This article covers Milvus’ approach to vector data management, as well as how the platform has been optimized for vector similarity search.

**Jump to:**
- [LSM trees keep dynamic data management efficient at massive scales](#lsm-trees-keep-dynamic-data-management-efficient-at-massive-scales)
- [Data management is optimized for rapid access and limited fragmentation](#data-management-is-optimized-for-rapid-access-and-limited-fragmentation)
- [Similarity searched is accelerated by indexing vector data](#similarity-searched-is-accelerated-by-indexing-vector-data)
- [Learn more about Milvus](#learn-more-about-Milvus)

### LSM trees keep dynamic data management efficient at massive scales

To provide efficient dynamic data management, Milvus uses a log-structured merge-tree (LSM tree) data structure. LSM trees are well suited for accessing data that has a high number of inserts and deletes. For detailed information on specific attributes of LSM trees that help ensure high-performance dynamic data management, see the [original research](http://paperhub.s3.amazonaws.com/18e91eb4db2114a06ea614f0384f2784.pdf) published by its inventors. LSM trees are the underlying data structure used by many popular databases, including [BigTable](https://cloud.google.com/bigtable), [Cassandra](https://cassandra.apache.org/), and [RocksDB](https://rocksdb.org/).

Vectors exist as entities in Milvus and are stored in segments. Each segment contains anywhere from one up to ~8 million entities. Each entity has fields for a unique ID and vector inputs, with the latter representing anywhere from 1 to 32768 dimensions.

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_2_492d31c7a0.png)
###### *A segment of 10-dimensional vectors in Milvus.*

### Data management is optimized for rapid access and limited fragmentation

When receiving an insert request, Milvus writes new data to the [write ahead log (WAL)](https://milvus.io/docs/v0.11.0/write_ahead_log.md). After the request is successfully recorded to the log file, the data is written to a mutable buffer. Finally, one of three triggers results in the buffer becoming immutable and flushing to disk:

1. **Timed intervals:** Data is regularly flushed to disk at defined intervals (1 second by default).
2. **Buffer size:** Accumulated data reaches the upper limit for the mutable buffer (128 MB).
3. **Manual trigger:** Data is manually flushed to disk when the client calls the flush function.

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_3_852dc2c9bb.png)
###### *An illustration of inserting vectors in Milvus.*

Users can add tens or millions of vectors at a time, generating data files of different sizes as new vectors are inserted. This results in fragmentation that can complicate data management and slow down vector similarity search. To prevent excessive data fragmentation, Milvus constantly merges data segments until the combined file size reaches a user configurable limit (e.g., 1 GB). For example, given an upper limit of 1 GB, inserting 100 million 512-dimensional vectors will result in just ~200 data files.

In incremental computation scenarios where vectors are inserted and searched concurrently, Milvus makes newly inserted vector data immediately available for search before merging it with other data. After data merges, the original data files will be removed and the newly created merged file will be used for search instead.

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_4.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_4_6bef3d914c.png)
###### *Queried data files before the merge.*

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_5.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_5_3851c2d789.png)
###### *Queried data files after the merge.*

### Similarity searched is accelerated by indexing vector data

By default, Milvus relies on brute-force search when querying vector data. Also known as exhaustive search, this approach checks all vector data each time a query is run. With datasets containing millions or billions of multi-dimensional vectors, this process is too slow to be useful in most similarity search scenarios. To help expedite query time, algorithms are used to build a vector index. The indexed data is clustered such that similar vectors are closer together, allowing the similarity search engine to query just a portion of the total data, drastically reducing query times while sacrificing accuracy.

Most of the vector index types supported by Milvus use approximate nearest neighbor (ANN) search algorithms. There are numerous ANN indexes, and each one comes with tradeoffs between performance, accuracy, and storage requirements. Milvus supports quantization-, graph, and tree-based indexes, all of which serve different application scenarios. See Milvus’ [technical documentation](https://milvus.io/docs/v0.11.0/index.md#CPU) for more information about building indexes and the specific types of vector indexes it supports.

Index building generates a lot of metadata. For example, indexing 100 million 512-dimensional vectors saved in 200 data files will result in an additional 200 index files. In order to efficiently check file statuses and delete or insert new files, an efficient metadata management system is required. Milvus uses online transactional processing (OLTP), a data processing technique that is well-suited for updating and/or deleting small amounts of data in a database. Milvus uses SQLite or MySQL to manage metadata.

### Learn more about Milvus

Milvus is an open-source vector data management platform currently in incubation at [LF AI & Data](https://lfaidata.foundation/), an umbrella organization of the Linux Foundation. Milvus was made open source in 2019 by [Zilliz](https://zilliz.com), the data science software company that initiated the project. More information about Milvus can be found on its [website](https://milvus.io/). If you’re interested in vector similarity search, or using AI to unlock the potential of unstructured data, please join our [open-source community](https://github.com/milvus-io) on GitHub.






large_Milvus Massive scale.jpg

medium_Milvus Massive scale.jpg

small_Milvus Massive scale.jpg

thumbnail_Milvus Massive scale.jpg

Milvus Massive scale.jpg

Explore the power of open-source in your next AI or machine learning project. Manage massive-scale vector data and power similarity search with Milvus. 

Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search

Artificial intelligence makes it possible to extract deep insights from massive unstructured datasets. By converting abstract data like video and audio into vectors, a conceptual task becomes a rigorous mathematical one well suited for machines. Milvus is a vector data management platform that is the ideal backbone for your next AI project.

Searching data using easily defined criteria, for example querying a movie database by actor, director, genre, or release date, is simple. A relational database is well equipped for these types of basic searches using a query language such as SQL. But when searches involve complex objects and more abstract queries, such as searching a video streaming library using natural language or a video clip, simple similarity metrics like matching words in a title or description are no longer sufficient.

[Artificial intelligence (AI)](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#d62b) has made computers significantly more capable of understanding the semantics of language, as well as helping people make sense of massive, difficult to analyze unstructured datasets (e.g., audio, video, documents, and social media data). AI makes it possible for Netflix to create sophisticated content recommendation engines, Google users to search the web by image, and pharmaceutical companies to discover new drugs.

### The challenge of searching large unstructured datasets

These feats of technology are accomplished by using AI algorithms to convert dense unstructured data into vectors, a numerical data format that is easily read by machines. Next, additional algorithms are used to calculate the similarity between vectors for a given search. The large size of unstructured datasets makes searching them in their entirety far too time consuming for most machine learning applications. To overcome this, approximate nearest neighbor (ANN) algorithms are used to cluster similar vectors together, then only search the portion of the dataset most likely to contain similar vectors to the target search vector.

This results in [drastically faster](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212#7a9a) (though slightly less accurate) similarity search, and is key to building useful AI tools. Thanks to vast public resources, it has never been easier or cheaper to build machine learning applications. However, AI-powered vector similarity search often requires interlacing different tools that vary in number and complexity depending on specific project requirements. Milvus is an open-source AI search engine that aims to simplify the process of building machine learning applications by providing robust functionality under a unified platform.

### What is Milvus?

[Milvus](https://milvus.io/) is an open-source data management platform built specifically for massive-scale vector data and streamlining machine learning operations (MLOps). Powered by Facebook AI Similarity Search (Faiss), Non-Metric Space Library (NMSLIB), and Annoy, Milvus brings a variety of powerful tools together in one place while extending their standalone functionality. The system was purpose built for storing, processing, and analyzing large vector datasets, and can be used to build AI applications spanning computer vision, recommendation engines, and much more.

![Blog_Milvus Is an Open-Source Scalable AI Search Engine_1.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Milvus_Is_an_Open_Source_Scalable_AI_Search_Engine_1_997255eb27.jpg)
###### *The structure of an AI-powered similarity search engine.*

### Milvus was made to power vector similarity search

Milvus was designed for flexibility, allowing developers to optimize the platform for their specific use case. Support for CPU/GPU-only and heterogeneous computing makes it possible to accelerate data processing and optimize resource requirements for any scenario. Data is stored in Milvus on a distributed architecture, making it trivial to scale data volumes. With support for various AI models, programming languages (e.g., C++, Java, and Python), and processor types (e.g., x86, ARM, GPU, TPU, and FPGA), Milvus offers high compatibility with a wide variety of hardware and software.

For more information about Milvus, check out the following resources:

- Explore Milvus' [technical documentation](https://milvus.io/) and learn more about the platform’s inner workings. 
- Learn how to launch Milvus, build applications, and more using [Milvus tutorials](https://tutorials.milvus.io/).
- Make contributions to the project and engage with Milvus' open-source community on [GitHub](https://github.com/milvus-io).





Build powerful machine learning applications and manage massive-scale vector data with Milvus.

Milvus Is an Open-Source Scalable Vector Database

Artificial intelligence influences what we read, the things we buy, who we become friends with, and more everyday human behavior. A huge part of what makes many modern AI applications possible is vector similarity search. Understanding how this underlying technology works can help demystify AI, providing insight into what computer intelligence is really made of. 

[Artificial intelligence (AI)](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#a291) has the potential to change the way even the most obscure things are done. For example, every year (before COVID, anyway) over 73,000 people congregate to compete in the Hong Kong Marathon. In order to properly sense and record finishing times for all race participants, organizers distribute 73,000 RFID chip timers to attach to each runner. Chip timing is a complex undertaking with obvious downsides. Materials (chips and electronic reading devices) must be purchased or rented from timing companies, and a registration area must be staffed for runners to collect chips on race day. Additionally, if sensors are installed only at the start and finish lines, it’s possible for unscrupulous runners to cut the course. 

![blog-1.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/blog_1_e55c133e05.jpeg)

###### Timing marathon runners is a logistical challenge few think about.

Now imagine a [video AI](https://cloud.google.com/video-intelligence) application capable of automatically identifying individual runners from footage captured at the finish line using a single photo. Rather than attach timing chips to each participant, runners simply upload a photo of themselves via an app after they cross the finish line. Instantly, a personalized highlight reel, race stats, and other relevant information is provided. Cameras installed at various points throughout the race can capture additional footage of participants and ensure each runner traverses the entire course. Which solution seems easier and more cost-effective to implement?

Although the Hong Kong Marathon doesn’t leverage machine learning to replace timing chips (yet), this example illustrates the potential AI has to drastically alter everything around us. For race timing, it reduces tens of thousands of chips to a few cameras paired with machine learning algorithms. But video AI is just one of many applications for vector similarity search, a process that uses artificial intelligence to analyze massive, trillion-scale unstructured datasets. This article provides an overview of vector search technology including what it is, how it can be used, as well as the open-source software and resources making it more accessible than ever before.





**Jump to:**

- [What is vector similarity search?](#what-is-vector-similarity-search)

- [What are some applications of vector similarity search?](#what-are-some-applications-of-vector-similarity-search)

- [Open-source vector similarity search software and resources.](#open-source-vector-similarity-search-software-and-resources)

### What is vector similarity search?

Video data is incredibly detailed and increasingly common, so logically it seems like it would be a great unsupervised learning signal for building video AI. In reality, this is not the case. Processing and analyzing video data, especially in large volumes, remains a [challenge for artificial intelligence](https://arxiv.org/pdf/1905.11954.pdf). Recent progress in this field, like much of the progress made in unstructured data analytics, is owed in large part to vector similarity search.

The problem with video, like all unstructured data, is that it doesn’t follow a predefined model or organizational structure, making it difficult to process and analyze at scale. Unstructured data includes things like images, audio, social media behavior, and documents, collectively accounting for an estimated 80-90%+ of all data. Companies are increasingly aware of the business-critical insights buried in massive, enigmatic unstructured datasets, driving demand for AI applications that can tap into this unrealized potential.

Using [neural networks](https://en.wikipedia.org/wiki/Neural_network) such as CNN, RNN, and BERT, unstructured data can be converted into feature vectors (aka embeddings), a machine-readable numerical data format. Algorithms are then used to calculate the similarity between vectors using measures like cosine similarity or Euclidean distance. Vector embedding and similarity search make it possible to analyze and build machine learning applications using previously indiscernible datasets.

Vector similarity is calculated using established algorithms however, unstructured datasets are typically massive. This means efficient and accurate search requires vast storage and compute power. To [accelerate similarity search](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212#7a9a) and reduce resource requirements, approximate nearest neighbor (ANN) search algorithms are used. By clustering similar vectors together, ANN algorithms make it possible to send queries to the clusters of vectors most likely to contain similar vectors, rather than searching the entire dataset. Although this approach is faster, it sacrifices some degree of accuracy. Leveraging ANN algorithms allows vector search to comb through billions of deep learning model insights in millisecond.


### What are some applications of vector similarity search?

Vector similarity search has applications spanning a wide variety of artificial intelligence, deep learning, and traditional vector calculation scenarios. The following provides a high-level overview of various vector similarity search applications:

**E-commerce:** Vector similarity search has broad applicability in e-commerce, including reverse image search engines that allow shoppers to search for products using an image captured on their smartphone or found online. Additionally, personalized recommendations based on user behavior, interests, purchase history, and more can be served by specialized recommender systems that rely on vector search.

**Physical & Cyber Security:** Video AI is just one of many applications for vector similarity search in the security field. Other scenarios include facial recognition, behavior tracing, identity authentication, intelligent access control, and more. Additionally, vector similarity search plays an important role in thwarting increasingly common and [sophisticated cyberattacks](https://www.wsj.com/articles/hack-suggests-new-scope-sophistication-for-cyberattacks-11608251360). For example, [code similarity search](https://medium.com/gsi-technology/application-of-ai-to-cybersecurity-part-3-19659bdb3422) can be used to identify security risks by comparing a piece of software to a database of known vulnerabilities or malware.

**Recommendation Engines:** Recommendation engines are systems that use machine learning and data analysis to suggest products, services, content, and information to users. User behavior, the behavior of similar users, and other data is processed using deep learning methods to generate recommendations. With enough data, algorithms can be trained to understand relationships between entities and invent ways to represent them autonomously. Recommendation systems have broad applicability and are something people already interact with every day, including content recommendations on Netflix, shopping recommendations on Amazon, and news feeds on Facebook.

**Chatbots:** Traditionally, chatbots are built using a regular knowledge graph that requires a large training dataset. However, chatbots built using deep learning models don’t need to preprocess data—instead, a map between frequent questions and answers is created. Using a pre-trained natural language processing (NLP) model, feature vectors can be extracted from the questions and then stored and queried using a [vector data management platform](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#92e0).

**Image or Video Search:** Deep learning networks have been used to recognize visual patterns since the late 1970s, and modern technology trends have made image and video search more powerful and accessible than ever before. 

**Chemical Similarity Search:** Chemical similarity is key to predicting the properties of chemical compounds and finding chemicals with specific attributes, making it indispensable to the development of new drugs. Fingerprints represented by feature vectors are created for each molecule, and then the distances between vectors are used to measure similarity. Using AI for new drug discovery is gaining momentum in the tech industry, with ByteDance (TikTok’s Chinese parent company) starting to [hire talent in the field](https://techcrunch.com/2020/12/23/bytedance-ai-drug/).


### Open-source vector similarity search software and resources.

Moore’s law, cloud computing, and declining resource costs are macro trends that have made artificial intelligence more accessible than ever. Thanks to open-source software and other publicly available resources, building AI/ML applications isn’t just for big tech companies. Below we provide a brief overview of Milvus, an open-source vector data management platform, and also highlight some publicly available datasets that help put AI within everyone’s reach.

#### Milvus, an open-source vector data management platform

[Milvus](https://milvus.io/) is an open-source vector data management platform built specifically for massive-scale vector data. Powered by Facebook AI Similarity Search (Faiss), Non-Metric Space Library (NMSLIB), and Annoy, Milvus brings a variety of powerful tools together under a single platform while extending their standalone functionality. The system was purpose built for storing, processing, and analyzing large vector datasets, and can be used to build all the AI applications (and more) mentioned above. 

More information about Milvus can be found on its [website](https://milvus.io/). Tutorials, instructions for setting up Milvus, benchmark testing, and information on building a variety of different applications is available in the [Milvus bootcamp](https://github.com/milvus-io/bootcamp). Developers interested in making contributions to the project can join Milvus' open-source community on [GitHub](https://github.com/milvus-io).

#### Public datasets for artificial intelligence and machine learning

It is no secret that technology giants like Google and Facebook have a data advantage over the little guys, with some pundits even advocating for a “[progressive data-sharing mandate](https://www.technologyreview.com/2019/06/06/135067/making-big-tech-companies-share-data-could-do-more-good-than-breaking-them-up/)” that would force companies that exceed a certain size to share some anonymized data with smaller rivals. Fortunately, there are thousands of publicly available datasets that can be used for AL/ML projects: 

- **The People’s Speech Dataset:** This [dataset from ML Commons](https://mlcommons.org/en/peoples-speech/) offers the largest speech dataset in the world, with over 87,000 hours of transcribed speech in 59 different languages. 

- **UC Irvine Machine Learning Repository:** The University of California at Irvine maintains [hundreds of public datasets](https://archive.ics.uci.edu/ml/index.php) in an effort to help the machine learning community. 

- **Data.gov:** The U.S. government offers [hundreds of thousands of open datasets](https://www.data.gov/) that span education, climate, COVID-19, and more. 

- **Eurostat:** The European Union’s statistical office provides [open datasets](https://ec.europa.eu/eurostat/data/database) spanning a variety of industries from economy and finance to population and social conditions. 

- **Harvard Dataverse:** [The Harvard Dataverse Repository](https://dataverse.harvard.edu/) is a free data repository open to researchers across disciplines. Many datasets are public, while others come with more restricted terms of use.

Although this list is by no means exhaustive, it is a good starting point for discovering the surprisingly wide variety of open datasets. For more information on public datasets as well as choosing the right data for your next ML or data science project, check out this [Medium post](https://altexsoft.medium.com/best-public-datasets-for-machine-learning-and-data-science-sources-and-advice-on-the-choice-636a0e754052).


## To learn more about vector similarity search, check out the following resources:
- [Thanks to Milvus, Anyone Can Build a Search Engine for 1+ Billion Images](https://zilliz.com/blog/Thanks-to-Milvus-Anyone-Can-Build-a-Vector-Database-for-1-Billion-Images)
- [Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search](https://zilliz.com/blog/Milvus-Was-Built-for-Massive-Scale-Think-Trillion-Vector-Similarity-Search)
- [Accelerating Similarity Search on Really Big Data with Vector Indexing](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing) 



large_plainview.jpg

medium_plainview.jpg

small_plainview.jpg

thumbnail_plainview.jpg

plainview.jpg

Discover what vector similarity search is, its various applications, and the public resources making artificial intelligence more accessible than ever.

Vector Similarity Search Hides in Plain View

Learn how to setup Milvus, an open-source vector data management platform, in Google Colaboratory. Thanks to Google Colab and Milvus, building machine learning applications has never been simpler.

Technological progress is perpetually making artificial intelligence (AI) and machine-scale analytics more accessible and easier to use. The [proliferation](https://techcrunch.com/2019/01/12/how-open-source-software-took-over-the-world/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAAL_qokucWT-HjbiOznq2RlO5TG78V6UYf332FKnWl3_knuMo6t5HZvZkIHFL3fhYX0nLzM6V1lVVAK4G3BXZHENX3zCXXgggGt39L9HKde3BufW1-iM2oKm0NIav2fcqgxvfpvx_7EPGstI7c_n99stI9oJf9sdsRPTQ6Wnu7DYX) of open-source software, public datasets, and other free tools are primary forces driving this trend. By pairing two free resources, [Milvus](https://milvus.io/) and [Google Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb#scrollTo=5fCEDCU_qrC0) (“Colab” for short), anyone can create powerful, flexible AI and data analytics solutions. This article provides instructions for setting up Milvus in Colab, as well as performing basic operations using the Python software development kit (SDK).

**Jump to:**

- [What is Milvus?](#what-is-milvus)
- [What is Google Colaboratory?](#what-is-google-colaboratory)
- [Getting started with Milvus in Google Colaboratory](#getting-started-with-milvus-in-google-colaboratory)
- [Run basic Milvus operations in Google Colab with Python](#run-basic-milvus-operations-in-google-colab-with-python)
- [Milvus and Google Colaboratory work beautifully together](#milvus-and-google-colaboratory-work-beautifully-together)


### What is Milvus?

[Milvus](https://milvus.io/) is an open-source vector similarity search engine that can integrate with widely adopted index libraries, including Faiss, NMSLIB, and Annoy. The platform also includes a comprehensive set of intuitive APIs. By pairing Milvus with artificial intelligence (AI) models, a wide variety of applications can be built including:

- Image, video, audio, and semantic text search engines.
- Recommendation systems and chatbots.
- New drug development, genetic screening, and other biomedical applications.


### What is Google Colaboratory?

[Google Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb#recent=true) is a product from the Google Research team that allows anyone to write and run python code from a web browser. Colab was built with machine learning and data analysis applications in mind, offers a free Jupyter notebook environment, syncs with Google Drive, and gives users access to powerful cloud computing resources (including GPUs). The platform supports many popular machine learning libraries and can be integrated with PyTorch, TensorFlow, Keras, and OpenCV.

### Getting started with Milvus in Google Colaboratory

Although Milvus recommends [using Docker](https://milvus.io/docs/v0.10.4/milvus_docker-cpu.md) to install and start the service, the current Google Colab cloud environment does not support Docker installation. Additionally, this tutorial aims to be as accessible as possible — and not everyone uses Docker. Install and start the system by [compiling Milvus’ source code](https://github.com/milvus-io/milvus/blob/0.10.4/INSTALL.md) to avoid using Docker.

### Download Milvus’ source code and create a new notebook in Colab

Google Colab comes with all supporting software for Milvus preinstalled, including required compilation tools GCC, CMake, and Git and drivers CUDA and NVIDIA, simplifying the installation and setup process for Milvus. To begin, download Milvus’ source code and create a new notebook in Google Colab:

1. Download Milvus’ source code: Milvus_tutorial.ipynb.

```Wget https://raw.githubusercontent.com/milvus-io/bootcamp/0.10.0/getting_started/basics/milvus_tutorial/Milvus_tutorial.ipynb```

2. Upload Milvus’ source code to [Google Colab](https://colab.research.google.com/notebooks/intro.ipynb#recent=true) and create a new notebook.

![Blog_Set Up Milvus in Google Colaboratory for Easy ML Application Building_2.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Set_Up_Milvus_in_Google_Colaboratory_for_Easy_ML_Application_Building_2_27809b0ce2.png)

### Compile Milvus from source code
#### Download Milvus source code

```git clone -b 0.10.3 https://github.com/milvus-io/milvus.git```

#### Install dependencies

```% cd /content/milvus/core ./ubuntu_build_deps.sh./ubuntu_build_deps.sh```

#### Build Milvus source code

```
% cd /content/milvus/core
!ls
!./build.sh -t Release
# To build GPU version, add -g option, and switch the notebook settings with GPU
#((Edit -> Notebook settings -> select GPU))
# !./build.sh -t Release -g
```

> Note: If the GPU version is correctly compiled, a “GPU resources ENABLED!” notice appears.

### Launch Milvus server

#### Add lib/ directory to LD_LIBRARY_PATH:

```
% cd /content/milvus/core/milvus
! echo $LD_LIBRARY_PATH
import os
os.environ['LD_LIBRARY_PATH'] +=":/content/milvus/core/milvus/lib"
! echo $LD_LIBRARY_PATH
```

#### Start and run Milvus server in the background:

```
% cd scripts
! ls
! nohup ./start_server.sh &
```

#### Show Milvus server status:

```
! ls
! cat nohup.out
```

> Note: If the Milvus server is launched successfully, the following prompt appears:

![Blog_Set Up Milvus in Google Colaboratory for Easy ML Application Building_3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Set_Up_Milvus_in_Google_Colaboratory_for_Easy_ML_Application_Building_3_b15138cd59.png)


###### *Milvus server started successfully!*

### Run basic Milvus operations in Google Colab with Python

After successfully launching in Google Colab, Milvus can provide a variety of API interfaces for Python, Java, Go, Restful, and C++. Below are instructions for using the Python interface to perform basic Milvus operations in Colab.

#### Install pymilvus:

```! pip install pymilvus==0.2.14```

#### Connect to the server:

```
# Connect to Milvus Server
milvus = Milvus(_HOST, _PORT)


# Return the status of the Milvus server.
server_status = milvus.server_status(timeout=10)
```

#### Create a collection/partition/index:

```
# Information needed to create a collection
param={'collection_name':collection_name, 'dimension': _DIM, 'index_file_size': _INDEX_FILE_SIZE, 'metric_type': MetricType.L2}

# Create a collection.
milvus.create_collection(param, timeout=10)

# Create a partition for a collection.
milvus.create_partition(collection_name=collection_name, partition_tag=partition_tag, timeout=10) 
ivf_param = {'nlist': 16384}

# Create index for a collection.
milvus.create_index(collection_name=collection_name, index_type=IndexType.IVF_FLAT, params=ivf_param)
```

#### Insert and flush:

```
# Insert vectors to a collection.
milvus.insert(collection_name=collection_name, records=vectors, ids=ids)

# Flush vector data in one collection or multiple collections to disk.
milvus.flush(collection_name_array=[collection_name], timeout=None)
```

#### Load and search:

```
# Load a collection for caching.
milvus.load_collection(collection_name=collection_name, timeout=None)

# Search vectors in a collection.
search_param = { "nprobe": 16 }
milvus.search(collection_name=collection_name,query_records=[vectors[0]],partition_tags=None,top_k=10,params=search_param)
```

#### Get collection/index information:

```
# Return information of a collection. milvus.get_collection_info(collection_name=collection_name, timeout=10)

# Show index information of a collection. milvus.get_index_info(collection_name=collection_name, timeout=10)
```


#### Get vectors by ID:

```
# List the ids in segment
# you can get the segment_name list by get_collection_stats() function.
milvus.list_id_in_segment(collection_name =collection_name, segment_name='1600328539015368000', timeout=None)

# Return raw vectors according to ids, and you can get the ids list by list_id_in_segment() function.
milvus.get_entity_by_id(collection_name=collection_name, ids=[0], timeout=None)
```


#### Get/set parameters:
```
# Get Milvus configurations. milvus.get_config(parent_key='cache', child_key='cache_size')

# Set Milvus configurations. milvus.set_config(parent_key='cache', child_key='cache_size', value='5G')
```

#### Delete index/vectors/partition/collection:
```
# Remove an index. milvus.drop_index(collection_name=collection_name, timeout=None)

# Delete vectors in a collection by vector ID. 
# id_array (list[int]) -- list of vector id milvus.delete_entity_by_id(collection_name=collection_name, id_array=[0], timeout=None)

# Delete a partition in a collection. milvus.drop_partition(collection_name=collection_name, partition_tag=partition_tag, timeout=None)

# Delete a collection by name. milvus.drop_collection(collection_name=collection_name, timeout=10)
```

 

### Milvus and Google Colaboratory work beautifully together

Google Colaboratory is a free and intuitive cloud service that greatly simplifies compiling Milvus from source code and running basic Python operations. Both resources are available for anyone to use, making AI and machine learning technology more accessible to everyone. For more information about Milvus, check out the following resources:

- For additional tutorials covering a wide variety of applications, visit [Milvus Bootcamp](https://github.com/milvus-io/bootcamp).
- For developers interested in making contributions or leveraging the system, find [Milvus on GitHub](https://github.com/milvus-io/milvus).
- For more information about the company that launched Milvus, visit [Zilliz.com](https://zilliz.com/).

large_3.jpg

medium_3.jpg

small_3.jpg

thumbnail_3.jpg

3.jpg

Google Colab makes developing and testing machine learning applications a breeze. Learn how to setup Milvus in Colab for better massive-scale vector data management.

Set Up Milvus in Google Colaboratory for Easy ML Application Building

In this article, we will demonstrate how to build a graph-based recommendation system by using PinSage (a GCN algorithm), DGL package, MovieLens datasets and Milvus.

## Background 

A recommendation system [1] (RS) can identify user preferences based on their historical data and suggest products or items to them accordingly. Companies will enjoy considerable economic benefits from a well-designed recommendation system.  

There are three elements in a complete set of recommendation system: user model, object model, and the core element, recommendation algorithm. Currently, established algorithms include collaborative filtering, implicit semantic modelling, graph-based modelling, combined recommendation, and more. In this article, we will provide some brief instructions on how to use Milvus to build a graph-based recommendation system.

## Graph Convolutional Neural (GCN) Networks

### PinSage

Users tag contents to their interest (pins) and related categories (boards) on Pinterest’s website, accumulating 2 billion pins, 1 billion boards and 18 billion edges (an edge is created only when the pin falls into a specific board). The following illustration is a pins-boards bipartite graph.

![1.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/1_742d28f7a9.png)
###### *A pins-boards bipartite graph.*

PinSage uses pins-boards bipartite graph to generate high-quality embeddings from pins for recommendations tasks such as pins recommendation. It has three key innovations:

### Dynamic convolutions

Unlike the traditional GCN algorithms, which perform convolutions on the feature matrices and the full graph, PinSage samples the neighborhood of the nodes, and performs more efficient local convolutions through dynamic construction of computational graph.

### Constructing convolutions with random walk modelling

Performing convolutions on the entire neighborhood of the node will result in a massive computational graph. To reduce the computation required, traditional GCN algorithms examine k-hop neighbor; PinSage simulates random walk to set the highly-visited contents as the key neighborhood and construct convolution based on it.

### Efficient MapReduce Inference

Performing local convolution on nodes takes with it the problem of repeated computation. This is because k-hop neighborhood overlaps. In each aggregate step, PinSage maps all nodes without repeated calculation, links them to the corresponding upper-level nodes, and then retrieves the embeddings of the upper-level nodes.

### DGL

Deep Graph Library (DGL)[4] is a Python package designed for building graph-based neural network model on top of existing deep learning frameworks, such as PyTorch, MXNet, Gluon, and more. With its easy-to-use backend interfaces, DGL can be readily implanted to frameworks that are based on tensor and supporting auto-generation. The PinSage algorithm that this article is dealing with is optimized based on DGL and PyTorch.
https://github.com/dmlc/dgl/tree/master/examples/pytorch/pinsage

### Milvus

The next thing to obtaining embeddings is to conduct similarity search in these embeddings to find items that might be of interest.

Milvus[5] is an open-source AI-powered similarity search engine supporting a wide variety of unstructured data-converted vectors. It has been adopted by 400+ enterprise users, and has applications spanning image processing, computer vision, natural language processing (NLP), speech recognition, recommendation engines, search engines, new drug development, gene analysis, and more. The following shows a general similarity search process using Milvus:

1. The user uses deep learning models to convert unstructured data to feature vectors and import them to Milvus.
2. Milvus stores and builds indexes for the feature vectors.
3. After receiving a vector query from the user, Milvus outputs a result similar to the input vector. Upon request, Milvus searches and returns vectors most similar to the input vectors.

![beike-intelligent-house-platform-diagram.jpg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/beike_intelligent_house_platform_diagram_6e278da118.jpg)
###### *Similarity search process using Milvus.*

## Implementation of Recommendation System

### System Overview

Here we will use the following figure to illustrate the basic process of building a graph-based recommendation system with Milvus. The basic process includes data preprocessing, PinSage model training, data loading, searching, and recommending.

![3-building-graph-based-recommender-system.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/3_building_graph_based_recommender_system_bf89770634.png)
###### *Building a graph-based recommendation system with Milvus.*

### Data Preprocessing

The recommendation system we build in this article is based on the open data sets MovieLens[5] (m1–1m), which contain 1,000,000 ratings of 4,000 movies by 6,000 users. Collected by GroupLens Research Labs, the data include movie information, user characteristics, and ratings of movies. In this article, we will use users’ movie history to build a graph with classification characteristics, a users-movies bipartite graph g.

### Build graph

    graph_builder = PandasGraphBuilder()
    graph_builder.add_entities(users, 'user_id', 'user')
    graph_builder.add_entities(movies_categorical, 'movie_id', 'movie')
    graph_builder.add_binary_relations(ratings, 'user_id', 'movie_id', 'watched')
    graph_builder.add_binary_relations(ratings, 'movie_id', 'user_id', 'watched-by')
    g = graph_builder.build()

### PinSage Model Training

The embedding vectors of pins generated by using the PinSage model are feature vectors of the acquired movie info. First, create a PinSage model according to the bipartite graph g and the customized movie feature vector dimensions (which is 256-dimension at default). Then, train the model with PyTorch to obtain the h_item embeddings of 4000 movies.

    # Define the model
    model = PinSAGEModel(g, item_ntype, textset, args.hidden_dims, args.num_layers).to(device)
    opt = torch.optim.Adam(model.parameters(), lr=args.lr)
    # Get the item embeddings
    for blocks in dataloader_test:
       for i in range(len(blocks)):
       blocks[i] = blocks[i].to(device)
       h_item_batches.append(model.get_repr(blocks))
    h_item = torch.cat(h_item_batches, 0)

### Data Loading

Load the movie embeddings h_item generated by the PinSage model into Milvus, and Milvus will return the corresponding IDs. Import the IDs and the corresponding movie information into MySQL.

    # Load data to Milvus and MySQL
    status, ids = milvus.insert(milvus_table, h_item)
    load_movies_to_mysql(milvus_table, ids_info)

### Searching

Get the corresponding embeddings in Milvus based on the movie IDs and carry out a similarity search with these embeddings in Milvus. Then, find the corresponding movie information in a MySQL database accordingly.

    # Get embeddings that users like
    _, user_like_vectors = milvus.get_entity_by_id(milvus_table, ids)
    # Get the information with similar movies
    _, ids = milvus.search(param = {milvus_table, user_like_vectors, top_k})
    sql = "select * from " + movies_table + " where milvus_id=" + ids + ";"
    results = cursor.execute(sql).fetchall()

### Recommendation

Finally, the system will recommend movies most similar to the search queries to the users.

Above is the main workflow of building a recommendation system. For more details, see Milvus-Bootcamp: https://github.com/milvus-io/bootcamp/tree/0.10.0/solutions/graph_based_recommend.

## System Demo

In addition to a FastAPI method, the project also has a front-end demo. By simulating the process of a user clicking on the movies to his liking, the demo makes movie recommendations.

The system also provides a FastAPI interface and front-end display that recommend movies catering to users’ tastes. You can simulate the process by logging into the Movie Recommendation System and mark the movies you like.

![4-system-demo-milvus-graph-based-recommender-system.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/4_system_demo_milvus_graph_based_recommender_system_83202824d2.png)

![5-system-demo-milvus-graph-based-recommender-system.gif](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/5_system_demo_milvus_graph_based_recommender_system_bd276f715b.gif)

## Conclusion

PinSage is a graph convolutional neural network that can be used for recommendation tasks. It generates high-quality embeddings of pins via a pins-boards bipartite graph.
We use the MovieLens datasets to create a users-movies bipartite graph, and the DGL open-source package and the PinSage model to generate feature vectors of movies.

The vectors are then stored in Milvus, a similarity embeddings search engine. Recommendations of movies are returned to users afterwards.

Milvus embedding vector similarity search engine can be integrated into a wide variety of deep learning platforms and multiple AI scenarios. By fully leveraging the optimized vector retrieval algorithms and integrated heterogeneous computing resources, Milvus can continually empower companies with vector retrieval capabilities.

## References

1. https://patentimages.storage.googleapis.com/0e/96/31/98058cb476cd77/CN105913296A.pdf
2. Graph Convolutional Neural Networks for Web-Scale Recommender Systems, arxiv: 1806.01973
3. https://medium.com/pinterest-engineering/pinsage-a-new-graph-convolutional-neural-network-for-web-scale-recommender-systems-88795a107f48
4. https://docs.dgl.ai/en/latest/
5. http://files.grouplens.org/datasets/movielens/ml-1m.zip

## 🤗 Join the Community
Don’t be a stranger, follow us on [Twitter](https://twitter.com/milvusio/) or join us on [Slack](https://milvusio.slack.com/join/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ)!

large_thisisengineering-raeng-z3cMjI6kP_I-unsplash.jpg

medium_thisisengineering-raeng-z3cMjI6kP_I-unsplash.jpg

small_thisisengineering-raeng-z3cMjI6kP_I-unsplash.jpg

thumbnail_thisisengineering-raeng-z3cMjI6kP_I-unsplash.jpg

thisisengineering-raeng-z3cMjI6kP_I-unsplash.jpg

Learn how to build a graph-based recommendation system using PinSage, Deep Graph Library (DGL), MovieLens datasets, and Milvus.

Graph-based Recommendation System with Milvus

**Interview subject:** Charles Xie


**To begin, how are you and your family doing amid the COVID-19 pandemic?** 

We are doing quite well, all things considered. Thanks for asking. The pandemic is largely under control here in China and daily life has some semblance of normality. That said, travel remains restricted and until vaccines are distributed globally, we must remain vigilant. We definitely feel fortunate for our circumstances.


**Tell us about yourself, your career, and founding Zilliz.**  

My technical background is in database systems and artificial intelligence. I spent over six years as a software engineer at Oracle’s U.S. headquarters where I became a founding member of the Oracle 12c Multi-tenant Database, a cloud enabled version of Oracle’s database management system.  

 
I founded Zilliz, an open-source software company specializing in developing AI-powered unstructured data processing and analysis platforms.  A commonly cited statistic is that around 80% of all data is unstructured, such as images, videos, audio, molecularstructures, and gene sequences. But just 1% of this data gets analyzed due to processing complexities. This means the vast majority of data people and businesses create is never utilized, which is clearly a waste. With Zilliz, we strive to advance data science for everyone by building accessible tools that extract value from unstructured data.
 

**How does Zilliz innovate?**  

Zilliz innovates in part thanks to the adoption of an open-source software development model. We listen closely to our users and the open-source community that makes our work possible, continuously iterating to maximize the utility of everything we build. Because unstructured data processing and analysis is somewhat of an emerging field, remaining agile has been paramount to our relevance and success.  


Innovation also happens through transparency, a core value in our company culture. We believe the best ideas are often found where you least expect them, and encourage Zillizers to always speak their minds. In practice, this means prioritizing impact over hierarchy and doing everything possible to support and develop our employees. 


**Can you tell us about your nickname and how you got it?** 

This is part of the company culture, which underscores teamwork. Everyone at Zilliz goes by nicknames. These can be any historic figure or even a cartoon character. The goal is not just for fun, it encourages everyone on the team to better understand each other, and to be a leader in his own area. Mine is Starlord, a reference to Marvel’s Guardians of the Galaxy. To me, this is more about working as a team. Through close teamwork, a team could achieve mission impossible. 


**Has the pandemic impacted your business? If so, what have you done to cope?**

We recently [closed our Series B funding round](https://zilliz.com/news/zilliz-raises-$43M-in-series-B-funding-to-build-AI-powered-unstructured-data-analytics) despite instability looming over the economy. So, from a financial perspective, we again feel very fortunate. However, our plans to expand Zilliz internationally have been complicated by the pandemic. That said, the setbacks are manageable and have largely meant changes to our timeline rather than any material shifts in strategy.  Still, we keep patient and practice prudence in times of uncertainty. 

Setbacks are basically negligible. We are an open-source company, and our distributed team can choose to work from home. Our pace remains unchanged despite the pandemic. We are also optimizing our tooling and workflow company-wide to improve work efficiency and to help the team keep a work-life balance. We foresee a big shift towards distributed working post-pandemic. Hopefully we will soon have more to share in this regard.


**How do you manage stress?** 


Diet, exercise, and overcoming fear of being wrong. For anyone running a business stress is par for the course, and lifestyle choices undoubtably impact your ability to handle pressure. But at the end of the day, a smart team that keeps you in check and is unafraid to point out your shortcomings is priceless. When stressful situations arise, I aim to react quickly and decidedly. By having trust in myself and the people around me, stress is kept at bay and less likely to cloud my judgement. I believe this is true not just in business, but life in general. Surround yourself with good people that help you be the best version of yourself.  
 

I believe all stress comes from fear of uncertainty. Rather than drawing back in fear, I would try to overcome fear of being wrong. Reading and deep thinking add a new dimension to my understanding of the world in times of uncertainty. No matter how busy I am, I would spare one hour every day for extensive reading from art, management, to literature. One should keep learning to improve his ability to handle stress and identify certainty from uncertainty.  


**Who are your competitors and how do you plan to stay ahead of them**?
 

Direct competitors, meaning companies also offering full-fledged unstructured data analysis and processing platforms, are few and far between. There are companies creating specialized tools that compete directly with software we’ve developed. For example, Milvus, an open-source vector similarity search engine initiated by Zilliz, faces competition from systems that offer comparable functionality. Our focus on breadth and depth is what sets us apart. By offering more tools, with greater functionality and better performance, we have become a top contender in the realm of unstructured data analysis. Although competition is intensifying, this field remains something of a “Blue Ocean.” 

 

**What does the future look like for Zilliz?** 

 

We will continue to focus on making meaningful contributions to the world of open-source software as well as developing cloud-based unstructured data services that emphasize security, reliability, usability, and affordability. Cloud services are the future of our revenue model, and open-source software development is how we plan to get there. Additionally, our vision from day one has been to be a global company. We are making that vision a reality by establishing Zilliz as an international leader in AI-powered unstructured data science software. 

large_CHAOGE.jpg

medium_CHAOGE.jpg

small_CHAOGE.jpg

thumbnail_CHAOGE.jpg

CHAOGE.jpg

An interview with Charles Xie

Making Sense of Unstructured Data with Zilliz Founder and CEO Charles Xie

Technological progress means machine-scale analytics and artificial intelligence are more accessible than ever before. Discover how Milvus, an open-source vector data management platform, unlocks the potential of unstructured data. With just one server and 10 lines of code, it is possible to build a real-time search engine capable of processing 1+ billion images. 

Rising compute power and declining compute costs have made machine-scale analytics and artificial intelligence (AI) more accessible than ever before. In practice, this means with just one server and 10 lines of code, it is possible to build a reverse image search engine capable of querying 1+ billion images in real time. This article explains how [Milvus](https://milvus.io/), an open-source vector data management platform, can be used to create powerful systems for unstructured data processing and analysis, as well as the underlying technology that makes this all possible.

**Jump to:**

- [How does AI enable unstructured data analytics?](#how-does-ai-enable-unstructured-data-analytics)
- [What are vector data management platforms?](#what-are-vector-data-management-platforms)
- [What are limitations of existing approaches to vector data management?](#what-are-limitations-of-existing-approaches-to-vector-data-management)
- [What are applications for vector data management platforms and vector similarity search?](#what-are-applications-for-vector-data-management-platforms-and-vector-similarity-search)


### How does AI enable unstructured data analytics?

An oft-cited statistic is that 80% of the world’s data is unstructured, but just 1% ever gets analyzed. Unstructured data, including images, video, audio, and natural language, doesn’t follow a predefined model or manner of organization. This makes processing and analyzing large unstructured datasets difficult. As the proliferation of smartphones and other connected devices pushes unstructured data production to new heights, businesses are increasingly aware of how important insights derived from this nebulous information can be.

For decades, computer scientists have developed indexing algorithms tailored for organizing, searching, and analyzing specific data types. For structured data, there is bitmap, hash tables, and B-tree, which are commonly used in relational databases developed by tech giants such as Oracle and IBM. For semi-structured data, inverted indexing algorithms are standard, and can be found in popular search engines like [Solr](http://www.solrtutorial.com/basic-solr-concepts.html) and [ElasticSearch](https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#inverted-indexes-and-index-terms). However, unstructured data indexing algorithms rely on compute-intensive artificial intelligence that has only become widely accessible in the past decade.

### Neural networks convert unstructured data into computer-friendly feature vectors

Using neural networks (e.g. [CNN](https://en.wikipedia.org/wiki/Convolutional_neural_network), [RNN](https://en.wikipedia.org/wiki/Recurrent_neural_network), and [BERT](https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270)) unstructured data can be converted into feature vectors (a.k.a., embeddings), which are a string of integers or floats. This numerical data format is far more readily processed and analyzed by machines. Applications spanning reverse image search, video search, natural language processing (NLP) and more can be built by embedding unstructured data into feature vectors, then calculating similarity between vectors using measures like Euclidean distance or cosine similarity.

![Blog_Thanks to AI, Anyone Can Build a Search Engine for 1+ Billion Images_2.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Thanks_to_AI_Anyone_Can_Build_a_Search_Engine_for_1_Billion_Images_2_db8c16aea4.jpeg)

###### *AI algorithms convert unstructured data to vectors*

Computing vector similarity is a relatively simple process that relies on established algorithms. However, unstructured datasets, even after being converted into feature vectors, are typically several orders of magnitude larger than traditional structured and semi-structured datasets. Vector similarity search is complicated by the sheer storage space and compute power required to efficiently and accurately query massive-scale unstructured data.
However, if some degree of accuracy can be sacrificed, there are various approximate nearest neighbor (ANN) search algorithms that can drastically improve query efficiency for massive datasets with high dimensionality. These ANN algorithms decrease storage requirements and computation load by clustering similar vectors together, resulting in faster vector search. Commonly used algorithms include tree-based, graph-based, and combined ANNs.

### What are vector data management platforms?

Vector data management platforms are purpose-built applications for storing, processing, and analyzing massive vector datasets. These tools are designed to easily interface with large amounts of data, and include functionality that streamlines vector data management. Unfortunately, few systems exist that are both flexible and powerful enough to solve modern big data challenges. Milvus, a vector data management platform initiated by [Zilliz](https://zilliz.com/) and released under an open-source license in 2019, attempts to fill this void.

### What are limitations of existing approaches to vector data management?

A common way to build an unstructured data analytics system is to pair algorithms like ANN with open-source implementation libraries such as Facebook AI Similarity Search (Faiss). Due to several limitations, these algorithm-library combinations are not equivalent to a full-fledged vector data management system like Milvus. Existing technology used for managing vector data faces the following problems:

1. **Flexibility:** By default, existing systems typically store all data in main memory, meaning they cannot be run across multiple machines and are poorly suited for handling massive datasets.
2. **Dynamic data handling:** Data is often assumed to be static once fed into existing systems, complicating processing for dynamic data and making near real-time search impossible.
3. **Advanced query processing:** Most tools do not support advanced query processing (e.g., attribute filtering and multi-vector queries), which is essential for building useful similarity search engines.
4. **Heterogeneous computing optimizations:** Few platforms offer optimizations for heterogenous system architectures on both CPUs and GPUs (excluding Faiss), leading to efficiency losses.

Milvus attempts to overcome all of these limitations. The system enhances flexibility by offering support for a variety of application interfaces (including SDKs in Python, Java, Go, C++ and RESTful APIs), multiple vector index types (e.g., quantization-based indexes and graph-based indexes), and advanced query processing. Milvus handles dynamic vector data using a log-structured merge-tree (LSM tree), keeping data insertions and deletions efficient and searches humming along in real time. Milvus also provides optimizations for heterogeneous computing architectures on modern CPUs and GPUs, allowing developers to adjust systems for specific scenarios, datasets, and application environments.

![Blog_Thanks to AI, Anyone Can Build a Search Engine for 1+ Billion Images_3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Thanks_to_AI_Anyone_Can_Build_a_Search_Engine_for_1_Billion_Images_3_380e31d32c.png)

###### *An overview of Milvus’ architecture.*

Using various ANN indexing techniques, Milvus is able to achieve a 99% top-5 recall rate. The system is also capable of loading 1+ million data entries per minute. This results in query time of less than one second when running a reverse image search on 1 billion images. As a cloud native application that can operate as a distributed system deployed across multiple nodes, Milvus can easily and reliably achieve similar performance on datasets that contain 10, or even 100, billion images. Additionally, the system is not limited to image data, with applications spanning computer vision, conversational AI, recommendation systems, new drug discovery, and more.

### What are applications for vector data management platforms and vector similarity search?

As outlined above, a capable vector data management platform like Milvus paired with approximate nearest neighbor algorithms enables similarity search on gigantic volumes of unstructured data. This technology can be used to develop applications that span a diverse array of fields. Below we briefly explain several common use cases for vector data management tools and vector similarity search.

### Reverse image search

Major search engines like Google already give users the option to search by image. Additionally, e-commerce platforms have realized the benefits this functionality offers online shoppers, with Amazon incorporating image search into its smartphone applications.

![Blog_Thanks to AI, Anyone Can Build a Search Engine for 1+ Billion Images_4.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Thanks_to_AI_Anyone_Can_Build_a_Search_Engine_for_1_Billion_Images_4_7884aabcd8.png)

###### *Google’s “search by image” feature.*

Open-source software like Milvus makes it possible for any business to create their own reverse image search system, lowering the barriers to entry for this increasingly in-demand feature. Developers can use pre-trained AI models to convert their own image datasets into vectors, and then leverage Milvus to enable searching for similar products by image.


#### Video recommendation systems

Major online video platforms like YouTube, which receives [500 hours of user generated content each minute](https://www.tubefilter.com/2019/05/07/number-hours-video-uploaded-to-youtube-per-minute/), present unique demands when it comes to content recommendation. In order to make relevant, real-time recommendations that take into consideration new uploads, video recommendation systems must offer lightning-fast query time and efficient dynamic data handling. By converting key frames into vectors and then feeding the results into Milvus, billions of videos can be searched and recommended in near real time.

#### Natural language processing (NLP)

Natural language processing is a branch of artificial intelligence that aims to build systems that can interpret human languages. After converting text data into vectors, Milvus can be used to quickly identify and remove duplicate text, power semantic search, or even [build an intelligent writing assistant](https://medium.com/unstructured-data-service/how-artificial-intelligence-empowered-professional-writing-f433c7e5b561%22%20/). An effective vector data management platform helps maximize the utility of any NLP system.

### Learn more about Milvus

If you would like to learn more about Milvus visit our [website](https://milvus.io/). Additionally, our [bootcamp](https://github.com/milvus-io/bootcamp) offers several tutorials, with instructions for setting up Milvus, benchmark testing, and building a variety of different applications. If you’re interested in vector data management, artificial intelligence, and big data challenges, please join our open-source community on [GitHub](https://github.com/milvus-io) and chat with us on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).

Want more information about building an image search system? Check out this case study:
- [The Journey to Optimizing Billion-scale Image Search (1/2)](https://dzone.com/articles/the-journey-to-optimize-billion-scale-image-search)
- [The Journey to Optimizing Billion-scale Image Search (2/2)](https://medium.com/unstructured-data-service/the-journey-to-optimizing-billion-scale-image-search-2-2-572a36d5d0d)


large_build search.jpg

medium_build search.jpg

small_build search.jpg

thumbnail_build search.jpg

build search.jpg

AI and open-source software make it possible to build a reverse image search engine with just one server and 10 lines of code. Search 1+ billion images in real time and more with Milvus, an open-source vector data management platform. 

Thanks to Milvus, Anyone Can Build a Vector Database for 1+ Billion Images 

This article tests and analyzes the performance of Milvus on both the AVX-512 and AVX2 instruction sets using different indexes

## Introduction  

An instruction is an order given to a computer processor by a computer program. At the lowest level, each instruction is a sequence of 0s and 1s describing a physical operation the computer is to perform. Generally speaking, in the computer assembler language, each language statement corresponds to a processor instruction. CPU relies on instructions for calculation and to control the computer system, and instruction execution capability is an important indicator of CPU’s performance. The instruction set is also closely related to CPU efficiency.  

Proposed by Intel in March 2008 and first supported by Intel’s Sandy Bridge processors in 2011, Advanced Vector Extensions (AVX) are an instruction set for microprocessors of the x86 architecture. AVX introduces new features, new instructions and a new coding scheme. AVX2 expands most integer operations to 256 bits and introduces fused multiply-accumulate (FMA) operations. AVX-512 expands AVX to 512-bit operations using a new EVEX prefix encoding.  

Milvus is an open-source vector similarity search engine that supports embedding of unstructured data using various AI models and provides vector search features for a wide range of applications including image processing, machine vision, natural language processing, speech recognition, and recommendation engines. As of v0.7.0, Milvus has added support for the AVX-512 instruction set. This means, theoretically, it can work on all CPUs containing the AVX-512 instructions.

## Performance Comparison  

### Configurations:   

- CPU: Intel(R) Platinum 8163 CPU @ 2.50GHz24 cores 48 threads
- Number of CPU: 2
- Graphics card, GeForce RTX 2080Ti 11GB 4 cards
- Mem: 768GB
- Disk: 2TB SSD  

### Milvus parameters:

- cahce.cahe_size: 25, The size of CPU memory used for caching data for faster query.
- nlist: 4096
- nprobe: 128

**Note:** nlist is the indexing parameter to create from the client; nprobe the searching parameter. Both IVF_FLAT and IVF_SQ8 use a clustering algorithm to partition a large number of vectors into buckets, nlist being the total number of buckets to partition during clustering. The first step in a query is to find the number of buckets that are closest to the target vector, and the second step is to find the top-k vectors in these buckets by comparing the distance of the vectors. nprobe refers to the number of buckets in the first step.

## Dataset  

We use the SIFT10M dataset, which contains one million 128-dimensional vectors and is often used for analyzing the performance of the corresponding nearest-neighbor search methods. We will compare the top-1 search time for nq = [1, 10, 100, 500, 1000] between the two instruction sets.

## Vector index  

Vector index is a time-efficient and space-efficient data structure built on the vector field of a collection using a certain mathematical model. Through vector index, we can efficiently query vectors that are most similar to the target vector.  

Since accurate retrieval is usually very time-consuming, most of the vector index types of Milvus use ANNS (Approximate Nearest Neighbors Search).
We will test three indexes with AVX512 and AVX2 this time: IVF_FLAT, IVF_SQ8 and HNSW.  

### IVF_FLAT  

IVF (Inverted File) is an index type based on quantization. IVF_FLAT is the most basic IVF index, and the encoded data stored in each unit is consistent with the original data.

![1-query-performance-IVF-FLAT-AVX-512-vs-AVX2-milvus.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/1_query_performance_IVF_FLAT_AVX_512_vs_AVX_2_milvus_9aaa652f2b.png)

### IVF_SQ8

IVF_SQ8 does scalar quantization for each vector placed in the unit based on IVF. Scalar quantization converts each dimension of the original vector from a 4-byte floating-point number to a 1-byte unsigned integer, so the IVF_SQ8 index file occupies much less space than the IVF_FLAT index file.

![2-query-performance-IVF-SQ9-AVX-512-vs-AVX2-milvus.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/2_query_performance_IVF_SQ_9_AVX_512_vs_AVX_2_milvus_73031cb612.png)

### HNSW

HNSW (Hierarchical Small World Graph) is a graph-based indexing algorithm. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then goes down to the next layer for another round of search. After multiple iterations, it can quickly approach the target position.

![3-query-performance-HNSW-AVX512-AVX2-milvus.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/3_query_performance_HNSW_AVX_512_AVX_2_milvus_f690551f7e.png)

## Summary

By analyzing the performance of the above three indexes on two instruction sets, we can see that vector retrieval is slightly faster on the AVX-512 instruction set than on AVX2. This is because the AVX-512 supports 512-bit computation compared to 256-bit computation on the AVX2, and at this level the AVX-512 should be twice as fast as the AVX2. However, Milvus has other time-consuming tasks in addition to its calculations, so the overall retrieval time of the AVX-512 is not twice as long as that of the AVX2.

## Index comparison

![4-index-comparison-IVF-flat-SQ8-AVX-512-AVX2-milvus.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/4_index_comparison_IVF_flat_SQ_8_AVX_512_AVX_2_milvus_a3a2551bb5.png)

![5-index-comparison-avx2-performance-milvus.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/5_index_comparison_avx2_performance_milvus_7f25741138.png)

It can be observed that HNSW retrieval is significantly faster than the other two indexes, while IVF_SQ8 retrieval is slightly faster than IVF_FLAT on both instruction sets. This is probably because IVF_SQ8 requires only 1/4 of the memory than IVF_FLAT: IVF_SQ8 loads 1 byte for each vector dimension, while IVF_FLAT loads 4 bytes per vector dimension. The time required for the calculation is most likely constrained by the memory bandwidth. As a result, IVF_SQ8 not only takes less space, but also requires less time to retrieve vectors.

## Conclusion

This article tests and analyzes the performance of Milvus on both the AVX-512 and AVX2 instruction sets using different indexes, and Milvus shows excellent performance with all types of indexes and performs better on the AVX-512 instruction set.

Milvus is compatible with a variety of deep learning platforms and is used in miscellaneous AI domains. In the latest v0.11.0, Milvus adds scalar field filtering and the ability to specify how distances are calculated during queries. Milvus will continue to add more features to facilitate unstructured data retrieval. For more information about Milvus, see our [official website](https://milvus.io/scenarios/). You can also follow us on [Twitter](https://twitter.com/milvusio/) or join us on [Slack](https://milvusio.slack.com/join/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).





small_header-milvus-performance-avx-512-vs-avx2.png

thumbnail_header-milvus-performance-avx-512-vs-avx2.png

header-milvus-performance-avx-512-vs-avx2.png

Milvus performance on AVX-512 vs. AVX2

Artificial Intelligence Offers a New Way to Find Your View in the CMA Collection

![Blog_ArtLens AI_ 1.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Art_Lens_AI_1_a7b1054511.jpeg)


 

Art is one way people reflect and shape the world they see, and the Cleveland Museum of Art has a new way to match your view to views created by some of history’s greatest artists.

The CMA’s Digital Innovation team has launched [ArtLens AI: Share Your View](https://www.clevelandart.org/art/collection/share-your-view). This interactive tool on the museum website and on [Twitter](https://twitter.com/ArtlensAI) matches what you see in your world to art from the CMA’s remarkable collection.

This easy point of entry into the CMA answers the fatigue of people learning new technologies to stay connected to friends, family, and co-workers. We also wanted to give the community a resource in which— without previous knowledge of history or art theory — someone can make visual connections between their world and a work of art at the click of a button.

The reverse image search tool uses [Artificial Intelligence (AI)](https://en.wikipedia.org/wiki/Artificial_intelligence) to recognize the shapes, patterns, and objects in your own photos to find surprising and delightful matches from the CMA’s encyclopedic collection. The tool can even recognize famous architecture in your photo.

![Blog_ArtLens AI_ 2.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Art_Lens_AI_2_66517b5302.jpeg)

###### *ArtLens AI matches your photo with artworks from the 30,000 in the CMA’s Open Access collection. Right: [Santa Maria Della Salute #1](https://www.clevelandart.org/art/1996.221), 1910. Mortimer Menpes (British, 1860–1938). Etching; sheet: 44.7 x 28 cm. The Cleveland Museum of Art, Gift of Carole W. and Charles B. Rosenblatt, 1996.221. Image courtesy of the Cleveland Museum of Art.*

 

When making a match on the website, you can shuffle through other close matches, download a composite image to share with your friends, or add the image to our visitor gallery. Links to the CMA’s [Collection Online](https://www.clevelandart.org/art/collection/search) offer you more than 36 fields of metadata about a work of art, such as an artist biography, citations, and curator videos. On [Twitter](https://twitter.com/ArtlensAI), attach a photo or an image to your tweet and mention @ArtLensAI to share your view. The Twitter bot will automatically reply with a matching artwork.

![Blog_ArtLens AI_ 3.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Art_Lens_AI_3_b77e68aef9.png)
###### *The interface on the CMA website looks like this. Upload an image to the left box, then watch ArtLens AI find the closest match in the right box. You can shuffle through many matches. Image courtesy of the Cleveland Museum of Art.*

 

![Blog_ArtLens AI_ 4.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Art_Lens_AI_4_f5cfe6af29.png)

###### *On Twitter, attach a photo or image to your tweet and mention @ArtLensAI to share your view. The Twitter bot will automatically reply with a matching artwork from the CMA’s collection. [Egham Lock](https://www.clevelandart.org/art/1961.300). Francis Seymour Haden (British, 1818–1910). Etching; The Cleveland Museum of Art, Gift of Mrs. T. Wingate Todd from the Collection of Dr. T. Wingate Todd, 1961.300. Image courtesy of the Cleveland Museum of Art.*

 

Using Machine Learning and the open source vector similarity engine [Milvus](https://milvus.io/), the CMA’s Digital Innovation development team created the AI tool with [Design IO](https://www.design-io.com/), who had previously collaborated on the museum’s [ArtLens Studio](https://www.clevelandart.org/artlens-gallery/artlens-studio) in 2016, and Jeff Schuler, a partner who leads the CMA team on website development. The new initiative builds on AI used in the museum’s on-site ArtLens Exhibition to read emotion and match a visitor’s face to portraits in the CMA’s collection.

We hope this will be another source of inspiration and fun in these uncertain times. We know that people need new, creative, and easy ways to engage with art. The CMA has grown as a digital museum during the COVID-19 pandemic, with many new gifts to the community. But CMA staffers feel the “Zoom fatigue” of remote work and video conferencing as much as anyone else after six months of self-isolation. Share Your View is one answer to the cautious steps we all are taking into a hybrid experience of life online and life in safe physical spaces.

ArtLens AI: Share Your View can help students, teachers, and parents feeling drained by the continuation of virtual learning. Schools in hybrid mode still rely on video calls and online platforms for classroom engagement. College and high school students are learning from their homes and dorms, without access to on-campus resources. The CMA’s education team has already begun to share this tool as a way to spark conversations about composition, palette, and pattern. Placing an artwork side-by-side with a contemporary view creates new comparisons and opportunities for critical thinking.

![Blog_ArtLens AI_ 5.jpeg](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Art_Lens_AI_5_6fc48ea113.jpeg)
###### *Right: Homage to Goya: [In My Dream I Saw in the Sky a Face of Mystery](https://www.clevelandart.org/art/1927.344.1), 1885. Odilon Redon (French, 1840–1916), printed by Lemercier & Cie.. Lithograph on China paper laid on wove paper; image: 29.1 x 23.9 cm. The Cleveland Museum of Art, Gift of The Print Club of Cleveland, 1927.344.1. Image courtesy of the Cleveland Museum of Art.*

 

Since the launch of [ARTLENS Gallery](https://www.clevelandart.org/artlens-gallery/about) in 2012, the CMA has been a leader in creating digital connections to art. This year we have added many [new curator videos](https://www.clevelandart.org/home-where-art-video-series) for online viewing and created the [ArtLens for Slack app](https://www.clevelandart.org/artlens-for-slack) so workplaces can share a daily moment of discussion and interaction centered around artwork in the museum’s [Open Access](https://www.clevelandart.org/open-access) collection. We hope the new AI feature will help you find yourself in some surprising and inspiring places inside some of the most beautiful art in the world.

![Blog_ArtLens AI_ 6.png](https://zillizstorage.blob.core.windows.net/zilliz-assets/zilliz-assets/assets/Blog_Art_Lens_AI_6_36d8922e8c.png)
###### *Right: [The Acropolis of Athens: Western Portico of the Parthenon](https://www.clevelandart.org/art/1993.34), 1882. William James Stillman (American, 1828–1901). Albumen print from gelatin dry plate negative; image: 38.2 x 36.7 cm. The Cleveland Museum of Art, John L. Severance Fund, 1993.34. Image courtesy of the Cleveland Museum of Art.*

 

*This blog article is reposted from https://medium.com/cma-thinker/artlens-ai-share-your-view-5a326c15943f*

Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.