What are vector databases? How do they differ from traditional databases? How can vector databases help companies from enterprises to startups? Find all that and more in the podcast interview with Frank Liu at Zilliz.

Traditional databases are great for handling structured data, such as information that can easily be captured and categorized in a table. However, [unstructured data](https://milvus.io/docs/v2.0.x/overview.md#Unstructured-data), accounting for around 80% of the world's data, is not so easily managed. Many companies are struggling with making sense of the massive scale of data they have. In the big data era, how can we [tackle the conundrum of meaningful data analysis](https://zilliz.com/learn/what-is-vector-database)? 

In the new episode of That Digital Show, [Frank Liu](https://twitter.com/frankzliu), AI Tech Lead at Zilliz, explains how AI and machine learning are making it possible for developers to understand and extract more value from unstructured data. This episode is hosted by [Jay Jenkins](https://www.linkedin.cn/incareer/in/jaydjenkins/), Tech Strategist and Evangelist at JAPAC - Google Cloud, and [Theo Davies](https://www.linkedin.cn/incareer/in/theohdavies/), Head of Cloud Sales Enablement at JAPAC - Google Cloud.

Below are some questions discussed in the conversation and their corresponding timestamps. To hear the answers, listen to the whole episode titled "[Using Al to Supercharge Data-Driven Applications with Zilliz](https://thatdigitalshow.withgoogle.com/post/apac-19-ai-and-data-driven-applications-with-zilliz/)".

- Where does the name Zilliz come from and what does Zilliz do? (04:00)

- What are some challenges that make data operational? (07:43)

- How have databases evolved to tackle the challenges that volume and complexity pose in meaningful data analysis? (09:00)

- What are vector databases and how do they differ from traditional databases? (09:50)

- How can vector databases help companies? (11:50)

- Could you share some interesting use cases using Milvus? (13:54)

- Why Milvus is open-source and free? (15:18)

large_What_Is_Vector_DB.png

medium_What_Is_Vector_DB.png

small_What_Is_Vector_DB.png

thumbnail_What_Is_Vector_DB.png

What_Is_Vector_DB.png

Frank Liu shares how vector databases tackle data challenges on That Digital Show.

Podcast: Using AI to Supercharge Data-Driven Applications with Zilliz

Feder allows you to visualize the indexing and search process during a reverse image search.

Applications of Vector Similarity Search

Reverse image search is one the most prevalent applications of vector search or approximate nearest neighbor search. When a user uploads an image to the search engine, a bunch of similar images will be returned. During the process, indexes are built to accelerate the search on large datasets, especially those billion- or even trillion-scale datasets. 

In the [previous blog](https://zilliz.com/blog/Visualize-Your-Approximate-Nearest-Neighbor-Search-with-Feder), we have introduced how to visualize your approximate nearest neighbor search with Feder by using the example of HNSW index visualization. In this article, we will take the example of reverse image search and continue to explain how you can use Feder to visualize the index building and search process. In this article, we use the IVF_FLAT index as it is the most commonly used index in reverse image search applications.


## How to visualize reverse image search with Feder
[Feder](https://github.com/zilliztech/feder) is built with JavaScript. To use Feder for visualization, you need to first build an index and save the index file from [Faiss](https://github.com/facebookresearch/faiss) or [Hnswlib](https://github.com/nmslib/hnswlib). Then Feder analyzes the uploaded file to obtain index information and gets ready for the visualization. During a vector similarity search, you need to provide a target vector and the configuration of search parameters. Then Feder visualizes the whole search process for you.

Learn more about how to use Feder by reading [Feder user guide](https://github.com/zilliztech/feder).

## A use case of visualizing search with IVF_FLAT index 
In this use case, we use [VOC 2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/), the classic ML image dataset that contains more than 17,000 images.

First, we use [Towhee](https://towhee.io/), an open-source ML pipeline to encode the images in the VOC 2012 dataset into vectors. Then we build an IVF_FLAT index with Faiss and save the index file. Finally, use Feder for visualization. 



### Build an IVF_Flat index
Indexes are built to accelerate the search process. An analogy can be drawn to a dictionary. All of the words are organized based on their initials. More specifically, words with the same initials are grouped together. And we all know that the number of entries under each initial is unequal. We have more words starting with the letter "E" than those starting with "Z". When looking up a word, we can quickly navigate to the section that only contains words with the same initial. This helps drastically boost the search speed. 

Similarly, the IVF_FLAT index divides vectors in the vector space into different clusters based on vector distance. Vectors close to each other are more likely to be put in the same cluster. And the vectors are not necessarily evenly distributed in each cluster. Therefore, each cluster contains a different amount of vectors.

In this use case, we used Faiss to build an IVF_FLAT index on the 17,000 images in the VOC 2012 dataset, with an `nlist` of 256. The 17,000 image vectors are divided into 256 clusters based on [K-means](https://en.wikipedia.org/wiki/K-means_clustering) clustering method. 

<iframe src="https://zilliz.com/feder-case?indexType=ivf_flat&source=faiss&viewType=overview&withLayout=false&height=600" style="width: 100%; height: 600px;" frameborder="0" scrolling="no"> </iframe>



With Feder, you can visualize the clustering of high-dimensional vector space in a 2D view. Feder supports viewing the details of each cluster while providing an interactive user experience. To have a better understanding of the IVF_FLAT index, you can click on one of the clusters in Feder, and then you will see a maximum of nine images represented by vectors within this cluster.

### Coarse search
When you input a target image and convert it into a target vector for reverse image search, the system first calculates the distance between the target vector and the centroid of each cluster to find the nearest clusters.

In this use case, `nlist` equals 256, which means that the whole vector space is divided into 265 cluster units. Therefore, in the coarse search process, the system compares the distance between the target vector and 256 cluster centroids. 

In IVF indexes, the vectors are clustered based on their relative distance to each other. This means that it is highly likely that the nearest neighbors of the target vector is located in its nearest clusters. We can control the number of cluster units to query with the parameter `nprobe`. In this use case, `nprobe` equals 8 meaning that the system will look for the nearest neighbor of the target vector within the top eight closest clusters.

<iframe src="https://zilliz.com/feder-case?indexType=ivf_flat&source=faiss&viewType=search&withLayout=false&targetId=1156&nprobe=8&k=12&height=600" style="width: 100%; height: 600px;" frameborder="0" scrolling="no"> </iframe>

The screenshot below is a detailed view of the closest clusters. In cluster-186 (the eighth closest cluster to the target vector) we can see it contains some vectors of car images. Though the cars are not at all similar to the airplane in our target image, the images in cluster-186 and the target do share some resemblance as the car tracks in cluster-186 images look very much alike the airport runway in the target image. In a much closer cluster, cluster-96, we can see it contains images of aircrafts in the sky.

![coarse.png](https://assets.zilliz.com/coarse_827557fcc9.png)

The clusters in this use case demonstrate that during embedding, the machine learning model accurately extracts the features including aircraft, runway, and sky in the target image. Then it divides vectors in the vector space based on these features. Cluster-186 shares the feature of "runway" while cluster-96 shares the feature of "aircraft".

### Fine search
After a coarse search, we can secure a number of nprobe clusters for a fine search. In this stage, the system compares the distance between the target vector and all vectors in the nprobe clusters. Then the topK closest vectors are returned as the final results.

In this use case, the system calculates the distance between the target vector and a total of 742 vectors in 8 clusters during the fine search process. 

Feder provides two visualization modes for the fine search process. One mode is visualization based on cluster and vector distance. The other is the projection for dimension reduction mode.

In the screenshot below, different clusters are shown in different colors. The white circle in the center represents the target vector. With the help of Feder, you can see the distance between each vector and the target vector in a clearer and more straight-forward way. You can click on each vector to see more detailed information like its distance to target vector, the image it represents, etc.

![fine-dist.png](https://assets.zilliz.com/fine_dist_ac4f81cec3.png)

The screenshot below is the projection for dimension reduction mode. Still, different clusters are shown in different colors. Currently, we only support [UMAP](https://arxiv.org/abs/1802.03426), one of the most popular method for dimension reduction. More project methods will be supported in future releases of Feder.

![fine-project.png](https://assets.zilliz.com/fine_project_67fe9622ab.png)


### Search performance analysis
When searching without an index, the system needs to calculate the distance between the target vector and all the 17,000 vectors in the database. However, by contrast, if we build an IVF_FLAT index, search efficiency is greatly boosted as the calculation volume is significantly reduced (the system only need to calculate the distance between the target vector and the 256 cluster centroids in coarse search and 742 vectors in fine search).

Also with Feder visualization, we will realize that the value of the index building parameters will influence how the vector space is divided. The `nprobe` parameter can be used to achieve tradeoff between search efficiency and accuracy. The higher the value of `nprobe`, the broader the search scope, and the more accurate the results are. But accordingly, search efficiency will be compromised as the calculation volume increases.

## What's next
- Try [Attu](https://github.com/zilliztech/attu/releases) to manage your vector database with [one-click simplicity](https://zilliz.com/blog/manage-your-milvus-vector-database-with-one-click-simplicity).
- Learn how to [visualize your nearest neighbor search with Feder](https://zilliz.com/blog/Visualize-Your-Approximate-Nearest-Neighbor-Search-with-Feder).

large_Frame 11.png

medium_Frame 11.png

small_Frame 11.png

thumbnail_Frame 11.png

Frame 11.png

See how image vectors are clustered in the vector space.

Visualize Reverse Image Search with Feder

See the actual structure of an index and the whole process of a vector similarity search.

With the help of machine learning (ML) models, we can easily encode unstructured data like photos and videos into [embeddings](https://zilliz.com/learn/what-is-vector-database) for vector similarity search. To accelerate the search, various [indexes](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing) like IVF_FLAT and HNSW are adopted. To select an index best suited for the application, users need to trade off between search speed and accuracy. 

To save the trouble for users, we are proud to announce [Feder](https://github.com/zilliztech/feder), a tool for visualizing ANNS algorithms. With Feder, users can understand different types of indexes and their parameters in a way that is unprecedentedly straight-forward. 

Feder enables users to observe how different indexes are structured, how data are organized using each type of index, and how different parameter configuration influences the indexing structure. In addition, Feder also helps visualize the whole process of vector similarity search and provides a detailed record of data access during the search. 

Currently, Feder only supports the HNSW from [hnswlib](https://github.com/nmslib/hnswlib). More indexes will be supported soon.

## Understanding Feder

[Feder](https://github.com/zilliztech/feder) is built with JavaScript. To use Feder for visualization, you need to first build an index and save the index file from Faiss or Hnswlib. Then Feder analyzes the uploaded file to obtain index information and gets ready for the visualization. During a vector similarity search, you need to provide a target vector and the configuration of search parameters. Then Feder visualizes the whole search process for you.

federjs consists of two parts: 
- Feder-Core
 - Analyzes index files to obtain detailed information about indexes.
 - Supports querying indexes, and keeps a detailed record of the vectors accessed during an index query.
- Feder-View
 - Enables visualization of the overall structure of different indexes.
 - Enables visualization of the whole similarity search process with different indexes.

In addition to federjs, Feder also provides federpy, a Python tool. With federpy, you can directly visualize index structure and search process under [IPython Notebook](https://ipython.org/). Or you can alternatively export the visualization to an HTML file and then use a browser to start the web service.

Learn more about how to use Feder by reading [Feder user guide](https://github.com/zilliztech/feder).

## A use case of visualizing HNSW index

In this use case, we use [VOC 2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/), the classic ML image dataset that contains more than 17,000 images.

First, we use [Towhee](https://towhee.io/), an open-source ML pipeline to encodes the images in the VOC 2012 dataset into vectors. Then we build an index with Hnswlib and save the index file. Finally, use Feder for visualization. 

The link [here](https://colab.research.google.com/drive/12L_oJPR-yFDlORpPondsqGNTPVsSsUwi?usp=sharing) provides an interactive user experience for you to see the visualization for HNSW.

### Visualizing the HNSW index structure

AN HNSW index is multi-layered and each layer is an interconnected network. The bottom layer captures all data objects in the database, and the data points/nodes become more sparse as it moves up to the uppermost layer. Let's draw an analogy to our modern transportation system. If you are visiting from San Francisco to a boutique shop tucked in the Upper East Side of the New York City, you probably first take a flight to JFK or LaGuardia where you find the most convenient metro to take you to Manhattan, and then you probably switch to a bus or even a Citi bike to get to that neighborhood. Similarly, if we want to quickly find the nearest node to your target, we will first start from searching in the uppermost layer because the search here is faster. However, one shortcoming is that, more often than not, the upper layers and networks cannot take us to the desired destination or help us find the expected results. Therefore, we turn to the next layer beneath for higher accuracy. 

When building an HNSW index, one node in the uppermost layer will be selected by algorithm as the entry point to start the search.

Below it shows the visualization of Layer 4, 3, and 2 in a five-layered HNSW index built on the VOC 2012 dataset.

<iframe src="https://zilliz.com/feder-case?indexType=hnsw&source=hnswlib&viewType=overview&withLayout=false&height=600" style="width: 100%; height: 600px;" frameborder="0" scrolling="no"> </iframe>

Feder provides an interactive user experience. Therefore, you can choose any node to take a closer observation. The path highlighted in yellow represents the shortest path with the least transit nodes from the entrance to reach the node you choose. The paths in white shows all other nodes your chosen node can reach. By zooming in, you can see more details and you will realize that the more the layers, the more similar the connected objects.

You can view relevant statistics in the overview panel on the upper-left side. The parameter M decides how many of other nodes the chosen node can reach in each layer. As we can see from the screenshot, m= 8. This means that starting from any random node, the maximum number of nodes this random node can reach is 8.

We can modify the value of the parameters to observe how the index structure is affected.

As the value of M increases, the HNSW structure becomes flatter. The outcome of modifying the value of ef is less obvious in the visualization. In fact the parameter ef [influences the generated links](https://arxiv.org/abs/1603.09320) during index building.

### Visualizing search process with the HNSW index

After you upload a target image for search, Feder will display the whole search process with animation. 

<iframe src="https://zilliz.com/feder-case?indexType=hnsw&source=hnswlib&viewType=search&withLayout=false&height=600" style="width: 100%; height: 600px;" frameborder="0" scrolling="no"> </iframe>


The animation that visualizes the whole process of vector similarity search.

The visualization displays a record of the data accessed in a vector similarity search. That is to say, you can see all the vectors that have been compared in terms of their distance to the target vector, while those not involved in this process are not displayed in the animation.

As we can see in the visualization, for HNSW indexes, the search starts from the uppermost layer, finds the closest node to the target in this layer, and then goes down to the next layer if all nodes accessible in this layer are not close enough to the target.

It should be noted that the search in the bottom layer proceeds in multiple paths. The parameter ef decides the choice of search path. For a detailed introduction to HNSW, read the paper "[Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs](https://arxiv.org/abs/1603.09320)".

Through the interactive visualization, we can see that the nodes at the beginning of the search path are less relevant. But as the search for the nearest neighbor proceeds, search accuracy rapidly rises. The statistics panel on the left demonstrates that only about 1% of the images (around 170 images) of a total of 17,000 images in the VOC 2012 dataset are actually accessed during the search. The tremendous acceleration in the search is made possible thanks to HNSW index.

You can also set different values for the index parameters and generate new index files to compare the structure and search efficiency. 

## What's next
- Try [Attu](https://github.com/zilliztech/attu/releases) to manage your vector database with [one-click simplicity](https://zilliz.com/blog/manage-your-milvus-vector-database-with-one-click-simplicity).

thumbnail_blog-cover-feder.png

blog-cover-feder.png

Feder: A Powerful Visualization Tool for Vector Similarity Search

Zilliz launched Attu - a GUI tool designed specifically for Milvus 2.0.


*This article is written by Zhen Chen and translated by Lichen Wang.*

Zilliz has been a pioneer who dedicated itself to enabling users in the face of rapidly growing demand for unstructured data processing. Zilliz has now open-sourced a graphical user interface - Attu - specifically for Milvus 2.0, an AI-oriented vector database system designed for massive production scenarios. In this article, we would like to show you step by step how to perform a vector similarity search with Attu and Milvus 2.0.

![map.png](https://assets.zilliz.com/map_aa1cda30d4.png)
###### Attu (/ ˈætu /) is an island on the west edge of Aleutian Islands. This uninhabited realm symbolizes an adventurous spirit.

In comparison with Milvus CLI which brings the uttermost simplicity of usage, Attu features more:
- Installers for Windows OS, macOS, and Linux OS;
- Intuitive GUI for easier usage of Milvus;
- Coverage of major functionalities of Milvus;
- Plugins for expansion of customized functionalities;
- Complete system topology information for easier understanding and administration of Milvus instance.

## Installation

You can find the newest release of Attu at [GitHub](https://github.com/zilliztech/attu/releases). Attu offers executable installers for different operating systems. It is an open-source project and welcomes contribution from everyone.

![installation.png](https://assets.zilliz.com/installation_bbe62873af.png)
###### Attu installers and source code.

You can also install Attu via Docker.

```shell
docker run -p 8000:3000 -e HOST_URL=http://{ attu IP }:8000 -e MILVUS_URL={milvus server IP}:19530 zilliz/attu:latest
```

`attu IP` is the IP address of the environment where Attu runs, and `milvus server IP` is IP address of the environment where Milvus runs.

Having installed Attu successfully, you can input the Milvus IP and Port in the interface to start Attu.

![connect.png](https://assets.zilliz.com/connect_1fde46d9d5.png)
###### Connect Milvus with Attu.

## Feature overview

![overview.png](https://assets.zilliz.com/overview_591e230514.png)
###### Attu Overview page.

Attu interface consists of **Overview** page, **Collection** page, **Vector Search** page, and **System View** page, corresponding to the four icons on the left-side navigation pane respectively.
The **Overview** page shows the loaded collections. While the **Collection** page lists all the collections and indicates if they are loaded or released.

![collection.png](https://assets.zilliz.com/collection_42656fe308.png)
###### Attu Collection page.

The **Vector Search** and **System View** pages are plugins of Attu. The concepts and usage of the plugins will be introduced in the final part of the blog.

You can perform vector similarity search in **Vector Search** page.

![vector search.png](https://assets.zilliz.com/vector_search_be7365687c.png)
###### Attu Vector Search page.

In **System View** page, you can check the topological structure of Milvus.

![system view.png](https://assets.zilliz.com/system_view_e1df15023d.png)
###### Attu System View page.

You can also check the detailed information of each node by clicking the node.

![node view.png](https://assets.zilliz.com/node_view_5bbc25f9b2.png)
###### Attu Node view page.

## Demonstration

Let's explore Attu with a test dataset.

Check our [GitHub repo](https://github.com/zilliztech/attu/tree/main/examples) for the dataset used in the following test.

First, create a collection named test with the following four fields:
- Field Name: id, primary key field
- Field Name: vector, vector field, float vector, Dimension: 128
- Field Name: brand, scalar field, Int64
- Field Name: color, scalar field, Int64

![create collection.png](https://assets.zilliz.com/create_collection_95dfa15354.png)
###### Create a collection with Attu.

Load the collection for search after it was successfully created.

![load collection.png](https://assets.zilliz.com/load_collection_fec39171df.png)
###### Load the collection with Attu.

You can now check the newly created collection in the **Overview** page.

![check collection.png](https://assets.zilliz.com/check_collection_163b05477e.png)
###### Check the collection with Attu.

Import the test dataset into Milvus.

![import data 1.png](https://assets.zilliz.com/import_data_1_f73d71be85.png)
![import data 2.png](https://assets.zilliz.com/import_data_2_4b3c3c3c25.png)
![import data 3.png](https://assets.zilliz.com/import_data_3_0def4e8550.png)
###### Import data with Attu.

Click the collection name in Overview or Collection page to enter query interface to check the imported data.
Add filter, specify the expression `id != 0`, click **Apply Filter**, and click **Query**.

![query data.png](https://assets.zilliz.com/query_data_24d9f71ccc.png)
###### Query data with Attu.

You will find all fifty entries of entities are imported successfully.

![query result.png](https://assets.zilliz.com/query_result_bcbbd17084.png)
###### Query result.

Let's try vector similarity search.

Copy one vector from the `search_vectors.csv` and paste it in **Vector Value** field. Choose the collection and field. Click **Search**.

![search data.png](https://assets.zilliz.com/search_data_5af3a1db53.png)
###### Search data with Attu.

You can then check the search result. Without compiling any scripts, you can search with Milvus easily.

![search result.png](https://assets.zilliz.com/search_result_961886efab.png)
###### Search result.

Finally, let's check the **System View** page.

With Metrics API encapsulated in Milvus Node.js SDK, you can check the system status, node relations, and node status.

As an exclusive feature of Attu, System Overview page includes a complete system topological graph. By clicking on each node, you can check its status (refresh every 10 seconds).

![topological graph.png](https://assets.zilliz.com/topological_graph_d0c5c17586.png)
###### Milvus node topological graph in Attu.

Click on each node to enter the **Node List View**. You can check all child nodes of a coord node. By sorting, you can identify the nodes with high CPU or memory usage quickly, and locate the problem with the system.

![node list.png](https://assets.zilliz.com/node_list_64fc610a8d.png)
###### Milvus node list.

## What's more

As mentioned earlier, the **Vector Search** and **System View** pages are plugins of Attu. We encourage users to develop their own plugins in Attu to suit their application scenarios. In the source code, there is folder built specifically for plugin codes.

![plugins.png](https://assets.zilliz.com/plugins_a2d98e4e5b.png)
###### Vector Search and System View pages are plugins of Attu.

You can refer to any of the plugin to learn how to build a plugin. By setting the following config file, you can add the plugin to Attu.

![add plugins.png](https://assets.zilliz.com/add_plugins_e3ef53cc0d.png)
###### Add plugins to Attu.

You can see [Attu GitHub Repo](https://github.com/zilliztech/attu/tree/main/doc) and [Milvus Technical Document](https://milvus.io/docs/v2.0.0/attu.md) for detailed instruction.

Attu is an open-source project. All contributions are welcome. You can also [file an issue](https://github.com/zilliztech/attu/issues) if you had any problem with Attu.

We sincerely hope that Attu can bring you a better user experience with Milvus. And if you like Attu, or have some feedbacks about the usage, you can complete this [Attu User Survey](https://wenjuan.feishu.cn/m/cfm?t=suw4QnODU1ui-ok7r) to help us optimize Attu for a better user experience.

large_attu.png

medium_attu.png

small_attu.png

thumbnail_attu.png

attu.png

Attu - a Milvus GUI Tool.

Manage Your Milvus Vector Database with One-click Simplicity

Charles, founder and CEO of Zilliz, was elected as the new LF AI & Data Foundation Governing Board General Member Representative.

Recently, LF AI & Data, the umbrella foundation of the Linux Foundation has successfully completed a new round of election and is excited to announce the new leaders. 

Charles Xie, founder and CEO of Zilliz, successfully completed his two-year service as the chairperson of the Governing Board. During this election, Junping Du, General Manager of Huawei Cloud & AI Open Source Department, was elected as the new chairperson of the LF AI and Data Governing Board. Charles was elected as the new LF AI & Data Foundation Governing Board General Member Representative, and will continue to serve LF AI & Data.

![image_upimage_upload_load_0.png](https://assets.zilliz.com/image_upimage_upload_load_0_265310269c.png)

###### Charles Xie, founder and CEO of Zilliz.

"We’ve witnessed the accelerated momentum in open source AI and the prosperity of the LF AI & Data community in the past three years. It’s really a privilege to continue serving the community. I hope to help bring the power of AI to every corporation, every organization, and every individual." said Charles.

LF AI & Data is an umbrella foundation of the Linux Foundation that supports open source innovation in artificial intelligence, machine learning, deep learning, and data. Its mission is to build and support an open AI community, and drive open source innovation in the AI, ML, DL and Data domains by enabling collaboration, sharing best practices, supporting development efforts, and the creation of new opportunities for all the members of the community. After four years of growth, LF AI & Data now has 49 member organizations and hosts 36 open-source projects, becoming one of the most influential open-source organizations in the field of AI.

During this election, Nancy Rausch, Senior Software Manager and Data Scientist in Research & Development at SAS Institute, was re-elected as the LF AI & Data Technical Advisory Council (TAC) Chairperson. Xiaoman (Charlotte) Hu, Director of MindSpore Community at Huawei, was elected as the LF AI & Data Outreach Committee Chairperson.

Zilliz sincerely congratulates LF AI & Data upon the new leadership and hopes that the LF AI & Data community can make greater contributions in developing open-source AI technologies and empowering more AI applications!

*Source: https://lfaidata.foundation/blog/2022/03/08/welcome-lf-ai-newly-elected-leaders-2/*

## About Zilliz

Zilliz is built by the engineers and scientists who created LF AI Milvus®, an open-source vector database for production ready AI. On a mission to unleash data insights with AI, the company builds the database and search technologies to help organizations create AI/ML applications stunningly easy and fast. By taking the burden of complex data infrastructure management off our users, Zilliz is committed to bringing the power of AI to every corporation, every organization and every individual.

Founded in 2017 by former Oracle engineers, Zilliz is backed by Hillhouse Capital, Pavilion Capital, 5Y Capital, Yunqi Partners, and others.


small_image_upimage_upload_load_0.png

thumbnail_image_upimage_upload_load_0.png

image_upimage_upload_load_0.png

Charles, founder and CEO of Zilliz, was elected as the new LF AI & Data Foundation Governing Board General Member Representative

 LF AI & Data Announces Newly Elected Leaders

Zilliz team has won the first place in the Disk-based ANN Search track in NeurIPS 2021.

On December 6th, 2021, the world's top AI academic conference, NeurIPS announced the result of its first Approximate Nearest Neighbor (ANN) Search Challenge. [Zilliz](https://zilliz.com/) research team has won the first place in the Disk-based ANN Search track with its Disk Performance Optimization Algorithm, leveraging the ANN search on billion-scale datasets to a higher altitude.

![email 1.png](https://assets.zilliz.com/email_1_b1453f6b5c.png)

The emergence of neural networks enables massive unstructured data such as speech, images, and videos to be embedded as vectors, making ANN search the key to understand these unstructured data. Led by experts and scholars from Microsoft Research, Facebook AI Research, Carnegie Mellon University, Yandex and other influential organizations, the first ANN Search Challenge attracted candidates from Tsinghua University, Nanjing University, Intel, NVIDIA, Kuaishou Technology, and more. A total of six billion-scale datasets were adopted as the example datasets in the challenge, and four of them were released by Facebook, Microsoft Turing, Microsoft Bing, and Yandex specifically for this event.

The Disk-based ANN Search Solution Block-based ANN (BBAnn), developed by the research team of Zilliz, ranked first in the ANN Search track of the Challenge. Its performance peaked during the search in the SimSearchNet++ dataset released by Facebook. This dataset simulates an accurate detection of subtle changes in image, posing a great challenge of retrieving all vectors within a certain radius around the target vector. To make the challenge tougher, the number of query results to return remained uncertain. According to the test result, Zilliz's solution retrieved 88.573% of all the relevant results in the dataset, far beyond the baseline of 16.274%, marking a huge breakthrough in billion-scale ANN search.

In the future, Zilliz will be dedicated to the implementation of this research achievement in Milvus, an open source vector database, to cater to the needs of users from different application scenarios. Milvus is a graduate project of the LF AI & Data Foundation. Milvus is capable of managing a large number of unstructured datasets, and has a wide range of applications in new drug discovery, recommender systems, chatbots, and more. Zilliz will continue to invest in unlocking the hidden value of massive unstructured data for enterprises through open source and cloud-native solutions.



large_1.png

medium_1.png

small_1.png

thumbnail_1.png

1.png

Zilliz Triumphed in Billion-Scale ANN Search Challenge of NeurIPS 2021

This article introduces Milvus_CLI and helps you complete common tasks.

In the age of information explosion, we are producing voice, images, videos, and other unstructured data all the time. How do we efficiently analyze this massive amount of data? The emergence of neural networks enables unstructured data to be embedded as vectors, and the Milvus database is a basic data service software, which helps complete the storage, search, and analysis of vector data.

But how can we use the Milvus vector database quickly?

Some users have complained that APIs are hard to memorize and hope there could be simple command lines to operate the Milvus database.

We're thrilled to introduce Milvus_CLI, a command-line tool dedicated to the Milvus vector database.

Milvus_CLI is a convenient database CLI for Milvus, supporting database connection, data import, data export, and vector calculation using interactive commands in shells. The latest version of Milvus_CLI has the following features.

- All platforms supported, including Windows, Mac, and Linux

- Online and offline installation with pip supported

- Portable, can be used anywhere

- Built on the Milvus SDK for Python

- Help docs included

- Auto-complete supported

## Installation

You can install Milvus_CLI either online or offline.

### Install Milvus_CLI online

Run the following command to install Milvus_CLI online with pip. Python 3.8 or later is required.

```
pip install milvus-cli
```

### Install Milvus_CLI offline

To install Milvus_CLI offline, [download](https://github.com/milvus-io/milvus_cli/releases) the latest tarball from the release page first.

![1.png](https://assets.zilliz.com/1_af0e832119.png)


After the tarball is downloaded, run the following command to install Milvus_CLI.

```
pip install milvus_cli-<version>.tar.gz
```

After Milvus_CLI is installed, run `milvus_cli`. The `milvus_cli >` prompt that appears indicates that the command line is ready.

![2.png](https://assets.zilliz.com/2_b50f5d2a5a.png)

If you're using a Mac with the M1 chip or a PC without a Python environment, you can choose to use a portable application instead. To accomplish this, [download](https://github.com/milvus-io/milvus_cli/releases) a file on the release page corresponding to your OS, run `chmod +x` on the file to make it executable, and run `./` on the file to run it.

#### **Example**

The following example makes `milvus_cli-v0.1.8-fix2-macOS` executable and runs it.

```
sudo chmod +x milvus_cli-v0.1.8-fix2-macOS
./milvus_cli-v0.1.8-fix2-macOS
```

## Usage

### Connect to Milvus

Before connecting to Milvus, ensure that Milvus is installed on your server. See [Install Milvus Standalone](https://milvus.io/docs/v2.0.0/install_standalone-docker.md) or [Install Milvus Cluster](https://milvus.io/docs/v2.0.0/install_cluster-docker.md) for more information.

If Milvus is installed on your localhost with the default port, run `connect`.


![3.png](https://assets.zilliz.com/3_f950d3739a.png)


Otherwise, run the following command with the IP address of your Milvus server. The following example uses `172.16.20.3` as the IP address and `19530` as the port number.

```
connect -h 172.16.20.3
```

![4.png](https://assets.zilliz.com/4_9ff2db9855.png)


### Create a collection

This section introduces how to create a collection. 

A collection consists of entities and is similar to a table in RDBMS. See [Glossary](https://milvus.io/docs/v2.0.0/glossary.md) for more information.


![5.png](https://assets.zilliz.com/5_95a88c1cbf.png)


#### Example 

The following example creates a collection named `car`. The `car` collection has four fields which are `id`, `vector`, `color`, and `brand`. The primary key field is `id`. See [create collection](https://milvus.io/docs/v2.0.0/cli_commands.md#create-collection) for more information.

```
create collection -c car -f id:INT64:primary_field -f vector:FLOAT_VECTOR:128 -f color:INT64:color -f brand:INT64:brand -p id -a -d 'car_collection'
```

### List collections

Run the following command to list all collections in this Milvus instance.

```
list collections
```

![6.png](https://assets.zilliz.com/6_1331f4c8bc.png)


Run the following command to check the details of the `car` collection.

```
describe collection -c car 
```


![7.png](https://assets.zilliz.com/7_1d70beee54.png)


### Calculate the distance between two vectors

Run the following command to import data into the `car` collection.

```
import -c car 'https://raw.githubusercontent.com/zilliztech/milvus_cli/main/examples/import_csv/vectors.csv'
```

![8.png](https://assets.zilliz.com/8_7609a4359a.png)


Run `query` and enter `car` as the collection name and `id>0` as the query expression when prompted. The IDs of the entities that meet the criteria are returned as shown in the following figure.

![9.png](https://assets.zilliz.com/9_f0755589f6.png)


Run `calc` and enter appropriate values when prompted to calculate the distances between vector arrays.

### Delete a collection

Run the following command to delete the `car` collection.

```
delete collection -c car
```

![10.png](https://assets.zilliz.com/10_16b2b01935.png)


## More 

Milvus_CLI is not limited to the preceding functions. Run `help` to view all commands that Milvus_CLI includes and the respective descriptions. Run `<command> --help` to view the details of a specified command.

![11.png](https://assets.zilliz.com/11_5f31ccb1e8.png)


**See also:**

[Milvus_CLI Command Reference](https://github.com/zilliztech/milvus_cli/blob/main/doc/en/cli_commands.md) on GitHub

[Milvus_CLI Command Reference](https://milvus.io/docs/v2.0.0/cli_commands.md) under Milvus Docs

We hope Milvus_CLI could help you easily use the Milvus vector database. We will keep optimizing Milvus_CLI and your contributions are welcome. 

If you have any questions, feel free to [file an issue](https://github.com/zilliztech/milvus_cli/issues) on GitHub.

large_carbon.png

medium_carbon.png

small_carbon.png

thumbnail_carbon.png

carbon.png

Learn how to install, connect to, and use Milvus_CLI.

Get started with Milvus_CLI

the minimal workflow of a recommender system

Accelerating Candidate Generation in Recommender Systems Using Milvus paired with PaddlePaddle

In the era of Big Data, what database technologies and applications will come into the limelight? What will be the next game-changer?

small_istockphoto-1286642964-170667a.jpeg

thumbnail_istockphoto-1286642964-170667a.jpeg

istockphoto-1286642964-170667a.jpeg

Design and Practice of AI-oriented General-purpose Vector Database Systems

Frustrated with New Data? Our Vector Database can Help

Withdrew from Oracle to zero in on unstructured data processing


![SYNC2021.png](https://assets.zilliz.com/SYNC_2021_16c77d7fb3.png)

SYNC 2021, hosted by [PingWest](https://en.pingwest.com/), culminated in success recently. Themed on "Reshape the Future", the conference is committed to establishing a new global consensus on the roaring ambition and resounding success of Chinese entrepreneurs in the global market.

In the session "New Opportunities: How Asian Entrepreneurs Change the World",  Charles Xie, CEO and founder of Zilliz, shared the  [stumble and fall while founding Zilliz and his key to business success](https://www.youtube.com/watch?v=K6in7Y2h_U8&t=5136s).  Other presenters included Brad Bao, Co-founder and Chairman of Lime, Jun Pei, CEO and Co-founder of Cepton, and Lake Dai, Partner at LDV Partners Adjunct.

Charles is an experienced database expert, immersing himself in database development for over 20 years. Prior to his entrepreneurial expedition with Zilliz, Charles worked for years at Oracle's US headquarters, where he was responsible for the development of Oracle relational database systems. He is a founding member of the Oracle 12c cloud database project, which proved to be a huge business success with cumulative revenue exceeding \$10 billion. 

Later, Charles seized the opportunity and founded Zilliz, an open source software company specializing in the development of AI unstructured data processing and analysis systems. Its core product, Milvus vector database, is a graduate project of the LF AI & Data Foundation. Milvus is widely used in intelligent question-answering systems, recommender systems, image and video retrieval, intelligent pathological analysis, new drug discovery, and many other fields. With the [$43 million financing](https://techcrunch.com/2020/11/16/zilliz-43-million-series-b/) led by Hillhouse Ventures, Zilliz set a record for the largest single Series B financing in the world of open source infrastructure software.

Behind the glamorous experience is the constant trial and error of Charles and his team. Charles sincerely shared the doubts, challenges, and difficulties he has faced with young entrepreneurs, and encouraged them to always challenge and stay true to themselves. He said, "Don't be afraid when you find that your idea goes against the grain. Will you set your foot on a totally different road if you persist on it?"

Currently, Zilliz is committed to developing a market and [actively hiring talents](https://app.mokahr.com/apply/zilliz/37974#/) in Silicon Valley.

Click [here](https://youtu.be/K6in7Y2h_U8?t=5136) to check out the conference footage.


medium_SYNC2021.png

small_SYNC2021.png

thumbnail_SYNC2021.png

SYNC2021.png

Withdrew from Oracle to zero in on unstructured data processing

Zilliz CEO Shared Start-up Experience in 2021 SYNC

Learn how Milvus powers the AI analysis of video contents.

*Shiyu Chen, a data engineer at Zilliz, graduated from Xidian University with a degree in Computer Science. Since joining Zilliz, she has been exploring solutions for Milvus in various fields, such as audio and video analysis, molecule formula retrieval, etc., which has greatly enriched the application scenarios of the community. She is currently exploring more interesting solutions. In her spare time, she loves sports and reading.*

When I was watching *Free Guy* last weekend, I felt that I'd seen the actor who plays Buddy, the security guard, somewhere before, yet couldn't recall any of his works. My head was stuffed with "who's this guy?" I was sure about having seen that face and was trying so hard to remember his name. A similar case is that once I saw the leading actor in a video having a drink I used to like a lot, but I ended up failing to recall the brand name.

The answer was on the tip of my tongue, but my brain felt completely stuck.

The tip of the tongue (TOT) phenomenon drives me crazy when watching movies. If only there was a reverse image search engine for videos that enables me to find videos and analyze video content. Before, I built a [reverse image search engine using Milvus](https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy). Considering that video content analysis somehow resembles image analysis, I decided to build a video content analysis engine based on Milvus.

## Object detection

### Overview

Before being analyzed, objects in a video should be detected first. Detecting objects in a video effectively and accurately is the main challenge of the task. It is also an important task for applications such as autopilot, wearable devices, and IoT.

Developed from traditional image processing algorithms to deep neural networks (DNN), today's mainstream models for object detection include R-CNN, FRCNN, SSD, and YOLO. The Milvus-based deep learning video analysis system introduced in this topic can detect objects intelligently and quickly.

### Implementation

To detect and recognize objects in a video, the system should first extract frames from a video and detect objects in the frame images using object detection, secondly, extract feature vectors from detected objects, and lastly, analyze the object based on the feature vectors.

- Frame extraction

Video analysis is converted to image analysis using frame extraction. Currently, frame extraction technology is very mature. Programs such as FFmpeg and OpenCV support extracting frames at specified intervals. This article introduces how to extract frames from a video every second using OpenCV.

- Object detection

Object detection is about finding objects in extracted frames and extracting screenshots of the objects according to their positions. As shown in the following figures, a bike, a dog, and a car were detected. This topic introduces how to detect objects using YOLOv3, which is commonly used for object detection.

![p1.png](https://assets.zilliz.com/p1_0913edccdc.png)


- Feature extraction

Feature extraction refers to converting unstructured data, which is difficult for machines to recognize, to feature vectors. For example, images can be converted to multi-dimensional feature vectors using deep learning models. Currently, the most popular image recognition AI models include VGG, GNN, and ResNet. This topic introduces how to extract features from detected objects using ResNet-50. 

![p2.png](https://assets.zilliz.com/p2_b398855741.png)

- Vector analysis

Extracted feature vectors are compared with library vectors, and the corresponding information to the most similar vectors is returned. For large-scale feature vector datasets, calculation is a huge challenge. This topic introduces how to analyze feature vectors using Milvus.

## Key technologies

### OpenCV 

Open Source Computer Vision Library (OpenCV) is a cross-platform computer vision library, which provides many universal algorithms for image processing and computer vision. OpenCV is commonly used in the computer vision field. 

The following example shows how to capture video frames at specified intervals and save them as images using OpenCV with Python.

```python
import cv2 
cap = cv2.VideoCapture(file_path)   
framerate = cap.get(cv2.CAP_PROP_FPS)   
allframes = int(cv2.VideoCapture.get(cap, int(cv2.CAP_PROP_FRAME_COUNT)))  
success, image = cap.read() 
cv2.imwrite(file_name, image)
```

### YOLOv3 

You Only Look Once, Version 3 (YOLOv3 [5]) is a one-stage object detection algorithm proposed in recent years. Compared to the traditional object detection algorithms with the same accuracy, YOLOv3 is twice as fast. YOLOv3 mentioned in this topic is the enhanced version from PaddlePaddle [6]. It uses multiple optimization methods with a higher inference speed.

### ResNet-50 

ResNet [7] is the winner of ILSVRC 2015 in image classification because of its simplicity and practicality. As the basis of many image analysis methods, ResNet proves to be a popular model specialized in image detection, segmentation, and recognition.

### Milvus  

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following procedures describe how Milvus works.

1. Unstructured data is converted to feature vectors by using deep learning models and is imported to Milvus.
2. Milvus stores and indexes the feature vectors.
3. Milvus returns the most similar vectors to the vector queried by users.

![p3.png](https://assets.zilliz.com/p3_00168a4c93.png)

## Deployment

Now you have some understanding of Milvus-based video analysis systems. The system mainly consists of two parts, as shown in the following figure.

- The red arrows indicate the data import process. Use ResNet-50 to extract feature vectors from the image dataset and import the feature vectors to Milvus.

- The black arrows indicate the video analysis process. First, extract frames from a video and save the frames as images. Second, detect and extract objects in the images using YOLOv3. Then, use ResNet-50 to extract feature vectors from the images. Lastly, Milvus searches and returns the information of the objects with the corresponding feature vectors.

![p4.png](https://assets.zilliz.com/p4_fdf0a9d93f.png)

For more information, see [Milvus Bootcamp: Video Object Detection System](https://github.com/milvus-io/bootcamp/tree/master/solutions/video_similarity_search/object_detection). 

**Data import**

The data import process is simple. Convert the data into 2,048-dimensional vectors and import the vectors into Milvus. 

```python
vector = image_encoder.execute(filename)
entities = [vector]
collection.insert(data=entities)
```

**Video analysis**

As introduced above, the video analysis process includes capturing video frames, detecting objects in each frame, extracting vectors from the objects, calculating vector similarity with Euclidean distance (L2) metrics, and searching for results using Milvus.

```python
images = extract_frame(filename, 1, prefix)   
detector = Detector()   
run(detector, DATA_PATH)       
vectors = get_object_vector(image_encoder, DATA_PATH)
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
results = collection.search(vectors, param=search_params, limit=10)
```

## Conclusion

Currently, more than 80% of the data is unstructured. With the rapid development of AI, an increasing number of deep learning models have been developed for analyzing unstructured data. Technologies such as object detection and image processing have achieved great breakthroughs in both academia and industry. Empowered by these technologies, more and more AI platforms have fulfilled practical requirements.

The video analysis system discussed in this topic is built with Milvus, which can quickly analyze video content.

As an open-source vector database, Milvus supports feature vectors extracted using various deep learning models. Integrated with libraries such as Faiss, NMSLIB, and Annoy, Milvus provides a set of intuitive APIs, supporting switching index types according to scenarios. Additionally, Milvus supports scalar filtering, which increases recall rate and search flexibility. Milvus has been applied to many fields such as image processing, computer vision, natural language processing, speech recognition, recommender system, and new drug discovery.

## References

[1] A. D. Bagdanov, L. Ballan, M. Bertini, A. Del Bimbo. “Trademark matching and retrieval in sports video databases.” Proceedings of the international workshop on Workshop on multimedia information retrieval, ACM, 2007. https://www.researchgate.net/publication/210113141_Trademark_matching_and_retrieval_in_sports_video_databases 

[2] J. Kleban, X. Xie, W.-Y. Ma. “Spatial pyramid mining for logo detection in natural scenes.” IEEE International Conference, 2008. https://ieeexplore.ieee.org/document/4607625 

[3] R. Boia, C. Florea, L. Florea, R. Dogaru. “Logo localization and recognition in natural images using homographic class graphs.” Machine Vision and Applications 27 (2), 2016. https://link.springer.com/article/10.1007/s00138-015-0741-7 

[4] R. Boia, C. Florea, L. Florea. “Elliptical asift agglomeration in class prototype for logo detection.” BMVC, 2015. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=5C87F52DE38AB0C90F8340DFEBB841F7?doi=10.1.1.707.9371&rep=rep1&type=pdf 

[5] https://arxiv.org/abs/1804.02767 

[6] https://paddlepaddle.org.cn/modelbasedetail/yolov3 

[7] https://arxiv.org/abs/1512.03385 

large_1011cover.png

medium_1011cover.png

small_1011cover.png

thumbnail_1011cover.png

1011cover.png

What Is It? Who Is She? Milvus Helps Analyze Videos Intelligently

Use ONNX to process multiple models of CV (computer vision), and combine multiple models with Milvus for similar vector retrieval to get similar images.

Open Neural Network Exchange (ONNX) is an open format built to represent machine learning models. Since it was open-sourced in 2017, ONNX has developed into a standard for AI, providing building blocks for machine learning and deep learning models. ONNX defines a common file format to enable AI developers to use models with various frameworks, tools, runtimes, and compilers, and helps increase the speed of innovation in the artificial intelligence community.

Milvus is an open-source vector database that is highly flexible, reliable and blazing fast. It supports adding, deleting, updating and near real-time search of vector. Milvus has a comprehensive set of intuitive APIs, and support for multiple widely adopted index libraries (e.g. Faiss, NMSLIB, and Annoy), simplifying the index selection for a given scenario. Milvus is simple to use, and it has been used in hundreds of organizations and institutions worldwide, including image, audio and video search, recommendation, chatbot, new drug search, etc.

This article will introduce you how to use multiple models for image search based on ONNX and Milvus. It takes VGG16 and ResNet50 models as examples, uses ONNX to run different AI models to generate feature vectors, and finally performs feature vector retrieval in Milvus to return similar images. There is an open sourced Jupyter Notebook code file for this project on [GitHub](https://github.com/milvus-io/bootcamp/blob/master/etc/onnx_and_milvus/milvus_with_different_models.ipynb), and next we will show you how to achieve.

## Process Models with ONNX
The ONNX format can be easily exchanged between AI models. For example, the TensorFlow model can be converted to ONNX format and run in the Caffe environment. In this example, we convert the pre-trained ResNet50 model under the Keras framework to the ONNX format, and then call the VGG16 model in ONNX format to analyze different models.

```
 from keras.applications.resnet50 import ResNet50
import tensorflow as tf

# load keras-resnet50 model and save as a floder
model_resnet50 = ResNet50(include_top=False, pooling='max', weights='imagenet')
tf.saved_model.save(model_resnet50, "keras_resnet50_model")

# convert resnet50 model to onnx
! python -m tf2onnx.convert --saved-model "keras_resnet50_model" --output "onnx_resnet50.onnx"
```

Note: When we used the interface `keras2onnx.convert_keras(model, model.name)` to convert the model, it will return the error `AttributeError:'KerasTensor' object has no attribute'graph'`. Then we can use Python's Bash command to convert according to the solution on Stack Overflow.

## Extract Feature Vectors using Models
After converting ResNet50 model into ONNX format, you can extract the feature vector of the picture directly through the inference.
Note: Feature vectors need to be normalized after extraction.

```
# get the image vectors with onnx model
def get_onnx_vectors(onnx_model, img_path):
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    
    sess = onnxruntime.InferenceSession(onnx_model)
    x = x if isinstance(x, list) else [x]
    feed = dict([(input.name, x[n]) for n, input in enumerate(sess.get_inputs())])
    feat = sess.run(None, feed)[0]
    
    norm_feat = feat[0] / LA.norm(feat[0])
    norm_feat = [i.item() for i in norm_feat]
    return norm_feat
```

Use ONNX-formatted VGG16 model to process image data:

```
# generate vectors with ResNet50 and VGG16 ONNX model
2vec_resnet = get_onnx_vectors("onnx_resnet50.onnx", "./pic/example.jpg")
3vec_vgg = get_onnx_vectors("onnx_vgg16.onnx", "./pic/example.jpg")
```

## Store Vector Data
Unstructured data such as pictures cannot be directly processed by computer, but it can be converted into vectors through AI model and then analyzed by a computer. Milvus vector database is designed power massive unstructured data analysis. It can store vector data and perform near real-time analysis.
First, create a collection of the corresponding model in Milvus, and then insert the image vectors.

```
from milvus import *

# create collections in Milvus
milvus.create_collection(resnet_collection_param)
milvus.create_collection(vgg_collection_param)

# insert data to Milvus and return ids
status, resnet_ids = milvus.insert(resnet_collection_name, resnet_vectors)
status, vgg_ids = milvus.insert(vgg_collection_name, vgg_vectors)
```

After inserting the data successfully, Milvus will return the ID corresponding to the vector, and then we can find the picture by ID. Since Milvus 1.1 used in this case does not support scalar filtering (which Milvus 2.0 now supports), Redis is used to store the vector ID and the key-value of the image path.

```
import redis
def img_ids_to_redis(img_directory, res_ids):
  for img, ids in zip(images, res_ids):
    redis.set(ids, img)
```

## Search for Similar Images
After storing the data, we can retrieve the vector. Milvus supports multiple distance calculation methods, including Euclidean, inner product and Hamming distance. The image similarity search in this article adopts the Euclidean distance calculation between the vectors in Milvus, returns the similar vector ID, and then finds the image corresponding to the ID in Redis.

```
# search in Milvus and return the similarly results with ids
def search_in_milvus(collection_name, search_vector):
    status, results = milvus.search(collection_name, TOP_K, [search_vector])
    print(status)
    re_ids = [x.id for x in results[0]]
    re_distance = [x.distance for x in results[0]]
    return re_ids, re_distance
    
# get the images according the result ids
def get_sim_imgs(collection_name, search_vector):
    ids, distance = search_in_milvus(collection_name, search_vector)
    img = [red.get(i).decode("utf-8") for i in ids]
    return ids, distance, img
```

Taking the VGG16 and ResNet50 models as examples, this article shows processing multiple models through ONNX and combining multiple models with Milvus for similar vector retrieval to get similar images. The above two models are based on the Keras framework, which can quickly extract feature vectors. It can be seen from the Notebook that although the results of Milvus's search for pictures on the COCO dataset based on these two models are similar, their Euclidean distances are not the same. You can also try to compare the search results of the two models using other datasets.

Milvus is a high-performance, highly available vector database that can be used to process feature vectors generated from massive unstructured data. For more solutions, you can refer to [Milvus bootcamp](https://github.com/milvus-io/bootcamp).

##  References
1. https://github.com/onnx/onnx
2. https://onnx.ai/
3. https://milvus.io/cn/
4. https://github.com/milvus-io/bootcamp



## About author
Shiyu Chen, a data engineer at Zilliz, graduated from Xidian University with a degree in Computer Science. Since joining Zilliz, she has been exploring solutions for Milvus in various fields, such as audio and video analysis, molecule formula retrieval, etc., which has greatly enriched the application scenarios of the community. She is currently exploring more interesting solutions. In her spare time, she loves sports and reading.


small_istockphoto-1224971139-170667a.jpeg

thumbnail_istockphoto-1224971139-170667a.jpeg

istockphoto-1224971139-170667a.jpeg

Combine AI Models for Image Search using ONNX and Milvus

“DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node” is a paper published on NeurIPS in 2019. The paper introduces a state-of-the-art method to perform index building and search on the billion-scale dataset using a single machine with only 64GB of RAM and a large enough SSD. 

The paper introduces a state-of-the-art method to perform index building and search on the billion-scale dataset using a single machine with only 64GB of RAM and a large enough SSD. 

DiskANN: A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale Dataset

Use Milvus, an open-source vector database, to recognize gene families of DNA sequences. Less space but higher accuracy.

> Author:
Mengjia Gu, a data engineer at Zilliz, graduated from McGill University with a Master degree in Information Studies. Her interests include AI applications and similarity search with vector databases. As a community member of open-source project Milvus, she has provided and improved various solutions, like recommendation system and DNA sequence classification model. She enjoys challenges and never gives up!

# Introduction

DNA sequence is a popular concept in both academic research and practical applications, such as gene traceability, species identification, and disease diagnosis. Whereas all industries starve for a more intelligent and efficient research method, artificial intelligence has attracted much attention especially from biological and medical domain. More and more scientists and researchers are contributing to machine learning and deep learning in bioinformatics. To make experimental results more convincing, one common option is increasing sample size. The collaboration with big data in genomics as well brings more possibilities of use case in reality. However, the traditional sequence alignment has limitations, which make it [unsuitable for large data](https://www.frontiersin.org/articles/10.3389/fbioe.2020.01032/full#h5). In order to make less trade-off in reality, vectorization is a good choice for a large dataset of DNA sequences.

The open source vector database [Milvus](https://milvus.io/docs/v2.0.0/overview.md) is friendly for massive data. It is able to store vectors of nucleic acid sequences and perform high-efficiency retrieval. It can also help reduce the cost of production or research. The DNA sequence classification system based on Milvus only takes milliseconds to do gene classification. Moreover, it shows a higher accuracy than other common classifiers in machine learning.

# Data Processing

A gene that encodes genetic information is made up of a small section of DNA sequences, which consists of 4 nucleotide bases [A, C, G, T]. There are about 30,000 genes in human genome, nearly 3 billion DNA base pairs, and each base pair has 2 corresponding bases. To support diverse uses, DNA sequences can be classified into various categories. In order to reduce the cost and make easier use of data of long DNA sequnces, [k-mer ](https://en.wikipedia.org/wiki/K-mer#:~:text=Usually%2C%20the%20term%20k%2Dmer,total%20possible%20k%2Dmers%2C%20where)is introduced to data preprocessing. Meanwhile, it makes DNA sequence data more similar to plain text. Furthermore, vectorized data can speed up calculation in data analysis or machine learning.

![1.png](https://assets.zilliz.com/1_a7469e9eac.png)

**k-mer**

The k-mer method is commonly used in DNA sequence preprocessing. It extracts a small section of length k starting from each base of the original sequence, thereby converting a long sequence of length s to (s-k+1) short sequences of length k. Adjusting the value of k will improve the model performance. Lists of short sequences are easier for data reading, feature extraction, and vectorization.

**Vectorization**

DNA sequences are vectorized in the form of text. A sequence transformed by k-mer becomes a list of short sequences, which looks like a list of individual words in a sentence. Therefore, most natural language processing models should work for DNA sequence data as well. Similar methodologies can be applied to model training, feature extraction, and encoding. Since each model has its own advantages and drawbacks, the selection of models depends on the feature of data and the purpose of research. For example, CountVectorizer, a bag-of-words model, implements feature extraction through straightforward tokenization. It sets no limit on data length, but the result returned is less obvious in terms of similarity comparison.

# Milvus Demo

Milvus can easily manage unstructured data and recall most similar results among trillions of vectors within an average delay of milliseconds. Its similarity search is based on Approximate Nearest Neighbor (ANN) search algorithm. These highlights make Milvus a great option to manage vectors of DNA sequences, hence promote the development and applications of bioinformatics.

Here is a demo showing how to build a DNA sequence classification system with Milvus. The [experimental dataset ](https://www.kaggle.com/nageshsingh/dna-sequence-dataset)includes 3 organisms and 7 gene families. All data are converted to lists of short sequences by k-mers. With a pre-trained CountVectorizer model, the system then encodes sequence data into vectors. The flow chart below depicts the system structure and the processes of inserting and searching.

![1.png](https://assets.zilliz.com/1_ebd89660f6.png)

Try out this demo at [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/dna_sequence_classification).

In Milvus, the system creates collection and inserts corresponding vectors of DNA sequences into the collection (or partition if enabled). When receiving a query request, Milvus will return distances between the vector of input DNA sequence and most similar results in database. The class of input sequence and similarity between DNA sequences can be determined by vector distances in results.

```
# Insert vectors to Milvus collection (partition "human")
DNA_human = collection.insert([human_ids, human_vectors], partition_name='human')
# Search topK results (in partition "human") for test vectors
res = collection.search(test_vectors, "vector_field", search_params, limit=topK, partition_names=['human'])
for results in res:
    res_ids = results.ids # primary keys of topK results
    res_distances = results.distances # distances between topK results & search input
```

**DNA Sequence Classification**
Searching for most similar DNA sequences in Milvus could imply the gene family of an unknown sample, thus learn about its possible functionality.[ If a sequence is classified as GPCRs, then it probably has influence in body functions. ](https://www.nature.com/scitable/topicpage/gpcr-14047471/)In this demo, Milvus has successfully enabled the system to identify the gene families of the human DNA sequences searched with.

![3.png](https://assets.zilliz.com/3_1616da5bb0.png)
![4.png](https://assets.zilliz.com/4_d719b22fc7.png)

**Genetic Similarity**

Average DNA sequence similarity between organisms illustrates how close between their genomes. The demo searches in human data for most similar DNA sequences as that of chimpanzee and dog respectively. Then it calculates and compares average inner product distances (0.97 for chimpanzee and 0.70 for dog), which proves that chimpanzee shares more similar genes with human than dog shares. With more complex data and system design, Milvus is able to support genetic research even on a higher level.

```
search_params = {"metric_type": "IP", "params": {"nprobe": 20}}
```

**Performance**

The demo trains the classification model with 80% human sample data (3629 in total) and uses the remaining as test data. It compares performance of the DNA sequence classification model which uses Milvus with the one powered by Mysql and 5 popular machine learning classifiers. The model based on Milvus outperforms its counterparts in accuracy.

```
from sklearn.model_selection import train_test_split
X, y = human_sequence_kmers, human_labels
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

![1.png](https://assets.zilliz.com/1_6541a7dec6.png)


# Further Exploration

With the development of big data technology, vectorization of DNA sequence will play a more important role in genetic research and practice. Combined with professional knowledge in bioinformatics, related studies can further benefit from the involvement of DNA sequence vectorization. Therefore, Milvus can present better results in practice. According to different scenarios and user needs, Milvus-powered similarity search and distance calculation show great potential and many possibilities.

- **Study unknown sequences**: [According to some researchers, vectorization can compress DNA sequence data.](https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012071/pdf) At the same time, it requires less effort to study structure, function, and evolution of unknown DNA sequences. Milvus can store and retrieve a huge number of DNA sequence vectors without losing accuracy.
- **Adapt devices**: Limited by traditional algorithms of sequence alignment, similarity search can barely benefit from device ([CPU](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7884812/)/[GPU](https://mjeer.journals.ekb.eg/article_146090.html)) improvement. Milvus, which supports both regular CPU computation and GPU acceleration, resolves this problem with approximate nearest neighbor algorithm.
- **Detect virus & trace origins**: [Scientists have compared genome sequences and reported that COVID19 virus of probable bat origin belongs to SARS-COV](https://www.nature.com/articles/s41586-020-2012-7?fbclid=IwAR2hxnXb9nLWgA8xexEoNrCNH8WHqvHhhbN38aSm48AaH6fTzGMB1BLljf4). Based on this conclusion, researchers can expand sample size for more evidence and patterns.
- **Diagnose diseases**: [Clinically, doctors could compare DNA sequences between patients and healthy group to identify variant genes that cause diseases](http://www.xinhuanet.com/science/2018-04/16/c_137114251.htm). It is possible to extract features and encode these data using proper algorithms. Milvus is able to return distances between vectors, which can be related to disease data. In addition to assisting diagnosis of disease, this application can also help to inspire the study of [targeted therapy](https://www.frontiersin.org/articles/10.3389/fgene.2021.680117/full).


# Learn more about Milvus
Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:
- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://milvusio.slack.com/join/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ#/shared-invite/email).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

large_11111.png

medium_11111.png

small_11111.png

thumbnail_11111.png

11111.png

DNA Sequence Classification based on Milvus

Applied ML & AI for Database Systems and Application

![VLDB-1.png](https://assets.zilliz.com/VLDB_1_1db3821fde.png)

What are the major breakthroughs in database industry in 2021?

This August, Zilliz was invited to VLDB Workshop 2021 to share its latest research progress and achievement in applying machine learning methods to database systems in company with Harvard University, Carnegie Mellon University, Tsinghua University, Microsoft and many other organizations. Zilliz senior researcher, Dr. Xiaomeng Yi introduced the development background, design ideas, challenges and technological innovations during the development of Milvus, an open source vector database.

VLDB, SIGMOD and ICDE are known as the three most prestigious conferences in the database field. Recently, the research result by Zilliz research team, [Milvus: A Purpose Built Vector Data Management System](https://zilliz.com/whitepapers/milvus-a-purpose-built-vector-data-management-system), was selected by SIGMOD and included as one of the 21 papers this year due to the excellent underlying functionality and perfect application in business scenarios of Milvus.

Every day, the world produces a huge amount data, of which more than 80% is unstructured data that is difficult to process. Whereas traditional databases can only store structured data, the Milvus engineer team proposed a hypothesis: all data can be uniformly represented by vectors at the semantic level regardless of the data types. Dr. Yi remarked that if the AI neural network model is used to extract the semantics of data and present them uniformly as vectors, the processing logic of many data can be directly executed on the level of vectors.

Milvus is designed specifically for analysis and retrieval of massive feature vectors. It provides a complete framework for vector data update, indexing and similarity search, which not only can be performed in real time on streaming data, but also meets the diverse search requirements in actual business scenarios. Until now, Milvus has released two versions: 1.0 and 2.0. The source code of Milvus is completely open source on [GitHub](https://github.com/milvus-io/milvus) and has been widely used in the frontier fields of artificial intelligence. Tested by 1,000 users around the world, its performance greatly surpasses the peer products.

![VLDB-2.png](https://assets.zilliz.com/VLDB_2_8d1a8074ad.png)

In the sharing, Dr. Yi introduced the design concept of Milvus 2.0: cloud-native, log-as-data, and unified batch-and-stream processing. On the basis of its predecessor, Milvus 2.0 provides users with an intelligent, stable and smooth search experience.

Facing the two major challenges of index selection and data tuning in database development, Dr. Yi believes that AI can open doors to best solutions. For example, new configuration can be first tested and evaluated on a small dataset, and then transferred to a large dataset. Use machine learning to analyze the existing dataset, and guide the configuration of the new dataset, so as to achieve better performance than that of the current BOHB algorithm.

Dr. Xiaomeng Yi (Ph.D. of Computer Architecture, Huazhong University of Science and Technology), Senior Researcher and Research team leader of Zilliz. His research concentrates on high-dimension data management, large-scale information retrieval, and resource allocation in distributed systems. Dr. Yi's research works have been published on leading journals and international conferences including IEEE Network Magazine, IEEE/ACM TON, ACM SIGMOD, IEEE ICDCS, and ACM TOMPECS.


large_VLDB-1.png

medium_VLDB-1.png

small_VLDB-1.png

thumbnail_VLDB-1.png

VLDB-1.png

Applied ML & AI for Database Systems and Application.

Zilliz attended VLDB Workshop 2021

The HM-ANN algorithm was recently introduced in a collaborative research paper co-written by authors from Microsoft Research and the University of California. This novel algorithm for graph-based similarity search offers distinct advantages over existing state-of-the-art ANN search solutions.

[HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogenous Memory](https://proceedings.neurips.cc/paper/2020/file/788d986905533aba051261497ecffcbb-Paper.pdf) is a research paper that was accepted at the 2020 Conference on Neural Information Processing Systems ([NeurIPS 2020](https://nips.cc/Conferences/2020)). In this paper, a novel algorithm for graph-based similarity search, called HM-ANN, is proposed. This algorithm considers both memory heterogeneity and data heterogeneity in a modern hardware setting. HM-ANN enables billion-scale similarity search on a single machine without compression technologies. Heterogeneous memory (HM) represents the combination of fast but small dynamic random-access memory (DRAM) and slow but large persistent memory (PMem). HM-ANN achieves low search latency and high search accuracy, especially when the dataset cannot fit into DRAM. The algorithm has a distinct advantage over the state-of-art approximate nearest neighbor (ANN) search solutions.


# Motivation

Since their inception, ANN search algorithms have posed a fundamental tradeoff between query accuracy and query latency due to the limited DRAM capacity. To store indexes in DRAM for fast query access, it is necessary to limit the number of data points or store compressed vectors, both of which hurt search accuracy. Graph-based indexes (e.g. Hierarchical Navigable Small World, HNSW) have superior query runtime performance and query accuracy. However, these indexes can also consume 1-TiB-level DRAM when operating on billion-scale datasets.

There are other workarounds to avoid letting DRAM store billion-scale datasets in raw format. When a dataset is too large to fit into memory on a single machine, compressed approaches such as product quantization of the dataset’s points are used. But the recall of those indexes with the compressed dataset is normally low because of the loss of precision during quantization. Subramanya et al. [1] explore leveraging solid-state drive (SSD) to achieve billion-scale ANN search using a single machine with an approach called Disk-ANN, where the raw dataset is stored on SSD and the compressed representation on DRAM.

# Introduction to Heterogeneous Memory

<img src="https://assets.zilliz.com/image_32_d26cfa9480.png" width="50%" />

Name of the pictureMemory/Storage Hierarchy with HM

Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-presentation_slides.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-presentation_slides.pdf)

Heterogeneous memory (HM) represents the combination of fast but small DRAM and slow but large PMem. DRAM is normal hardware that can be found in every modern server, and its access is relatively fast. New PMem technologies, such as Intel® Optane™ DC Persistent Memory Modules, bridge the gap between NAND-based flash (SSD) and DRAM, eliminating the I/O bottleneck. PMem is durable like SSD, and directly addressable by the CPU, like memory. Renen et al. [2] discover that the PMem read bandwidth is 2.6× lower, and the write bandwidth 7.5× lower, than DRAM in the configured experiment environment.

# HM-ANN Design

HM-ANN is an accurate and fast billion-scale ANN search algorithm that runs on a single machine without compression. The design of HM-ANN generalizes the idea of HNSW, whose hierarchical structure naturally fits into HM. HNSW consists of multiple layers—only layer 0 contains the whole dataset, and each remaining layer contains a subset of elements from the layer directly underneath it.

![2.png](https://assets.zilliz.com/2_25a1836e8b.png)

An example of HNSW with 3 layers

Source: [https://arxiv.org/pdf/1603.09320.pdf](https://arxiv.org/pdf/1603.09320.pdf)

- Elements in the upper layers, which include only subsets of the dataset, consume a small portion of the whole storage. This observation makes them decent candidates to be placed in DRAM. In this way, the majority of searches on HM-ANN are expected to happen in the upper layers, which maximizes the utilization of the fast access characteristic of DRAM. However, in the cases of HNSW, most searches happen in the bottom layer.
- The bottom-most layer carries the whole dataset, which makes it suitable to be placed in PMem. Since accessing layer 0 is slower, it is preferable to have only a small portion accessed by each query and the access frequency reduced.

## Graph Construction Algorithm

![3.png](https://assets.zilliz.com/3_dd9627c753.png)

An example of graph construction of HM-ANN

Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf)

The key idea of HM-ANN’s construction is to create high-quality upper layers, in order to provide better navigation for search at layer 0. Thus most memory access happens in DRAM, and access in PMem is reduced. To make this possible, the construction algorithm of HM-ANN has a top-down insertion phase and a bottom-up promotion phase.

The top-down insertion phase builds a navigable small-world graph as the bottom-most layer is placed on the PMem.

The bottom-up promotion phase promotes pivot points from the bottom layer to form upper layers that are placed on DRAM without losing much accuracy. If a high-quality projection of elements from layer 0 is created in layer 1, search in layer 0 finds the accurate nearest neighbors of the query with only a few hops.

- Instead of using HNSW’s random selection for promotion, HM-ANN uses a high-degree promotion strategy to promote elements with the highest degree in layer 0 into layer 1. For higher layers, HM-ANN promotes high-degree nodes to the upper layer based on a promotion rate.
- HM-ANN promotes more nodes from layer 0 to layer 1 and sets a larger maximum number of neighbors for each element in layer 1. The number of nodes in the upper layers is decided by the available DRAM space. Since layer 0 is not stored in DRAM, making each layer stored in DRAM denser increases the search quality.

## Graph Seach Algorithm

![4.png](https://assets.zilliz.com/4_a5a7f29c93.png)

An example of graph seach of HM-ANN

Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf)

The search algorithm consists of two phases: fast memory search and parallel layer-0 search with prefetching.

### Fast memory search

As the same as in HNSW, the search in DRAM begins at the entry point in the very top layer and then performs 1-greedy search from top to layer 2. To narrow down the search space in layer 0, HM-ANN performs the search in layer 1 with a search budget with ```efSearchL1```, which limits the size of the candidate list in layer 1. Those candidates of the list are used as multiple entry points for search in layer 0, to enhance search quality in layer 0. While HNSW using only one entry point, the gap between layer 0 and layer 1 is more specially handled in HM-ANN than gaps between any other two layers.

### Parallel layer-0 search with prefetching

In the bottom layer, HM-ANN evenly partitions the aforementioned candidates from searching layer 1 and sees them as entry points to perform a parallel multi-start 1-greedy search with threads. The top candidates from each search are collected to find the best candidates. As known, going down from layer 1 to layer 0 is exactly going to PMem. Parallel search hides the latency of PMem and makes the best use of memory bandwidth, to improve search quality without increasing search time.

HM-ANN implements a software-managed buffer in DRAM to prefetch data from PMem before the memory access happens. When searching layer 1, HM-ANN asynchronously copies neighbor elements of those candidates in ```efSearchL1``` and the neighbor elements’ connections in layer 1 from PMem to the buffer. When the search in layer 0 happens, a portion of to-be-accessed data is already prefetched in DRAM, which hides the latency to access PMem and leads to shorter query time. It matches the design goal of HM-ANN, where most memory accesses happen in DRAM and memory accesses in PMem are reduced.

# Evaluation

In this paper, an extensive evaluation is conducted. All experiments are done on a machine with Intel Xeon Gold 6252 CPU@2.3GHz. It uses DDR4 (96GB) as fast memory and Optane DC PMM (1.5TB) as slow memory. Five datasets are evaluated: BIGANN, DEEP1B, SIFT1M, DEEP1M, and GIST1M. For billion-scale tests, the following schemes are included: billion-scale quantization-based methods (IMI+OPQ and L&C), the non-compression-based methods (HNSW and NSG).

## Billion-scale algorithm comparison

![5.png](https://assets.zilliz.com/5_4297db66a9.png)

In table 1, the build time and storage of different graph-based indexes are compared. HNSW takes the shortest build time and HM-ANN needs 8% additional time than HNSW. In terms of whole storage usage, HM-ANN indexes are 5–13% larger than HSNW, because it promotes more nodes from layer 0 to layer 1.

![6.png](https://assets.zilliz.com/6_f363e64d3f.png)

In Figure 1, the query performance of different indexes is analyzed. Figure 1 (a) and (b) show that HM-ANN achieves the top-1 recall of > 95% within 1ms. Figures 1 (c) and (d) show that HM-ANN obtains top-100 recall of > 90% within 4 ms. HM-ANN provides the best latency-vs-recall performance than all other approaches.

## Million-scale algorithm comparison

![7.png](https://assets.zilliz.com/7_a5c23de240.png)

In Figure 2, the query performance of different indexes is analyzed in a pure DRAM setting. HNSW, NSG, and HM-ANN are evaluated with the three million-scale datasets fitting in DRAM. HM-ANN still achieves better query performance than HNSW. The reason is that the total number of distance computations from HM-ANN is lower (on average 850/query) than that of HNSW (on average 900/query) to achieve 99% recall target.

<img src="https://assets.zilliz.com/image_33_f99d31f322.png" width="35%" />


## Effectiveness of high-degree promotion

In Figure 3, the random promotion and high-degree promotion strategies are compared in the same configuration. The high-degree promotion outperforms the baseline. The high-degree promotion performs 1.8x, 4.3x, and 3.9x faster than the random promotion to reach 95%, 99%, and 99.5% recall targets, respectively.

<img src="https://assets.zilliz.com/image_34_3af47e0842.png" width="35%" />


## Performance benefit of memory management techniques

Figure 5 contains a series of steps between HNSW and HM-ANN to show how each optimization of HM-ANN contributes to its improvements. BP stands for the Bottom-up Promotion while building index. PL0 represents for Parallel layer-0 search, while DP for data prefetching from PMem to DRAM. Step by step, HM-ANN’s search performance is pushed further.

# Conclusion

A new graph-based indexing and search algorithm, called HM-ANN, maps the hierarchical design of the graph-based ANNs with memory heterogeneity in HM. Evaluations show that HM-ANN belongs to the new state-of-the-art indexes in billion point datasets.

We notice a trend in academia as well as in industry, where building indexes on persistent storage devices is focused on. To offload the pressure of DRAM, Disk-ANN [1] is an index built on SSD, whose throughput is significantly lower than PMem. However, the building of HM-ANN still takes few days, where no huge differences compared with Disk-ANN is established. We believe it is possible to optimize the build time of HM-ANN, when we utilize PMem’s characteristics more carefully, e.g. to be aware of PMem's granularity (256 Bytes) and to use streaming instruction to bypass cachelines. We also believe there would be more approaches with durable storage devices are proposed in the future.

# Reference

[1]: Suhas Jayaram Subramanya and Devvrit and Rohan Kadekodi and Ravishankar Krishaswamy and Ravishankar Krishaswamy: DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node, NIPS, 2019

[DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node - Microsoft Research](https://www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/)

[DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node](https://papers.nips.cc/paper/2019/hash/09853c7fb1d3f8ee67a61b6bf4a7f8e6-Abstract.html)

[2]: Alexander van Renen and Lukas Vogel and Viktor Leis and Thomas Neumann and Alfons Kemper: Persistent Memory I/O Primitives, CoRR & DaMoN, 2019

[https://dl.acm.org/doi/abs/10.1145/3329785.3329930](https://dl.acm.org/doi/abs/10.1145/3329785.3329930)

[Persistent Memory I/O Primitives](https://arxiv.org/abs/1904.01614)

small_blog cover.png

thumbnail_blog cover.png

blog cover.png

HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory

Paper Reading｜HM-ANN: When ANNS Meets Heterogeneous Memory

Create a personalized product recommender system using Milvus, an open-source vector database.

With the explosive growth of Internet data scale, the product quantity as well as category in the current mainstream e-commerce platform increase on the one hand, the difficulty for users to find the products they need surges on the other hand.

[Vipshop](https://www.vip.com/) is a leading online discount retailer for brands in China. The Company offers high-quality and popular branded products to consumers throughout China at a significant discount from retail prices. To optimize the shopping experience for their customers, the company decided to build a personalized search recommendation system based on user query keywords and user portraits.

The core function of the e-commerce search recommendation system is to retrieve suitable products from a large number of products and display them to users according to their search intent and preference. In this process, the system needs to calculate the similarity between products and users' search intent & preference, and recommends the TopK products with the highest similarity to users.

Data such as product information, user search intent, and user preferences are all unstructured data. We tried to calculate the similarity of such data using CosineSimilarity(7.x) of the search engine Elasticsearch (ES), but this approach has the following drawbacks.

- Long computational response time - the average latency to retrieve TopK results from millions of items is around 300 ms.

- High maintenance cost of ES indexes - the same set of indexes is used for both commodity feature vectors and other related data, which hardly facilitates the index construction, but produces a massive amount of data.

We tried to develop our own locally sensitive hash plug-in to accelerate the CosineSimilarity calculation of ES. Although the performance and throughput were significantly improved after the acceleration, the latency of 100+ ms was still difficult to meet the actual online product retrieval requirements.

After althorough research, we decided to use Milvus, an open source vector database, which is advantaged with the support for distributed deployment, multi-language SDKs, read/write separation, etc. compared to the commonly used standalone Faiss.

Using various deep learning models, we convert massive unstructured data into feature vectors, and import the vectors into Milvus. With the excellent performance of Milvus, our e-commerce search recommendation system can efficiently query the TopK vectors that are similar to the target vectors.

## Overall Architecture
![Architecture](https://assets.zilliz.com/1_01551e7b2b.jpg)
As shown in the diagram, the system overall architecture consists of two main parts.

- Write process: the item feature vectors (hereinafter referred to as item vectors) generated by the deep learning model are normalized and written into MySQL. MySQL then reads the processed item feature vectors using the data synchronization tool (ETL) and import them into the vector database Milvus.

- Read process: The search service obtains user preference feature vectors (hereinafter referred to as user vectors) based on user query keywords and user portraits, queries similar vectors in Milvus and recalls TopK item vectors.

Milvus supports both incremental data update and entire data update. Each incremental update has to delete the existing item vector and insert a new item vector, meaning that every newly updated collection will be re-indexed. It better suits the scenario with more reads and fewer writes. Therefore, we choose the entire data update method. Moreover, it takes only a few minutes to write the entire data in batches of multiple partitions, which is equivalent to near real-time updates.

Milvus write nodes perform all write operations, including creating data collections, building indexes, inserting vectors, etc., and provide services to the public with write domain names. Milvus read nodes perform all read operations and provide services to the public with read-only domain names.

Whereas the current version of Milvus does not support switching collection aliases, we introduce Redis to seamlessly switch aliases between multiple entire data collections.

The read node only needs to read existing metadata information and vector data or indexes from MySQL, Milvus, and GlusterFS distributed file system, so the read capability can be horizontally extended by deploying multiple instances.

## Implementation Details

### Data Update

The data update service includes not only writing vector data, but also data volume detection of vectors, index construction, index pre-loading, alias control, etc. The overall process is as follows.
![Process](https://assets.zilliz.com/2_6052b01334.jpg)
1. Assume that before building the entire data, CollectionA provides data service to the public, and the entire data being used is directed to CollectionA (`redis key1 = CollectionA`). The purpose of constructing entire data is to create a new collection CollectionB.

2. Commodity data check - check the item number of commodity data in the MySQL table, compare the commodity data with the existing data in CollectionA. Alert can be set in accordance with quantity or percentage. If the set quantity (percentage) is not reached, the entire data will not be built, and it will be regarded as the failure of this building operation, triggering the alert; once the set quantity (percentage) is reached, the entire data building process starts.

3. Start building the entire data - initialize the alias of the entire data being built, and update Redis. After updating, the alias of the entire data being built is directed to CollectionB (`redis key2 = CollectionB`).

4. Create a new entire collection - determine if CollectionB exists. If it does, delete it before creating a new one.

5. Data batch write-in - calculate the partition ID of each commodity data with its own ID using modulo operation, and write the data to multiple partitions to the newly created collection in batches.

6. Build and pre-load index - Create index (`createIndex()`) for the new collection. The index file is stored in distributed storage server GlusterFS. The system automatically simulates query on the new collection and pre-load the index for query warm-up. 

7. Collection data check - check the item number of data in the new collection, compare the data with the existing collection, and set alarms based on the quantity and percentage. If the set number (percentage) is not reached, the collection will not be switched and the building process will be regarded as a failure, triggering the alert.

8. Switching collection - Alias control. After updating Redis, the entire data alias being used is directed to CollectionB (`redis key1 = CollectionB`), the original Redis key2 is deleted, and the building process is completed.

### Data Recall

The Milvus partition data is called several times to calculate the similarity between user vectors, obtained based on user query keywords and user portrait, and item vector, and the TopK item vectors are returned after merging. The overall workflow schematic is as follow:
![workflow](https://assets.zilliz.com/3_93518602b1.jpg)
The following table lists the main services involved in this process. It can be seen that the average latency for recalling TopK vectors is about 30 ms.

| **Service**              | **Role**                                                | **Input Parameters**                                         | **Output parameters** | **Response latency** |
| ------------------------ | ------------------------------------------------------- | ------------------------------------------------------------ | --------------------- | -------------------- |
| User vectors acquisition | Obtain user vector                                      | user info + query                                            | user vector           | 10 ms                |
| Milvus Search            | Calculate the vector similarity and return TopK results | user vector                                                  | item vector           | 10 ms                |
| Scheduling Logic         | Concurrent result recalling and merging                 | Multi-channel recalled item vectors and the similarity score | TopK items            | 10 ms                |

**Implementation process:**

1. Based on the user query keywords and user portrait, the user vector is calculated by the deep learning model.
2. Obtain the collection alias of the entire data being used from Redis currentInUseKeyRef and get Milvus CollectionName. This process is data synchronization service, i.e. switching alias to Redis after entire data update.
3. Milvus is called concurrently and asynchronously with the user vector to obtain data from different partitions of the same collection, and Milvus calculates the similarity between the user vector and the item vector, and returns the TopK similar item vectors in each partition.
4. Merge the TopK item vectors returned from each partition, and rank the results in the reverse order of similarity distance, which are calculated using the IP inner product (the greater the distance between the vectors, the more similar they are). The final TopK item vectors are returned.

## Looking Ahead

At present, Milvus-based vector search can be used steadily in the search of recommendation scenarios, and its high performance gives us more room to play in the dimensionality of the model and algorithm selection.

Milvus will play a crucial role as the middleware for more scenarios, including recall of main site search and all-scenario recommendations.

The three most anticipated features of Milvus in the future are as follows.

- Logic for collection alias switching - coordinate the switching across collections without external conponents.
- Filtering mechanism - Milvus v0.11.0 only supports ES DSL filtering mechanism in standalone version. The newly released Milvus 2.0 supports scalar filtering, and read/write separation.
- Storage support for Hadoop Distributed File System (HDFS) - The Milvus v0.10.6 we are using only supports POSIX file interface, and we have deployed GlusterFS with FUSE support as the storage backend. However, HDFS is a better choice in terms of performance and ease of scaling.

## Lessons Learned and Best Practices

1. For applications where read operations are the primary focus, a read-write separation deployment can significantly increase the processing power and improve performance.
2. The Milvus Java client lacks a reconnection mechanism because the Milvus client used by the recall service is resident in memory. We have to build our own connection pool to ensure the availability of the connection between the Java client and the server through heartbeat test.
3. Slow queries occur occasionally on Milvus. This is due to insufficient warm-up of the new collection. By simulating the query on the new collection, the index file is loaded into the memory to achieve the  index warm-up.
4. nlist is the index building parameter and nprobe is the query parameter. You need to get a reasonable threshold value according to your business scenario through pressure testing experiments to balance the retrieval performance and accuracy.
5. For static data scenario, it is more efficient to import all data into the collection first and build indexes later. 

large_blog_shopping.jpg

medium_blog_shopping.jpg

small_blog_shopping.jpg

thumbnail_blog_shopping.jpg

blog_shopping.jpg

Milvus makes it easy to provide the personalized recommendation service to users.

Building a Personalized Product Recommender System with Vipshop and Milvus

Create an audio retrieval system using Milvus, an open-source vector database. Classify and analyze sound data in real time.

Sound is an information dense data type. Although it may feel antiquated in the era of video content, audio remains a primary information source for many people. Despite long-term decline in listeners, 83% of Americans ages 12 or older listened to terrestrial (AM/FM) radio in a given week in 2020 (down from 89% in 2019). Conversely, online audio has seen a steady rise in listeners over the past two decades, with 62% of Americans reportedly listening to some form of it on a weekly basis according to the same [Pew Research Center study](https://www.journalism.org/fact-sheet/audio-and-podcasting/).

As a wave, sound includes four properties: frequency, amplitude, wave form, and duration. In musical terminology, these are called pitch, dynamic, tone, and duration. Sounds also help humans and other animals perceive and understand our environment, providing context clues for the location and movement of objects in our surroundings.

As an information carrier, audio can be classified into three categories:

1. **Speech:** A communication medium composed of words and grammar. With speech recognition algorithms, speech can be converted to text.
2. **Music:** Vocal and/or instrumental sounds combined to produce a composition comprised of melody, harmony, rhythm, and timbre. Music can be represented by a score.
3. **Waveform:** A digital audio signal obtained by digitizing analog sounds. Waveforms can represent speech, music, and natural or synthesized sounds.

Audio retrieval can be used to search and monitor online media in real-time to crack down on infringement of intellectual property rights. It also assumes an important role in the classification and statistical analysis of audio data.

## Processing Technologies

Speech, music, and other generic sounds each have unique characteristics and demand different processing methods. Typically, audio is separated into groups that contain speech and groups that do not:

- Speech audio is processed by automatic speech recognition.
- Non-speech audio, including musical audio, sound effects, and digitized speech signals, are processed using audio retrieval systems. 

This article focuses on how to use an audio retrieval system to process non-speech audio data. Speech recognition is not covered in this article

### Audio feature extraction

Feature extraction is the most important technology in audio retrieval systems as it enables audio similarity search. Methods for extracting audio features are divided into two categories: 

- Traditional audio feature extraction models such as Gaussian mixture models (GMMs) and hidden Markov models (HMMs);
- Deep learning-based audio feature extraction models such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, encoding-decoding frameworks, attention mechanisms, etc.

Deep learning-based models have an error rate that is an order of magnitude lower than traditional models, and therefore are gaining momentum as core technology in the field of audio signal processing.

Audio data is usually represented by the extracted audio features. The retrieval process searches and compares these features and attributes rather than the audio data itself. Therefore, the effectiveness of audio similarity retrieval largely depends on the feature extraction quality.

In this article, [large-scale pre-trained audio neural networks for audio pattern recognition (PANNs)](https://github.com/qiuqiangkong/audioset_tagging_cnn) are used to extract feature vectors for its mean average accuracy (mAP) of 0.439 (Hershey et al., 2017).

After extracting the feature vectors of the audio data, we can implement high-performance feature vector analysis using Milvus.

### Vector similarity search

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following diagram depicts the general similarity search process using Milvus: 
   
![how-does-milvus-work.png](https://assets.zilliz.com/how_does_milvus_work_6926180543.png)
###### *Milvus' vector similarity search process.*

1. Unstructured data are converted to feature vectors by deep learning models and inserted into Milvus.
2. Milvus stores and indexes these feature vectors.
3. Upon request, Milvus searches and returns vectors most similar to the query vector.

## System overview

The audio retrieval system mainly consists of two parts: insert (black line) and search(red line).

![audio-retrieval-system.png](https://assets.zilliz.com/audio_retrieval_system_663a911c95.png)
###### *Audio retrieval system powered by Milvus.*

The sample dataset used in this project contains open-source game sounds, and the code is detailed in the [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search).

### Step 1: Insert data

Below is the example code for generating audio embeddings with the pre-trained PANNs-inference model and inserting them into Milvus, which assigns a unique ID to each vector embedding.

```
1 wav_name, vectors_audio = get_audio_embedding(audio_path)  
2 if vectors_audio:   
3     embeddings.append(vectors_audio)  
4     wav_names.append(wav_name)  
5 ids_milvus = insert_vectors(milvus_client, table_name, embeddings)  
6 
```

Then the returned **ids_milvus** are stored along with other relevant information (e.g. **wav_name**) for the audio data held in a MySQL database for subsequent processing. 

```
1 get_ids_correlation(ids_milvus, wav_name)  
2 load_data_to_mysql(conn, cursor, table_name)   
3 
```

### Step 2: Audio search

Milvus calculates the inner product distance between the pre-stored feature vectors and the input feature vectors, extracted from the query audio data using the PANNs-inference model, and returns the **ids_milvus** of similar feature vectors, which correspond to the audio data searched.

```
1 _, vectors_audio = get_audio_embedding(audio_filename)   
2 results = search_vectors(milvus_client, table_name, [vectors_audio], METRIC_TYPE, TOP_K)  
3 ids_milvus = [x.id for x in results[0]]  
4 audio_name = search_by_milvus_ids(conn, cursor, ids_milvus, table_name)   
5
````

## API reference and demo

### API

This audio retrieval system is built with open-source code. Its main features are audio data insertion and deletion. All APIs can be viewed by typing **127.0.0.1:<port>/docs** in the browser. 





### Demo

We host a [live demo](https://zilliz.com/solutions ) of the Milvus-based audio retrieval system online that you can try out with your own audio data. 

![audio-search-demo.png](https://assets.zilliz.com/audio_search_demo_cae60625db.png)
###### *Audio search demo powered by Milvus.*

## Conclusion

Living in the era of big data, people find their lives abound with all sorts of information. To make better sense of it, traditional text retrieval no long cuts it. Today's information retrieval technology is in urgent need of the retrieval of various unstructured data types, such as videos, images, and audio.

Unstructured data, which is difficult for computers to process, can be converted into feature vectors using deep learning models. This converted data can easily be processed by machines, enabling us to analyze unstructured data in ways our predecessors were never able to. Milvus, an open-source vector database, can efficiently process the feature vectors extracted by AI models and provides a variety of common vector similarity calculations.

## References
Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B. and Slaney, M., 2017, March. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131-135, 2017 

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).

- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).

- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_audio search.jpg

medium_blog_audio search.jpg

small_blog_audio search.jpg

thumbnail_blog_audio search.jpg

blog_audio search.jpg

Audio retrieval with Milvus makes it possible to classify and analyze sound data in real time.

Audio Retrieval Based on Milvus

A new and improved Milvus bootcamp offers updated guides and easier to follow code examples. Discover how to create your own similarity search solutions with the world's most advanced vector database.

With the release of Milvus 2.0, the team has revamped the Milvus [bootcamp](https://github.com/milvus-io/bootcamp). The new and improved bootcamp offers updated guides and easier to follow code examples for a variety of use cases and deployments. Additionally, this new version is updated for [Milvus 2.0](https://zilliz.com/blog/milvus2.0-redefining-vector-database), a reimagined version of the world's most advanced vector databse.
 

Jump to:
- [Stress test your system against 1M and 100M dataset benchmarks](#stress-test-your-system-against-1m-and-100m-dataset-benchmarks)
- [Explore and build popular vector similarity search solutions](#explore-and-build-popular-vector-similarity-search-solutions)
- [Quickly deploy a fully built application on any system](#quickly-deploy-a-fully-built-application-on-any-system)
- [Use scenario specific notebooks to easily deploy pre-configured applications](#use-scenario-specific-notebooks-to-easily-deploy-pre-configured-applications)
- [Learn how to deploy Milvus in different environments](#learn-how-to-deploy-milvus-in-different-environments)

 

### Stress test your system against 1M and 100M dataset benchmarks 
The [benchmark directory](https://github.com/milvus-io/bootcamp/tree/master/benchmark_test) contains 1 million and 100 million vector benchmark tests that indicate how your system will react to differently sized datasets.

 

### Explore and build popular vector similarity search solutions

The [solution directory](https://github.com/milvus-io/bootcamp/tree/master/solutions) includes the most popular vector similarity search use cases. Each use case contains a notebook solution and a docker deployable solution. Use cases include:

- [Image similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search)
- [Video similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/video_similarity_search)
- [Audio similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search)
- [Recommendation system](https://github.com/milvus-io/bootcamp/tree/master/solutions/recommendation_system)
- [Molecular search](https://github.com/milvus-io/bootcamp/tree/master/solutions/molecular_similarity_search)
- [Question answering system](https://github.com/milvus-io/bootcamp/tree/master/solutions/question_answering_system)

 

### Quickly deploy a fully built application on any system

The quick deploy solutions are dockerized solutions that allow users to deploy fully built applications on any system. These solutions are ideal for brief demos, but require additional work to customize and understand compared to notebooks.

 

### Use scenario specific notebooks to easily deploy pre-configured applications

The notebooks contain a simple example of deploying Milvus to solve the problem in a given use case. Each of the examples are able to be run from start to finish without the need to manage files or configurations. Each notebook is also easy to follow and modifiable, making them ideal base files for other projects.

 

### Image similarity search notebook example
Image similarity search is one of the core ideas behind many different technologies, including autonomous cars recognizing objects. This example explains how to easily build computer vision programs with Milvus.


This [notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/reverse_image_search/reverse_image_search.ipynb) revolves around three things:
- Milvus server 
- Redis server (for metadata storage)
- Pretrained Resnet-18 model. 


#### Step 1: Download required packages
Begin by downloading all the required packages for this project. This notebook includes a table listing the packages to use.

```
pip install -r requirements.txt
```

#### Step 2: Server startup
After the packages are installed, start the servers and ensure both are running properly. Be sure to follow the correct instructions for starting the [Milvus](https://milvus.io/docs/v2.0.0/install_standalone-docker.md) and [Redis](https://hub.docker.com/_/redis) servers.

#### Step 3: Download project data
By default, this notebook pulls a snippet of the VOCImage data for use as an example, but any directory with images should work as long as it follows the file structure that can be seen at the top of the notebook.

```
! gdown "https://drive.google.com/u/1/uc?id=1jdudBiUu41kL-U5lhH3ari_WBRXyedWo&export=download"
! tar -xf 'VOCdevkit.zip'
! rm 'VOCdevkit.zip'
```

#### Step 4: Connect to the servers
In this example, the servers are running on the default ports on the localhost.

```
connections.connect(host="127.0.0.1", port=19537)
red = redis.Redis(host = '127.0.0.1', port=6379, db=0)
```

#### Step 5: Create a collection
After starting the servers, create a collection in Milvus for storing all the vectors. In this example, the dimension size is set to 512, the size of the resnet-18 output, and the similarity metric is set to the Euclidean distance (L2). Milvus supports a variety of different [similarity metrics](https://milvus.io/docs/metric.md#floating).

```
collection_name = "image_similarity_search"
dim = 512
default_fields = [
 schema.FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
 schema.FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
]
default_schema = schema.CollectionSchema(fields=default_fields, description="Image test collection")
collection = Collection(name=collection_name, schema=default_schema)
```

#### Step 6: Build an index for the collection
Once the collection is made, build an index for it. In this case, the IVF_SQ8 index is used. This index requires the 'nlist' parameter, which tells Milvus how many clusters to make within each datafile (segment). Different [indices](https://milvus.io/docs/index.md#CPU) require different parameters.

```
default_index = {"index_type": "IVF_SQ8", "params": {"nlist": 2048}, "metric_type": "L2"}
collection.create_index(field_name="vector", index_params=default_index)
collection.load()
```
#### Step 7: Set up model and data loader
After the IVF_SQ8 index is built, set up the neural network and data loader. The pretrained pytorch resnet-18 used in this example is sans its last layer, which compresses vectors for classification and may lose valuable information.

```
model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet18', pretrained=True)
encoder = torch.nn.Sequential(*(list(model.children())[:-1]))
```

The dataset and data loader needs to be modified so that they are able to preprocess and batch the images while also providing the file paths of the images. This can be done with a slightly modified torchvision dataloader. For preprocessing, the images need to be cropped and normalized due to the resnet-18 model being trained on a specific size and value range. 

```
dataset = ImageFolderWithPaths(data_dir, transform=transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]))

dataloader = torch.utils.data.DataLoader(dataset, num_workers=0, batch_si
```

#### Step 8: Insert vectors into the collection
With the collection setup, the images can be processed and loaded into the created collection. First the images are pulled by the dataloader and run through the resnet-18 model. The resulting vector embeddings are then inserted into Milvus, which returns a unique ID for each vector. The vector IDs and image file paths are then inserted as key-value pairs into the Redis server.

```
steps = len(dataloader)
step = 0
for inputs, labels, paths in dataloader:
 with torch.no_grad():
 output = encoder(inputs).squeeze()
 output = output.numpy()

 mr = collection.insert([output.tolist()])
 ids = mr.primary_keys
 for x in range(len(ids)):
 red.set(str(ids[x]), paths[x])
 if step%5 == 0:
 print("Insert Step: " + str(step) + "/" + str(steps))
 step += 1
```

#### Step 9: Conduct a vector similarity search
Once all of the data is inserted into Milvus and Redis, the actual vector similarity search can be performed. For this example, three randomly selected images are pulled out of the Redis server for a vector similarity search. 

```
random_ids = [int(red.randomkey()) for x in range(3)]
search_images = [x.decode("utf-8") for x in red.mget(random_ids)]
```
These images first go through the same preprocessing that is found in Step 7 and are then pushed through the resnet-18 model. 

```
transform_ops = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
 
embeddings = [transform_ops(Image.open(x)) for x in search_images]
embeddings = torch.stack(embeddings, dim=0)
 
with torch.no_grad():
 embeddings = encoder(embeddings).squeeze().numpy()
```
Then the resulting vector embeddings are used to perform a search. First, set the search parameters, including the name of the collection to search, nprobe (the number of the clusters to search), and top_k (the number of returned vectors). In this example, the search should be very quick. 

```
search_params = {"metric_type": "L2", "params": {"nprobe": 32}}
start = time.time()
results = collection.search(embeddings, "vector", param=search_params, limit=3, expr=None)
end = time.time() - start
```

#### Step 10: Image search results 
The vector IDs returned from the queries are used to find the corresponding images. Matplotlib is then used to display the image search results.
 

![pic1.png](https://assets.zilliz.com/pic1_c8652c7fae.png)
![pic2.png](https://assets.zilliz.com/pic2_355b054161.png)
![pic3.png](https://assets.zilliz.com/pic3_01780c6aac.png)

 

### Learn how to deploy Milvus in different enviroments
The [deployments section](https://github.com/milvus-io/bootcamp/tree/master/deployments) of the new bootcamp contains all the information for using Milvus in different environments and setups. It includes deploying Mishards, using Kubernetes with Milvus, load balancing, and more. Each environment has a detailed step by step guide explaining how to get Milvus working in it.

 

### Don't be a stranger
- Read our our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to Milvus, the world’s most popular vector database, on [Github](http://page/).

thumbnail_cover.png

cover.png

Build, test, and customize vector similarity search solutions with Milvus, an open-source vector database.

Quickly Test and Deploy Vector Search Solutions with the Milvus 2.0 Bootcamp

Mozat is a Singapore-based startup that specializes in social media, lifestyle, and communication applications. Learn how the company uses Milvus to create an AI-powered image search system for its fashion app, Stylepedia.

![stylepedia-1.png](https://assets.zilliz.com/stylepedia_1_5f239a8d48.png)

Founded in 2003, [Mozat](http://www.mozat.com/home) is a start-up headquartered in Singapore with offices in China and Saudi Arabia. The company specializes in building social media, communication, and lifestyle applications. [Stylepedia](https://stylepedia.com/) is a wardrobe app built by Mozat that helps users discover new styles and connect with other people that are passionate about fashion. Its key features include the ability to curate a digital closet, personalized style recommendations, social media functionality, and an image search tool for finding similar items to something seen online or in real life.

[Milvus](https://milvus.io) is used to power the image search system within Stylepedia. The app deals with three image types: user images, product images, and fashion photographs. Each image can include one or more items, further complicating each query. To be useful, an image search system must be accurate, fast, and stable, features that lay a solid technical foundation for adding new functionality to the app such as outfit suggestions and fashion content recommendations.

## System overview

![stylepedia-system-process.png](https://assets.zilliz.com/stylepedia_system_process_8e7e2ab3e4.png)
###### *System process diagram.*

The image search system is divided into offline and online components. 

Offline, images are vectorized and inserted into a vector database (Milvus). In the data workflow, relevant product images and fashion photographs are converted into 512-dimensional feature vectors using object detection and feature extraction models. The vector data is then indexed and added to the vector database.

Online, the image database is queried and similar images are returned to the user. Similar to the off-line component, a query image is processed by object detection and feature extraction models to obtain a feature vector. Using the feature vector, Milvus searches for TopK similar vectors and obtains their corresponding image IDs. Finally, after post-processing (filtering, sorting, etc.), a collection of images similar to the query image are returned.

## Implementation

The implementation breaks down into four modules:
1. Garment detection
2. Feature extraction
3. Vector similarity search
4. Post-processing

### Garment detection

In the garment detection module, [YOLOv5](https://pytorch.org/hub/ultralytics_yolov5/), a one-stage, anchor-based target detection framework, is used as the object detection model for its small size and real-time inference. It offers four model sizes (YOLOv5s/m/l/x), and each specific size has pros and cons. The larger models will perform better (higher precision) but require a lot more computing power and run slower. Because the objects in this case are relatively large items and easy to detect, the smallest model, YOLOv5s, suffices.

Clothing items in each image are recognized and cropped out to serve as the feature extraction model inputs used in subsequent processing. Simultaneously, the object detection model also predicts the garment classification according to predefined classes (tops, outerwear, trousers, skirts, dresses, and rompers).

### Feature extraction

The key to similarity search is the feature extraction model. Cropped clothes images are embedded into 512-dimensional floating point vectors that represent their attributes in a machine readable numeric data format. The [deep metric learning (DML)](https://github.com/kdhht2334/Survey_of_Deep_Metric_Learning) methodology is adopted with [EfficientNet](https://arxiv.org/abs/1905.11946) as the backbone model.

Metric learning aims to train a CNN-based nonlinear feature extraction module (or an encoder) to reduce the distance between the feature vectors corresponding to the same class of samples, and increase the distance between the feature vectors corresponding to different classes of samples. In this scenario, the same class of samples refers to the same piece of clothing.

EfficientNet takes into account both speed and precision when uniformly scaling network width, depth, and resolution. EfficientNet-B4 is used as the feature extraction network, and the output of the ultimate fully connected layer is the image features needed to conduct vector similarity search.

### Vector similarity search

Milvus is an open-source vector database that supports create, read, update, and delete (CRUD) operations as well as near real-time search on trillion-byte datasets. In Stylepedia, it is used for large-scale vector similarity search because it is highly elastic, stable, reliable, and lightening fast. Milvus extends the capabilities of widely used vector index libraries (Faiss, NMSLIB, Annoy, etc.), and provides a set of simple and intuitive APIs that allow users to select the ideal index type for a given scenario.

Given the scenario requirements and data scale, Stylepedia's developers used the CPU-only distribution of Milvus paired with the HNSW index. Two indexed collections, one for products and the other for fashion photographs, are built to power different application functionalities. Each collection is further divided into six partitions based on the detection and classification results to narrow the search scope. Milvus performs search on tens of millions of vectors in milliseconds, providing optimal performance while keeping development costs low and minimizing resource consumption.

### Post-processing

To improve the similarity between the image retrieval results and the query image, we use color filtering and key label (sleeve length, clothes length, collar style, etc.) filtering to filter out ineligible images. In addition, an image quality assessment algorithm is used to make sure that higher quality images are presented to users first.

## Application

### User uploads and image search

Users can take pictures of their own clothes and upload them to their Stylepedia digital closet, then retrieve product images most similar to their uploads.

![stylepedia-search-results.png](https://assets.zilliz.com/stylepedia_search_results_0568e20dc0.png)
###### *Search results of a denim jacket image.*

### Outfit suggestions

By conducting similarity search on the Stylepedia database, users can find fashion photographs that contain a specific fashion item. These could be new garments someone is thinking about purchasing, or something from their own collection that could be worn or paired differently. Then, through the clustering of the items it is often paired with, outfit suggestions are generated. For example, a black biker jacket can go with a variety of items, such as a pair of black skinny jeans. Users can then browse relevant fashion photographs where this match occurs in the selected formula.

![stylepedia-jacket-outfit.png](https://assets.zilliz.com/stylepedia_jacket_outfit_e84914da9e.png)
###### *Outfit ideas for a black biker jacket.*
![stylepedia-jacket-snapshot.png](https://assets.zilliz.com/stylepedia_jacket_snapshot_25f53cc09b.png)
###### *A spread of snapshots featuring a black biker jacket + black skinny jeans match.*

### Fashion photograph recommendations

Based on a user's browsing history, likes, and the contents of their digital closet, the system calculates similarity and provides customized fashion photograph recommendations that may be of interest.

![stylepedia-user-wardrobe.png](https://assets.zilliz.com/stylepedia_user_wardrobe_6770c856b9.png)
###### *Items in user's digital closet.*
![stylepedia-streetsnap-rec.png](https://assets.zilliz.com/stylepedia_streetsnap_rec_901601a34d.png)
###### *Fashion photograph recommendations that match user preferences and tastes.*

By combining deep learning and computer vision methodologies, Mozat was able to build a fast, stable, and accurate image similarity search system using Milvus to power various features in the Stylepedia app.

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).


large_mozat-blog.jpg

medium_mozat-blog.jpg

small_mozat-blog.jpg

thumbnail_mozat-blog.jpg

mozat-blog.jpg

Discover how Milvus, an open-source vector database, is used by Mozat to power a fashion app that offers personalized style recommendations and an image search system.

Building a Wardrobe and Outfit Planning App with Milvus

Learn how to build a Milvus cluster based on JuiceFS, a shared file system designed for cloud-native environments.

Collaborations between open-source communities are a magical thing. Not only do passionate, intelligent, and creative volunteers keep open-source solutions innovative, they also work to bring different tools together in interesting and useful ways. [Milvus](https://milvus.io/), the world's most popular vector database, and [JuiceFS](https://github.com/juicedata/juicefs), a shared file system designed for cloud-native environments, were united in this spirit by their respective open-source communities. This article explains what JuiceFS is, how to build a Milvus cluster based on JuiceFS shared file storage, and the performance users can expect using this solution.

## **What is JuiceFS?**

JuiceFS is a high-performance, open-source distributed POSIX file system, which can be built on top of Redis and S3. It was designed for cloud-native environments and supports managing, analyzing, archiving, and backing up data of any type. JuiceFS is commonly used for solving big data challenges, building artificial intelligence (AI) applications, and log collection. The system also supports data sharing across multiple clients and can be used directly as shared storage in Milvus.

After data, and its corresponding metadata, are persisted to object storage and [Redis](https://redis.io/) respectively, JuiceFS serves as a stateless middleware. Data sharing is realized by enabling different applications to dock with each other seamlessly through a standard file system interface. JuiceFS relies on Redis, an open-source in-memory data store, for metadata storage. Redis is used because it guarantees atomicity and provides high performance metadata operations. All data is stored in object storage through the JuiceFS client. The architecture diagram is as follows:

![juicefs-architecture.png](https://assets.zilliz.com/juicefs_architecture_2023b37a4e.png)
###### *Overall architecture of JuiceFS.*

## **Build a Milvus cluster based on JuiceFS**

A Milvus cluster built with JuiceFS (see architecture diagram below) works by splitting upstream requests using Mishards, a cluster sharding middleware, to cascade the requests down to its sub-modules. When inserting data, Mishards allocates upstream requests to the Milvus write node, which stores newly inserted data in JuiceFS. When reading data, Mishards loads the data from JuiceFS through a Milvus read node to memory for processing, then collects and returns results from sub-services upstream.

![milvus-cluster-built-with-juicefs.png](https://assets.zilliz.com/milvus_cluster_built_with_juicefs_3a43cd262c.png)
###### *Architecture of Milvus cluster built with JuiceFS.*

### **Step 1: Launch MySQL service**

Launch the MySQL service on **any** node in the cluster. For details, see [Manage Metadata with MySQL](https://milvus.io/docs/v1.1.0/data_manage.md).

### **Step 2: Create a JuiceFS file system**

For demonstration purposes, the pre-compiled binary JuiceFS program is used. Download the correct [installation package](https://github.com/juicedata/juicefs/releases) for your system and follow the JuiceFS [Quick Start Guide](https://github.com/juicedata/juicefs/blob/main/docs/en/quick_start_guide.md) for detailed install instructions. To create a JuiceFS file system, first set up a Redis database for metadata storage. It is recommended that for public cloud deployments you host the Redis service on the same cloud as the application. Additionally, set up object storage for JuiceFS. In this example, Azure Blob Storage is used; however, JuiceFS supports almost all object services. Select the object storage service that best suits the demands of your scenario.

After configuring the Redis service and object storage, format a new file system and mount JuiceFS to the local directory:

```
1 $ export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=XXX;AccountKey=XXX;EndpointSuffix=core.windows.net"
2 $ ./juicefs format \
3 --storage wasb \
4 --bucket https://<container> \
5 ... \
6 localhost test #format
7 $ ./juicefs mount -d localhost ~/jfs #mount
8
```

> If the Redis server is not running locally, replace the localhost with the following address: `redis://<user:password>@host:6379/1`.

When the installation succeeds, JuiceFS returns the shared storage page **/root/jfs**.

![installation-success.png](https://assets.zilliz.com/installation_success_9d05279ecd.png)

### **Step 3: Start Milvus**

All the nodes in the cluster should have Milvus installed, and each Milvus node should be configured with read or write permission. Only one Milvus node can be configured as write node, and the rest must be read nodes. First, set the parameters of sections `cluster` and `general` in the Milvus system configuration file **server_config.yaml**:

**Section** `cluster` 

| **Parameter** | **Description** | **Configuration** |
| :------------ | :----------------------------- | :---------------- |
| `enable` | Whether to enable cluster mode | `true` |
| `role` | Milvus deployment role | `rw`/`ro` |

**Section** `general` 

```
# meta_uri is the URI for metadata storage, using MySQL (for Milvus Cluster). Format: mysql://<username:password>@host:port/database
general:
 timezone: UTC+8
 meta_uri: mysql://root:milvusroot@host:3306/milvus
```


During installation, the configured JuiceFS shared storage path is set as **/root/jfs/milvus/db**.

```
1 sudo docker run -d --name milvus_gpu_1.0.0 --gpus all \ 
2 -p 19530:19530 \ 
3 -p 19121:19121 \ 
4 -v /root/jfs/milvus/db:/var/lib/milvus/db \ #/root/jfs/milvus/db is the shared storage path 
5 -v /home/$USER/milvus/conf:/var/lib/milvus/conf \ 
6 -v /home/$USER/milvus/logs:/var/lib/milvus/logs \ 
7 -v /home/$USER/milvus/wal:/var/lib/milvus/wal \ 
8 milvusdb/milvus:1.0.0-gpu-d030521-1ea92e 
9
```

After the installation completes, start Milvus and confirm that it is launched properly.
Finally, start the Mishards service on **any** of the nodes in the cluster. The image below shows a successful launch of Mishards. For more information, refer to the GitHub [tutorial](https://github.com/milvus-io/bootcamp/tree/new-bootcamp/deployments/juicefs).

![mishards-launch-success.png](https://assets.zilliz.com/mishards_launch_success_921695d3a8.png)

## **Performance benchmarks**

Shared storage solutions are usually implemented by network-attached storage (NAS) systems. Commonly used NAS systems types include Network File System (NFS) and Server Message Block (SMB). Public cloud platforms generally provide managed storage services compatible with these protocols, such as Amazon Elastic File System (EFS).

Unlike traditional NAS systems, JuiceFS is implemented based on Filesystem in Userspace (FUSE), where all data reading and writing takes place directly on the application side, further reducing access latency. There are also features unique to JuiceFS that cannot be found in other NAS systems, such as data compression and caching.

Benchmark testing reveals that JuiceFS offers major advantages over EFS. In the metadata benchmark (Figure 1), JuiceFS sees I/O operations per second (IOPS) up to ten times higher than EFS. Additionally, the I/O throughput benchmark (Figure 2) shows JuiceFS outperforms EFS in both single- and multi-job scenarios.

![performance-benchmark-1.png](https://assets.zilliz.com/performance_benchmark_1_b7fcbb4439.png) 
###### *Figure 1. Metadata benchmark*
![performance-benchmark-2.png](https://assets.zilliz.com/performance_benchmark_2_e311098123.png)
###### *Figure 2. Sequential read/write benchmark.*

Additionally, benchmark testing shows first query retrieval time, or time to load newly inserted data from disk to memory, for the JuiceFS-based Milvus cluster is just 0.032 seconds on average, indicating that data is loaded from disk to memory almost instantaneously. For this test, first query retrieval time is measured using one million rows of 128-dimensional vector data inserted in batches of 100k at intervals of 1 to 8 seconds.

JuiceFS is a stable and reliable shared file storage system, and the Milvus cluster built on JuiceFS offers both high performance and flexible storage capacity.

## **Learn more about Milvus**

Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

![writer bio-changjian gao.png](https://assets.zilliz.com/writer_bio_changjian_gao_68018f7716.png)
![writer bio-jingjing jia.png](https://assets.zilliz.com/writer_bio_jingjing_jia_a85d1c2e3b.png)

large_JuiceFS-blog-cover.jpg

medium_JuiceFS-blog-cover.jpg

small_JuiceFS-blog-cover.jpg

thumbnail_JuiceFS-blog-cover.jpg

JuiceFS-blog-cover.jpg

Building a Milvus Cluster Based on JuiceFS

Building a news content recommendation system with semantic vectors is easy with Milvus, an open-source vector database. 

With [71% of Americans](https://www.socialmediatoday.com/news/new-research-shows-that-71-of-americans-now-get-news-content-via-social-pl/593255/) getting their news recommendations from social platforms, personalized content has quickly become how new media is discovered. Whether people are searching for specific topics, or interacting with recommended content, everything users see is optimized by algorithms to improve click-through rates (CTR), engagement, and relevance. Sohu is a NASDAQ-listed Chinese online media, video, search and gaming group. It leveraged [Milvus](https://milvus.io/), an open-source vector database built by [Zilliz](https://zilliz.com/), to build a semantic vector search engine inside its news app. This article explains how the company used user profiles to fine-tune personalized content recommendations over time, improving user experience and engagement.

## Recommending content using semantic vector search

Sohu News user profiles are built from browsing history and adjusted as users search for, and interact with, news content. Sohu’s recommender system uses semantic vector search to find relevant news articles. The system works by identifying a set of tags that are expected to be of interest to each user based on browsing history. It then quickly searches for relevant articles and sorts the results by popularity (measured by average CTR), before serving them to users. 

The New York Times alone publishes [230 pieces of content](https://www.theatlantic.com/technology/archive/2016/05/how-many-stories-do-newspapers-publish-per-day/483845/) a day, which offers a glimpse at the magnitude of new content an effective recommendation system must be capable of processing. Ingesting large volumes of news demands millisecond similarity search and hourly matching of tags to new content. Sohu chose Milvus because it processes massive datasets efficiently and accurately, reduces memory usage during search, and supports high-performance deployments.

## Understanding a news recommendation system workflow

Sohu’s semantic vector search-based content recommendation relies on the Deep Structured Semantic Model (DSSM), which uses two neural networks to represent user queries and news articles as vectors. The model calculates the cosine similarity of the two semantic vectors, then the most similar batch of news is sent to the recommendation candidates pool. Next, news articles are ranked based on their estimated CTR, and those with the highest predicted click-through rate are displayed to users.

### Encoding news articles into semantic vectors with BERT-as-service

To encode news articles into semantic vectors, the system uses the [BERT-as-service](https://github.com/hanxiao/bert-as-service.git) tool. If the word count of any piece of content exceeds 512 while using this model, information loss occurs during the embedding process. To help overcome this, the system first extracts a summary and encodes it into a 768-dimensional semantic vector. Then the two most relevant topics from each news article are extracted, and the corresponding pre-trained topic vectors (200-dimensions) are identified based on topic ID. Next the topic vectors are spliced into the 768-dimensional semantic vector extracted from the article summary, forming a 968-dimensional semantic vector. 

New content continuously comes in through Kafta, and is converted into semantic vectors before being inserted into the Milvus database.

### Extracting semantically similar tags from user profiles with BERT-as-service

The other neural network of the model is user semantic vector. Semantically similar tags (e.g., coronavirus, covid, COVID-19, pandemic, novel strain, pneumonia) are extracted from user profiles based on interests, search queries, and browsing history. The list of acquired tags is sorted by weight, and the top 200 are divided into different semantic groups. Permutations of the tags within each semantic group are used to generate new tag phrases, which are then encoded into semantic vectors through BERT-as-service

For each user profile, sets of tag phrases have a [corresponding set of topics](https://github.com/baidu/Familia) that are marked by a weight indicating a user's interest level. The top two topics out of all relevant topics are selected and encoded by the machine learning (ML) model to be spliced into the corresponding tag semantic vector, forming a 968-dimensional user semantic vector. Even if the system generates the same tags for different users, different weights for tags and their corresponding topics, as well as explicit variance between each user's topic vectors, ensures recommendations are unique

The system is able to make personalized news recommendations by calculating the cosine similarity of the semantic vectors extracted from both user profiles and news articles.

![Sohu01.jpg](https://assets.zilliz.com/Sohu01_1e466fe0c3.jpg)
###### *Sohu News DSSM structure.*

### Computing new semantic user profile vectors and inserting them to Milvus

Semantic user profile vectors are computed daily, with data from the previous 24-hour period processed the following evening. Vectors are inserted into Milvus individually and run through the query process to serve relevant news results to users. News content is inherently topical, requiring computation to be run hourly to generate a current newsfeed that contains content that has a high predicted click-through rate and is relevant to users. News content is also sorted into partitions by date, and old news is purged daily.

### Decreasing semantic vector extraction time from days to hours

Retrieving content using semantic vectors requires converting tens of millions of tag phrases into semantic vectors every day. This is a time-consuming process that would require days to complete even when running on graphics processing units (GPU), which accelerate this type of computation. To overcome this technical issue, semantic vectors from the previous embedding must be optimized so that when similar tag phrases surface corresponding semantic vectors are directly retrieved.

The semantic vector of the existing set of tag phrases is stored, and a new set of tag phrases that is generated daily gets encoded into MinHash vectors. [Jaccard distance](https://milvus.io/docs/metric.md) is used to compute similarity between the MinHash vector of the new tag phrase and the saved tag phrase vector. If the Jaccard distance exceeds a pre-defined threshold, the two sets are considered similar. If the similarity threshold is met, new phrases can leverage the semantic information from previous embeddings. Tests suggest a distance above 0.8 should guarantee enough accuracy for most situations.

Through this process, the daily conversion of the tens of millions of vectors mentioned above is reduced from days to around two hours. Although other methods of storing semantic vectors might be more appropriate depending on specific project requirements, computing similarity between two tag phrases using Jaccard distance in a Milvus database remains an efficient and accurate method in a wide variety of scenarios.

![Sohu02.jpg](https://assets.zilliz.com/Sohu02_d50fccc538.jpg)
###### *Using Jaccard distance to compute similarity between two sets.*

## Overcoming “bad cases” of short text classification

When classifying news text, short news articles have fewer features for extraction than longer ones. Because of this, classification algorithms fail when content of varying lengths is run through the same classifier. Milvus helps solve this issue by searching for multiple pieces of long text classification information with similar semantics and reliable scores, then using a voting mechanism to modify short text classification. 

### Identifying and resolving misclassified short text 

Precise classification of each news article is crucial to providing useful content recommendations. Since short news articles have fewer features, applying the same classifier for news with different lengths results in a higher error rate for short text classification. Human labeling is too slow and inaccurate for this task, so BERT-as-service and Milvus are used to quickly identify misclassified short text in batches, correctly reclassify them, then use batches of data as a corpus for training against this problem.

BERT-as-service is used to encode a total number of five million long news articles with a classifier score greater than 0.9 into semantic vectors. After inserting the long text articles into Milvus, short text news is encoded into semantic vectors. Each short news semantic vector is used to query the Milvus database and obtain the top 20 long news articles with the highest cosine similarity to the target short news. If 18 of the top 20 semantically similar long news appear to be in the same classification and it differs from that of the query short news, then the short news classification is considered incorrect and must be adjusted to align with the 18 long news articles.

This process quickly identifies and corrects inaccurate short text classifications. Random sampling statistics show that after short text classifications are corrected, the overall accuracy of text classification exceeds 95%. By leveraging the classification of high-confidence long text to correct the classification of short text, the majority of bad classification cases are corrected in a short amount of time. This also offers a good corpus for training a short text classifier.

![Sohu03.jpg](https://assets.zilliz.com/Sohu03_a43074cf5f.jpg)
###### *Flowchart of discovery of "bad cases" of short text classification.*

## Milvus can power real-time news content recommendation and more

Milvus greatly improved the real-time performance of Sohu’s news recommendation system, and also bolstered the efficiency of identifying misclassified short text. If you’re interested in learning more about Milvus and its various applications:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

large_blog_SohuNews.jpg

medium_blog_SohuNews.jpg

small_blog_SohuNews.jpg

thumbnail_blog_SohuNews.jpg

blog_SohuNews.jpg

Learn how Milvus was used to build an intelligent news recommendation system inside an app.

Building an Intelligent News Recommendation System Inside Sohu News App

Discover how dependency decoupling & containerization can be used to accelerate compilation times for large-scale AI and MLOps projects. 

Compile time can be compounded by complex internal and external dependencies that evolve throughout the development process, as well as changes in compilation environments such as the operating system or hardware architectures. Following are common issues one may encounter when working on large-scale AI or MLOps projects: 

**Prohibitively long compilation** - Code integration is done hundreds of times each day. With hundreds of thousands of lines of code in place, even a small change could result in a full compilation that typically takes one or more hours. 

**Complex compilation environment** - The project code needs to be compiled under different environments, which involve different operating systems, such as CentOS and Ubuntu, underlying dependencies, such as GCC, LLVM, and CUDA, and hardware architectures. And compilation under a specific environment normally may not work under a different environment. 

**Complex dependencies** - Project compilation involves more than 30 between-component and third-party dependencies. Project development often leads to changes in dependencies, inevitably causing dependency conflicts. The version control between dependencies is so complex that updating version of dependencies will easily affect other components. 

**Third-party dependency download is slow or fails** - Network delays or unstable third-party dependency libraries cause slow resource downloads or access failures, seriously affecting code integration. 

By decoupling dependencies and implementing testing containerization, we managed to decrease average compile time by 60% while working on the open-source embeddings similarity search project [Milvus](https://milvus.io/). 

 

### Decouple the dependencies of the project

Project compilation usually involves a large number of internal and external component dependencies. The more dependencies a project has, the more complex it becomes to manage them. As software grows, it becomes more difficult and costly to change or remove dependencies, as well as identify the effects of doing so. Regular maintenance is required throughout the development process to ensure the dependencies functions properly.
Poor maintenance, complex dependencies, or faulty dependencies can cause conflicts that slow or stall development. In practice, this can mean lagging resource downloads, access failures that negatively impact code integration, and more. Decoupling project dependencies can mitigate defects and reduce compile time, accelerating system testing and avoiding unnecessary drag on software development.

Therefore, we recommend decouple dependencies your project:
- Split up components with complex dependencies
- Use different repositories for version management.
- Use configuration files to manage version information, compilation options, dependencies, etc.
- Add the configuration files to the component libraries so that they are updated as the project iterates.

**Compile optimization between components** — Pull and compile the relevant component according to the dependencies and the compile options recorded in the configuration files. Tag and pack the binary compilation results and the corresponding manifest files, and then upload them to your private repository. If no change is made to a component or the components it depends on, playback its compilation results according to the manifest files. For issues such as network delays or unstable third-party dependency libraries, try setting up an internal repository or using mirrored repositories.

To optimize compilation between components:

1.Create dependency relationship graph — Use the configuration files in the component libraries to create dependency relationship graph. Use the dependency relationship to retrieve the version information (Git Branch, Tag, and Git commit ID) and compilation options and more of both upstream and downstream dependent components.

![1.png](https://assets.zilliz.com/1_949dffec32.png)

2.**Check for dependencies** — Generate alerts for circular dependencies, version conflicts, and other issues that arise between components.

3.**Flatten dependencies** — Sort dependencies by Depth First Search (DFS) and front-merge components with duplicate dependencies to form a dependency graph.

![2.png](https://assets.zilliz.com/2_45130c55e4.png)

4.Use MerkleTree algorithm to generate a hash (Root Hash) containing dependencies of each component based on version information, compilation options, and more. Combined with information such as component name, the algorithm forms a unique tag for each component.

![3.png](https://assets.zilliz.com/3_6a4fcdf4e3.png)

5.Based on the component’s unique tag information, check if a corresponding compilation archive exists in the private repo. If a compilation archive is retrieved, unzip it to get the manifest file for playback; if not, compile the component, mark up the generated compilation object files and manifest file, and upload them to the private repo.

 

**Implement compilation optimizations within components** — Choose a language-specific compilation cache tool to cache the compiled object files, and upload and store them in your private repository. For C/C++ compilation, choose a compilation cache tool like CCache to cache the C/C++ compilation intermediate files, and then archive the local CCache cache after compilation. Such compile cache tools simply cache the changed code files one by one after compilation, and copy the compiled components of the unchanged code file so that they can be directly involved in the final compilation.
Optimization of the compilation within components includes the following steps:

1. Add the necessary compilation dependencies to Dockerfile. Use Hadolint to perform compliance checks on Dockerfile to ensure that the image conforms to Docker’s best practices.
2. Mirror the compilation environment according to the project sprint version (version + build), operating system, and other information.
3. Run the mirrored compilation environment container, and transfer the image ID to the container as an environment variable. Here’s an example command for getting image ID: “docker inspect ‘ — type=image’ — format ‘{{.ID}}’ repository/build-env:v0.1-centos7”.
4. Choose the appropriate compile cache tool: Enter your containter to integrate and compile your codes and check in your private repository if an appropriate compile cache exists. If yes, download and extract it to the specified directory. After all components are compiled, the cache generated by the compile cache tool is packaged and uploaded to your private repository based on the project version and image ID.

 

### Further compilation optimization

Our initially-built occupies too much disk space and network bandwidth, and takes a long time to deploy, we took the following measures:
1. Choose the leanest base image to reduce the image size, e.g. alpine, busybox, etc.
2. Reduce the number of image layers. Reuse dependencies as much as possible. Merge multiple commands with “&&”.
3. Clean up the intermediate products during image building.
4. Use image cache to build image as much as possible.

As our project continues to progress, disk usage and network resource began to soar as the compilation cache increases, while some of the compilation caches are underutilized. We then made the following adjustments:

**Regularly clean up cache files** — Regularly check the private repository (using scripts for example), and clean up cache files that have not changed for a while or have not been downloaded much.

**Selective compile caching** — Only cache resource-demanding compiles, and skip caching compiles that do not require much resource.

 

### Leveraging containerized testing to reduce errors, improve stability and reliability

Codes have to be compiled in different environments, which involve variety of operating systems (e.g. CentOS and Ubuntu), underlying dependencies (e.g. GCC, LLVM, and CUDA), and specific hardware architectures. Code that successfully compiles under a specific environment fail in a different environment. By running tests inside containers, the testing process becomes faster and more accurate.

Containerization ensures that the test environment is consistent, and that an application is working as expected. The containerized testing approach packages tests as image containers and builds a truly-isolated test environment. Our testers found that this approach pretty useful, which ended up reducing compile times by as much as 60%.

**Ensure a consistent compile environment** — As the compiled products are sensitive to changes in the system environment, unknown errors may occur in different operating systems. We have to tag and archive the compiled product cache according to the changes in the compile environment, but they are difficult to categorize. So we introduced containerization technology to unify the compile environment to solve such issues.

 

### Conclusion

By analyzing project dependencies, this article introduces different methods for compilation optimization between and within components, providing ideas and best practices for building stable and efficient continuous code integration. These methods helped solve slow code integration caused by complex dependencies, unify operations inside the container to ensure the consistency of the environment, and improve compilation efficiency through the playback of the compilation results and the use of compilation cache tools to cache the intermediate compilation results.

This above-mentioned practices have reduced the compile time of the project by 60% on average, greatly improving the overall efficiency of code integration. Moving forward, we will continue parallelizing compilation between and within components to further reduce compilation times.

 

*The following sources were used for this article:*

- “Decoupling Source Trees into Build-Level Components”
- “[Factors to consider when adding third party dependencies to a project](https://dev.to/brpaz/factors-to-consider-when-adding-third-party-dependencies-to-a-project-46hf)”
- “[Surviving Software Dependencies](https://queue.acm.org/detail.cfm?id=3344149)”
- “[Understanding Dependencies: A Study of the Coordination Challenges in Software Development](https://www.cc.gatech.edu/~beki/t1.pdf)”

 

### About the author

Zhifeng Zhang is a senior DevOps engineer at Zilliz.com working on Milvus, an open-source vector database, and authorized instructor of the LF open-source software university in China. He received his bachelor’s degree in Internet of Things (IOT) from Software Engineering Institute of Guangzhou. He spends his career participating in and leading projects in the area of CI/CD, DevOps, IT infrastructure management, Cloud-Native toolkit, containerization, and compilation process optimization.

small_cover.jpeg

thumbnail_cover.jpeg

cover.jpeg

Discover how zilliz to reduce compile times 2.5x using dependency decoupling and containerization techniques for large-scale AI and MLOps projects.  

Accelerating Compilation 2.5X with Dependency Decoupling & Testing Containerization

The finance industry has long leveraged open-source software to extract insights from large, unstructured datasets. Discover how Milvus, an open-source vector database, can power AI applications including banking chatbots, recommender systems, and semantic text mining. 

Banks and other financial institutions have long been early adopters of open-source software for big data processing and analytics. In 2010, Morgan Stanley [began using](https://www.forbes.com/sites/tomgroenfeldt/2012/05/30/morgan-stanley-takes-on-big-data-with-hadoop/?sh=19f4f8cd16db) the open-source Apache Hadoop framework as part of a small experiment. The company was struggling to successfully scale traditional databases to the massive volumes of data its scientists wanted to leverage, so it decided to explore alternative solutions. Hadoop is now a staple at Morgan Stanley, helping with everything from managing CRM data to portfolio analysis. Other open-source relational database software such as MySQL, MongoDB, and PostgreSQL have been indispensable tools for making sense of big data in the finance industry.

Technology is what gives the financial services industry a competitive edge, and artificial intelligence (AI) is rapidly becoming the standard approach to extracting valuable insights from big data and analyzing activity in real-time across the banking, asset management, and insurance sectors. By using AI algorithms to convert unstructured data such as images, audio, or video to vectors, a machine-readable numeric data format, it is possible to run similarity searches on massive million, billion, or even trillion vector datasets. Vector data is stored in high-dimensional space, and similar vectors are found using similarity search, which requires a dedicated infrastructure called a vector database.

![01 (1).jpg](https://assets.zilliz.com/01_1_cb99f15886.jpg)
###### *Development of open source big data technologies.*

[Milvus](https://github.com/milvus-io/milvus) is an open-source vector database built specifically for managing vector data, which means engineers and data scientists can focus on building AI applications or conducting analysis—instead of the underlying data infrastructure. The platform was built around AI application development workflows and is optimized to streamline machine learning operations (MLOps). For more information about Milvus and its underlying technology, check out our [blog](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View).

Common applications of AI in the financial services industry include algorithmic trading, portfolio composition and optimization, model validation, backtesting, Robo-advising, virtual customer assistants, market impact analysis, regulatory compliance, and stress testing. This article covers three specific areas where vector data is leveraged as one of the most valuable assets for banking and financial companies:

1. Enhancing customer experience with banking chatbots
2. Boosting financial services sales and more with recommender systems
3. Analyzing earnings reports and other unstructured financial data with semantic text mining

 

### Enhancing customer experience with banking chatbots

Banking chatbots can improve customer experiences by helping consumers select investments, banking products, and insurance policies. Digital services are rising rapidly in popularity in part due to trends accelerated by the coronavirus pandemic. Chatbots work by using natural language processing (NLP) to convert user-submitted questions into semantic vectors to search for matching answers. Modern banking chatbots offer a personalized natural experience for users and speak in a conversational tone. Milvus provides a data fabric well suited for creating chatbots using real-time vector similarity search.

Learn more in our demo that covers building [chatbots with Milvus](https://zilliz.com/blog/building-intelligent-chatbot-with-nlp-and-milvus).

![02 (1).jpg](https://assets.zilliz.com/02_1_8c298c45e5.jpg)

 

#### Boosting financial services sales and more with recommender systems:

The private banking sector uses recommender systems to increase sales of financial products through personalized recommendations based on customer profiles. Recommender systems can also be leveraged in financial research, business news, stock selection, and trading support systems. Thanks to deep learning models, every user and item is described as an embedding vector. A vector database offers an embedding space where similarities between users and items can be calculated.

Learn more from our [demo](https://zilliz.com/blog/graph-based-recommendation-system-with-milvus) covering graph-based recommendation systems with Milvus.

 

#### Analyzing earnings reports and other unstructured financial data with semantic text mining:

Text mining techniques had a substantial impact on the financial industry. As financial data grows exponentially, text mining has emerged as an important field of research in the domain of finance.

Deep learning models are currently applied to represent financial reports through word vectors capable of capturing numerous semantic aspects. A vector database like Milvus is able to store massive semantic word vectors from millions of reports, then conduct similarity searches on them in milliseconds.

Learn more about how to [use deepset's Haystack with Milvus](https://medium.com/deepset-ai/semantic-search-with-milvus-knowledge-graph-qa-web-crawlers-and-more-837451eae9fa).

 

### Don’t be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_tyler-franta-iusJ25iYu1c-unsplash.jpg

medium_tyler-franta-iusJ25iYu1c-unsplash.jpg

small_tyler-franta-iusJ25iYu1c-unsplash.jpg

thumbnail_tyler-franta-iusJ25iYu1c-unsplash.jpg

tyler-franta-iusJ25iYu1c-unsplash.jpg

Milvus can be used to build AI applications for the finance industry including chatbots, recommender systems, and more.

Accelerating AI in Finance with Milvus, an Open-Source Vector Database

VOVA, an emerging e-commerce platform, prioritizes convenient and positive user experiences. Learn how the company used Milvus to build an image search tool to help shoppers find items they encounter in real life.

Jump to:
 
- [How does image search work?](#how-does-image-search-work)
- [Target detection using the YOLO model](#target-detection-using-the-yolo-model)
- [Image feature vector extraction with ResNet](#image-feature-vector-extraction-with-resnet)
- [Vector similarity search powered by Milvus](#vector-similarity-search-powered-by-milvus)
- [VOVA's shop by image tool](#vovas-shop-by-image-tool)

Online shopping surged in 2020, [up 44%](https://www.digitalcommerce360.com/2021/02/15/ecommerce-during-coronavirus-pandemic-in-charts/) in large part due to the coronavirus pandemic. As people sought to socially distance and avoid contact with strangers, no-contact delivery became an incredibly desirable option for many consumers. This popularity has also led to people buying a greater variety of goods online, including niche items that can be hard to describe using a traditional keyword search.

To help users overcome the limitations of keyword-based queries, companies can build image search engines that allow users to use images instead of words for search. Not only does this allow users to find items that are difficult to describe, but it also helps them shop for things they encounter in real life. This functionality helps build a unique user experience and offers general convenience that customers appreciate.

VOVA is an emerging e-commerce platform that focuses on affordability and offering a positive shopping experience to its users, with listings covering millions of products and support for 20 languages and 35 major currencies. To enhance the shopping experience for its users, the company used Milvus to build image search functionality into its e-commerce platform. The article explores how VOVA successfully built an image search engine with Milvus. 

 

### How does image search work?

VOVA's shop by image system searches the company's inventory for product images that are similar to user uploads. The following chart shows the two stages of the system process, the data import stage (blue) and the query stage (orange):

1. Use the YOLO model to detect targets from uploaded photos;
2. Use ResNet to extract feature vectors from the detected targets; 
3. Use Milvus for vector similarity search.

![Vova-1.png](https://assets.zilliz.com/Vova_1_47ee6f2da9.png)
###### *System process of VOVA's search by image functionality.*

 

### Target detection using the YOLO model

VOVA's mobile apps on Android and iOS currently support image search. The company uses a state-of-the-art, real-time object detection system called YOLO (You only look once) to detect objects in user uploaded images. The YOLO model is currently in its fifth iteration. 

YOLO is a one-stage model, using only one convolutional neural network (CNN) to predict categories and positions of different targets. It is small, compact, and well suited for mobile use. 

YOLO uses convolutional layers to extract features and fully-connected layers to obtain predicted values. Drawing inspiration from the GooLeNet model, YOLO’s CNN includes 24 convolutional layers and two fully-connected layers.

As the following illustration shows, a 448 &times; 448 input image is converted by a number of convolutional layers and pooling layers to a 7 &times; 7 &times; 1024-dimensional tensor (depicted in the third to last cube below), and then converted by two fully-connected layers to a 7 &times; 7 &times; 30-dimensional tensor output.

The predicted output of YOLO P is a two-dimensional tensor, whose shape is [batch,7 &times;7 &times;30]. Using slicing, P[:,0:7&times;7&times;20] is the category probability, P[:,7&times;7&times;20:7&times;7&times;(20+2)] is the confidence, and P[:,7&times;7&times;(20+2)]:] is the predicted result of the bounding box.

![vova-2.png](https://assets.zilliz.com/vova_2_1ccf38f721.png)
###### *YOLO network architecture.*

 

### Image feature vector extraction with ResNet

VOVA adopted the residual neural network (ResNet) model to extract feature vectors from an extensive product image library and user uploaded photos. ResNet is limited because as the depth of a learning network increases, the accuracy of the network decreases. The image below depicts ResNet running the VGG19 model (a variant of the VGG model) modified to include a residual unit through the short circuit mechanism. VGG was proposed in 2014 and includes just 14 layers, while ResNet came out a year later and can have up to 152.

The ResNet structure is easy to modify and scale. By changing the number of channels in the block and the number of stacked blocks, the width and depth of the network can be easily adjusted to obtain networks with different expressive capabilities. This effectively solves the network degeneration effect, where accuracy declines as the depth of learning increases. With sufficient training data, a model with improving expressive performance can be obtained while gradually deepening the network. Through model training, features are extracted for each picture and converted to 256-dimensional floating point vectors.

![vova-3.png](https://assets.zilliz.com/vova_3_df4b810281.png)
###### *ResNet structure.*

 

### Vector similarity search powered by Milvus
 
VOVA's product image database includes 30 million pictures and is growing rapidly. To quickly retrieve the most similar product images from this massive dataset, Milvus is used to conduct vector similarity search. Thanks to a number of optimizations, Milvus offers a fast and streamlined approach to managing vector data and building machine learning applications. Milvus offers integration with popular index libraries (e.g., Faiss, Annoy), supports multiple index types and distance metrics, has SDKs in multiple languages, and provides rich APIs for managing vector data.

Milvus can conduct similarity search on trillion-vector datasets in milliseconds, with a query time under 1.5 seconds when nq=1 and an average batch query time under 0.08 seconds. To build its image search engine, VOVA referred to the design of Mishards, Milvus' sharding middleware solution (see the chart below for its system design), to implement a highly available server cluster. By leveraging the horizontal scalability of a Milvus cluster, the project requirement for high query performance on massive datasets was met.

![vova-4.png](https://assets.zilliz.com/vova_4_e305f1955c.png)
###### *Mishards architecture in Milvus.*


### VOVA's shop by image tool
 
The screenshots below show the VOVA search by image shopping tool on the company's Android app.

![vova-5.png](https://assets.zilliz.com/vova_5_c4c25a3bae.png)
###### *Screenshots of VOVA's search by image shopping tool.*
 
As more users search for products and upload photos, VOVA will continue to optimize the models that power the system. Additionally, the company will incorporate new Milvus functionality that can further enhance the online shopping experience of its users.

### Reference

**YOLO:**

https://arxiv.org/pdf/1506.02640.pdf

https://arxiv.org/pdf/1612.08242.pdf

**ResNet:**

https://arxiv.org/abs/1512.03385

**Milvus:**

https://milvus.io/docs/overview.md

large_vova-thumbnail.jpg

medium_vova-thumbnail.jpg

small_vova-thumbnail.jpg

thumbnail_vova-thumbnail.jpg

vova-thumbnail.jpg

Discover how Milvus, an open-source vector database, was used by e-commerce platform VOVA to power shopping by image. 

Building a Search by Image Shopping Experience with VOVA and Milvus

Learn how Milvus is used to mitigate threats to critical data and strengthen cybersecurity with real-time virus detection.

Cybersecurity remains a persistent threat to both individuals and businesses, with data privacy concerns increasing for [86% of companies](https://www.getapp.com/resources/annual-data-security-report/) in 2020 and just [23% of consumers](https://www.firstdata.com/downloads/pdf/digital-commerce-cybersecurity-ebook.pdf) believing their personal data is very secure. As malware becomes steadily more omnipresent and sophisticated, a proactive approach to threat detection has become essential. [Trend Micro](https://www.trendmicro.com/en_us/business.html) is a global leader in hybrid cloud security, network defense, small business security, and endpoint security. To protect Android devices from viruses, the company built Trend Micro Mobile Security—a mobile app that compares APKs (Android Application Package) from the Google Play Store to a database of known malware. The virus detection system works as follows:

- External APKs (Android application package) from the Google Play Store are crawled.
- Known malware is converted into vectors and stored in [Milvus](https://www.milvus.io/docs/v1.0.0/overview.md).
- New APKs are also converted into vectors, then compared to the malware database using similarity search.
- If an APK vector is similar to any of the malware vectors, the app provides users with detailed information about the virus and its threat level.

To work, the system has to perform highly efficient similarity search on massive vector datasets in real time. Initially, Trend Micro used [MySQL](https://www.mysql.com/). However, as its business expanded so did the number of APKs with nefarious code stored in its database. The company’s algorithm team began searching for alternative vector similarity search solutions after quickly outgrowing MySQL.

 

### Comparing vector similarity search solutions

There are a number of vector similarity search solutions available, many of which are open source. Although the circumstances vary from project to project, most users benefit from leveraging a vector database built for unstructured data processing and analytics rather than a simple library that requires extensive configuration. Below we compare some popular vector similarity search solutions and explain why Trend Micro chose Milvus.

#### Faiss
[Faiss](https://ai.facebook.com/tools/faiss/) is a library developed by Facebook AI Research that enables efficient similarity search and clustering of dense vectors. The algorithms it contains search vectors of any size in sets. Faiss is written in C++ with wrappers for Python/numpy, and supports a number of indexes including IndexFlatL2, IndexFlatIP, HNSW, and IVF.

Although Faiss is an incredibly useful tool, it has limitations. It only works as a basic algorithm library, not a database for managing vector datasets. Additionally, it does not offer a distributed version, monitoring services, SDKs, or high availability, which are the key features of most cloud-based services. 

#### Plug-ins based on Faiss & other ANN search libraries
There are several plug-ins built on top of Faiss, NMSLIB, and other ANN search libraries that are designed to enhance the basic functionality of the underlying tool that powers them. Elasticsearch (ES) is a search engine based on the Lucene library with a number of such plugins. Below is an architecture diagram of an ES plug-in:

![1.png](https://assets.zilliz.com/1_3ce4e516c3.png)

###### *Architecture diagram of an Elasticsearch plug-in.*

Built in support for distributed systems is a major advantage of an ES solution. This saves developers time and companies money thanks to code that doesn’t have to be written. ES plug-ins are technically advanced and prevalent. Elasticsearch provides a QueryDSL (domain-specific language), which defines queries based on JSON and is easy to grasp. A full set of ES services makes it possible to conduct vector/text search and filter scalar data simultaneously.

Amazon, Alibaba, and Netease are a few large tech companies that currently rely on Elasticsearch plug-ins for vector similarity search. The primary downsides with this solution are high memory consumption and no support for performance tuning. In contrast, [JD.com](http://jd.com/) has developed its own distributed solution based on Faiss called [Vearch](https://github.com/vearch/vearch). However, Vearch is still an incubation-stage project and its open-source community is relatively inactive.

#### Milvus 
[Milvus](https://www.milvus.io/) is an open-source vector database created by [Zilliz](https://zilliz.com). It is highly flexible, reliable, and blazing fast. By encapsulating multiple widely adopted index libraries, such as Faiss, NMSLIB, and Annoy, Milvus provides a comprehensive set of intuitive APIs, allowing developers to choose the ideal index type for their scenario. It also provides distributed solutions and monitoring services. Milvus has a highly active open-source community and over 5.5K stars on [Github](https://github.com/milvus-io/milvus).

#### Milvus bests the competition
We compiled a number of different test results from the various vector similarity search solutions mentioned above. As we can see in the following comparison table, Milvus was significantly faster than the competition despite being tested on a dataset of 1 billion 128-dimensional vectors.

| **Engine** | **Performance (ms)** | **Dataset Size (million)** |
| :---- | :---- | :---- |
| ES | 600 | 1 |
| ES + Alibaba Cloud | 900 | 20 |
| Milvus | 27 | 1000+ |
| SPTAG | Not good | |
| ES + nmslib, faiss | 90 | 150 |

###### *A comparison of vector similarity search solutions.*

After weighing the pros and cons of each solution, Trend Micro settled on Milvus for its vector retrieval model. With exceptional performance on massive, billion-scale datasets, it's obvious why the company chose Milvus for a mobile security service that requires real-time vector similarity search.

 

### Designing a system for real-time virus detection
Trend Micro has more than 10 million malicious APKs stored in its MySQL database, with 100k new APKs added each day. The system works by extracting and calculating Thash values of different components of an APK file, then uses the Sha256 algorithm to transform it into binary files and generate 256-bit Sha256 values that differentiate the APK from others. Since Sha256 values vary with APK files, one APK can have one combined Thash value and one unique Sha256 value.

Sha256 values are only used to differentiate APKs, and Thash values are used for vector similarity retrieval. Similar APKs may have the same Thash values but different Sha256 values.

To detect APKs with nefarious code, Trend Micro developed its own system for retrieving similar Thash values and corresponding Sha256 values. Trend Micro chose Milvus to conduct instantaneous vector similarity search on massive vector datasets converted from Thash values. After similarity search is run, the corresponding Sha256 values are queried in MySQL. A Redis caching layer is also added to the architecture to map Thash values to Sha256 values, significantly reducing query time.

Below is the architecture diagram of Trend Micro’s mobile security system.

![image-20210118-022039.png](https://assets.zilliz.com/image_20210118_022039_ae824b663c.png)

###### *Architecture diagram for Trend Micro Mobile Security.*

 

Choosing an appropriate distance metric helps improve vector classification and clustering performance. The following table shows the [distance metrics](https://www.milvus.io/docs/v1.0.0/metric.md#binary) and the corresponding indexes that work with binary vectors. 

| **Distance Metrics** | **Index Types** |
| ---- | ---- |
| - Jaccard - Tanimoto - Hamming | - FLAT - IVF_FLAT |
| - Superstructure - Substructure | FLAT |

###### *Distance metrics and indexes for binary vectors.*

 

Trend Micro converts Thash values into binary vectors and stores them in Milvus. For this scenario, Trend Micro is using Hamming distance to compare vectors.

Milvus will soon support string vector ID, and integer IDs won’t have to be mapped to the corresponding name in string format. This makes the Redis caching layer unnecessary and the system architecture less bulky.

Trend Micro adopts a cloud-based solution and deploys many tasks on [Kubernetes](https://kubernetes.io/). To achieve high availability, Trend Micro uses [Mishards](https://www.milvus.io/docs/v1.0.0/mishards.md), a Milvus cluster sharding middleware developed in Python. 

![image-20210118-022104.png](https://assets.zilliz.com/image_20210118_022104_3001950ee8.png)

###### *Mishards architecture in Milvus.*

 

Trend Micro separates storage and distance calculation by storing all vectors in the [EFS](https://aws.amazon.com/efs/) (Elastic File System) provided by [AWS](https://aws.amazon.com/). This practice is a popular trend in the industry. Kubernetes is used to start multiple reading nodes, and develops LoadBalancer services on these reading nodes to ensure high availability.

To maintain data consistency Mishards supports just one writing node. However, a distributed version of Milvus with support for multiple writing nodes will be available in the coming months. 

 

### Monitoring and Alert Functions
Milvus is compatible with monitoring systems built on [Prometheus](https://prometheus.io/), and uses [Grafana](https://grafana.com/), an open-source platform for time-series analytics, to visualize various performance metrics.

Prometheus monitors and stores the following metrics:

- Milvus performance metrics including insertion speed, query speed, and Milvus uptime. 
- System performance metrics including CPU/GPU usage, network traffic, and disk access speed.
- Hardware storage metrics including data size and total file number.

The monitoring and alert system works as follows: 

- A Milvus client pushes customized metrics data to Pushgateway.
- The Pushgateway ensures short-lived, ephemeral metric data is safely sent to Prometheus.
- Prometheus keeps pulling data from Pushgateway. 
- Alertmanager sets the alert threshold for different metrics and raises alarms through emails or messages.

 

### System Performance
A couple months have passed since the ThashSearch service built on Milvus was first launched. The graph below shows that end-to-end query latency is less than 95 milliseconds.

![image-20210118-022116.png](https://assets.zilliz.com/image_20210118_022116_a0c735ce20.png)

###### *Query latency for Thash search service built on Milvus.*

 

Insertion is also fast. It takes around 10 seconds to insert 3 million 192-dimensional vectors. With help from Milvus, the system performance was able to meet the performance criteria set by Trend Micro. 

 

### Don’t be a stranger
- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_TrendMicro.jpg

medium_blog_TrendMicro.jpg

small_blog_TrendMicro.jpg

thumbnail_blog_TrendMicro.jpg

blog_TrendMicro.jpg

Making with Milvus: Detecting Android Viruses in Real Time for Trend Micro

Learn more about using semantic machine learning methodologies to power more relevant search results across your organization.

[Semantic search](https://lucidworks.com/post/what-is-semantic-search/) is a great tool to help your customers—or your employees—find the right products or information. It can even surface difficult-to-index information for better results. That said, if your semantic methodologies aren’t being deployed to work fast, they won’t do you any good. The customer or employee isn’t just going to sit around while the system takes its time responding to their query—and a thousand others are likely being ingested at the same time.

How can you make semantic search fast? Slow semantic search isn’t going to cut it.

Fortunately, this is the kind of problem Lucidworks loves to solve. We recently tested a modest-sized cluster—read on for more details—that resulted in 1500 RPS (requests per second) against a collection of over one million documents, with an average response time of roughly 40 milliseconds. Now that’s some serious speed.

 

### Implementing Semantic Search
To make lightning-fast, machine learning magic happen, Lucidworks has implemented semantic search using the semantic vector search approach. There are two critical parts.

 

#### Part One: The Machine Learning Model

First, you need a way to encode text into a numerical vector. The text could be a product description, a user search query, a question, or even an answer to a question. A semantic search model is trained to encode text such that text that is semantically similar to other text is encoded into vectors that are numerically “close” to one another. This encoding step needs to be fast in order to support the thousand or more possible customer searches or user queries coming in every second.

 

#### Part Two: The Vector Search Engine
Second, you need a way to quickly find the best matches to the customer search or user query. The model will have encoded that text into a numerical vector. From there, you need to compare that to all the numerical vectors in your catalog or lists of questions and answers to find those best matches—the vectors that are “closest” to the query vector. For that, you will need a vector engine that can handle all of that information effectively and at lightning speed. The engine could contain millions of vectors and you really just want the best twenty or so matches to your query. And of course, it needs to handle a thousand or so such queries every second.

To tackle these challenges, we added the vector search engine [Milvus](https://doc.lucidworks.com/fusion/5.3/8821/milvus) in our [Fusion 5.3 release](https://lucidworks.com/post/enhance-personalization-efforts-with-new-features-in-fusion/). Milvus is open-source software and it is fast. Milvus uses FAISS ([Facebook AI Similarity Search](https://ai.facebook.com/tools/faiss/)), the same technology Facebook uses in production for its own machine learning initiatives. When needed, it can run even faster on [GPU](https://en.wikipedia.org/wiki/Graphics_processing_unit). When Fusion 5.3 (or higher) is installed with the machine learning component, Milvus is automatically installed as part of that component so you can turn on all of these capabilities with ease.

The size of the vectors in a given collection, specified when the collection is created, depends on the model that produces those vectors. For example, a given collection could store the vectors created from encoding (via a model) all of the product descriptions in a product catalog. Without a vector search engine like Milvus, similarity searches would not be feasible across the entire vector space. So, similarity searches would have to be limited to pre-selected candidates from the vector space (for example, 500) and would have both slower performance and lower quality results. Milvus can store hundreds of billions of vectors across multiple collections of vectors to ensure that search is fast and results are relevant.

 

### Using Semantic Search

Let’s get back to the semantic search workflow, now that we’ve learned a little about why Milvus might be so important. Semantic search has three stages. During the first stage, the machine learning model is loaded and/or trained. Afterwards, data is indexed into Milvus and Solr. The final stage is the query stage, when the actual search occurs. We’ll focus on those last two stages below.

 

### Indexing into Milvus
![Lucidworks-1.png](https://assets.zilliz.com/Lucidworks_1_47a9221723.png)
###### *Architectural diagram for indexing into Milvus.*

As shown in the above diagram, the query stage begins similarly to the indexing stage, just with queries coming in instead of documents. For each query:

1. The query is sent to the [Smart Answers](https://lucidworks.com/products/smart-answers/) index pipeline.
2. The query is then sent to the ML model.
3. The ML model returns a numeric vector (encrypted from the query). Again, the type of model determines the size of the vector.
4. The vector is sent to Milvus, which then determines which vectors, in the specified Milvus collection, best match the provided vector.
5. Milvus returns a list of unique IDs and distances corresponding to the vectors determined in step four.
6. A query containing those IDs and distances is sent to Solr.
7. Solr then returns an ordered list of the documents associated with those IDs.

 

### Scale Testing
In order to prove that our semantic search flows are running at the efficiency we require for our customers, we run scale tests using Gatling scripts on the Google Cloud Platform using a Fusion cluster with eight replicas of the ML model, eight replicas of the query service, and a single instance of Milvus. Tests were run using the Milvus FLAT and HNSW indexes. The FLAT index has 100% recall, but is less efficient – except when the datasets are small. The HNSW (Hierarchical Small World Graph) index still has high quality results and it has improved performance on larger datasets.

Let’s jump into some numbers from a recent example we ran:

![Lucidworks-2.png](https://assets.zilliz.com/Lucidworks_2_3162113560.png)
###### *Performance of Milvus FLAT and HNSW indexes on a small dataset.*

![Lucidworks-3.png](https://assets.zilliz.com/Lucidworks_3_3dc17f0ed8.png)
###### *Performance of Milvus FLAT and HNSW indexes on a medium dataset.*

![Lucidworks-4.png](https://assets.zilliz.com/Lucidworks_4_8a6edd2f59.png)
###### *Performance of Milvus FLAT and HNSW indexes on a large dataset.*

 

### Getting Started

The [Smart Answers](https://lucidworks.com/products/smart-answers/) pipelines are designed to be easy-to-use. Lucidworks has [pre-trained models that are easy-to-deploy](https://doc.lucidworks.com/how-to/734/set-up-a-pre-trained-cold-start-model-for-smart-answers) and generally have good results—though training your own models, in tandem with pre-trained models, will offer the best results. Contact us today to learn how you can implement these initiatives into your search tools to power more effective and delightful results.


> This blog is reposted from: https://lucidworks.com/post/how-to-build-fast-semantic-search/?utm_campaign=Oktopost-Blog+Posts&utm_medium=organic_social&utm_source=linkedin

Products

Resources

Open Source

About

Blog