the minimal workflow of a recommender system

large_1.png

medium_1.png

small_1.png

thumbnail_1.png

1.png

Accelerating Candidate Generation in Recommender Systems Using Milvus paired with PaddlePaddle

This tutorial introduces how to run Milvus 2.0 on Windows. 

This tutorial introduces how to run Milvus 2.0 on Windows. If you want to get started with Milvus 2.0 using Windows, or simply have to run Milvus on Windows due to environment limits, this tutorial can be a good reference. This article is originally written by [@matrixji](https://github.com/matrixji).

There are two ways to run Milvus 2.0 on Windows:

- [Compile from source code](#Compile-from-source-code)
- [Install from the binary package](#Install-from-the-binary-package)



## Compile from source code

### Prerequisites

You need to install MinGW64/MSYS toolchain before compiling Milvus 2.0 on Windows. See [MSYS2](https://www.msys2.org/) for the installation guide. The compilation in this tutorial is completed in the shell of MSYS2 MinGW 64-bit. You can run `$ pacman -Su` to update the toolchain. 

We also recommend that you run `$ pacman -S git` to install git. 

### Step 1. Clone code 

Run the following command to clone the code from the **windows-dev** branch in the **matrixji/milvus** repository. Currently, all commits can be traced in **windows-dev**, which rebases commits from the upstream main branch [**milvus-io/milvus**](https://github.com/milvus-io/milvus) on a weekly basis.

```python
$ git clone git@github.com:matrixji/milvus.git -b windows-dev
```

### Step 2. Compile Milvus

#### 1. Install dependencies and toolchain

Run the following command to install the required dependencies and toolchain. After the installation, you need to restart the MinGW/MYSY shell to ensure the configurations of some tools are properly applied.

```python
$ cd milvus
$ scripts/install_deps_msys.sh
```

#### 2. Compile Milvus

Run the following command to compile and package Milvus.

```python
$ make
$ sh scripts/package_windows.sh
```

If successful, you will find a zip file in the subdirectory of `windows_package`. The zip file contains all the files for running `milvus.exe`, including `minio.exe` and `etcd.exe`.

Please note that Milvus 2.0 is developed in Go. You might need to configure [Goproxy](https://goproxy.cn/) to install third-party modules.



## Install from the binary package

You can also download the binary package that has already been compiled from [Releases - matrixji/milvus](https://github.com/matrixji/milvus/releases). Each release has **windows-test-** as the prefix, for instance, **windows-test-v8**. Find the release you want and download the corresponding zip package.

### Step 1. Start Milvus 

#### 1. Unzip the package

Unzip the package, and you will find a `milvus` directory, which contains all the files required.

#### 2. Start a MinIO service

Double-click the `run_minio.bat` file to start a MinIO service with default configurations. Data will be stored in the subdirectory `s3data`.

#### 3. Start an etcd service

Double-click the `run_etcd.bat` file to start an etcd service with default configurations.

#### 4. Start Milvus service

Double-click the `run_milvus.bat` file to start the Milvus service.



### Step 2. Run `hello_milvus.py`

After starting the Milvus service, you can test by running `hello_milvus.py `. See [Hello Milvus](https://milvus.io/docs/v2.0.0/example_code.md) for more information.

medium_screenshot-20211119-104605.png

small_screenshot-20211119-104605.png

thumbnail_screenshot-20211119-104605.png

screenshot-20211119-104605.png

Run Milvus 2.0 on Windows

How to modify the configuration of Milvus deployed on Kubernetes

*Yufen Zong, a Zilliz Test Development Engineer, graduated from Huazhong University of Science and Technology with a master's degree in computer technology. She is currently engaged in the quality assurance of Milvus vector database, including but not limited to interface integration testing, SDK testing, Benchmark testing, etc. Yufen is an enthusiastic problem-shooter in the test and development of Milvus, and a huge fan of chaos engineering theory and fault drill practice.*

## Background

While using Milvus vector database, you will need to modify the default configuration to satisfy the requirements of different scenarios. Previously, a Milvus user shared on [how to modify the configuration of Milvus deployed using Docker Compose] (https://zilliz.com/blog/apply-configuration-changes-on-milvus-2). And in this article, I would like to share with you on how to modify the configuration of Milvus deployed on Kubernetes.

## Modify configuration of Milvus on Kubernetes

You may choose different modification plans according to the configuration parameters you wish to modify. All Milvus configuration files are stored under **milvus/configs**. While installing Milvus on Kubernetes, a Milvus Helm Chart repository will be added locally. By running `helm show values milvus/milvus`, you can check the parameters that can be modified directly with Chart. For the modifiable parameters with Chart, you can pass the parameter using `--values` or `--set`. For more information, see [Milvus Helm Chart](https://artifacthub.io/packages/helm/milvus/milvus) and [Helm](https://helm.sh/docs/).

If the parameters you expect to modify are not on the list, you can follow the instruction below.

In the following steps, the parameter `rootcoord.dmlChannelNum` in **/milvus/configs/advanced/root_coord.yaml** is to be modified for demonstration purposes. Configuration file management of Milvus on Kubernetes is implemented through ConfigMap resource object. To change the parameter, you should first update the ConfigMap object of corresponding Chart release, and then modify the deployment resource files of corresponding pods. 

Beware that this method only applies to parameter modification on deployed Milvus application. To modify the parameters in **/milvus/configs/advanced/\*.yaml** before deployment, you will need to re-develop the Milvus Helm Chart.

### Modify ConfigMap YAML

As shown below, your Milvus release running on Kubernetes corresponds to a ConfigMap object with the same name of the release. The `data` section of the ConfigMap object only includes configurations in **milvus.yaml**. To change the `rootcoord.dmlChannelNum` in **root_coord.yaml**, you must add the parameters in **root_coord.yaml** to the `data` section in the ConfigMap YAML and change the specific parameter. 

```
kind: ConfigMap
apiVersion: v1
metadata:
  name: milvus-chaos
  ...
data:
  milvus.yaml: >
    ......
  root_coord.yaml: |
    rootcoord:
      dmlChannelNum: 128
      maxPartitionNum: 4096
      minSegmentSizeToEnableIndex: 1024
      timeout: 3600 # time out, 5 seconds
      timeTickInterval: 200 # ms
```

### Modify Deployment YAML

The data stored in a ConfigMap can be referenced in a volume of type configMap and then consumed by containerized applications running in a pod. To direct the pods to the new configuration files, you must modify the pod templates that need to load the configurations in **root_coord.yaml**. Specifically, you need to add a mount declaration under the `spec.template.spec.containers.volumeMounts` section in deployment YAML.

Taking the deployment YAML of rootcoord pod as an example, a `configMap` type volume named **milvus-config** is specified in `.spec.volumes` section. And, in `spec.template.spec.containers.volumeMounts` section, the volume is declared to mount **milvus.yaml** of your Milvus release on **/milvus/configs/milvus.yaml**. Similarly, you only need to add a mount declaration specifically for rootcoord container to mount the **root_coord.yaml** on **/milvus/configs/advanced/root_coord.yaml**, and thus the container can access the new configuration file.

```yaml
spec:
  replicas: 1
  selector:
    ......
  template:
    metadata:
      ...
    spec:
      volumes:
        - name: milvus-config
          configMap:
            name: milvus-chaos
            defaultMode: 420
      containers:
        - name: rootcoord
          image: 'milvusdb/milvus-dev:master-20210906-86afde4'
          args:
            ...
          ports:
            ...
          resources: {}
          volumeMounts:
            - name: milvus-config
              readOnly: true
              mountPath: /milvus/configs/milvus.yaml
              subPath: milvus.yaml
            - name: milvus-config
              readOnly: true
              mountPath: /milvus/configs/advanced/`root_coord.yaml
              subPath: root_coord.yaml
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
```

### Verify the result

The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. When the ConfigMap consumed in the volume is updated, projected keys are automatically updated as well. When the new pod is running again, you can verify if the modification is successful in the pod. Commands to check the parameter `rootcoord.dmlChannelNum` are shared below.

```bash
$ kctl exec -ti milvus-chaos-rootcoord-6f56794f5b-xp2zs -- sh
# cd configs/advanced
# pwd
/milvus/configs/advanced
# ls
channel.yaml  common.yaml  data_coord.yaml  data_node.yaml  etcd.yaml  proxy.yaml  query_node.yaml  root_coord.yaml
# cat root_coord.yaml
rootcoord:
  dmlChannelNum: 128
  maxPartitionNum: 4096
  minSegmentSizeToEnableIndex: 1024
  timeout: 3600 # time out, 5 seconds
  timeTickInterval: 200 # ms
# exit
```



Above is the method to modify the advanced configurations in Milvus deployed on Kubernetes. Future release of Milvus will integrate all configurations in one file, and will support updating configuration via helm chart. But before that, I hope this article can help you as a temporary solution.



## Engage with our open-source community:

- Find or contribute to Milvus on [GitHub](https://bit.ly/3EUGtcd).

- Interact with the community via [Forum](https://bit.ly/3o14tDG).

- Connect with us on [Twitter](https://bit.ly/3qvSeSD).

  

small_blog cover.png

thumbnail_blog cover.png

blog cover.png

Technical Sharing: How to Modify Milvus Advanced Configurations

In the era of Big Data, what database technologies and applications will come into the limelight? What will be the next game-changer?

large_milvuscover.png

medium_milvuscover.png

small_milvuscover.png

thumbnail_milvuscover.png

milvuscover.png

Design and Practice of AI-oriented General-purpose Vector Database Systems

Frustrated with New Data? Our Vector Database can Help

Apply Configuration Changes on Milvus 2.0 using Docker Compose

*Jingjing Jia, Zilliz Data Engineer, graduated from Xi’an Jiaotong University with a degree in Computer Science. After joining Zilliz, she mainly works on data pre-processing, AI model deployment, Milvus related technology research, and helping community users to implement application scenarios. SHe is very patient, likes to communicate with community partners, and enjoys listening to music and watching anime.*


As a frequent user of Milvus, I was very excited about the newly released Milvus 2.0 RC. According to the introduction on the official website, Milvus 2.0 seems to outmatch its predecessors by a large margin. I was so eager to try it out myself.



And I did.  However, when I truly got my hands on Milvus 2.0, I realized that I wasn't able to modify the configuration file in Milvus 2.0 as easily as I did with Milvus 1.1.1. I couldn't change the configuration file inside the docker container of Milvus 2.0 started with Docker Compose, and even force change wouldn't take effect. Later, I learned that Milvus 2.0 RC was unable to detect changes to the configuration file after installation. And future stable release will fix this issue.



Having tried different approaches, I've found a reliable way to apply changes to configuration files for Milvus 2.0 standalone & cluster, and here is how.



Note that all changes to configuration must be made before restarting Milvus using Docker Compose.

## Modify configuration file in Milvus standalone

First, you will need to [download](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml) a copy of **milvus.yaml** file to your local device.



Then you can change the configurations in the file. For instance, you can change the log format as `.json`. 

![1.1.png](https://assets.zilliz.com/1_1_ee4a16a3ee.png)



Once **milvus.yaml** file is modified, you will also need to [download](https://github.com/milvus-io/milvus/blob/master/deployments/docker/standalone/docker-compose.yml) and modify in **docker-compose.yaml** file for standalone by mapping the local path to milvus.yaml onto the corresponding docker container path to configuration file `/milvus/configs/milvus.yaml` under the `volumes` section.

![1.2.png](https://assets.zilliz.com/1_2_5e7c73708c.png)

Lastly, start Milvus standalone using `docker-compose up -d` and check if the modifications are successful. For instance, run `docker logs` to check the log format.

![1.3.png](https://assets.zilliz.com/1_3_a0406df3ab.png)



## Modify configuration file in Milvus cluster

First, [download](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml) and modify the **milvus.yaml** file to suit your needs.

![1.4.png](https://assets.zilliz.com/1_4_758b182846.png)


Then you will need to [download](https://github.com/milvus-io/milvus/blob/master/deployments/docker/cluster/docker-compose.yml) and modify the cluster **docker-compose.yml** file by mapping the local path to **milvus.yaml** onto the corresponding path to configuration files in all components, i.e. root coord, data coord, data node, query coord, query node, index coord, index node, and proxy.

![1.5.png](https://assets.zilliz.com/1_5_80e15811b8.png)


![1.6.png](https://assets.zilliz.com/1_6_b2f3e4e47f.png)
![1.7.png](https://assets.zilliz.com/1_7_4d1eb5e1e5.png)


Finally, you can start Milvus cluster using `docker-compose up -d` and check if the modifications are successful.

## Change log file path in configuration file

First, [download](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml) the **milvus.yaml** file, and change the `rootPath` section as the directory where you expect to store the log files in Docker container. 


![1.8.png](https://assets.zilliz.com/1_8_e3bdc4843f.png)




After that, download the corresponding **docker-compose.yml** file for Milvus [standalone](https://github.com/milvus-io/milvus/blob/master/deployments/docker/standalone/docker-compose.yml) or [cluster](https://github.com/milvus-io/milvus/blob/master/deployments/docker/cluster/docker-compose.yml). 

For standalone, you need to map the local path to **milvus.yaml** onto the corresponding docker container path to configuration file `/milvus/configs/milvus.yaml`, and map the local log file directory onto the Docker container directory you created previously. 

For cluster, you will need to map both paths in every component.

![1.9.png](https://assets.zilliz.com/1_9_22d8929d92.png)

Lastly, start Milvus standalone or cluster using `docker-compose up -d` and check the log files to see if the modification is successful.

small_pc-blog.jpg

thumbnail_pc-blog.jpg

pc-blog.jpg

Technical Sharing: Apply Configuration Changes on Milvus 2.0 using Docker Compose

Learn how Milvus powers the AI analysis of video contents.

*Shiyu Chen, a data engineer at Zilliz, graduated from Xidian University with a degree in Computer Science. Since joining Zilliz, she has been exploring solutions for Milvus in various fields, such as audio and video analysis, molecule formula retrieval, etc., which has greatly enriched the application scenarios of the community. She is currently exploring more interesting solutions. In her spare time, she loves sports and reading.*

When I was watching *Free Guy* last weekend, I felt that I'd seen the actor who plays Buddy, the security guard, somewhere before, yet couldn't recall any of his works. My head was stuffed with "who's this guy?" I was sure about having seen that face and was trying so hard to remember his name. A similar case is that once I saw the leading actor in a video having a drink I used to like a lot, but I ended up failing to recall the brand name.

The answer was on the tip of my tongue, but my brain felt completely stuck.

The tip of the tongue (TOT) phenomenon drives me crazy when watching movies. If only there was a reverse image search engine for videos that enables me to find videos and analyze video content. Before, I built a [reverse image search engine using Milvus](https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy). Considering that video content analysis somehow resembles image analysis, I decided to build a video content analysis engine based on Milvus.

## Object detection

### Overview

Before being analyzed, objects in a video should be detected first. Detecting objects in a video effectively and accurately is the main challenge of the task. It is also an important task for applications such as autopilot, wearable devices, and IoT.

Developed from traditional image processing algorithms to deep neural networks (DNN), today's mainstream models for object detection include R-CNN, FRCNN, SSD, and YOLO. The Milvus-based deep learning video analysis system introduced in this topic can detect objects intelligently and quickly.

### Implementation

To detect and recognize objects in a video, the system should first extract frames from a video and detect objects in the frame images using object detection, secondly, extract feature vectors from detected objects, and lastly, analyze the object based on the feature vectors.

- Frame extraction

Video analysis is converted to image analysis using frame extraction. Currently, frame extraction technology is very mature. Programs such as FFmpeg and OpenCV support extracting frames at specified intervals. This article introduces how to extract frames from a video every second using OpenCV.

- Object detection

Object detection is about finding objects in extracted frames and extracting screenshots of the objects according to their positions. As shown in the following figures, a bike, a dog, and a car were detected. This topic introduces how to detect objects using YOLOv3, which is commonly used for object detection.

![p1.png](https://assets.zilliz.com/p1_0913edccdc.png)


- Feature extraction

Feature extraction refers to converting unstructured data, which is difficult for machines to recognize, to feature vectors. For example, images can be converted to multi-dimensional feature vectors using deep learning models. Currently, the most popular image recognition AI models include VGG, GNN, and ResNet. This topic introduces how to extract features from detected objects using ResNet-50. 

![p2.png](https://assets.zilliz.com/p2_b398855741.png)

- Vector analysis

Extracted feature vectors are compared with library vectors, and the corresponding information to the most similar vectors is returned. For large-scale feature vector datasets, calculation is a huge challenge. This topic introduces how to analyze feature vectors using Milvus.

## Key technologies

### OpenCV 

Open Source Computer Vision Library (OpenCV) is a cross-platform computer vision library, which provides many universal algorithms for image processing and computer vision. OpenCV is commonly used in the computer vision field. 

The following example shows how to capture video frames at specified intervals and save them as images using OpenCV with Python.

```python
import cv2 
cap = cv2.VideoCapture(file_path)   
framerate = cap.get(cv2.CAP_PROP_FPS)   
allframes = int(cv2.VideoCapture.get(cap, int(cv2.CAP_PROP_FRAME_COUNT)))  
success, image = cap.read() 
cv2.imwrite(file_name, image)
```

### YOLOv3 

You Only Look Once, Version 3 (YOLOv3 [5]) is a one-stage object detection algorithm proposed in recent years. Compared to the traditional object detection algorithms with the same accuracy, YOLOv3 is twice as fast. YOLOv3 mentioned in this topic is the enhanced version from PaddlePaddle [6]. It uses multiple optimization methods with a higher inference speed.

### ResNet-50 

ResNet [7] is the winner of ILSVRC 2015 in image classification because of its simplicity and practicality. As the basis of many image analysis methods, ResNet proves to be a popular model specialized in image detection, segmentation, and recognition.

### Milvus  

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following procedures describe how Milvus works.

1. Unstructured data is converted to feature vectors by using deep learning models and is imported to Milvus.
2. Milvus stores and indexes the feature vectors.
3. Milvus returns the most similar vectors to the vector queried by users.

![p3.png](https://assets.zilliz.com/p3_00168a4c93.png)

## Deployment

Now you have some understanding of Milvus-based video analysis systems. The system mainly consists of two parts, as shown in the following figure.

- The red arrows indicate the data import process. Use ResNet-50 to extract feature vectors from the image dataset and import the feature vectors to Milvus.

- The black arrows indicate the video analysis process. First, extract frames from a video and save the frames as images. Second, detect and extract objects in the images using YOLOv3. Then, use ResNet-50 to extract feature vectors from the images. Lastly, Milvus searches and returns the information of the objects with the corresponding feature vectors.

![p4.png](https://assets.zilliz.com/p4_fdf0a9d93f.png)

For more information, see [Milvus Bootcamp: Video Object Detection System](https://github.com/milvus-io/bootcamp/tree/master/solutions/video_similarity_search/object_detection). 

**Data import**

The data import process is simple. Convert the data into 2,048-dimensional vectors and import the vectors into Milvus. 

```python
vector = image_encoder.execute(filename)
entities = [vector]
collection.insert(data=entities)
```

**Video analysis**

As introduced above, the video analysis process includes capturing video frames, detecting objects in each frame, extracting vectors from the objects, calculating vector similarity with Euclidean distance (L2) metrics, and searching for results using Milvus.

```python
images = extract_frame(filename, 1, prefix)   
detector = Detector()   
run(detector, DATA_PATH)       
vectors = get_object_vector(image_encoder, DATA_PATH)
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
results = collection.search(vectors, param=search_params, limit=10)
```

## Conclusion

Currently, more than 80% of the data is unstructured. With the rapid development of AI, an increasing number of deep learning models have been developed for analyzing unstructured data. Technologies such as object detection and image processing have achieved great breakthroughs in both academia and industry. Empowered by these technologies, more and more AI platforms have fulfilled practical requirements.

The video analysis system discussed in this topic is built with Milvus, which can quickly analyze video content.

As an open-source vector database, Milvus supports feature vectors extracted using various deep learning models. Integrated with libraries such as Faiss, NMSLIB, and Annoy, Milvus provides a set of intuitive APIs, supporting switching index types according to scenarios. Additionally, Milvus supports scalar filtering, which increases recall rate and search flexibility. Milvus has been applied to many fields such as image processing, computer vision, natural language processing, speech recognition, recommender system, and new drug discovery.

## References

[1] A. D. Bagdanov, L. Ballan, M. Bertini, A. Del Bimbo. “Trademark matching and retrieval in sports video databases.” Proceedings of the international workshop on Workshop on multimedia information retrieval, ACM, 2007. https://www.researchgate.net/publication/210113141_Trademark_matching_and_retrieval_in_sports_video_databases 

[2] J. Kleban, X. Xie, W.-Y. Ma. “Spatial pyramid mining for logo detection in natural scenes.” IEEE International Conference, 2008. https://ieeexplore.ieee.org/document/4607625 

[3] R. Boia, C. Florea, L. Florea, R. Dogaru. “Logo localization and recognition in natural images using homographic class graphs.” Machine Vision and Applications 27 (2), 2016. https://link.springer.com/article/10.1007/s00138-015-0741-7 

[4] R. Boia, C. Florea, L. Florea. “Elliptical asift agglomeration in class prototype for logo detection.” BMVC, 2015. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=5C87F52DE38AB0C90F8340DFEBB841F7?doi=10.1.1.707.9371&rep=rep1&type=pdf 

[5] https://arxiv.org/abs/1804.02767 

[6] https://paddlepaddle.org.cn/modelbasedetail/yolov3 

[7] https://arxiv.org/abs/1512.03385 

large_1011cover.png

medium_1011cover.png

small_1011cover.png

thumbnail_1011cover.png

1011cover.png

What Is It? Who Is She? Milvus Helps Analyze Videos Intelligently

Use ONNX to process multiple models of CV (computer vision), and combine multiple models with Milvus for similar vector retrieval to get similar images.

Open Neural Network Exchange (ONNX) is an open format built to represent machine learning models. Since it was open-sourced in 2017, ONNX has developed into a standard for AI, providing building blocks for machine learning and deep learning models. ONNX defines a common file format to enable AI developers to use models with various frameworks, tools, runtimes, and compilers, and helps increase the speed of innovation in the artificial intelligence community.

Milvus is an open-source vector database that is highly flexible, reliable and blazing fast. It supports adding, deleting, updating and near real-time search of vector. Milvus has a comprehensive set of intuitive APIs, and support for multiple widely adopted index libraries (e.g. Faiss, NMSLIB, and Annoy), simplifying the index selection for a given scenario. Milvus is simple to use, and it has been used in hundreds of organizations and institutions worldwide, including image, audio and video search, recommendation, chatbot, new drug search, etc.

This article will introduce you how to use multiple models for image search based on ONNX and Milvus. It takes VGG16 and ResNet50 models as examples, uses ONNX to run different AI models to generate feature vectors, and finally performs feature vector retrieval in Milvus to return similar images. There is an open sourced Jupyter Notebook code file for this project on [GitHub](https://github.com/milvus-io/bootcamp/blob/master/etc/onnx_and_milvus/milvus_with_different_models.ipynb), and next we will show you how to achieve.

## Process Models with ONNX
The ONNX format can be easily exchanged between AI models. For example, the TensorFlow model can be converted to ONNX format and run in the Caffe environment. In this example, we convert the pre-trained ResNet50 model under the Keras framework to the ONNX format, and then call the VGG16 model in ONNX format to analyze different models.

```
 from keras.applications.resnet50 import ResNet50
import tensorflow as tf

# load keras-resnet50 model and save as a floder
model_resnet50 = ResNet50(include_top=False, pooling='max', weights='imagenet')
tf.saved_model.save(model_resnet50, "keras_resnet50_model")

# convert resnet50 model to onnx
! python -m tf2onnx.convert --saved-model "keras_resnet50_model" --output "onnx_resnet50.onnx"
```

Note: When we used the interface `keras2onnx.convert_keras(model, model.name)` to convert the model, it will return the error `AttributeError:'KerasTensor' object has no attribute'graph'`. Then we can use Python's Bash command to convert according to the solution on Stack Overflow.

## Extract Feature Vectors using Models
After converting ResNet50 model into ONNX format, you can extract the feature vector of the picture directly through the inference.
Note: Feature vectors need to be normalized after extraction.

```
# get the image vectors with onnx model
def get_onnx_vectors(onnx_model, img_path):
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    
    sess = onnxruntime.InferenceSession(onnx_model)
    x = x if isinstance(x, list) else [x]
    feed = dict([(input.name, x[n]) for n, input in enumerate(sess.get_inputs())])
    feat = sess.run(None, feed)[0]
    
    norm_feat = feat[0] / LA.norm(feat[0])
    norm_feat = [i.item() for i in norm_feat]
    return norm_feat
```

Use ONNX-formatted VGG16 model to process image data:

```
# generate vectors with ResNet50 and VGG16 ONNX model
2vec_resnet = get_onnx_vectors("onnx_resnet50.onnx", "./pic/example.jpg")
3vec_vgg = get_onnx_vectors("onnx_vgg16.onnx", "./pic/example.jpg")
```

## Store Vector Data
Unstructured data such as pictures cannot be directly processed by computer, but it can be converted into vectors through AI model and then analyzed by a computer. Milvus vector database is designed power massive unstructured data analysis. It can store vector data and perform near real-time analysis.
First, create a collection of the corresponding model in Milvus, and then insert the image vectors.

```
from milvus import *

# create collections in Milvus
milvus.create_collection(resnet_collection_param)
milvus.create_collection(vgg_collection_param)

# insert data to Milvus and return ids
status, resnet_ids = milvus.insert(resnet_collection_name, resnet_vectors)
status, vgg_ids = milvus.insert(vgg_collection_name, vgg_vectors)
```

After inserting the data successfully, Milvus will return the ID corresponding to the vector, and then we can find the picture by ID. Since Milvus 1.1 used in this case does not support scalar filtering (which Milvus 2.0 now supports), Redis is used to store the vector ID and the key-value of the image path.

```
import redis
def img_ids_to_redis(img_directory, res_ids):
  for img, ids in zip(images, res_ids):
    redis.set(ids, img)
```

## Search for Similar Images
After storing the data, we can retrieve the vector. Milvus supports multiple distance calculation methods, including Euclidean, inner product and Hamming distance. The image similarity search in this article adopts the Euclidean distance calculation between the vectors in Milvus, returns the similar vector ID, and then finds the image corresponding to the ID in Redis.

```
# search in Milvus and return the similarly results with ids
def search_in_milvus(collection_name, search_vector):
    status, results = milvus.search(collection_name, TOP_K, [search_vector])
    print(status)
    re_ids = [x.id for x in results[0]]
    re_distance = [x.distance for x in results[0]]
    return re_ids, re_distance
    
# get the images according the result ids
def get_sim_imgs(collection_name, search_vector):
    ids, distance = search_in_milvus(collection_name, search_vector)
    img = [red.get(i).decode("utf-8") for i in ids]
    return ids, distance, img
```

Taking the VGG16 and ResNet50 models as examples, this article shows processing multiple models through ONNX and combining multiple models with Milvus for similar vector retrieval to get similar images. The above two models are based on the Keras framework, which can quickly extract feature vectors. It can be seen from the Notebook that although the results of Milvus's search for pictures on the COCO dataset based on these two models are similar, their Euclidean distances are not the same. You can also try to compare the search results of the two models using other datasets.

Milvus is a high-performance, highly available vector database that can be used to process feature vectors generated from massive unstructured data. For more solutions, you can refer to [Milvus bootcamp](https://github.com/milvus-io/bootcamp).

##  References
1. https://github.com/onnx/onnx
2. https://onnx.ai/
3. https://milvus.io/cn/
4. https://github.com/milvus-io/bootcamp



## About author
Shiyu Chen, a data engineer at Zilliz, graduated from Xidian University with a degree in Computer Science. Since joining Zilliz, she has been exploring solutions for Milvus in various fields, such as audio and video analysis, molecule formula retrieval, etc., which has greatly enriched the application scenarios of the community. She is currently exploring more interesting solutions. In her spare time, she loves sports and reading.


Combine AI Models for Image Search using ONNX and Milvus

“DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node” is a paper published on NeurIPS in 2019. The paper introduces a state-of-the-art method to perform index building and search on the billion-scale dataset using a single machine with only 64GB of RAM and a large enough SSD. 

DiskANN: A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale Dataset

Use Milvus, an open-source vector database, to recognize gene families of DNA sequences. Less space but higher accuracy.

> Author:
Mengjia Gu, a data engineer at Zilliz, graduated from McGill University with a Master degree in Information Studies. Her interests include AI applications and similarity search with vector databases. As a community member of open-source project Milvus, she has provided and improved various solutions, like recommendation system and DNA sequence classification model. She enjoys challenges and never gives up!

# Introduction

DNA sequence is a popular concept in both academic research and practical applications, such as gene traceability, species identification, and disease diagnosis. Whereas all industries starve for a more intelligent and efficient research method, artificial intelligence has attracted much attention especially from biological and medical domain. More and more scientists and researchers are contributing to machine learning and deep learning in bioinformatics. To make experimental results more convincing, one common option is increasing sample size. The collaboration with big data in genomics as well brings more possibilities of use case in reality. However, the traditional sequence alignment has limitations, which make it [unsuitable for large data](https://www.frontiersin.org/articles/10.3389/fbioe.2020.01032/full#h5). In order to make less trade-off in reality, vectorization is a good choice for a large dataset of DNA sequences.

The open source vector database [Milvus](https://milvus.io/docs/v2.0.0/overview.md) is friendly for massive data. It is able to store vectors of nucleic acid sequences and perform high-efficiency retrieval. It can also help reduce the cost of production or research. The DNA sequence classification system based on Milvus only takes milliseconds to do gene classification. Moreover, it shows a higher accuracy than other common classifiers in machine learning.

# Data Processing

A gene that encodes genetic information is made up of a small section of DNA sequences, which consists of 4 nucleotide bases [A, C, G, T]. There are about 30,000 genes in human genome, nearly 3 billion DNA base pairs, and each base pair has 2 corresponding bases. To support diverse uses, DNA sequences can be classified into various categories. In order to reduce the cost and make easier use of data of long DNA sequnces, [k-mer ](https://en.wikipedia.org/wiki/K-mer#:~:text=Usually%2C%20the%20term%20k%2Dmer,total%20possible%20k%2Dmers%2C%20where)is introduced to data preprocessing. Meanwhile, it makes DNA sequence data more similar to plain text. Furthermore, vectorized data can speed up calculation in data analysis or machine learning.

![1.png](https://assets.zilliz.com/1_a7469e9eac.png)

**k-mer**

The k-mer method is commonly used in DNA sequence preprocessing. It extracts a small section of length k starting from each base of the original sequence, thereby converting a long sequence of length s to (s-k+1) short sequences of length k. Adjusting the value of k will improve the model performance. Lists of short sequences are easier for data reading, feature extraction, and vectorization.

**Vectorization**

DNA sequences are vectorized in the form of text. A sequence transformed by k-mer becomes a list of short sequences, which looks like a list of individual words in a sentence. Therefore, most natural language processing models should work for DNA sequence data as well. Similar methodologies can be applied to model training, feature extraction, and encoding. Since each model has its own advantages and drawbacks, the selection of models depends on the feature of data and the purpose of research. For example, CountVectorizer, a bag-of-words model, implements feature extraction through straightforward tokenization. It sets no limit on data length, but the result returned is less obvious in terms of similarity comparison.

# Milvus Demo

Milvus can easily manage unstructured data and recall most similar results among trillions of vectors within an average delay of milliseconds. Its similarity search is based on Approximate Nearest Neighbor (ANN) search algorithm. These highlights make Milvus a great option to manage vectors of DNA sequences, hence promote the development and applications of bioinformatics.

Here is a demo showing how to build a DNA sequence classification system with Milvus. The [experimental dataset ](https://www.kaggle.com/nageshsingh/dna-sequence-dataset)includes 3 organisms and 7 gene families. All data are converted to lists of short sequences by k-mers. With a pre-trained CountVectorizer model, the system then encodes sequence data into vectors. The flow chart below depicts the system structure and the processes of inserting and searching.

![1.png](https://assets.zilliz.com/1_ebd89660f6.png)

Try out this demo at [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/dna_sequence_classification).

In Milvus, the system creates collection and inserts corresponding vectors of DNA sequences into the collection (or partition if enabled). When receiving a query request, Milvus will return distances between the vector of input DNA sequence and most similar results in database. The class of input sequence and similarity between DNA sequences can be determined by vector distances in results.

```
# Insert vectors to Milvus collection (partition "human")
DNA_human = collection.insert([human_ids, human_vectors], partition_name='human')
# Search topK results (in partition "human") for test vectors
res = collection.search(test_vectors, "vector_field", search_params, limit=topK, partition_names=['human'])
for results in res:
    res_ids = results.ids # primary keys of topK results
    res_distances = results.distances # distances between topK results & search input
```

**DNA Sequence Classification**
Searching for most similar DNA sequences in Milvus could imply the gene family of an unknown sample, thus learn about its possible functionality.[ If a sequence is classified as GPCRs, then it probably has influence in body functions. ](https://www.nature.com/scitable/topicpage/gpcr-14047471/)In this demo, Milvus has successfully enabled the system to identify the gene families of the human DNA sequences searched with.

![3.png](https://assets.zilliz.com/3_1616da5bb0.png)
![4.png](https://assets.zilliz.com/4_d719b22fc7.png)

**Genetic Similarity**

Average DNA sequence similarity between organisms illustrates how close between their genomes. The demo searches in human data for most similar DNA sequences as that of chimpanzee and dog respectively. Then it calculates and compares average inner product distances (0.97 for chimpanzee and 0.70 for dog), which proves that chimpanzee shares more similar genes with human than dog shares. With more complex data and system design, Milvus is able to support genetic research even on a higher level.

```
search_params = {"metric_type": "IP", "params": {"nprobe": 20}}
```

**Performance**

The demo trains the classification model with 80% human sample data (3629 in total) and uses the remaining as test data. It compares performance of the DNA sequence classification model which uses Milvus with the one powered by Mysql and 5 popular machine learning classifiers. The model based on Milvus outperforms its counterparts in accuracy.

```
from sklearn.model_selection import train_test_split
X, y = human_sequence_kmers, human_labels
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

![1.png](https://assets.zilliz.com/1_6541a7dec6.png)


# Further Exploration

With the development of big data technology, vectorization of DNA sequence will play a more important role in genetic research and practice. Combined with professional knowledge in bioinformatics, related studies can further benefit from the involvement of DNA sequence vectorization. Therefore, Milvus can present better results in practice. According to different scenarios and user needs, Milvus-powered similarity search and distance calculation show great potential and many possibilities.

- **Study unknown sequences**: [According to some researchers, vectorization can compress DNA sequence data.](https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012071/pdf) At the same time, it requires less effort to study structure, function, and evolution of unknown DNA sequences. Milvus can store and retrieve a huge number of DNA sequence vectors without losing accuracy.
- **Adapt devices**: Limited by traditional algorithms of sequence alignment, similarity search can barely benefit from device ([CPU](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7884812/)/[GPU](https://mjeer.journals.ekb.eg/article_146090.html)) improvement. Milvus, which supports both regular CPU computation and GPU acceleration, resolves this problem with approximate nearest neighbor algorithm.
- **Detect virus & trace origins**: [Scientists have compared genome sequences and reported that COVID19 virus of probable bat origin belongs to SARS-COV](https://www.nature.com/articles/s41586-020-2012-7?fbclid=IwAR2hxnXb9nLWgA8xexEoNrCNH8WHqvHhhbN38aSm48AaH6fTzGMB1BLljf4). Based on this conclusion, researchers can expand sample size for more evidence and patterns.
- **Diagnose diseases**: [Clinically, doctors could compare DNA sequences between patients and healthy group to identify variant genes that cause diseases](http://www.xinhuanet.com/science/2018-04/16/c_137114251.htm). It is possible to extract features and encode these data using proper algorithms. Milvus is able to return distances between vectors, which can be related to disease data. In addition to assisting diagnosis of disease, this application can also help to inspire the study of [targeted therapy](https://www.frontiersin.org/articles/10.3389/fgene.2021.680117/full).


# Learn more about Milvus
Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:
- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://milvusio.slack.com/join/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ#/shared-invite/email).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

large_11111.png

medium_11111.png

small_11111.png

thumbnail_11111.png

11111.png

DNA Sequence Classification based on Milvus

The HM-ANN algorithm was recently introduced in a collaborative research paper co-written by authors from Microsoft Research and the University of California. This novel algorithm for graph-based similarity search offers distinct advantages over existing state-of-the-art ANN search solutions.

[HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogenous Memory](https://proceedings.neurips.cc/paper/2020/file/788d986905533aba051261497ecffcbb-Paper.pdf) is a research paper that was accepted at the 2020 Conference on Neural Information Processing Systems ([NeurIPS 2020](https://nips.cc/Conferences/2020)). In this paper, a novel algorithm for graph-based similarity search, called HM-ANN, is proposed. This algorithm considers both memory heterogeneity and data heterogeneity in a modern hardware setting. HM-ANN enables billion-scale similarity search on a single machine without compression technologies. Heterogeneous memory (HM) represents the combination of fast but small dynamic random-access memory (DRAM) and slow but large persistent memory (PMem). HM-ANN achieves low search latency and high search accuracy, especially when the dataset cannot fit into DRAM. The algorithm has a distinct advantage over the state-of-art approximate nearest neighbor (ANN) search solutions.


# Motivation

Since their inception, ANN search algorithms have posed a fundamental tradeoff between query accuracy and query latency due to the limited DRAM capacity. To store indexes in DRAM for fast query access, it is necessary to limit the number of data points or store compressed vectors, both of which hurt search accuracy. Graph-based indexes (e.g. Hierarchical Navigable Small World, HNSW) have superior query runtime performance and query accuracy. However, these indexes can also consume 1-TiB-level DRAM when operating on billion-scale datasets.

There are other workarounds to avoid letting DRAM store billion-scale datasets in raw format. When a dataset is too large to fit into memory on a single machine, compressed approaches such as product quantization of the dataset’s points are used. But the recall of those indexes with the compressed dataset is normally low because of the loss of precision during quantization. Subramanya et al. [1] explore leveraging solid-state drive (SSD) to achieve billion-scale ANN search using a single machine with an approach called Disk-ANN, where the raw dataset is stored on SSD and the compressed representation on DRAM.

# Introduction to Heterogeneous Memory

<img src="https://assets.zilliz.com/image_32_d26cfa9480.png" width="50%" />

Name of the pictureMemory/Storage Hierarchy with HM

Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-presentation_slides.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-presentation_slides.pdf)

Heterogeneous memory (HM) represents the combination of fast but small DRAM and slow but large PMem. DRAM is normal hardware that can be found in every modern server, and its access is relatively fast. New PMem technologies, such as Intel® Optane™ DC Persistent Memory Modules, bridge the gap between NAND-based flash (SSD) and DRAM, eliminating the I/O bottleneck. PMem is durable like SSD, and directly addressable by the CPU, like memory. Renen et al. [2] discover that the PMem read bandwidth is 2.6× lower, and the write bandwidth 7.5× lower, than DRAM in the configured experiment environment.

# HM-ANN Design

HM-ANN is an accurate and fast billion-scale ANN search algorithm that runs on a single machine without compression. The design of HM-ANN generalizes the idea of HNSW, whose hierarchical structure naturally fits into HM. HNSW consists of multiple layers—only layer 0 contains the whole dataset, and each remaining layer contains a subset of elements from the layer directly underneath it.

![2.png](https://assets.zilliz.com/2_25a1836e8b.png)

An example of HNSW with 3 layers

Source: [https://arxiv.org/pdf/1603.09320.pdf](https://arxiv.org/pdf/1603.09320.pdf)

- Elements in the upper layers, which include only subsets of the dataset, consume a small portion of the whole storage. This observation makes them decent candidates to be placed in DRAM. In this way, the majority of searches on HM-ANN are expected to happen in the upper layers, which maximizes the utilization of the fast access characteristic of DRAM. However, in the cases of HNSW, most searches happen in the bottom layer.
- The bottom-most layer carries the whole dataset, which makes it suitable to be placed in PMem. Since accessing layer 0 is slower, it is preferable to have only a small portion accessed by each query and the access frequency reduced.

## Graph Construction Algorithm

![3.png](https://assets.zilliz.com/3_dd9627c753.png)

An example of graph construction of HM-ANN

Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf)

The key idea of HM-ANN’s construction is to create high-quality upper layers, in order to provide better navigation for search at layer 0. Thus most memory access happens in DRAM, and access in PMem is reduced. To make this possible, the construction algorithm of HM-ANN has a top-down insertion phase and a bottom-up promotion phase.

The top-down insertion phase builds a navigable small-world graph as the bottom-most layer is placed on the PMem.

The bottom-up promotion phase promotes pivot points from the bottom layer to form upper layers that are placed on DRAM without losing much accuracy. If a high-quality projection of elements from layer 0 is created in layer 1, search in layer 0 finds the accurate nearest neighbors of the query with only a few hops.

- Instead of using HNSW’s random selection for promotion, HM-ANN uses a high-degree promotion strategy to promote elements with the highest degree in layer 0 into layer 1. For higher layers, HM-ANN promotes high-degree nodes to the upper layer based on a promotion rate.
- HM-ANN promotes more nodes from layer 0 to layer 1 and sets a larger maximum number of neighbors for each element in layer 1. The number of nodes in the upper layers is decided by the available DRAM space. Since layer 0 is not stored in DRAM, making each layer stored in DRAM denser increases the search quality.

## Graph Seach Algorithm

![4.png](https://assets.zilliz.com/4_a5a7f29c93.png)

An example of graph seach of HM-ANN

Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf)

The search algorithm consists of two phases: fast memory search and parallel layer-0 search with prefetching.

### Fast memory search

As the same as in HNSW, the search in DRAM begins at the entry point in the very top layer and then performs 1-greedy search from top to layer 2. To narrow down the search space in layer 0, HM-ANN performs the search in layer 1 with a search budget with ```efSearchL1```, which limits the size of the candidate list in layer 1. Those candidates of the list are used as multiple entry points for search in layer 0, to enhance search quality in layer 0. While HNSW using only one entry point, the gap between layer 0 and layer 1 is more specially handled in HM-ANN than gaps between any other two layers.

### Parallel layer-0 search with prefetching

In the bottom layer, HM-ANN evenly partitions the aforementioned candidates from searching layer 1 and sees them as entry points to perform a parallel multi-start 1-greedy search with threads. The top candidates from each search are collected to find the best candidates. As known, going down from layer 1 to layer 0 is exactly going to PMem. Parallel search hides the latency of PMem and makes the best use of memory bandwidth, to improve search quality without increasing search time.

HM-ANN implements a software-managed buffer in DRAM to prefetch data from PMem before the memory access happens. When searching layer 1, HM-ANN asynchronously copies neighbor elements of those candidates in ```efSearchL1``` and the neighbor elements’ connections in layer 1 from PMem to the buffer. When the search in layer 0 happens, a portion of to-be-accessed data is already prefetched in DRAM, which hides the latency to access PMem and leads to shorter query time. It matches the design goal of HM-ANN, where most memory accesses happen in DRAM and memory accesses in PMem are reduced.

# Evaluation

In this paper, an extensive evaluation is conducted. All experiments are done on a machine with Intel Xeon Gold 6252 CPU@2.3GHz. It uses DDR4 (96GB) as fast memory and Optane DC PMM (1.5TB) as slow memory. Five datasets are evaluated: BIGANN, DEEP1B, SIFT1M, DEEP1M, and GIST1M. For billion-scale tests, the following schemes are included: billion-scale quantization-based methods (IMI+OPQ and L&C), the non-compression-based methods (HNSW and NSG).

## Billion-scale algorithm comparison

![5.png](https://assets.zilliz.com/5_4297db66a9.png)

In table 1, the build time and storage of different graph-based indexes are compared. HNSW takes the shortest build time and HM-ANN needs 8% additional time than HNSW. In terms of whole storage usage, HM-ANN indexes are 5–13% larger than HSNW, because it promotes more nodes from layer 0 to layer 1.

![6.png](https://assets.zilliz.com/6_f363e64d3f.png)

In Figure 1, the query performance of different indexes is analyzed. Figure 1 (a) and (b) show that HM-ANN achieves the top-1 recall of > 95% within 1ms. Figures 1 (c) and (d) show that HM-ANN obtains top-100 recall of > 90% within 4 ms. HM-ANN provides the best latency-vs-recall performance than all other approaches.

## Million-scale algorithm comparison

![7.png](https://assets.zilliz.com/7_a5c23de240.png)

In Figure 2, the query performance of different indexes is analyzed in a pure DRAM setting. HNSW, NSG, and HM-ANN are evaluated with the three million-scale datasets fitting in DRAM. HM-ANN still achieves better query performance than HNSW. The reason is that the total number of distance computations from HM-ANN is lower (on average 850/query) than that of HNSW (on average 900/query) to achieve 99% recall target.

<img src="https://assets.zilliz.com/image_33_f99d31f322.png" width="35%" />


## Effectiveness of high-degree promotion

In Figure 3, the random promotion and high-degree promotion strategies are compared in the same configuration. The high-degree promotion outperforms the baseline. The high-degree promotion performs 1.8x, 4.3x, and 3.9x faster than the random promotion to reach 95%, 99%, and 99.5% recall targets, respectively.

<img src="https://assets.zilliz.com/image_34_3af47e0842.png" width="35%" />


## Performance benefit of memory management techniques

Figure 5 contains a series of steps between HNSW and HM-ANN to show how each optimization of HM-ANN contributes to its improvements. BP stands for the Bottom-up Promotion while building index. PL0 represents for Parallel layer-0 search, while DP for data prefetching from PMem to DRAM. Step by step, HM-ANN’s search performance is pushed further.

# Conclusion

A new graph-based indexing and search algorithm, called HM-ANN, maps the hierarchical design of the graph-based ANNs with memory heterogeneity in HM. Evaluations show that HM-ANN belongs to the new state-of-the-art indexes in billion point datasets.

We notice a trend in academia as well as in industry, where building indexes on persistent storage devices is focused on. To offload the pressure of DRAM, Disk-ANN [1] is an index built on SSD, whose throughput is significantly lower than PMem. However, the building of HM-ANN still takes few days, where no huge differences compared with Disk-ANN is established. We believe it is possible to optimize the build time of HM-ANN, when we utilize PMem’s characteristics more carefully, e.g. to be aware of PMem's granularity (256 Bytes) and to use streaming instruction to bypass cachelines. We also believe there would be more approaches with durable storage devices are proposed in the future.

# Reference

[1]: Suhas Jayaram Subramanya and Devvrit and Rohan Kadekodi and Ravishankar Krishaswamy and Ravishankar Krishaswamy: DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node, NIPS, 2019

[DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node - Microsoft Research](https://www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/)

[DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node](https://papers.nips.cc/paper/2019/hash/09853c7fb1d3f8ee67a61b6bf4a7f8e6-Abstract.html)

[2]: Alexander van Renen and Lukas Vogel and Viktor Leis and Thomas Neumann and Alfons Kemper: Persistent Memory I/O Primitives, CoRR & DaMoN, 2019

[https://dl.acm.org/doi/abs/10.1145/3329785.3329930](https://dl.acm.org/doi/abs/10.1145/3329785.3329930)

[Persistent Memory I/O Primitives](https://arxiv.org/abs/1904.01614)

HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory

Paper Reading｜HM-ANN: When ANNS Meets Heterogeneous Memory

Create a personalized product recommender system using Milvus, an open-source vector database.

With the explosive growth of Internet data scale, the product quantity as well as category in the current mainstream e-commerce platform increase on the one hand, the difficulty for users to find the products they need surges on the other hand.

[Vipshop](https://www.vip.com/) is a leading online discount retailer for brands in China. The Company offers high-quality and popular branded products to consumers throughout China at a significant discount from retail prices. To optimize the shopping experience for their customers, the company decided to build a personalized search recommendation system based on user query keywords and user portraits.

The core function of the e-commerce search recommendation system is to retrieve suitable products from a large number of products and display them to users according to their search intent and preference. In this process, the system needs to calculate the similarity between products and users' search intent & preference, and recommends the TopK products with the highest similarity to users.

Data such as product information, user search intent, and user preferences are all unstructured data. We tried to calculate the similarity of such data using CosineSimilarity(7.x) of the search engine Elasticsearch (ES), but this approach has the following drawbacks.

- Long computational response time - the average latency to retrieve TopK results from millions of items is around 300 ms.

- High maintenance cost of ES indexes - the same set of indexes is used for both commodity feature vectors and other related data, which hardly facilitates the index construction, but produces a massive amount of data.

We tried to develop our own locally sensitive hash plug-in to accelerate the CosineSimilarity calculation of ES. Although the performance and throughput were significantly improved after the acceleration, the latency of 100+ ms was still difficult to meet the actual online product retrieval requirements.

After althorough research, we decided to use Milvus, an open source vector database, which is advantaged with the support for distributed deployment, multi-language SDKs, read/write separation, etc. compared to the commonly used standalone Faiss.

Using various deep learning models, we convert massive unstructured data into feature vectors, and import the vectors into Milvus. With the excellent performance of Milvus, our e-commerce search recommendation system can efficiently query the TopK vectors that are similar to the target vectors.

## Overall Architecture
![Architecture](https://assets.zilliz.com/1_01551e7b2b.jpg)
As shown in the diagram, the system overall architecture consists of two main parts.

- Write process: the item feature vectors (hereinafter referred to as item vectors) generated by the deep learning model are normalized and written into MySQL. MySQL then reads the processed item feature vectors using the data synchronization tool (ETL) and import them into the vector database Milvus.

- Read process: The search service obtains user preference feature vectors (hereinafter referred to as user vectors) based on user query keywords and user portraits, queries similar vectors in Milvus and recalls TopK item vectors.

Milvus supports both incremental data update and entire data update. Each incremental update has to delete the existing item vector and insert a new item vector, meaning that every newly updated collection will be re-indexed. It better suits the scenario with more reads and fewer writes. Therefore, we choose the entire data update method. Moreover, it takes only a few minutes to write the entire data in batches of multiple partitions, which is equivalent to near real-time updates.

Milvus write nodes perform all write operations, including creating data collections, building indexes, inserting vectors, etc., and provide services to the public with write domain names. Milvus read nodes perform all read operations and provide services to the public with read-only domain names.

Whereas the current version of Milvus does not support switching collection aliases, we introduce Redis to seamlessly switch aliases between multiple entire data collections.

The read node only needs to read existing metadata information and vector data or indexes from MySQL, Milvus, and GlusterFS distributed file system, so the read capability can be horizontally extended by deploying multiple instances.

## Implementation Details

### Data Update

The data update service includes not only writing vector data, but also data volume detection of vectors, index construction, index pre-loading, alias control, etc. The overall process is as follows.
![Process](https://assets.zilliz.com/2_6052b01334.jpg)
1. Assume that before building the entire data, CollectionA provides data service to the public, and the entire data being used is directed to CollectionA (`redis key1 = CollectionA`). The purpose of constructing entire data is to create a new collection CollectionB.

2. Commodity data check - check the item number of commodity data in the MySQL table, compare the commodity data with the existing data in CollectionA. Alert can be set in accordance with quantity or percentage. If the set quantity (percentage) is not reached, the entire data will not be built, and it will be regarded as the failure of this building operation, triggering the alert; once the set quantity (percentage) is reached, the entire data building process starts.

3. Start building the entire data - initialize the alias of the entire data being built, and update Redis. After updating, the alias of the entire data being built is directed to CollectionB (`redis key2 = CollectionB`).

4. Create a new entire collection - determine if CollectionB exists. If it does, delete it before creating a new one.

5. Data batch write-in - calculate the partition ID of each commodity data with its own ID using modulo operation, and write the data to multiple partitions to the newly created collection in batches.

6. Build and pre-load index - Create index (`createIndex()`) for the new collection. The index file is stored in distributed storage server GlusterFS. The system automatically simulates query on the new collection and pre-load the index for query warm-up. 

7. Collection data check - check the item number of data in the new collection, compare the data with the existing collection, and set alarms based on the quantity and percentage. If the set number (percentage) is not reached, the collection will not be switched and the building process will be regarded as a failure, triggering the alert.

8. Switching collection - Alias control. After updating Redis, the entire data alias being used is directed to CollectionB (`redis key1 = CollectionB`), the original Redis key2 is deleted, and the building process is completed.

### Data Recall

The Milvus partition data is called several times to calculate the similarity between user vectors, obtained based on user query keywords and user portrait, and item vector, and the TopK item vectors are returned after merging. The overall workflow schematic is as follow:
![workflow](https://assets.zilliz.com/3_93518602b1.jpg)
The following table lists the main services involved in this process. It can be seen that the average latency for recalling TopK vectors is about 30 ms.

| **Service**              | **Role**                                                | **Input Parameters**                                         | **Output parameters** | **Response latency** |
| ------------------------ | ------------------------------------------------------- | ------------------------------------------------------------ | --------------------- | -------------------- |
| User vectors acquisition | Obtain user vector                                      | user info + query                                            | user vector           | 10 ms                |
| Milvus Search            | Calculate the vector similarity and return TopK results | user vector                                                  | item vector           | 10 ms                |
| Scheduling Logic         | Concurrent result recalling and merging                 | Multi-channel recalled item vectors and the similarity score | TopK items            | 10 ms                |

**Implementation process:**

1. Based on the user query keywords and user portrait, the user vector is calculated by the deep learning model.
2. Obtain the collection alias of the entire data being used from Redis currentInUseKeyRef and get Milvus CollectionName. This process is data synchronization service, i.e. switching alias to Redis after entire data update.
3. Milvus is called concurrently and asynchronously with the user vector to obtain data from different partitions of the same collection, and Milvus calculates the similarity between the user vector and the item vector, and returns the TopK similar item vectors in each partition.
4. Merge the TopK item vectors returned from each partition, and rank the results in the reverse order of similarity distance, which are calculated using the IP inner product (the greater the distance between the vectors, the more similar they are). The final TopK item vectors are returned.

## Looking Ahead

At present, Milvus-based vector search can be used steadily in the search of recommendation scenarios, and its high performance gives us more room to play in the dimensionality of the model and algorithm selection.

Milvus will play a crucial role as the middleware for more scenarios, including recall of main site search and all-scenario recommendations.

The three most anticipated features of Milvus in the future are as follows.

- Logic for collection alias switching - coordinate the switching across collections without external conponents.
- Filtering mechanism - Milvus v0.11.0 only supports ES DSL filtering mechanism in standalone version. The newly released Milvus 2.0 supports scalar filtering, and read/write separation.
- Storage support for Hadoop Distributed File System (HDFS) - The Milvus v0.10.6 we are using only supports POSIX file interface, and we have deployed GlusterFS with FUSE support as the storage backend. However, HDFS is a better choice in terms of performance and ease of scaling.

## Lessons Learned and Best Practices

1. For applications where read operations are the primary focus, a read-write separation deployment can significantly increase the processing power and improve performance.
2. The Milvus Java client lacks a reconnection mechanism because the Milvus client used by the recall service is resident in memory. We have to build our own connection pool to ensure the availability of the connection between the Java client and the server through heartbeat test.
3. Slow queries occur occasionally on Milvus. This is due to insufficient warm-up of the new collection. By simulating the query on the new collection, the index file is loaded into the memory to achieve the  index warm-up.
4. nlist is the index building parameter and nprobe is the query parameter. You need to get a reasonable threshold value according to your business scenario through pressure testing experiments to balance the retrieval performance and accuracy.
5. For static data scenario, it is more efficient to import all data into the collection first and build indexes later. 

large_blog_shopping.jpg

medium_blog_shopping.jpg

small_blog_shopping.jpg

thumbnail_blog_shopping.jpg

blog_shopping.jpg

Milvus makes it easy to provide the personalized recommendation service to users.

Building a Personalized Product Recommender System with Vipshop and Milvus

Create an audio retrieval system using Milvus, an open-source vector database. Classify and analyze sound data in real time.

Sound is an information dense data type. Although it may feel antiquated in the era of video content, audio remains a primary information source for many people. Despite long-term decline in listeners, 83% of Americans ages 12 or older listened to terrestrial (AM/FM) radio in a given week in 2020 (down from 89% in 2019). Conversely, online audio has seen a steady rise in listeners over the past two decades, with 62% of Americans reportedly listening to some form of it on a weekly basis according to the same [Pew Research Center study](https://www.journalism.org/fact-sheet/audio-and-podcasting/).

As a wave, sound includes four properties: frequency, amplitude, wave form, and duration. In musical terminology, these are called pitch, dynamic, tone, and duration. Sounds also help humans and other animals perceive and understand our environment, providing context clues for the location and movement of objects in our surroundings.

As an information carrier, audio can be classified into three categories:

1. **Speech:** A communication medium composed of words and grammar. With speech recognition algorithms, speech can be converted to text.
2. **Music:** Vocal and/or instrumental sounds combined to produce a composition comprised of melody, harmony, rhythm, and timbre. Music can be represented by a score.
3. **Waveform:** A digital audio signal obtained by digitizing analog sounds. Waveforms can represent speech, music, and natural or synthesized sounds.

Audio retrieval can be used to search and monitor online media in real-time to crack down on infringement of intellectual property rights. It also assumes an important role in the classification and statistical analysis of audio data.

## Processing Technologies

Speech, music, and other generic sounds each have unique characteristics and demand different processing methods. Typically, audio is separated into groups that contain speech and groups that do not:

- Speech audio is processed by automatic speech recognition.
- Non-speech audio, including musical audio, sound effects, and digitized speech signals, are processed using audio retrieval systems. 

This article focuses on how to use an audio retrieval system to process non-speech audio data. Speech recognition is not covered in this article

### Audio feature extraction

Feature extraction is the most important technology in audio retrieval systems as it enables audio similarity search. Methods for extracting audio features are divided into two categories: 

- Traditional audio feature extraction models such as Gaussian mixture models (GMMs) and hidden Markov models (HMMs);
- Deep learning-based audio feature extraction models such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, encoding-decoding frameworks, attention mechanisms, etc.

Deep learning-based models have an error rate that is an order of magnitude lower than traditional models, and therefore are gaining momentum as core technology in the field of audio signal processing.

Audio data is usually represented by the extracted audio features. The retrieval process searches and compares these features and attributes rather than the audio data itself. Therefore, the effectiveness of audio similarity retrieval largely depends on the feature extraction quality.

In this article, [large-scale pre-trained audio neural networks for audio pattern recognition (PANNs)](https://github.com/qiuqiangkong/audioset_tagging_cnn) are used to extract feature vectors for its mean average accuracy (mAP) of 0.439 (Hershey et al., 2017).

After extracting the feature vectors of the audio data, we can implement high-performance feature vector analysis using Milvus.

### Vector similarity search

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following diagram depicts the general similarity search process using Milvus: 
   
![how-does-milvus-work.png](https://assets.zilliz.com/how_does_milvus_work_6926180543.png)
###### *Milvus' vector similarity search process.*

1. Unstructured data are converted to feature vectors by deep learning models and inserted into Milvus.
2. Milvus stores and indexes these feature vectors.
3. Upon request, Milvus searches and returns vectors most similar to the query vector.

## System overview

The audio retrieval system mainly consists of two parts: insert (black line) and search(red line).

![audio-retrieval-system.png](https://assets.zilliz.com/audio_retrieval_system_663a911c95.png)
###### *Audio retrieval system powered by Milvus.*

The sample dataset used in this project contains open-source game sounds, and the code is detailed in the [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search).

### Step 1: Insert data

Below is the example code for generating audio embeddings with the pre-trained PANNs-inference model and inserting them into Milvus, which assigns a unique ID to each vector embedding.

```
1 wav_name, vectors_audio = get_audio_embedding(audio_path)  
2 if vectors_audio:   
3     embeddings.append(vectors_audio)  
4     wav_names.append(wav_name)  
5 ids_milvus = insert_vectors(milvus_client, table_name, embeddings)  
6 
```

Then the returned **ids_milvus** are stored along with other relevant information (e.g. **wav_name**) for the audio data held in a MySQL database for subsequent processing. 

```
1 get_ids_correlation(ids_milvus, wav_name)  
2 load_data_to_mysql(conn, cursor, table_name)   
3 
```

### Step 2: Audio search

Milvus calculates the inner product distance between the pre-stored feature vectors and the input feature vectors, extracted from the query audio data using the PANNs-inference model, and returns the **ids_milvus** of similar feature vectors, which correspond to the audio data searched.

```
1 _, vectors_audio = get_audio_embedding(audio_filename)   
2 results = search_vectors(milvus_client, table_name, [vectors_audio], METRIC_TYPE, TOP_K)  
3 ids_milvus = [x.id for x in results[0]]  
4 audio_name = search_by_milvus_ids(conn, cursor, ids_milvus, table_name)   
5
````

## API reference and demo

### API

This audio retrieval system is built with open-source code. Its main features are audio data insertion and deletion. All APIs can be viewed by typing **127.0.0.1:<port>/docs** in the browser. 





### Demo

We host a [live demo](https://zilliz.com/solutions ) of the Milvus-based audio retrieval system online that you can try out with your own audio data. 

![audio-search-demo.png](https://assets.zilliz.com/audio_search_demo_cae60625db.png)
###### *Audio search demo powered by Milvus.*

## Conclusion

Living in the era of big data, people find their lives abound with all sorts of information. To make better sense of it, traditional text retrieval no long cuts it. Today's information retrieval technology is in urgent need of the retrieval of various unstructured data types, such as videos, images, and audio.

Unstructured data, which is difficult for computers to process, can be converted into feature vectors using deep learning models. This converted data can easily be processed by machines, enabling us to analyze unstructured data in ways our predecessors were never able to. Milvus, an open-source vector database, can efficiently process the feature vectors extracted by AI models and provides a variety of common vector similarity calculations.

## References
Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B. and Slaney, M., 2017, March. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131-135, 2017 

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).

- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).

- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_audio search.jpg

medium_blog_audio search.jpg

small_blog_audio search.jpg

thumbnail_blog_audio search.jpg

blog_audio search.jpg

Audio retrieval with Milvus makes it possible to classify and analyze sound data in real time.

Audio Retrieval Based on Milvus

MilMil helps Milvus users solve problems faster, and was built by and for the Milvus open-source community using Milvus—an advanced vector database built for AI and similarity search. 

Open-source communities are creative and collaborative spaces. In that vein, the Milvus open-source community recently created MilMil—a Milvus FAQ chatbot built by and for Milvus users. MilMil is available 24/7 at [Milvus.io](https://milvus.io/) to answer common questions about Milvus, the world's most advanced open-source vector database. 

This question answering system not only helps solve common problems Milvus users encounter more quickly, but identifies new problems based on user submissions. MilMil's database includes questions users have asked since the project was first released under an open-source license in 2019. Questions are stored in two collections, one for Milvus 1.x and earlier and another for Milvus 2.0.

MilMil is currently only available in English. 

## How does MilMil work?

MilMil relies on the *sentence-transformers/paraphrase-mpnet-base-v2* model to obtain vector representations of the FAQ database, then Milvus is used for vector similarity retrieval to return semantically similar questions. 

First, the FAQ data is converted into semantic vectors using BERT, a natural language processing (NLP) model. The embeddings are then inserted into Milvus and each one assigned a unique ID. Finally, the questions and answers are inserted into PostgreSQL, a relational database, together with their vector IDs.

When users submit a question, the system converts it into a feature vector using BERT. Next it searches Milvus for five vectors that are most similar to the query vector and retrieves their IDs. Finally, the questions and answers that correspond with the retrieved vector IDs are returned to the user.

![system-process.png](https://assets.zilliz.com/system_process_dca67a80a6.png)
###### *The blue line represents the data insertion process, and the black line represents the query process.*

See the [question answering system](https://github.com/milvus-io/bootcamp/tree/master/solutions/question_answering_system) project in the Milvus bootcamp to explore the code used to build AI chatbots.

## Ask MilMil about Milvus

To chat with MilMil, navigate to any page on [Milvus.io](https://milvus.io/) and click the bird icon in the lower-right corner. Type your question into the text input box and hit send. MilMil will get back to you in milliseconds! Additionally, the dropdown list in the upper-left corner can be used to switch between technical documentation for different versions of Milvus. 

![milvus-chatbot-icon.png](https://assets.zilliz.com/milvus_chatbot_icon_f3c25708ca.png)
###### *Click on the Milvus icon in the lower right corner to use Milvus FAQ Chatbot.*

After submitting a question, the bot immediately returns three questions that are semantically similar to the query question. You can click "See answer" to browse potential answers to your question, or click "See more" to view more questions related to your search. If a suitable answer is unavailable,  click "Put in your feedback here" to ask your question along with an email address. Help from the Milvus community will arrive shortly!

![chatbot_UI.png](https://assets.zilliz.com/chatbot_UI_0f4a7655d4.png)
###### *MilMil's search results of "The way to install Milvus."*

Give MilMil a try and let us know what you think. All questions, comments, or any form of feedback are welcome.

## Don't be a stranger
- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_milmil.jpg

medium_milmil.jpg

small_milmil.jpg

thumbnail_milmil.jpg

milmil.jpg

Using open-source vector search tools to build a question answering service. 

MilMil: A Milvus-powered FAQ Chatbot that Answers Questions About Milvus

A new and improved Milvus bootcamp offers updated guides and easier to follow code examples. Discover how to create your own similarity search solutions with the world's most advanced vector database.

With the release of Milvus 2.0, the team has revamped the Milvus [bootcamp](https://github.com/milvus-io/bootcamp). The new and improved bootcamp offers updated guides and easier to follow code examples for a variety of use cases and deployments. Additionally, this new version is updated for [Milvus 2.0](https://zilliz.com/blog/milvus2.0-redefining-vector-database), a reimagined version of the world's most advanced vector databse.
 

Jump to:
- [Stress test your system against 1M and 100M dataset benchmarks](#stress-test-your-system-against-1m-and-100m-dataset-benchmarks)
- [Explore and build popular vector similarity search solutions](#explore-and-build-popular-vector-similarity-search-solutions)
- [Quickly deploy a fully built application on any system](#quickly-deploy-a-fully-built-application-on-any-system)
- [Use scenario specific notebooks to easily deploy pre-configured applications](#use-scenario-specific-notebooks-to-easily-deploy-pre-configured-applications)
- [Learn how to deploy Milvus in different environments](#learn-how-to-deploy-milvus-in-different-environments)

 

### Stress test your system against 1M and 100M dataset benchmarks 
The [benchmark directory](https://github.com/milvus-io/bootcamp/tree/master/benchmark_test) contains 1 million and 100 million vector benchmark tests that indicate how your system will react to differently sized datasets.

 

### Explore and build popular vector similarity search solutions

The [solution directory](https://github.com/milvus-io/bootcamp/tree/master/solutions) includes the most popular vector similarity search use cases. Each use case contains a notebook solution and a docker deployable solution. Use cases include:

- [Image similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search)
- [Video similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/video_similarity_search)
- [Audio similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search)
- [Recommendation system](https://github.com/milvus-io/bootcamp/tree/master/solutions/recommendation_system)
- [Molecular search](https://github.com/milvus-io/bootcamp/tree/master/solutions/molecular_similarity_search)
- [Question answering system](https://github.com/milvus-io/bootcamp/tree/master/solutions/question_answering_system)

 

### Quickly deploy a fully built application on any system

The quick deploy solutions are dockerized solutions that allow users to deploy fully built applications on any system. These solutions are ideal for brief demos, but require additional work to customize and understand compared to notebooks.

 

### Use scenario specific notebooks to easily deploy pre-configured applications

The notebooks contain a simple example of deploying Milvus to solve the problem in a given use case. Each of the examples are able to be run from start to finish without the need to manage files or configurations. Each notebook is also easy to follow and modifiable, making them ideal base files for other projects.

 

### Image similarity search notebook example
Image similarity search is one of the core ideas behind many different technologies, including autonomous cars recognizing objects. This example explains how to easily build computer vision programs with Milvus.


This [notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/reverse_image_search/reverse_image_search.ipynb) revolves around three things:
- Milvus server 
- Redis server (for metadata storage)
- Pretrained Resnet-18 model. 


#### Step 1: Download required packages
Begin by downloading all the required packages for this project. This notebook includes a table listing the packages to use.

```
pip install -r requirements.txt
```

#### Step 2: Server startup
After the packages are installed, start the servers and ensure both are running properly. Be sure to follow the correct instructions for starting the [Milvus](https://milvus.io/docs/v2.0.0/install_standalone-docker.md) and [Redis](https://hub.docker.com/_/redis) servers.

#### Step 3: Download project data
By default, this notebook pulls a snippet of the VOCImage data for use as an example, but any directory with images should work as long as it follows the file structure that can be seen at the top of the notebook.

```
! gdown "https://drive.google.com/u/1/uc?id=1jdudBiUu41kL-U5lhH3ari_WBRXyedWo&export=download"
! tar -xf 'VOCdevkit.zip'
! rm 'VOCdevkit.zip'
```

#### Step 4: Connect to the servers
In this example, the servers are running on the default ports on the localhost.

```
connections.connect(host="127.0.0.1", port=19537)
red = redis.Redis(host = '127.0.0.1', port=6379, db=0)
```

#### Step 5: Create a collection
After starting the servers, create a collection in Milvus for storing all the vectors. In this example, the dimension size is set to 512, the size of the resnet-18 output, and the similarity metric is set to the Euclidean distance (L2). Milvus supports a variety of different [similarity metrics](https://milvus.io/docs/metric.md#floating).

```
collection_name = "image_similarity_search"
dim = 512
default_fields = [
 schema.FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
 schema.FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
]
default_schema = schema.CollectionSchema(fields=default_fields, description="Image test collection")
collection = Collection(name=collection_name, schema=default_schema)
```

#### Step 6: Build an index for the collection
Once the collection is made, build an index for it. In this case, the IVF_SQ8 index is used. This index requires the 'nlist' parameter, which tells Milvus how many clusters to make within each datafile (segment). Different [indices](https://milvus.io/docs/index.md#CPU) require different parameters.

```
default_index = {"index_type": "IVF_SQ8", "params": {"nlist": 2048}, "metric_type": "L2"}
collection.create_index(field_name="vector", index_params=default_index)
collection.load()
```
#### Step 7: Set up model and data loader
After the IVF_SQ8 index is built, set up the neural network and data loader. The pretrained pytorch resnet-18 used in this example is sans its last layer, which compresses vectors for classification and may lose valuable information.

```
model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet18', pretrained=True)
encoder = torch.nn.Sequential(*(list(model.children())[:-1]))
```

The dataset and data loader needs to be modified so that they are able to preprocess and batch the images while also providing the file paths of the images. This can be done with a slightly modified torchvision dataloader. For preprocessing, the images need to be cropped and normalized due to the resnet-18 model being trained on a specific size and value range. 

```
dataset = ImageFolderWithPaths(data_dir, transform=transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]))

dataloader = torch.utils.data.DataLoader(dataset, num_workers=0, batch_si
```

#### Step 8: Insert vectors into the collection
With the collection setup, the images can be processed and loaded into the created collection. First the images are pulled by the dataloader and run through the resnet-18 model. The resulting vector embeddings are then inserted into Milvus, which returns a unique ID for each vector. The vector IDs and image file paths are then inserted as key-value pairs into the Redis server.

```
steps = len(dataloader)
step = 0
for inputs, labels, paths in dataloader:
 with torch.no_grad():
 output = encoder(inputs).squeeze()
 output = output.numpy()

 mr = collection.insert([output.tolist()])
 ids = mr.primary_keys
 for x in range(len(ids)):
 red.set(str(ids[x]), paths[x])
 if step%5 == 0:
 print("Insert Step: " + str(step) + "/" + str(steps))
 step += 1
```

#### Step 9: Conduct a vector similarity search
Once all of the data is inserted into Milvus and Redis, the actual vector similarity search can be performed. For this example, three randomly selected images are pulled out of the Redis server for a vector similarity search. 

```
random_ids = [int(red.randomkey()) for x in range(3)]
search_images = [x.decode("utf-8") for x in red.mget(random_ids)]
```
These images first go through the same preprocessing that is found in Step 7 and are then pushed through the resnet-18 model. 

```
transform_ops = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
 
embeddings = [transform_ops(Image.open(x)) for x in search_images]
embeddings = torch.stack(embeddings, dim=0)
 
with torch.no_grad():
 embeddings = encoder(embeddings).squeeze().numpy()
```
Then the resulting vector embeddings are used to perform a search. First, set the search parameters, including the name of the collection to search, nprobe (the number of the clusters to search), and top_k (the number of returned vectors). In this example, the search should be very quick. 

```
search_params = {"metric_type": "L2", "params": {"nprobe": 32}}
start = time.time()
results = collection.search(embeddings, "vector", param=search_params, limit=3, expr=None)
end = time.time() - start
```

#### Step 10: Image search results 
The vector IDs returned from the queries are used to find the corresponding images. Matplotlib is then used to display the image search results.
 

![pic1.png](https://assets.zilliz.com/pic1_c8652c7fae.png)
![pic2.png](https://assets.zilliz.com/pic2_355b054161.png)
![pic3.png](https://assets.zilliz.com/pic3_01780c6aac.png)

 

### Learn how to deploy Milvus in different enviroments
The [deployments section](https://github.com/milvus-io/bootcamp/tree/master/deployments) of the new bootcamp contains all the information for using Milvus in different environments and setups. It includes deploying Mishards, using Kubernetes with Milvus, load balancing, and more. Each environment has a detailed step by step guide explaining how to get Milvus working in it.

 

### Don't be a stranger
- Read our our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to Milvus, the world’s most popular vector database, on [Github](http://page/).

thumbnail_cover.png

cover.png

Build, test, and customize vector similarity search solutions with Milvus, an open-source vector database.

Quickly Test and Deploy Vector Search Solutions with the Milvus 2.0 Bootcamp

Mozat is a Singapore-based startup that specializes in social media, lifestyle, and communication applications. Learn how the company uses Milvus to create an AI-powered image search system for its fashion app, Stylepedia.

![stylepedia-1.png](https://assets.zilliz.com/stylepedia_1_5f239a8d48.png)

Founded in 2003, [Mozat](http://www.mozat.com/home) is a start-up headquartered in Singapore with offices in China and Saudi Arabia. The company specializes in building social media, communication, and lifestyle applications. [Stylepedia](https://stylepedia.com/) is a wardrobe app built by Mozat that helps users discover new styles and connect with other people that are passionate about fashion. Its key features include the ability to curate a digital closet, personalized style recommendations, social media functionality, and an image search tool for finding similar items to something seen online or in real life.

[Milvus](https://milvus.io) is used to power the image search system within Stylepedia. The app deals with three image types: user images, product images, and fashion photographs. Each image can include one or more items, further complicating each query. To be useful, an image search system must be accurate, fast, and stable, features that lay a solid technical foundation for adding new functionality to the app such as outfit suggestions and fashion content recommendations.

## System overview

![stylepedia-system-process.png](https://assets.zilliz.com/stylepedia_system_process_8e7e2ab3e4.png)
###### *System process diagram.*

The image search system is divided into offline and online components. 

Offline, images are vectorized and inserted into a vector database (Milvus). In the data workflow, relevant product images and fashion photographs are converted into 512-dimensional feature vectors using object detection and feature extraction models. The vector data is then indexed and added to the vector database.

Online, the image database is queried and similar images are returned to the user. Similar to the off-line component, a query image is processed by object detection and feature extraction models to obtain a feature vector. Using the feature vector, Milvus searches for TopK similar vectors and obtains their corresponding image IDs. Finally, after post-processing (filtering, sorting, etc.), a collection of images similar to the query image are returned.

## Implementation

The implementation breaks down into four modules:
1. Garment detection
2. Feature extraction
3. Vector similarity search
4. Post-processing

### Garment detection

In the garment detection module, [YOLOv5](https://pytorch.org/hub/ultralytics_yolov5/), a one-stage, anchor-based target detection framework, is used as the object detection model for its small size and real-time inference. It offers four model sizes (YOLOv5s/m/l/x), and each specific size has pros and cons. The larger models will perform better (higher precision) but require a lot more computing power and run slower. Because the objects in this case are relatively large items and easy to detect, the smallest model, YOLOv5s, suffices.

Clothing items in each image are recognized and cropped out to serve as the feature extraction model inputs used in subsequent processing. Simultaneously, the object detection model also predicts the garment classification according to predefined classes (tops, outerwear, trousers, skirts, dresses, and rompers).

### Feature extraction

The key to similarity search is the feature extraction model. Cropped clothes images are embedded into 512-dimensional floating point vectors that represent their attributes in a machine readable numeric data format. The [deep metric learning (DML)](https://github.com/kdhht2334/Survey_of_Deep_Metric_Learning) methodology is adopted with [EfficientNet](https://arxiv.org/abs/1905.11946) as the backbone model.

Metric learning aims to train a CNN-based nonlinear feature extraction module (or an encoder) to reduce the distance between the feature vectors corresponding to the same class of samples, and increase the distance between the feature vectors corresponding to different classes of samples. In this scenario, the same class of samples refers to the same piece of clothing.

EfficientNet takes into account both speed and precision when uniformly scaling network width, depth, and resolution. EfficientNet-B4 is used as the feature extraction network, and the output of the ultimate fully connected layer is the image features needed to conduct vector similarity search.

### Vector similarity search

Milvus is an open-source vector database that supports create, read, update, and delete (CRUD) operations as well as near real-time search on trillion-byte datasets. In Stylepedia, it is used for large-scale vector similarity search because it is highly elastic, stable, reliable, and lightening fast. Milvus extends the capabilities of widely used vector index libraries (Faiss, NMSLIB, Annoy, etc.), and provides a set of simple and intuitive APIs that allow users to select the ideal index type for a given scenario.

Given the scenario requirements and data scale, Stylepedia's developers used the CPU-only distribution of Milvus paired with the HNSW index. Two indexed collections, one for products and the other for fashion photographs, are built to power different application functionalities. Each collection is further divided into six partitions based on the detection and classification results to narrow the search scope. Milvus performs search on tens of millions of vectors in milliseconds, providing optimal performance while keeping development costs low and minimizing resource consumption.

### Post-processing

To improve the similarity between the image retrieval results and the query image, we use color filtering and key label (sleeve length, clothes length, collar style, etc.) filtering to filter out ineligible images. In addition, an image quality assessment algorithm is used to make sure that higher quality images are presented to users first.

## Application

### User uploads and image search

Users can take pictures of their own clothes and upload them to their Stylepedia digital closet, then retrieve product images most similar to their uploads.

![stylepedia-search-results.png](https://assets.zilliz.com/stylepedia_search_results_0568e20dc0.png)
###### *Search results of a denim jacket image.*

### Outfit suggestions

By conducting similarity search on the Stylepedia database, users can find fashion photographs that contain a specific fashion item. These could be new garments someone is thinking about purchasing, or something from their own collection that could be worn or paired differently. Then, through the clustering of the items it is often paired with, outfit suggestions are generated. For example, a black biker jacket can go with a variety of items, such as a pair of black skinny jeans. Users can then browse relevant fashion photographs where this match occurs in the selected formula.

![stylepedia-jacket-outfit.png](https://assets.zilliz.com/stylepedia_jacket_outfit_e84914da9e.png)
###### *Outfit ideas for a black biker jacket.*
![stylepedia-jacket-snapshot.png](https://assets.zilliz.com/stylepedia_jacket_snapshot_25f53cc09b.png)
###### *A spread of snapshots featuring a black biker jacket + black skinny jeans match.*

### Fashion photograph recommendations

Based on a user's browsing history, likes, and the contents of their digital closet, the system calculates similarity and provides customized fashion photograph recommendations that may be of interest.

![stylepedia-user-wardrobe.png](https://assets.zilliz.com/stylepedia_user_wardrobe_6770c856b9.png)
###### *Items in user's digital closet.*
![stylepedia-streetsnap-rec.png](https://assets.zilliz.com/stylepedia_streetsnap_rec_901601a34d.png)
###### *Fashion photograph recommendations that match user preferences and tastes.*

By combining deep learning and computer vision methodologies, Mozat was able to build a fast, stable, and accurate image similarity search system using Milvus to power various features in the Stylepedia app.

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).


large_mozat-blog.jpg

medium_mozat-blog.jpg

small_mozat-blog.jpg

thumbnail_mozat-blog.jpg

mozat-blog.jpg

Discover how Milvus, an open-source vector database, is used by Mozat to power a fashion app that offers personalized style recommendations and an image search system.

Building a Wardrobe and Outfit Planning App with Milvus

Learn how to build a Milvus cluster based on JuiceFS, a shared file system designed for cloud-native environments.

Collaborations between open-source communities are a magical thing. Not only do passionate, intelligent, and creative volunteers keep open-source solutions innovative, they also work to bring different tools together in interesting and useful ways. [Milvus](https://milvus.io/), the world's most popular vector database, and [JuiceFS](https://github.com/juicedata/juicefs), a shared file system designed for cloud-native environments, were united in this spirit by their respective open-source communities. This article explains what JuiceFS is, how to build a Milvus cluster based on JuiceFS shared file storage, and the performance users can expect using this solution.

## **What is JuiceFS?**

JuiceFS is a high-performance, open-source distributed POSIX file system, which can be built on top of Redis and S3. It was designed for cloud-native environments and supports managing, analyzing, archiving, and backing up data of any type. JuiceFS is commonly used for solving big data challenges, building artificial intelligence (AI) applications, and log collection. The system also supports data sharing across multiple clients and can be used directly as shared storage in Milvus.

After data, and its corresponding metadata, are persisted to object storage and [Redis](https://redis.io/) respectively, JuiceFS serves as a stateless middleware. Data sharing is realized by enabling different applications to dock with each other seamlessly through a standard file system interface. JuiceFS relies on Redis, an open-source in-memory data store, for metadata storage. Redis is used because it guarantees atomicity and provides high performance metadata operations. All data is stored in object storage through the JuiceFS client. The architecture diagram is as follows:

![juicefs-architecture.png](https://assets.zilliz.com/juicefs_architecture_2023b37a4e.png)
###### *Overall architecture of JuiceFS.*

## **Build a Milvus cluster based on JuiceFS**

A Milvus cluster built with JuiceFS (see architecture diagram below) works by splitting upstream requests using Mishards, a cluster sharding middleware, to cascade the requests down to its sub-modules. When inserting data, Mishards allocates upstream requests to the Milvus write node, which stores newly inserted data in JuiceFS. When reading data, Mishards loads the data from JuiceFS through a Milvus read node to memory for processing, then collects and returns results from sub-services upstream.

![milvus-cluster-built-with-juicefs.png](https://assets.zilliz.com/milvus_cluster_built_with_juicefs_3a43cd262c.png)
###### *Architecture of Milvus cluster built with JuiceFS.*

### **Step 1: Launch MySQL service**

Launch the MySQL service on **any** node in the cluster. For details, see [Manage Metadata with MySQL](https://milvus.io/docs/v1.1.0/data_manage.md).

### **Step 2: Create a JuiceFS file system**

For demonstration purposes, the pre-compiled binary JuiceFS program is used. Download the correct [installation package](https://github.com/juicedata/juicefs/releases) for your system and follow the JuiceFS [Quick Start Guide](https://github.com/juicedata/juicefs/blob/main/docs/en/quick_start_guide.md) for detailed install instructions. To create a JuiceFS file system, first set up a Redis database for metadata storage. It is recommended that for public cloud deployments you host the Redis service on the same cloud as the application. Additionally, set up object storage for JuiceFS. In this example, Azure Blob Storage is used; however, JuiceFS supports almost all object services. Select the object storage service that best suits the demands of your scenario.

After configuring the Redis service and object storage, format a new file system and mount JuiceFS to the local directory:

```
1 $ export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=XXX;AccountKey=XXX;EndpointSuffix=core.windows.net"
2 $ ./juicefs format \
3 --storage wasb \
4 --bucket https://<container> \
5 ... \
6 localhost test #format
7 $ ./juicefs mount -d localhost ~/jfs #mount
8
```

> If the Redis server is not running locally, replace the localhost with the following address: `redis://<user:password>@host:6379/1`.

When the installation succeeds, JuiceFS returns the shared storage page **/root/jfs**.

![installation-success.png](https://assets.zilliz.com/installation_success_9d05279ecd.png)

### **Step 3: Start Milvus**

All the nodes in the cluster should have Milvus installed, and each Milvus node should be configured with read or write permission. Only one Milvus node can be configured as write node, and the rest must be read nodes. First, set the parameters of sections `cluster` and `general` in the Milvus system configuration file **server_config.yaml**:

**Section** `cluster` 

| **Parameter** | **Description** | **Configuration** |
| :------------ | :----------------------------- | :---------------- |
| `enable` | Whether to enable cluster mode | `true` |
| `role` | Milvus deployment role | `rw`/`ro` |

**Section** `general` 

```
# meta_uri is the URI for metadata storage, using MySQL (for Milvus Cluster). Format: mysql://<username:password>@host:port/database
general:
 timezone: UTC+8
 meta_uri: mysql://root:milvusroot@host:3306/milvus
```


During installation, the configured JuiceFS shared storage path is set as **/root/jfs/milvus/db**.

```
1 sudo docker run -d --name milvus_gpu_1.0.0 --gpus all \ 
2 -p 19530:19530 \ 
3 -p 19121:19121 \ 
4 -v /root/jfs/milvus/db:/var/lib/milvus/db \ #/root/jfs/milvus/db is the shared storage path 
5 -v /home/$USER/milvus/conf:/var/lib/milvus/conf \ 
6 -v /home/$USER/milvus/logs:/var/lib/milvus/logs \ 
7 -v /home/$USER/milvus/wal:/var/lib/milvus/wal \ 
8 milvusdb/milvus:1.0.0-gpu-d030521-1ea92e 
9
```

After the installation completes, start Milvus and confirm that it is launched properly.
Finally, start the Mishards service on **any** of the nodes in the cluster. The image below shows a successful launch of Mishards. For more information, refer to the GitHub [tutorial](https://github.com/milvus-io/bootcamp/tree/new-bootcamp/deployments/juicefs).

![mishards-launch-success.png](https://assets.zilliz.com/mishards_launch_success_921695d3a8.png)

## **Performance benchmarks**

Shared storage solutions are usually implemented by network-attached storage (NAS) systems. Commonly used NAS systems types include Network File System (NFS) and Server Message Block (SMB). Public cloud platforms generally provide managed storage services compatible with these protocols, such as Amazon Elastic File System (EFS).

Unlike traditional NAS systems, JuiceFS is implemented based on Filesystem in Userspace (FUSE), where all data reading and writing takes place directly on the application side, further reducing access latency. There are also features unique to JuiceFS that cannot be found in other NAS systems, such as data compression and caching.

Benchmark testing reveals that JuiceFS offers major advantages over EFS. In the metadata benchmark (Figure 1), JuiceFS sees I/O operations per second (IOPS) up to ten times higher than EFS. Additionally, the I/O throughput benchmark (Figure 2) shows JuiceFS outperforms EFS in both single- and multi-job scenarios.

![performance-benchmark-1.png](https://assets.zilliz.com/performance_benchmark_1_b7fcbb4439.png) 
###### *Figure 1. Metadata benchmark*
![performance-benchmark-2.png](https://assets.zilliz.com/performance_benchmark_2_e311098123.png)
###### *Figure 2. Sequential read/write benchmark.*

Additionally, benchmark testing shows first query retrieval time, or time to load newly inserted data from disk to memory, for the JuiceFS-based Milvus cluster is just 0.032 seconds on average, indicating that data is loaded from disk to memory almost instantaneously. For this test, first query retrieval time is measured using one million rows of 128-dimensional vector data inserted in batches of 100k at intervals of 1 to 8 seconds.

JuiceFS is a stable and reliable shared file storage system, and the Milvus cluster built on JuiceFS offers both high performance and flexible storage capacity.

## **Learn more about Milvus**

Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

![writer bio-changjian gao.png](https://assets.zilliz.com/writer_bio_changjian_gao_68018f7716.png)
![writer bio-jingjing jia.png](https://assets.zilliz.com/writer_bio_jingjing_jia_a85d1c2e3b.png)

large_JuiceFS-blog-cover.jpg

medium_JuiceFS-blog-cover.jpg

small_JuiceFS-blog-cover.jpg

thumbnail_JuiceFS-blog-cover.jpg

JuiceFS-blog-cover.jpg

Building a Milvus Cluster Based on JuiceFS

Building a news content recommendation system with semantic vectors is easy with Milvus, an open-source vector database. 

With [71% of Americans](https://www.socialmediatoday.com/news/new-research-shows-that-71-of-americans-now-get-news-content-via-social-pl/593255/) getting their news recommendations from social platforms, personalized content has quickly become how new media is discovered. Whether people are searching for specific topics, or interacting with recommended content, everything users see is optimized by algorithms to improve click-through rates (CTR), engagement, and relevance. Sohu is a NASDAQ-listed Chinese online media, video, search and gaming group. It leveraged [Milvus](https://milvus.io/), an open-source vector database built by [Zilliz](https://zilliz.com/), to build a semantic vector search engine inside its news app. This article explains how the company used user profiles to fine-tune personalized content recommendations over time, improving user experience and engagement.

## Recommending content using semantic vector search

Sohu News user profiles are built from browsing history and adjusted as users search for, and interact with, news content. Sohu’s recommender system uses semantic vector search to find relevant news articles. The system works by identifying a set of tags that are expected to be of interest to each user based on browsing history. It then quickly searches for relevant articles and sorts the results by popularity (measured by average CTR), before serving them to users. 

The New York Times alone publishes [230 pieces of content](https://www.theatlantic.com/technology/archive/2016/05/how-many-stories-do-newspapers-publish-per-day/483845/) a day, which offers a glimpse at the magnitude of new content an effective recommendation system must be capable of processing. Ingesting large volumes of news demands millisecond similarity search and hourly matching of tags to new content. Sohu chose Milvus because it processes massive datasets efficiently and accurately, reduces memory usage during search, and supports high-performance deployments.

## Understanding a news recommendation system workflow

Sohu’s semantic vector search-based content recommendation relies on the Deep Structured Semantic Model (DSSM), which uses two neural networks to represent user queries and news articles as vectors. The model calculates the cosine similarity of the two semantic vectors, then the most similar batch of news is sent to the recommendation candidates pool. Next, news articles are ranked based on their estimated CTR, and those with the highest predicted click-through rate are displayed to users.

### Encoding news articles into semantic vectors with BERT-as-service

To encode news articles into semantic vectors, the system uses the [BERT-as-service](https://github.com/hanxiao/bert-as-service.git) tool. If the word count of any piece of content exceeds 512 while using this model, information loss occurs during the embedding process. To help overcome this, the system first extracts a summary and encodes it into a 768-dimensional semantic vector. Then the two most relevant topics from each news article are extracted, and the corresponding pre-trained topic vectors (200-dimensions) are identified based on topic ID. Next the topic vectors are spliced into the 768-dimensional semantic vector extracted from the article summary, forming a 968-dimensional semantic vector. 

New content continuously comes in through Kafta, and is converted into semantic vectors before being inserted into the Milvus database.

### Extracting semantically similar tags from user profiles with BERT-as-service

The other neural network of the model is user semantic vector. Semantically similar tags (e.g., coronavirus, covid, COVID-19, pandemic, novel strain, pneumonia) are extracted from user profiles based on interests, search queries, and browsing history. The list of acquired tags is sorted by weight, and the top 200 are divided into different semantic groups. Permutations of the tags within each semantic group are used to generate new tag phrases, which are then encoded into semantic vectors through BERT-as-service

For each user profile, sets of tag phrases have a [corresponding set of topics](https://github.com/baidu/Familia) that are marked by a weight indicating a user's interest level. The top two topics out of all relevant topics are selected and encoded by the machine learning (ML) model to be spliced into the corresponding tag semantic vector, forming a 968-dimensional user semantic vector. Even if the system generates the same tags for different users, different weights for tags and their corresponding topics, as well as explicit variance between each user's topic vectors, ensures recommendations are unique

The system is able to make personalized news recommendations by calculating the cosine similarity of the semantic vectors extracted from both user profiles and news articles.

![Sohu01.jpg](https://assets.zilliz.com/Sohu01_1e466fe0c3.jpg)
###### *Sohu News DSSM structure.*

### Computing new semantic user profile vectors and inserting them to Milvus

Semantic user profile vectors are computed daily, with data from the previous 24-hour period processed the following evening. Vectors are inserted into Milvus individually and run through the query process to serve relevant news results to users. News content is inherently topical, requiring computation to be run hourly to generate a current newsfeed that contains content that has a high predicted click-through rate and is relevant to users. News content is also sorted into partitions by date, and old news is purged daily.

### Decreasing semantic vector extraction time from days to hours

Retrieving content using semantic vectors requires converting tens of millions of tag phrases into semantic vectors every day. This is a time-consuming process that would require days to complete even when running on graphics processing units (GPU), which accelerate this type of computation. To overcome this technical issue, semantic vectors from the previous embedding must be optimized so that when similar tag phrases surface corresponding semantic vectors are directly retrieved.

The semantic vector of the existing set of tag phrases is stored, and a new set of tag phrases that is generated daily gets encoded into MinHash vectors. [Jaccard distance](https://milvus.io/docs/metric.md) is used to compute similarity between the MinHash vector of the new tag phrase and the saved tag phrase vector. If the Jaccard distance exceeds a pre-defined threshold, the two sets are considered similar. If the similarity threshold is met, new phrases can leverage the semantic information from previous embeddings. Tests suggest a distance above 0.8 should guarantee enough accuracy for most situations.

Through this process, the daily conversion of the tens of millions of vectors mentioned above is reduced from days to around two hours. Although other methods of storing semantic vectors might be more appropriate depending on specific project requirements, computing similarity between two tag phrases using Jaccard distance in a Milvus database remains an efficient and accurate method in a wide variety of scenarios.

![Sohu02.jpg](https://assets.zilliz.com/Sohu02_d50fccc538.jpg)
###### *Using Jaccard distance to compute similarity between two sets.*

## Overcoming “bad cases” of short text classification

When classifying news text, short news articles have fewer features for extraction than longer ones. Because of this, classification algorithms fail when content of varying lengths is run through the same classifier. Milvus helps solve this issue by searching for multiple pieces of long text classification information with similar semantics and reliable scores, then using a voting mechanism to modify short text classification. 

### Identifying and resolving misclassified short text 

Precise classification of each news article is crucial to providing useful content recommendations. Since short news articles have fewer features, applying the same classifier for news with different lengths results in a higher error rate for short text classification. Human labeling is too slow and inaccurate for this task, so BERT-as-service and Milvus are used to quickly identify misclassified short text in batches, correctly reclassify them, then use batches of data as a corpus for training against this problem.

BERT-as-service is used to encode a total number of five million long news articles with a classifier score greater than 0.9 into semantic vectors. After inserting the long text articles into Milvus, short text news is encoded into semantic vectors. Each short news semantic vector is used to query the Milvus database and obtain the top 20 long news articles with the highest cosine similarity to the target short news. If 18 of the top 20 semantically similar long news appear to be in the same classification and it differs from that of the query short news, then the short news classification is considered incorrect and must be adjusted to align with the 18 long news articles.

This process quickly identifies and corrects inaccurate short text classifications. Random sampling statistics show that after short text classifications are corrected, the overall accuracy of text classification exceeds 95%. By leveraging the classification of high-confidence long text to correct the classification of short text, the majority of bad classification cases are corrected in a short amount of time. This also offers a good corpus for training a short text classifier.

![Sohu03.jpg](https://assets.zilliz.com/Sohu03_a43074cf5f.jpg)
###### *Flowchart of discovery of "bad cases" of short text classification.*

## Milvus can power real-time news content recommendation and more

Milvus greatly improved the real-time performance of Sohu’s news recommendation system, and also bolstered the efficiency of identifying misclassified short text. If you’re interested in learning more about Milvus and its various applications:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

large_blog_SohuNews.jpg

medium_blog_SohuNews.jpg

small_blog_SohuNews.jpg

thumbnail_blog_SohuNews.jpg

blog_SohuNews.jpg

Learn how Milvus was used to build an intelligent news recommendation system inside an app.

Building an Intelligent News Recommendation System Inside Sohu News App

Milvus v1.1.0 is available now. Discover new features, improvements, and bug fixes made to the world' most popular vector database.

[Milvus](https://github.com/milvus-io) is an ongoing open-source software (OSS) project focused on building the world's fastest and most reliable vector database. New features inside Milvus v1.1.0 are the first of many updates to come, thanks to long-term support from the open-source community and sponsorship from Zilliz. This blog article covers the new features, improvements, and bug fixes included with Milvus v1.1.0.

**Jump to:**

- [New features](#new-features)
- [Improvements](#improvements)
- [Bug fixes](#bug-fixes)

 

## New features

Like any OSS project, Milvus is a perpetual work in progress. We strive to listen to our users and the open-source community to prioritize the features that matter most. The latest update, Milvus v1.1.0, offers the following new features:

### Specify partitions with `get_entity_by_id()` method calls

To further accelerate vector similarity search, Milvus 1.1.0 now supports retrieving vectors from a specified partition. Generally, Milvus supports querying vectors through specified vector IDs. In Milvus 1.0, calling the method `get_entity_by_id()` searches the entire collection, which can be time consuming for large datasets. As we can see from the code below, `GetVectorsByIdHelper` uses a `FileHolder` structure to loop through and find a specific vector. 

```
std::vector<meta::CollectionSchema> collection_array; 
 auto status = meta_ptr_->ShowPartitions(collection.collection_id_, collection_array); 
 
 collection_array.push_back(collection); 
 status = meta_ptr_->FilesByTypeEx(collection_array, file_types, files_holder); 
 if (!status.ok()) { 
 std::string err_msg = "Failed to get files for GetVectorByID: " + status.message(); 
 LOG_ENGINE_ERROR_ << err_msg; 
 return status; 
 } 
 
 if (files_holder.HoldFiles().empty()) { 
 LOG_ENGINE_DEBUG_ << "No files to get vector by id from"; 
 return Status(DB_NOT_FOUND, "Collection is empty"); 
 } 
 
 cache::CpuCacheMgr::GetInstance()->PrintInfo(); 
 status = GetVectorsByIdHelper(id_array, vectors, files_holder); 
DBImpl::GetVectorsByIdHelper(const IDNumbers& id_array, std::vector<engine::VectorsData>& vectors, 
 meta::FilesHolder& files_holder) { 
 // attention: this is a copy, not a reference, since the files_holder.UnMarkFile will change the array internal 
 milvus::engine::meta::SegmentsSchema files = files_holder.HoldFiles(); 
 LOG_ENGINE_DEBUG_ << "Getting vector by id in " << files.size() << " files, id count = " << id_array.size(); 
 
 // sometimes not all of id_array can be found, we need to return empty vector for id not found 
 // for example: 
 // id_array = [1, -1, 2, -1, 3] 
 // vectors should return [valid_vector, empty_vector, valid_vector, empty_vector, valid_vector] 
 // the ID2RAW is to ensure returned vector sequence is consist with id_array 
 using ID2VECTOR = std::map<int64_t, VectorsData>; 
 ID2VECTOR map_id2vector; 
 
 vectors.clear(); 
 
 IDNumbers temp_ids = id_array; 
 for (auto& file : files) { 
```

However, this structure is not filtered by any partitions in `FilesByTypeEx()`. In Milvus v1.1.0, it is possible for the system to pass partition names to the `GetVectorsIdHelper` loop so that the `FileHolder` only contains segments from specified partitions. Put differently, if you know exactly which partition the vector for a search belongs to, you can specify the partition name in a `get_entity_by_id()` method call to accelerate the search process.

We not only made modifications to code controlling system queries at the Milvus server level, but also updated all our SDKs(Python, Go, C++, Java, and RESTful) by adding a parameter for specifying partition names. For example, in pymilvus, the definition of `get_entity_by_id` `def get_entity_by_id(self, collection_name, ids, timeout=None)` is changed to `def get_entity_by_id(self, collection_name, partition_tags=None, ids, timeout=None)`.

 

### Specify partitions with `delete_entity_by_id()` method calls

To make vector management more efficient, Milvus v1.1.0 now supports specifying partition names when deleting a vector in a collection. In Milvus 1.0, vectors in a collection can only be deleted by ID. When calling the delete method, Milvus will scan all vectors in the collection. However, it is far more efficient to scan only relevant partitions when working with massive million, billion, or even trillion vector datasets. Similar to the new feature for specifying partitions with `get_entity_by_id()` method calls, modifications were made to the Milvus code using the same logic.

 

### New method `release_collection()`

To free up memory Milvus used to load collections at runtime, a new method `release_collection()` has been added in Milvus v1.1.0 to manually unload specific collections from cache.

 

## Improvements

Although new features are usually all the rage, it's also important to improve what we already have. What follows are upgrades and other general improvements over Milvus v1.0.

 

### Improved performance of `get_entity_by_id()` method call

The chart below is a comparison of vector search performance between Milvus v1.0 and Milvus v1.1.0:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Segment file size = 1024 MB 
> Row count = 1,000,000 
> Dim = 128 

| Query ID Num | v 1.0.0 | v1.1.0 |
| :-----------: | :-----------: | :-----------: |
| 10 | 9 ms | 2 ms |
| 100 | 149 ms | 19 ms |

 

### Hnswlib upgraded to v0.5.0

Milvus adopts multiple widely used index libraries, including Faiss, NMSLIB, Hnswlib, and Annoy to simplify the process of choosing the right index type for a given scenario.

Hnswlib has been upgraded from v0.3.0 to v0.5.0 in Milvus 1.1.0 due to a bug detected in the earlier version. Additionally, upgrading Hnswlib improves `addPoint()` performance in index building.

A Zilliz developer created a pull request (PR) to improve Hnswlib performance while building indexes in Milvus. See [PR #298](https://github.com/nmslib/hnswlib/pull/298) for details.

The chart below is a comparison of `addPoint()` performance between Hnswlib 0.5.0 and the proposed PR:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Dataset: sift_1M (row count = 1000000, dim = 128, space = L2)

| | 0.5.0 | PR-298 |
| :-----------: | :-----------: | :-----------: |
| M = 16, ef_construction = 100 | 274406 ms | 265631 ms |
| M = 16, ef_construction = 200 | 522411 ms | 499639 ms |

 

### Improved IVF index training performance

Creating an index includes training, inserting and writing data to disk. Milvus 1.1.0 improves the training component of index building. The chart below is a comparison of IVF index training performance between Milvus 1.0 and Milvus 1.1.0:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Dataset: sift_1m (row_count = 1000000, dim = 128, metric_type = L2)

| | v1.0.0 (ms) | v1.1.0 (ms) |
| :-----------: | :-----------: | :-----------: |
| ivf_flat (nlist = 2048) | 90079 | 81544 |
| ivf_pq (nlist = 2048, m=16) | 103535 | 97115 |
| ivf_pq (nlist = 2048, m=32) | 108638 | 104558 |
| ivf_flat (nlist = 4096) | 340643 | 310685 |
| ivf_pq (nlist = 4096, m=16) | 351982 | 323758 |
| ivf_pq (nlist = 4096, m=32) | 357359 | 330887 |


 

## Bug fixes

We also fixed some bugs to make Milvus more stable and efficient when managing vector datasets. See [Fixed Issues](https://milvus.io/docs/v1.1.0/release_notes.md#Fixed-issues) for more details.

large_v1.1 cover.jpeg

medium_v1.1 cover.jpeg

small_v1.1 cover.jpeg

thumbnail_v1.1 cover.jpeg

v1.1 cover.jpeg

Milvus v1.1.0 has arrived! New features, improvements, and bug fixes are available now.

Inside Milvus 1.1.0

The finance industry has long leveraged open-source software to extract insights from large, unstructured datasets. Discover how Milvus, an open-source vector database, can power AI applications including banking chatbots, recommender systems, and semantic text mining. 

Banks and other financial institutions have long been early adopters of open-source software for big data processing and analytics. In 2010, Morgan Stanley [began using](https://www.forbes.com/sites/tomgroenfeldt/2012/05/30/morgan-stanley-takes-on-big-data-with-hadoop/?sh=19f4f8cd16db) the open-source Apache Hadoop framework as part of a small experiment. The company was struggling to successfully scale traditional databases to the massive volumes of data its scientists wanted to leverage, so it decided to explore alternative solutions. Hadoop is now a staple at Morgan Stanley, helping with everything from managing CRM data to portfolio analysis. Other open-source relational database software such as MySQL, MongoDB, and PostgreSQL have been indispensable tools for making sense of big data in the finance industry.

Technology is what gives the financial services industry a competitive edge, and artificial intelligence (AI) is rapidly becoming the standard approach to extracting valuable insights from big data and analyzing activity in real-time across the banking, asset management, and insurance sectors. By using AI algorithms to convert unstructured data such as images, audio, or video to vectors, a machine-readable numeric data format, it is possible to run similarity searches on massive million, billion, or even trillion vector datasets. Vector data is stored in high-dimensional space, and similar vectors are found using similarity search, which requires a dedicated infrastructure called a vector database.

![01 (1).jpg](https://assets.zilliz.com/01_1_cb99f15886.jpg)
###### *Development of open source big data technologies.*

[Milvus](https://github.com/milvus-io/milvus) is an open-source vector database built specifically for managing vector data, which means engineers and data scientists can focus on building AI applications or conducting analysis—instead of the underlying data infrastructure. The platform was built around AI application development workflows and is optimized to streamline machine learning operations (MLOps). For more information about Milvus and its underlying technology, check out our [blog](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View).

Common applications of AI in the financial services industry include algorithmic trading, portfolio composition and optimization, model validation, backtesting, Robo-advising, virtual customer assistants, market impact analysis, regulatory compliance, and stress testing. This article covers three specific areas where vector data is leveraged as one of the most valuable assets for banking and financial companies:

1. Enhancing customer experience with banking chatbots
2. Boosting financial services sales and more with recommender systems
3. Analyzing earnings reports and other unstructured financial data with semantic text mining

 

### Enhancing customer experience with banking chatbots

Banking chatbots can improve customer experiences by helping consumers select investments, banking products, and insurance policies. Digital services are rising rapidly in popularity in part due to trends accelerated by the coronavirus pandemic. Chatbots work by using natural language processing (NLP) to convert user-submitted questions into semantic vectors to search for matching answers. Modern banking chatbots offer a personalized natural experience for users and speak in a conversational tone. Milvus provides a data fabric well suited for creating chatbots using real-time vector similarity search.

Learn more in our demo that covers building [chatbots with Milvus](https://zilliz.com/blog/building-intelligent-chatbot-with-nlp-and-milvus).

![02 (1).jpg](https://assets.zilliz.com/02_1_8c298c45e5.jpg)

 

#### Boosting financial services sales and more with recommender systems:

The private banking sector uses recommender systems to increase sales of financial products through personalized recommendations based on customer profiles. Recommender systems can also be leveraged in financial research, business news, stock selection, and trading support systems. Thanks to deep learning models, every user and item is described as an embedding vector. A vector database offers an embedding space where similarities between users and items can be calculated.

Learn more from our [demo](https://zilliz.com/blog/graph-based-recommendation-system-with-milvus) covering graph-based recommendation systems with Milvus.

 

#### Analyzing earnings reports and other unstructured financial data with semantic text mining:

Text mining techniques had a substantial impact on the financial industry. As financial data grows exponentially, text mining has emerged as an important field of research in the domain of finance.

Deep learning models are currently applied to represent financial reports through word vectors capable of capturing numerous semantic aspects. A vector database like Milvus is able to store massive semantic word vectors from millions of reports, then conduct similarity searches on them in milliseconds.

Learn more about how to [use deepset's Haystack with Milvus](https://medium.com/deepset-ai/semantic-search-with-milvus-knowledge-graph-qa-web-crawlers-and-more-837451eae9fa).

 

### Don’t be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_03 (1).jpg

medium_03 (1).jpg

small_03 (1).jpg

thumbnail_03 (1).jpg

03 (1).jpg

Milvus can be used to build AI applications for the finance industry including chatbots, recommender systems, and more.

Accelerating AI in Finance with Milvus, an Open-Source Vector Database

VOVA, an emerging e-commerce platform, prioritizes convenient and positive user experiences. Learn how the company used Milvus to build an image search tool to help shoppers find items they encounter in real life.

Jump to:
 
- [How does image search work?](#how-does-image-search-work)
- [Target detection using the YOLO model](#target-detection-using-the-yolo-model)
- [Image feature vector extraction with ResNet](#image-feature-vector-extraction-with-resnet)
- [Vector similarity search powered by Milvus](#vector-similarity-search-powered-by-milvus)
- [VOVA's shop by image tool](#vovas-shop-by-image-tool)

Online shopping surged in 2020, [up 44%](https://www.digitalcommerce360.com/2021/02/15/ecommerce-during-coronavirus-pandemic-in-charts/) in large part due to the coronavirus pandemic. As people sought to socially distance and avoid contact with strangers, no-contact delivery became an incredibly desirable option for many consumers. This popularity has also led to people buying a greater variety of goods online, including niche items that can be hard to describe using a traditional keyword search.

To help users overcome the limitations of keyword-based queries, companies can build image search engines that allow users to use images instead of words for search. Not only does this allow users to find items that are difficult to describe, but it also helps them shop for things they encounter in real life. This functionality helps build a unique user experience and offers general convenience that customers appreciate.

VOVA is an emerging e-commerce platform that focuses on affordability and offering a positive shopping experience to its users, with listings covering millions of products and support for 20 languages and 35 major currencies. To enhance the shopping experience for its users, the company used Milvus to build image search functionality into its e-commerce platform. The article explores how VOVA successfully built an image search engine with Milvus. 

 

### How does image search work?

VOVA's shop by image system searches the company's inventory for product images that are similar to user uploads. The following chart shows the two stages of the system process, the data import stage (blue) and the query stage (orange):

1. Use the YOLO model to detect targets from uploaded photos;
2. Use ResNet to extract feature vectors from the detected targets; 
3. Use Milvus for vector similarity search.

![Vova-1.png](https://assets.zilliz.com/Vova_1_47ee6f2da9.png)
###### *System process of VOVA's search by image functionality.*

 

### Target detection using the YOLO model

VOVA's mobile apps on Android and iOS currently support image search. The company uses a state-of-the-art, real-time object detection system called YOLO (You only look once) to detect objects in user uploaded images. The YOLO model is currently in its fifth iteration. 

YOLO is a one-stage model, using only one convolutional neural network (CNN) to predict categories and positions of different targets. It is small, compact, and well suited for mobile use. 

YOLO uses convolutional layers to extract features and fully-connected layers to obtain predicted values. Drawing inspiration from the GooLeNet model, YOLO’s CNN includes 24 convolutional layers and two fully-connected layers.

As the following illustration shows, a 448 &times; 448 input image is converted by a number of convolutional layers and pooling layers to a 7 &times; 7 &times; 1024-dimensional tensor (depicted in the third to last cube below), and then converted by two fully-connected layers to a 7 &times; 7 &times; 30-dimensional tensor output.

The predicted output of YOLO P is a two-dimensional tensor, whose shape is [batch,7 &times;7 &times;30]. Using slicing, P[:,0:7&times;7&times;20] is the category probability, P[:,7&times;7&times;20:7&times;7&times;(20+2)] is the confidence, and P[:,7&times;7&times;(20+2)]:] is the predicted result of the bounding box.

![vova-2.png](https://assets.zilliz.com/vova_2_1ccf38f721.png)
###### *YOLO network architecture.*

 

### Image feature vector extraction with ResNet

VOVA adopted the residual neural network (ResNet) model to extract feature vectors from an extensive product image library and user uploaded photos. ResNet is limited because as the depth of a learning network increases, the accuracy of the network decreases. The image below depicts ResNet running the VGG19 model (a variant of the VGG model) modified to include a residual unit through the short circuit mechanism. VGG was proposed in 2014 and includes just 14 layers, while ResNet came out a year later and can have up to 152.

The ResNet structure is easy to modify and scale. By changing the number of channels in the block and the number of stacked blocks, the width and depth of the network can be easily adjusted to obtain networks with different expressive capabilities. This effectively solves the network degeneration effect, where accuracy declines as the depth of learning increases. With sufficient training data, a model with improving expressive performance can be obtained while gradually deepening the network. Through model training, features are extracted for each picture and converted to 256-dimensional floating point vectors.

![vova-3.png](https://assets.zilliz.com/vova_3_df4b810281.png)
###### *ResNet structure.*

 

### Vector similarity search powered by Milvus
 
VOVA's product image database includes 30 million pictures and is growing rapidly. To quickly retrieve the most similar product images from this massive dataset, Milvus is used to conduct vector similarity search. Thanks to a number of optimizations, Milvus offers a fast and streamlined approach to managing vector data and building machine learning applications. Milvus offers integration with popular index libraries (e.g., Faiss, Annoy), supports multiple index types and distance metrics, has SDKs in multiple languages, and provides rich APIs for managing vector data.

Milvus can conduct similarity search on trillion-vector datasets in milliseconds, with a query time under 1.5 seconds when nq=1 and an average batch query time under 0.08 seconds. To build its image search engine, VOVA referred to the design of Mishards, Milvus' sharding middleware solution (see the chart below for its system design), to implement a highly available server cluster. By leveraging the horizontal scalability of a Milvus cluster, the project requirement for high query performance on massive datasets was met.

![vova-4.png](https://assets.zilliz.com/vova_4_e305f1955c.png)
###### *Mishards architecture in Milvus.*


### VOVA's shop by image tool
 
The screenshots below show the VOVA search by image shopping tool on the company's Android app.

![vova-5.png](https://assets.zilliz.com/vova_5_c4c25a3bae.png)
###### *Screenshots of VOVA's search by image shopping tool.*
 
As more users search for products and upload photos, VOVA will continue to optimize the models that power the system. Additionally, the company will incorporate new Milvus functionality that can further enhance the online shopping experience of its users.

### Reference

**YOLO:**

https://arxiv.org/pdf/1506.02640.pdf

https://arxiv.org/pdf/1612.08242.pdf

**ResNet:**

https://arxiv.org/abs/1512.03385

**Milvus:**

https://milvus.io/docs/overview.md

large_vova-thumbnail.jpg

medium_vova-thumbnail.jpg

small_vova-thumbnail.jpg

thumbnail_vova-thumbnail.jpg

vova-thumbnail.jpg

Discover how Milvus, an open-source vector database, was used by e-commerce platform VOVA to power shopping by image. 

Building a Search by Image Shopping Experience with VOVA and Milvus

We are proud to announce Milvus v1.0, a stable, long-term support version of  the world’s most popular open-source vector database. Discover the highlight features that Milvus v1.0 has to offer.

![Milvus.jpeg](https://assets.zilliz.com/Milvus_510cf50aee.jpeg)

Milvus is an open-source vector database designed to manage massive million, billion, or even trillion vector datasets. Milvus has broad applications spanning new drug discovery, computer vision, autonomous driving, recommendation engines, chatbots, and much more. 

In March, 2021 Zilliz, the company behind Milvus, released the platform's first long-term support version—Milvus v1.0. After months of extensive testing, a stable, production ready version of the world's most popular vector database is ready for prime time. This blog article covers some Milvus fundamentals as well as key features of v1.0.

 

### Milvus distributions

Milvus is available in CPU-only and GPU-enabled distributions. The former relies exclusively on CPU for index building and search; the latter enables CPU and GPU hybrid search and index building that further accelerates Milvus. For example, using the hybrid distribution, CPU can be used for search and GPU for index building, further improving query efficiency.

Both Milvus distributions are available in Docker. You can either compile Milvus from Docker (if your operating system supports it) or compile Milvus from source code on Linux (other operating systems are not supported). 

 

### Embedding vectors

Vectors are stored in Milvus as entities. Each entity has one vector ID field and one vector field. Milvus v1.0 supports integer vector IDs only. When creating a collection within Milvus, vector IDs can be automatically generated or manually defined. Milvus ensures auto-generated vector IDs are unique however, manually defined IDs can be duplicated within Milvus. If manually defining IDs, users are responsible for making sure all IDs are unique.

 

### Partitions

Milvus supports creating partitions in a collection. In situations where data is inserted regularly and historical data isn't significant (e.g., streaming data), partitions can be used to accelerate vector similarity search. One collection can have up to 4,096 partitions. Specifying a vector search within a specific partition narrows the search and may significantly reduce query time, particularly for collections that contain more than a trillion vectors.

 

### Index algorithm optimizations
Milvus is built on top of multiple widely-adopted index libraries, including Faiss, NMSLIB, and Annoy. Milvus is far more than a basic wrapper for these index libraries. Here are some of the major enhancements that have been made to the underlying libraries:

- IVF index performance optimizations using the Elkan k-means algorithm.
- FLAT search optimizations.
- IVF_SQ8H hybrid index support, which can reduce index file sizes by up to 75% without sacrificing data accuracy. IVF_SQ8H is built upon IVF_SQ8, with identical recall but much faster query speed. It was designed specifically for Milvus to harnesses the parallel processing capacity of GPUs, and the potential for synergy between CPU/GPU co-processing.
- Dynamic instruction set compatibility.

 

### Search, index building, and other Milvus optimizations

The following optimizations have been made to Milvus to improve search and index building performance. 

- Search performance is optimized in situations when the number of queries (nq) is less than the number of CPU threads. 
- Milvus combines search requests from a client that take the same topK and search parameters. 
- Index building is suspended when search requests come in. 
- Milvus automatically preloads collections to memory at start.
- Multiple GPU devices can be assigned to accelerate vector similarity search.

 

### Distance metrics

Milvus is a vector database built to power vector similarity search. The platform was built with MLOps and production level AI applications in mind. Milvus supports a wide range of distance metrics for calculating similarity, such as Euclidean distance (L2), inner product (IP), Jaccard distance, Tanimoto, Hamming distance, superstructure, and substructure. The last two metrics are commonly used in molecular search and AI-powered new drug discovery.

 

### Logging

Milvus supports log rotation. In the system configuration file, milvus.yaml, you can set the size of a single log file, the number of log files, and log output to stdout.

 

### Distributed solution

Mishards, a Milvus sharding middleware, is the distributed solution for Milvus With one write node and an unlimited number of read nodes, Mishards unleashes the computational potential of server cluster. Its features include request forwarding, read/write splitting, dynamic/horizontal scaling, and more.

 

### Monitoring

Milvus is compatible with Prometheus, an open-source system monitoring and alerts toolkit. Milvus adds support for Pushgateway in Prometheus, making it possible for Prometheus to acquire short-lived batch metrics. The monitoring and alerts system works as follows:

- The Milvus server pushes customized metrics data to Pushgateway.
- Pushgateway ensures ephemeral metric data is safely sent to Prometheus.
- Prometheus continues pulling data from Pushgateway.
- Alertmanager is used to set the alert threshold for different indicators and send alerts via email or message.

 

### Metadata management

Milvus uses SQLite for metadata management by default. SQLite is implemented in Milvus and does not require configuration. In a production environment, it is recommended that you use MySQL for metadata management. 

 

### Engage with our open-source community:

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_Milvus.jpeg

medium_Milvus.jpeg

small_Milvus.jpeg

thumbnail_Milvus.jpeg

Milvus.jpeg

	 Milvus v1.0 is available now. This stable, long-term support version powers image/video search, chatbots, and many more AI applications.

What's Inside Milvus 1.0?

Learn how Milvus is used to mitigate threats to critical data and strengthen cybersecurity with real-time virus detection.

Cybersecurity remains a persistent threat to both individuals and businesses, with data privacy concerns increasing for [86% of companies](https://www.getapp.com/resources/annual-data-security-report/) in 2020 and just [23% of consumers](https://www.firstdata.com/downloads/pdf/digital-commerce-cybersecurity-ebook.pdf) believing their personal data is very secure. As malware becomes steadily more omnipresent and sophisticated, a proactive approach to threat detection has become essential. [Trend Micro](https://www.trendmicro.com/en_us/business.html) is a global leader in hybrid cloud security, network defense, small business security, and endpoint security. To protect Android devices from viruses, the company built Trend Micro Mobile Security—a mobile app that compares APKs (Android Application Package) from the Google Play Store to a database of known malware. The virus detection system works as follows:

- External APKs (Android application package) from the Google Play Store are crawled.
- Known malware is converted into vectors and stored in [Milvus](https://www.milvus.io/docs/v1.0.0/overview.md).
- New APKs are also converted into vectors, then compared to the malware database using similarity search.
- If an APK vector is similar to any of the malware vectors, the app provides users with detailed information about the virus and its threat level.

To work, the system has to perform highly efficient similarity search on massive vector datasets in real time. Initially, Trend Micro used [MySQL](https://www.mysql.com/). However, as its business expanded so did the number of APKs with nefarious code stored in its database. The company’s algorithm team began searching for alternative vector similarity search solutions after quickly outgrowing MySQL.

 

### Comparing vector similarity search solutions

There are a number of vector similarity search solutions available, many of which are open source. Although the circumstances vary from project to project, most users benefit from leveraging a vector database built for unstructured data processing and analytics rather than a simple library that requires extensive configuration. Below we compare some popular vector similarity search solutions and explain why Trend Micro chose Milvus.

#### Faiss
[Faiss](https://ai.facebook.com/tools/faiss/) is a library developed by Facebook AI Research that enables efficient similarity search and clustering of dense vectors. The algorithms it contains search vectors of any size in sets. Faiss is written in C++ with wrappers for Python/numpy, and supports a number of indexes including IndexFlatL2, IndexFlatIP, HNSW, and IVF.

Although Faiss is an incredibly useful tool, it has limitations. It only works as a basic algorithm library, not a database for managing vector datasets. Additionally, it does not offer a distributed version, monitoring services, SDKs, or high availability, which are the key features of most cloud-based services. 

#### Plug-ins based on Faiss & other ANN search libraries
There are several plug-ins built on top of Faiss, NMSLIB, and other ANN search libraries that are designed to enhance the basic functionality of the underlying tool that powers them. Elasticsearch (ES) is a search engine based on the Lucene library with a number of such plugins. Below is an architecture diagram of an ES plug-in:

![1.png](https://assets.zilliz.com/1_3ce4e516c3.png)

###### *Architecture diagram of an Elasticsearch plug-in.*

Built in support for distributed systems is a major advantage of an ES solution. This saves developers time and companies money thanks to code that doesn’t have to be written. ES plug-ins are technically advanced and prevalent. Elasticsearch provides a QueryDSL (domain-specific language), which defines queries based on JSON and is easy to grasp. A full set of ES services makes it possible to conduct vector/text search and filter scalar data simultaneously.

Amazon, Alibaba, and Netease are a few large tech companies that currently rely on Elasticsearch plug-ins for vector similarity search. The primary downsides with this solution are high memory consumption and no support for performance tuning. In contrast, [JD.com](http://jd.com/) has developed its own distributed solution based on Faiss called [Vearch](https://github.com/vearch/vearch). However, Vearch is still an incubation-stage project and its open-source community is relatively inactive.

#### Milvus 
[Milvus](https://www.milvus.io/) is an open-source vector database created by [Zilliz](https://zilliz.com). It is highly flexible, reliable, and blazing fast. By encapsulating multiple widely adopted index libraries, such as Faiss, NMSLIB, and Annoy, Milvus provides a comprehensive set of intuitive APIs, allowing developers to choose the ideal index type for their scenario. It also provides distributed solutions and monitoring services. Milvus has a highly active open-source community and over 5.5K stars on [Github](https://github.com/milvus-io/milvus).

#### Milvus bests the competition
We compiled a number of different test results from the various vector similarity search solutions mentioned above. As we can see in the following comparison table, Milvus was significantly faster than the competition despite being tested on a dataset of 1 billion 128-dimensional vectors.

| **Engine** | **Performance (ms)** | **Dataset Size (million)** |
| :---- | :---- | :---- |
| ES | 600 | 1 |
| ES + Alibaba Cloud | 900 | 20 |
| Milvus | 27 | 1000+ |
| SPTAG | Not good | |
| ES + nmslib, faiss | 90 | 150 |

###### *A comparison of vector similarity search solutions.*

After weighing the pros and cons of each solution, Trend Micro settled on Milvus for its vector retrieval model. With exceptional performance on massive, billion-scale datasets, it's obvious why the company chose Milvus for a mobile security service that requires real-time vector similarity search.

 

### Designing a system for real-time virus detection
Trend Micro has more than 10 million malicious APKs stored in its MySQL database, with 100k new APKs added each day. The system works by extracting and calculating Thash values of different components of an APK file, then uses the Sha256 algorithm to transform it into binary files and generate 256-bit Sha256 values that differentiate the APK from others. Since Sha256 values vary with APK files, one APK can have one combined Thash value and one unique Sha256 value.

Sha256 values are only used to differentiate APKs, and Thash values are used for vector similarity retrieval. Similar APKs may have the same Thash values but different Sha256 values.

To detect APKs with nefarious code, Trend Micro developed its own system for retrieving similar Thash values and corresponding Sha256 values. Trend Micro chose Milvus to conduct instantaneous vector similarity search on massive vector datasets converted from Thash values. After similarity search is run, the corresponding Sha256 values are queried in MySQL. A Redis caching layer is also added to the architecture to map Thash values to Sha256 values, significantly reducing query time.

Below is the architecture diagram of Trend Micro’s mobile security system.

![image-20210118-022039.png](https://assets.zilliz.com/image_20210118_022039_ae824b663c.png)

###### *Architecture diagram for Trend Micro Mobile Security.*

 

Choosing an appropriate distance metric helps improve vector classification and clustering performance. The following table shows the [distance metrics](https://www.milvus.io/docs/v1.0.0/metric.md#binary) and the corresponding indexes that work with binary vectors. 

| **Distance Metrics** | **Index Types** |
| ---- | ---- |
| - Jaccard - Tanimoto - Hamming | - FLAT - IVF_FLAT |
| - Superstructure - Substructure | FLAT |

###### *Distance metrics and indexes for binary vectors.*

 

Trend Micro converts Thash values into binary vectors and stores them in Milvus. For this scenario, Trend Micro is using Hamming distance to compare vectors.

Milvus will soon support string vector ID, and integer IDs won’t have to be mapped to the corresponding name in string format. This makes the Redis caching layer unnecessary and the system architecture less bulky.

Trend Micro adopts a cloud-based solution and deploys many tasks on [Kubernetes](https://kubernetes.io/). To achieve high availability, Trend Micro uses [Mishards](https://www.milvus.io/docs/v1.0.0/mishards.md), a Milvus cluster sharding middleware developed in Python. 

![image-20210118-022104.png](https://assets.zilliz.com/image_20210118_022104_3001950ee8.png)

###### *Mishards architecture in Milvus.*

 

Trend Micro separates storage and distance calculation by storing all vectors in the [EFS](https://aws.amazon.com/efs/) (Elastic File System) provided by [AWS](https://aws.amazon.com/). This practice is a popular trend in the industry. Kubernetes is used to start multiple reading nodes, and develops LoadBalancer services on these reading nodes to ensure high availability.

To maintain data consistency Mishards supports just one writing node. However, a distributed version of Milvus with support for multiple writing nodes will be available in the coming months. 

 

### Monitoring and Alert Functions
Milvus is compatible with monitoring systems built on [Prometheus](https://prometheus.io/), and uses [Grafana](https://grafana.com/), an open-source platform for time-series analytics, to visualize various performance metrics.

Prometheus monitors and stores the following metrics:

- Milvus performance metrics including insertion speed, query speed, and Milvus uptime. 
- System performance metrics including CPU/GPU usage, network traffic, and disk access speed.
- Hardware storage metrics including data size and total file number.

The monitoring and alert system works as follows: 

- A Milvus client pushes customized metrics data to Pushgateway.
- The Pushgateway ensures short-lived, ephemeral metric data is safely sent to Prometheus.
- Prometheus keeps pulling data from Pushgateway. 
- Alertmanager sets the alert threshold for different metrics and raises alarms through emails or messages.

 

### System Performance
A couple months have passed since the ThashSearch service built on Milvus was first launched. The graph below shows that end-to-end query latency is less than 95 milliseconds.

![image-20210118-022116.png](https://assets.zilliz.com/image_20210118_022116_a0c735ce20.png)

###### *Query latency for Thash search service built on Milvus.*

 

Insertion is also fast. It takes around 10 seconds to insert 3 million 192-dimensional vectors. With help from Milvus, the system performance was able to meet the performance criteria set by Trend Micro. 

 

### Don’t be a stranger
- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_TrendMicro.jpg

medium_blog_TrendMicro.jpg

small_blog_TrendMicro.jpg

thumbnail_blog_TrendMicro.jpg

blog_TrendMicro.jpg

Making with Milvus: Detecting Android Viruses in Real Time for Trend Micro

Learn more about using semantic machine learning methodologies to power more relevant search results across your organization.

[Semantic search](https://lucidworks.com/post/what-is-semantic-search/) is a great tool to help your customers—or your employees—find the right products or information. It can even surface difficult-to-index information for better results. That said, if your semantic methodologies aren’t being deployed to work fast, they won’t do you any good. The customer or employee isn’t just going to sit around while the system takes its time responding to their query—and a thousand others are likely being ingested at the same time.

How can you make semantic search fast? Slow semantic search isn’t going to cut it.

Fortunately, this is the kind of problem Lucidworks loves to solve. We recently tested a modest-sized cluster—read on for more details—that resulted in 1500 RPS (requests per second) against a collection of over one million documents, with an average response time of roughly 40 milliseconds. Now that’s some serious speed.

 

### Implementing Semantic Search
To make lightning-fast, machine learning magic happen, Lucidworks has implemented semantic search using the semantic vector search approach. There are two critical parts.

 

#### Part One: The Machine Learning Model

First, you need a way to encode text into a numerical vector. The text could be a product description, a user search query, a question, or even an answer to a question. A semantic search model is trained to encode text such that text that is semantically similar to other text is encoded into vectors that are numerically “close” to one another. This encoding step needs to be fast in order to support the thousand or more possible customer searches or user queries coming in every second.

 

#### Part Two: The Vector Search Engine
Second, you need a way to quickly find the best matches to the customer search or user query. The model will have encoded that text into a numerical vector. From there, you need to compare that to all the numerical vectors in your catalog or lists of questions and answers to find those best matches—the vectors that are “closest” to the query vector. For that, you will need a vector engine that can handle all of that information effectively and at lightning speed. The engine could contain millions of vectors and you really just want the best twenty or so matches to your query. And of course, it needs to handle a thousand or so such queries every second.

To tackle these challenges, we added the vector search engine [Milvus](https://doc.lucidworks.com/fusion/5.3/8821/milvus) in our [Fusion 5.3 release](https://lucidworks.com/post/enhance-personalization-efforts-with-new-features-in-fusion/). Milvus is open-source software and it is fast. Milvus uses FAISS ([Facebook AI Similarity Search](https://ai.facebook.com/tools/faiss/)), the same technology Facebook uses in production for its own machine learning initiatives. When needed, it can run even faster on [GPU](https://en.wikipedia.org/wiki/Graphics_processing_unit). When Fusion 5.3 (or higher) is installed with the machine learning component, Milvus is automatically installed as part of that component so you can turn on all of these capabilities with ease.

The size of the vectors in a given collection, specified when the collection is created, depends on the model that produces those vectors. For example, a given collection could store the vectors created from encoding (via a model) all of the product descriptions in a product catalog. Without a vector search engine like Milvus, similarity searches would not be feasible across the entire vector space. So, similarity searches would have to be limited to pre-selected candidates from the vector space (for example, 500) and would have both slower performance and lower quality results. Milvus can store hundreds of billions of vectors across multiple collections of vectors to ensure that search is fast and results are relevant.

 

### Using Semantic Search

Let’s get back to the semantic search workflow, now that we’ve learned a little about why Milvus might be so important. Semantic search has three stages. During the first stage, the machine learning model is loaded and/or trained. Afterwards, data is indexed into Milvus and Solr. The final stage is the query stage, when the actual search occurs. We’ll focus on those last two stages below.

 

### Indexing into Milvus
![Lucidworks-1.png](https://assets.zilliz.com/Lucidworks_1_47a9221723.png)
###### *Architectural diagram for indexing into Milvus.*

As shown in the above diagram, the query stage begins similarly to the indexing stage, just with queries coming in instead of documents. For each query:

1. The query is sent to the [Smart Answers](https://lucidworks.com/products/smart-answers/) index pipeline.
2. The query is then sent to the ML model.
3. The ML model returns a numeric vector (encrypted from the query). Again, the type of model determines the size of the vector.
4. The vector is sent to Milvus, which then determines which vectors, in the specified Milvus collection, best match the provided vector.
5. Milvus returns a list of unique IDs and distances corresponding to the vectors determined in step four.
6. A query containing those IDs and distances is sent to Solr.
7. Solr then returns an ordered list of the documents associated with those IDs.

 

### Scale Testing
In order to prove that our semantic search flows are running at the efficiency we require for our customers, we run scale tests using Gatling scripts on the Google Cloud Platform using a Fusion cluster with eight replicas of the ML model, eight replicas of the query service, and a single instance of Milvus. Tests were run using the Milvus FLAT and HNSW indexes. The FLAT index has 100% recall, but is less efficient – except when the datasets are small. The HNSW (Hierarchical Small World Graph) index still has high quality results and it has improved performance on larger datasets.

Let’s jump into some numbers from a recent example we ran:

![Lucidworks-2.png](https://assets.zilliz.com/Lucidworks_2_3162113560.png)
###### *Performance of Milvus FLAT and HNSW indexes on a small dataset.*

![Lucidworks-3.png](https://assets.zilliz.com/Lucidworks_3_3dc17f0ed8.png)
###### *Performance of Milvus FLAT and HNSW indexes on a medium dataset.*

![Lucidworks-4.png](https://assets.zilliz.com/Lucidworks_4_8a6edd2f59.png)
###### *Performance of Milvus FLAT and HNSW indexes on a large dataset.*

 

### Getting Started

The [Smart Answers](https://lucidworks.com/products/smart-answers/) pipelines are designed to be easy-to-use. Lucidworks has [pre-trained models that are easy-to-deploy](https://doc.lucidworks.com/how-to/734/set-up-a-pre-trained-cold-start-model-for-smart-answers) and generally have good results—though training your own models, in tandem with pre-trained models, will offer the best results. Contact us today to learn how you can implement these initiatives into your search tools to power more effective and delightful results.


> This blog is reposted from: https://lucidworks.com/post/how-to-build-fast-semantic-search/?utm_campaign=Oktopost-Blog+Posts&utm_medium=organic_social&utm_source=linkedin

medium_lucidworks.png

small_lucidworks.png

thumbnail_lucidworks.png

lucidworks.png

Build Semantic Search at Speed

Milvus accelerates machine learning application development and machine learning operations (MLOps). With Milvus, you can rapidly develop a minimum viable product (MVP) while keeping costs at lower limits.

![blog cover.png](https://assets.zilliz.com/blog_cover_4a9807b9e0.png)

[Milvus](https://milvus.io/) is an open-source vector database. It supports adding, deleting, updating, and near real-time search of massive vector datasets created by extracting feature vectors from unstructured data using AI models. With a comprehensive set of intuitive APIs, and support for multiple widely adopted index libraries (e.g., Faiss, NMSLIB, and Annoy), Milvus accelerates machine learning application development and machine learning operations (MLOps). With Milvus, you can rapidly develop a minimum viable product (MVP) while keeping costs at lower limits.

"What resources are available for developing an AI application with Milvus?” is commonly asked in the Milvus community. Zilliz, the [company](https://zilliz.com/) behind Milvus, developed a number of demos that leverage Milvus to conduct lightening-fast similarity search that powers intelligent applications. Source code of Milvus solutions can be found at [zilliz-bootcamp](https://github.com/zilliz-bootcamp). The following interactive scenarios demonstrate natural language processing (NLP), reverse image search, audio search, and computer vision.


Feel free to try out the solutions to gain some hands-on experience with specific scenarios! Share your own application scenarios via: 
- [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ)
- [GitHub](https://github.com/milvus-io/milvus/discussions)

 

**Jump to:**

- [Natural language processing (chatbots)](#natural-language-processing-chatbots)
- [Reverse image search](#reverse-image-search-systems)
- [Audio search](#audio-search-systems)
- [Video object detection (computer vision)](#video-object-detection-computer-vision)

 

### Natural language processing (chatbots)
Milvus can be used to build chatbots that use natural language processing to simulate a live operator, answer questions, route users to relevant information, and reduce labor costs. To demonstrate this application scenario, Zilliz built an AI-powered chatbot that understands semantic language by combining Milvus with [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)), a machine learning (ML) model developed for NLP pre-training.

👉Source code：[zilliz-bootcamp/intelligent_question_answering_v2](https://github.com/zilliz-bootcamp/intelligent_question_answering_v2)

![1.png](https://assets.zilliz.com/1_c301a9e4bd.png)
###### *AI-powered chatbot built with Milvus and BERT.*

#### How to use

1. Upload a dataset that includes question-answer pairs. Format questions and answers in two separate columns. Alternatively, a [sample dataset](https://zilliz.com/solutions/qa) is available for download.

2. After typing in your question, a list of similar questions will be retrieved from the uploaded dataset.

3. Reveal the answer by selecting the question most similar to your own.

👉Video：[[Demo] QA System Powered by Milvus](https://www.youtube.com/watch?v=ANgoyvgAxgU)

#### How it works

Questions are converted into feature vectors using Google’s BERT model, then Milvus is used to manage and query the dataset.

**Data processing:**

1. BERT is used to convert the uploaded question-answer pairs into 768-dimensional feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Question, and corresponding answer, vector IDs are stored in PostgreSQL.

**Searching for similar questions:**

1. BERT is used to extract feature vectors from a user's input question.
2. Milvus retrieves vector IDs for questions that are most similar to the input question.
3. The system looks up the corresponding answers in PostgreSQL.

 

### Reverse image search systems
Reverse image search is transforming e-commerce through personalized product recommendations and similar product lookup tools that can boost sales. In this application scenario, Zilliz built a reverse image search system by combining Milvus with [VGG](https://towardsdatascience.com/how-to-use-a-pre-trained-model-vgg-for-image-classification-8dd7c4a4a517), an ML model that can extract image features.

👉Source code：[zilliz-bootcamp/image_search](https://github.com/zilliz-bootcamp/image_search)

![2.jpeg](https://assets.zilliz.com/2_09000e2e2e.jpeg)
###### *Reverse image search system built with Milvus and VGG.*

#### How to use
1. Upload a zipped image dataset comprised of .jpg images only (other image file types are not accepted). Alternatively, a [sample dataset](https://zilliz.com/solutions/image-search) is available for download.
2. Upload an image to use as the search input for finding similar images.

👉Video: [[Demo] Image Search Powered by Milvus](https://www.youtube.com/watch?v=mTO8YdQObKY)

#### How it works
Images are converted into 512-dimensional feature vectors using the VGG model, then Milvus is used to manage and query the dataset.

**Data processing:**

1. The VGG model is used to convert the uploaded image dataset to feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Image feature vectors, and corresponding image file paths, are stored in CacheDB.

**Searching for similar images:**

1. VGG is used to convert a user’s uploaded image into feature vectors.
2. Vector IDs of images most similar to the input image are retrieved from Milvus.
3. The system looks up the corresponding image file paths in CacheDB.

 

### Audio search systems
Speech, music, sound effects, and other types of audio search makes it possible to quickly query massive volumes of audio data and surface similar sounds. Applications include identifying similar sound effects, minimizing IP infringement, and more. To demonstrate this application scenario, Zilliz built a highly efficient audio similarity search system by combining Milvus with [PANNs](https://arxiv.org/abs/1912.10211)—a large-scale pretrained audio neural networks built for audio pattern recognition.

👉Source code：[zilliz-bootcamp/audio_search](https://github.com/zilliz-bootcamp/audio_search)
![3.png](https://assets.zilliz.com/3_419bac3dd2.png)
###### *Audio search system built with Milvus and PANNs.*

#### How to use
1. Upload a zipped audio dataset comprised of .wav files only (other audio file types are not accepted). Alternatively, a [sample dataset](https://zilliz.com/solutions/audio-search) is available for download.
2. Upload a .wav file to use as the search input for finding similar audio.

👉Video: [[Demo] Audio Search Powered by Milvus](https://www.youtube.com/watch?v=0eQHeqriCXw)
#### How it works
Audio is converted into feature vectors using PANNs, large-scale pre-trained audio neural networks built for audio pattern recognition. Then Milvus is used to manage and query the dataset.

**Data processing:**

1. PANNs converts audio from the uploaded dataset to feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Audio feature vector IDs and their corresponding .wav file paths are stored in PostgreSQL.

**Searching for similar audio:**

1. PANNs is used to convert a user’s uploaded audio file into feature vectors.
2. Vector IDs of audio most similar to the uploaded file are retrieved from Milvus by calculating inner product (IP) distance.
3. The system looks up the corresponding audio file paths in MySQL.

 

### Video object detection (computer vision)
Video object detection has applications in computer vision, image retrieval, autonomous driving, and more. To demonstrate this application scenario, Zilliz built a video object detection system by combining Milvus with technologies and algorithms including [OpenCV](https://en.wikipedia.org/wiki/OpenCV), [YOLOv3](https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b), and [ResNet50](https://www.mathworks.com/help/deeplearning/ref/resnet50.html).

👉Source code: [zilliz-bootcamp/video_analysis](https://github.com/zilliz-bootcamp/video_analysis)

![4.png](https://assets.zilliz.com/4_54b4ceb2ad.png)
###### *Video object detection system with Milvus.*

#### How to use
1. Upload a zipped image dataset comprised of .jpg files only (other image file types are not accepted). Ensure that each image file is named by the object it depicts. Alternatively, a [sample dataset](https://zilliz.com/solutions/video-obj-analysis) is available for download. 
2. Upload a video to use for analysis.
3. Click the play button to view the uploaded video with object detection results shown in real time.

👉Video: [[Demo] Video Object Detection System Powered by Milvus](https://www.youtube.com/watch?v=m9rosLClByc)

#### How it works
Object images are converted into 2048-dimensional feature vectors using ResNet50. Then Milvus is used to manage and query the dataset.

**Data processing:**

1. ResNet50 converts object images to 2048-dimensional feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Audio feature vector IDs and their corresponding image file paths are stored in MySQL.

**Detecting objects in video:**

1. OpenCV is used to trim the video.
2. YOLOv3 is used to detect objects in the video.
3. ResNet50 converts detected object images into 2048-dimensional feature vectors.

Milvus searches for the most similar object images in the uploaded dataset. Corresponding object names and image file paths are retrieved from MySQL.

How to Make 4 Popular AI Applications with Milvus

Ensuring investments in artificial intelligence are successful requires effectively managing the life cycle of AI models and the applications they are used in. This article explains the concept of machine learning operations, or MLOps, and how the open-source vector database Milvus can be used to advance MLOps programs of varying complexity.

Building machine learning (ML) applications is a complex and iterative process. As more companies realize the untapped potential of unstructured data, the demand for [AI-powered data processing and analytics](https://zilliz.com/blog/Thanks-to-Milvus-Anyone-Can-Build-a-Vector-Database-for-1-Billion-Images) will continue to rise. Without effective machine learning operations, or MLOps, most ML application investments will wither on the vine. Research has found that [as little as 5%](https://www.forbes.com/sites/cognitiveworld/2020/03/31/modelops-is-the-key-to-enterprise-ai/?sh=44c0f5066f5a) of the AI adoptions companies plan to deploy actually reach deployment. Many organizations incur "model debt," where changes in market conditions, and failure to adapt to them, result in unrealized investments in models that linger unrefreshed (or worse, never get deployed at all).

This article explains MLOps, a systemic approach to AI model life cycle management, and how the open-source vector data management platform [Milvus](https://milvus.io/) can be used to operationalize AI at scale.

 

### What is MLOps?

Machine learning operations (MLOps), also known as model operations (ModelOps) or AI model operationalization, is necessary to build, maintain, and deploy AI applications at scale. As companies seek to apply the AI models they develop to hundreds of different scenarios, it is mission critical that models in use, and those under development, are operationalized across the entire organization. MLOps involves monitoring a machine learning model throughout its lifecycle, and governing everything from underlying data to the effectiveness of a production system that relies on a particular model.

![01.jpg](https://assets.zilliz.com/01_362a07d156.jpg)

###### *MLOps governs the life cycle of a machine learning model.*

Gartner [defines](https://www.gartner.com/en/information-technology/glossary/modelops) ModelOps as the governance and life cycle management of a wide range of operationalized artificial intelligence and decision models. The core functionality of MLOps can be broken down as follows:

- **Continuous integration/continuous delivery (CI/CD):** A set of best practices borrowed from developer operations (DevOps), CI/CD is a method for delivering code changes more frequently and more reliably. [Continuous integration](https://www.gartner.com/en/information-technology/glossary/continuous-integration-ci) promotes implementing code changes in small batches while monitoring them with strict version control. [Continuous delivery](https://www.gartner.com/smarterwithgartner/5-steps-to-master-continuous-delivery/) automates the delivery of applications to various environments (e.g., testing and development environments). 

- **Model development environments (MDE):** A complex process for building, reviewing, documenting, and examining models, MDE help ensure models are created iteratively, documented as they are developed, trusted, and reproducible. Effective MDE ensures models can be explored, researched, and experimented on in a controlled manner.

- **Champion-challenger testing:** Similar to A/B testing methodology used by marketers, [champion-challenger testing](https://medium.com/decision-automation/what-is-champion-challenger-and-how-does-it-enable-choosing-the-right-decision-f57b8b653149) involves experimenting with different solutions to aid the decision-making process that proceeds committing to a single approach. This technique involves monitoring and measuring performance in real time to identify which deviation works best.

- **Model versioning:** As with any complex system, machine learning models are developed in steps by many different people—resulting in data management questions around versions of data and ML models. Model versioning helps manage and govern the iterative process of ML development where data, models, and code may evolve at different rates.

- **Model store and rollback:** When a model is deployed, its corresponding image file must be stored. Rollback and recovery abilities allow MLOps teams to revert to a previous model version if needed.

Using just one model in a production application presents a number of difficult challenges. MLOps is a structured, repeatable method that relies on tools, technology, and best practices to overcome technical or business problems that arise during a machine learning model’s life cycle. Successful MLOps maintains efficiency across teams working to build, deploy, monitor, retrain, and govern AI models and their use in production systems. 

 

### Why is MLOps necessary?

As depicted in the ML model life cycle above, building a machine learning model is an iterative process that involves incorporating new data, retraining models, and dealing with general model decay over time. These are all issues that traditional developer operations, or DevOps, does not address or provide solutions for. MLOps has become necessary as a way to manage investment in AI models and ensure a productive model life cycle. Because machine learning models will be leveraged by a variety of different production systems, MLOps becomes integral to making sure requirements can be met across different environments and amid varying scenarios.

![02.jpg](https://assets.zilliz.com/02_403e7f2fe2.jpg)

###### *A machine learning model being deployed in a cloud environment that feeds into an application.*

 

The simple illustration above depicts a machine learning model being deployed in a cloud environment that feeds into an application. In this basic scenario, a number of problems could arise that MLOps helps overcome. Because the production application relies on a specific cloud environment, there are latency requirements that the data scientists who developed the ML model don’t have access to. Operationalizing the model life cycle would make it possible for data scientists or engineers with deep knowledge of the model to identify and troubleshoot problems that arise in specific production environments. 

Not only are machine learning models trained in different environments from the production applications they are used in, but they also often rely on historical datasets that differ from the data used in production applications. With MLOps the entire data science team, from those developing the model to people working at the application level, have a means of sharing and requesting information and assistance. The rate at which data and markets change make it imperative that there is as little friction as possible between all key stakeholders and contributors that will come to depend on a given machine learning model. 

### Supporting the transition to Software 2.0

[Software 2.0](https://karpathy.medium.com/software-2-0-a64152b37c35) is the idea that software development will experience a paradigm shift as artificial intelligence increasingly plays a central role in writing AI models that power software applications. Under Software 1.0, development involves programmers writing explicit instructions using a specific programming language (e.g., Python, C++). Software 2.0 is far more abstract. Although people provide input data and set parameters, neural networks are difficult for humans to understand due to their sheer complexity— with typical networks containing millions of weights that impact outcomes (and sometimes billions or trillions). 

DevOps was built around Software 1.0’s reliance on specific instructions dictated by programmers using languages, but never considered the life cycle of a machine learning model that powers a variety of different applications. MLOps addresses the need for the process of managing software development to change alongside the software under development. As Software 2.0 becomes the new standard for computer-based problem solving, having the right tools and processes for managing model life cycles will make or break investments in new technology. Milvus is an open-source vector similarity search engine built to support the transition to Software 2.0 and manage model life cycles with MLOps.

![03.jpg](https://assets.zilliz.com/03_c63c501995.jpg)

###### *How Milvus supports the transition to Software 2.0.*

 

### Operationalizing AI at scale with Milvus

Milvus is a vector data management platform that was made specifically for storing, querying, updating, and maintaining massive, trillion-scale vector datasets. The platform powers vector similarity search and can integrate with widely adopted index libraries, including Faiss, NMSLIB, and Annoy. By pairing AI models that convert unstructured data to vectors with Milvus, applications spanning new drug development, biometric analysis, recommendation systems and much more can be created. 

[Vector similarity search](https://blog.milvus.io/vector-similarity-search-hides-in-plain-view-654f8152f8ab) is the go-to solution for unstructured data data processing and analytics, and vector data is quickly emerging as a core data type. A comprehensive data management system like Milvus facilitates operationalizing AI in many ways, including:

- Providing an environment for model training that ensures more aspects of development are done in one place, facilitating cross team collaboration, model governance, and more. 

- Offering a comprehensive set of APIs that support popular frameworks such as Python, Java, and Go, making it easy to integrate a common set of ML models. 

- Compatibility with Google Colaboratory, a Jupyter notebook environment that runs in a browser, simplifies the process of compiling Milvus from source code and running basic Python operations.

- Automated machine learning (AutoML) functionality makes it possible to automate the tasks associated with applying machine learning to real world problems. Not only does AutoML lead to efficiency improvements, but it makes it possible for non-experts to take advantage of machine learning models and techniques.


Regardless of the machine learning applications you’re building today, or the plans you have for applications in the future, Milvus is a flexible data management platform created with Software 2.0 and MLOps in mind. To learn more about Milvus or make contributions, find the project on [Github](https://github.com/milvus-io). To get involved with the community or ask questions, join our [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ) channel. Hungry for more content? Check out the following resources:

- [Milvus Is an Open-Source Scalable Vector Database](https://zilliz.com/blog/Milvus-Is-an-Open-Source-Scalable-Vector-Database)
- [Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search](https://zilliz.com/blog/Milvus-Was-Built-for-Massive-Scale-Think-Trillion-Vector-Similarity-Search)
- [Set Up Milvus in Google Colaboratory for Easy ML Application Building](https://zilliz.com/blog/Set-Up-Milvus-in-Google-Colaboratory-for-Easy-ML-Application-Building)

large_milvus.jpg

medium_milvus.jpg

small_milvus.jpg

thumbnail_milvus.jpg

milvus.jpg

MLOps is superseding DevOps as we make the transition to Software 2.0. Find  what model operations is, and how the open-source vector database Milvus supports it. 

Operationalize AI at Scale with Software 2.0, MLOps, and Milvus

Beike wanted to create a personalized home search experience. Discover how they leveraged Milvus to build an intelligent real estate search platform capable of providing tailored recommendations in near real-time. 

Artificial intelligence (AI) has [powerful applications](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#d62b) in real estate that are transforming the home search process. Tech savvy real estate professionals have been taking advantage of AI for years, recognizing its ability to help clients find the right home faster and simplify the process of purchasing property. The coronavirus pandemic has [accelerated](https://www.pwc.com/ca/en/industries/real-estate/emerging-trends-in-real-estate-2021/shifting-landscape-proptech.html) interest, adoption, and investement in property technology (or proptech) worldwide, suggesting it will play an increasingly greater role in the real estate industry moving forward. 

This article explores how Beike used vector similarity search to build a house hunting platform that provides personalized results and recommends listings in near real-time.

**Jump to:**

- [What is vector similarity search?](#what-is-vector-similarity-search)

- [How does Milvus manage vector data?](#how-does-milvus-manage-vector-data)

- [How does Beike use Milvus to make house hunting smarter?](#how-does-beike-use-milvus-to-make-house-hunting-smarter)

 

### What is vector similarity search?

[Vector similarity search](https://medium.com/unstructured-data-service/vector-similarity-search-hides-in-plain-view-654f8152f8ab) has applications spanning a wide variety of artificial intelligence, deep learning, and traditional vector calculation scenarios. The proliferation of AI technology is in part attributed to vector search and its ability to make sense of unstructured data, which includes things like images, video, audio, behavior data, documents, and much more.

Unstructured data makes up an estimated 80-90% of all data, and extracting insights from it is quickly becoming a requrement for businesses that want to remain competitive in an ever-changing world. Increasing demand for unstructured data analytics, rising compute power, and declining compute costs have made AI-enabled vector search more accessible than ever.

![beike-blog-img1.jpg](https://assets.zilliz.com/beike_blog_img1_2dc95cac08.jpg)
###### *Understanding the difference between structured and unstructured data.*

Traditionally, unstructured data has been a challenge to process and analyze at scale because it doesn't follow a predefined model or organizational structure. Neural networks (e.g., CNN, RNN, and BERT) make it possible to convert unstructured data into feature vectors, a numerical data format that can be easily interpreted by computers. Algorithms are then used to calculate similarity between vectors using metrics like cosine similarity or Euclidean distance.

Ultimately, vector similarity search is a broad term that desribes techniques for identifying similar things in massive datasets. Beike uses this technology to power an intelligent home search engine that automatically recommends listings based on individual user preferences, search history, and property criteria—accelerating the real estate search and purchase process. Milvus is an open-source vector database that connects information with algorithms, enabling Beike to develop and manage its AI real estate platform.

 

### How does Milvus manage vector data?

Milvus was built specifically for large-scale vector data management, and has applications spanning image and video search, chemical similarity analysis, personalized recommendation systems, conversational AI, and much more. Vector datasets stored in Milvus can be efficiently queried, with most implementations following this general process:

![beike-blog-img2.jpg](https://assets.zilliz.com/beike_blog_img2_d5abb58f95.jpg)
###### *How does Milvus manage vector data?*

 

### How does Beike use Milvus to make house hunting smarter?

Commonly described as China's answer to Zillow, [Beike](https://bj.ke.com/) is an online platform that allows real estate agents to list properties for rent or sale. To help improve the home search experience for house hunters, and to help agents close deals faster, the company built an AI-powered search engine for its listing database. Beike’s real estate listing database was converted into feature vectors then fed into Milvus for indexing and storage. Milvus is then used to conduct similarity search based on an input listing, search criteria, user profile, or other criteria.

For example, when searching for more homes similar to a given listing, features such as floor plan, size, orientation, interior finishings, paint colors, and more are extracted. Since the original database of property listings listing data has been [indexed](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212), searches can be conducted in mere milliseconds. Beike’s final product had an average query time of 113 milliseconds on a dataset containing over 3 million vectors. However, Milvus is capable of maintaining efficient speeds on trillion-scale datasets—making light work of this relatively small real estate database. In general, the system follows the following process:

1. Deep learning models (e.g., CNN, RNN, or BERT) convert unstructured data to feature vectors, which are then imported to Milvus.

2. Milvus stores and indexes the feature vectors.

3. Milvus returns similarity search results based on user queries. 

![milvus-overview-diagram.png](https://assets.zilliz.com/milvus_overview_diagram_d17cda0e47.png)

###### *An overview of Milvus.*

 

Beike’s intelligent real estate search platform is powered by a recommendation algorithm that calculates vector similarity using cosine distance. The system finds similar homes based on favorite listings and search criteria. At a high level, it works as follows:

1. Based on an input listing, characteristics such as floor plan, size, and orientation are used to extract 4 collections of feature vectors.

2. The extracted feature collections are used to perform similarity search in Milvus. Results of the query for each collection of vectors are a measure of similarity between the input listing and other similar listings. 

3. The search results from each of the 4 vector collections are compared then used to recommend similar homes. 

![beike-intelligent-house-platform-diagram.jpg](https://assets.zilliz.com/beike_intelligent_house_platform_diagram_6e278da118.jpg)

###### *An overview of Beike’s intelligent house hunting platform.*

 

As the figure above shows, the system implements an A/B table switching mechanism for updating data. Milvus stores the data for the first T days in table A, on day T+1 it starts to store data in table B, on day 2T+1, it starts to rewrite table A, and so forth. 

 

### To learn more about making things with Milvus, check out the following resources:

- [Building an AI-Powered Writing Assistant for WPS Office](https://zilliz.com/blog/Building-an-AI-Powered-Writing-Assistant-with-WPS-Office)

- [Making with Milvus: AI-Powered News Recommendation Inside Xiaomi's Mobile Browser](https://zilliz.com/blog/Making-with-Milvus-AI-Powered-News-Recommendation-Inside-Xiaomi-Mobile-Browser)

large_blog_realistate search.jpg

medium_blog_realistate search.jpg

small_blog_realistate search.jpg

thumbnail_blog_realistate search.jpg

blog_realistate search.jpg

AI is transforming the real estate industry, discover how intelligent proptech accelerates the home search and purchase process. 

Making With Milvus: AI-Infused Proptech for Personalized Real Estate Search

Learn how to use Milvus data migration tool to greatly improve efficiency of data management and reduce DevOps costs.

### Overview
[MilvusDM](https://github.com/milvus-io/milvus-tools) (Milvus Data Migration) is an open-source tool designed specifically for importing and exporting data files with Milvus. MilvusDM can greatly improve data mangement efficiency and reduce DevOps costs in the following ways:

- [Faiss to Milvus](#faiss-to-milvus): Import unzipped data from Faiss to Milvus.

- [HDF5 to Milvus](#hdf5-to-milvus): Import HDF5 files to Milvus.

- [Milvus to Milvus](#milvus-to-milvus): Migrate data from a source Milvus to a different target Milvus.

- [Milvus to HDF5](#milvus-to-hdf5): Save data in Milvus as HDF5 files.

![milvusdm blog 1.png](https://assets.zilliz.com/milvusdm_blog_1_199cbdebe7.png)
###### *Importing and exporting data files with Milvus.*

MilvusDM is hosted on [Github](https://github.com/milvus-io/milvus-tools) and can be easily installed by running the command line ```pip3 install pymilvusdm```. MilvusDM allows you to migrate data in a specific collection or partition. In the following sections, we will explain how to use each data migration type. 

 


### Faiss to Milvus

#### Steps
1.Download **F2M.yaml**: 

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/F2
```

2.Set the following parameters:

- ``` data_path ```: Data path (vectors and their corresponding IDs) in Faiss.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: Delete data before insertion if the collection or partition already exists.

- ```dest_collection_name```: Name of receiving collection for data import.

- ```dest_partition_name```: Name of receiving partition for data import. 

- ```collection_parameter```: Collection-specific information such as vector dimension, index file size, and distance metric.

```
F2M:
 milvus_version: 1.0.0
 data_path: '/home/data/faiss.index'
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'append' # 'skip/append/overwrite'
 dest_collection_name: 'test'
 dest_partition_name: ''
 collection_parameter:
 dimension: 256
 index_file_size: 1024
 metric_type: 'L2'
```
3.Run **F2M.yaml:**

```
$ milvusdm --yaml F2M.yaml
```

#### Sample Code

1.Read Faiss files to retrieve vectors and their corresponding IDs.

```
ids, vectors = faiss_data.read_faiss_data()
```

2.Insert the retrieved data into Milvus: 

```
insert_milvus.insert_data(vectors, self.dest_collection_name, self.collection_parameter, self.mode, ids, self.dest_partition_name)
```

 

### HDF5 to Milvus
#### Steps
1.Download **H2M.yaml**.

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/H2M.yaml
```

2.Set the following parameters:

- ```data_path```: Path to the HDF5 files. 

- ```data_dir```: Directory holding the HDF5 files.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: Delete data before insertion if the collection or partition already exists.

- ```dest_collection_name```: Name of receiving collection for data import.

- ```dest_partition_name```: Name of receiving partition for data import.

- ```collection_parameter```: Collection-specific information such as vector dimension, index file size, and distance metric. 

> Set either ```data_path``` or ```data_dir```. Do **not** set both. Use ```data_path``` to specify multiple file paths, or ```data_dir``` to specify the directory holding your data file.

```
H2M:
 milvus-version: 1.0.0
 data_path:
 - /Users/zilliz/float_1.h5
 - /Users/zilliz/float_2.h5
 data_dir:
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'overwrite' # 'skip/append/overwrite'
 dest_collection_name: 'test_float'
 dest_partition_name: 'partition_1'
 collection_parameter:
 dimension: 128
 index_file_size: 1024
 metric_type: 'L2'
```
3.Run **H2M.yaml:**

```
$ milvusdm --yaml H2M.yaml
```

#### Sample Code

1.Read the HDF5 files to retrieve vectors and their corresponding IDs:

```
vectors, ids = self.file.read_hdf5_data()
```

2.Insert the retrieved data into Milvus:

```
ids = insert_milvus.insert_data(vectors, self.c_name, self.c_param, self.mode, ids,self.p_name)
```

 

### Milvus to Milvus
#### Steps
1.Download **M2M.yaml**.

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2M.yaml
```

2.Set the following parameters:

- ```source_milvus_path```: Source Milvus work path.

- ```mysql_parameter```: Source Milvus MySQL settings. If MySQL is not used, set mysql_parameter as ''.

- ```source_collection```: Names of the collection and its partitions in the source Milvus.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: If the collection or partition already exists, delete the data before inserting it.Delete data before insertion if the collection or partition already exists.

```
M2M:
 milvus_version: 1.0.0
 source_milvus_path: '/home/user/milvus'
 mysql_parameter:
 host: '127.0.0.1'
 user: 'root'
 port: 3306
 password: '123456'
 database: 'milvus'
 source_collection:
 test:
 - 'partition_1'
 - 'partition_2'
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'skip' # 'skip/append/overwrite'
```

3.Run **M2M.yaml.**

```
$ milvusdm --yaml M2M.yaml
```

#### Sample Code

1.According to a specified collection or partition's metadata, read the files under **milvus/db** on your local drive to retrieve vectors and their corresponding IDs from the source Milvus. 

```
collection_parameter, _ = milvus_meta.get_collection_info(collection_name)
r_vectors, r_ids, r_rows = milvusdb.read_milvus_file(self.milvus_meta, collection_name, partition_tag) 
```

2.Insert the retrieved data into the target Milvus.


```
milvus_insert.insert_data(r_vectors, collection_name, collection_parameter, self.mode, r_ids, partition_tag)
```

 

### Milvus to HDF5
#### Steps
1.Download **M2H.yaml**:

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2H.yaml
```


2.Set the following parameters:

- ```source_milvus_path```: Source Milvus work path.

- ```mysql_parameter```: Source Milvus MySQL settings. If MySQL is not used, set mysql_parameter as ''.

- ```source_collection```: Names of the collection and its partitions in the source Milvus.

- ```data_dir```: Directory for holding the saved HDF5 files.

```
M2H:
 milvus_version: 1.0.0
 source_milvus_path: '/home/user/milvus'
 mysql_parameter:
 host: '127.0.0.1'
 user: 'root'
 port: 3306
 password: '123456'
 database: 'milvus'
 source_collection: # specify the 'partition_1' and 'partition_2' partitions of the 'test' collection.
 test:
 - 'partition_1'
 - 'partition_2'
 data_dir: '/home/user/data'
```

3.Run **M2H.yaml**:

```
$ milvusdm --yaml M2H.yaml
```

#### Sample Code
1.According to a specified collection or partition's metadata, read the files under **milvus/db** on your local drive to retrieve vectors and their corresponding IDs.

```
collection_parameter, version = milvus_meta.get_collection_info(collection_name)
r_vectors, r_ids, r_rows = milvusdb.read_milvus_file(self.milvus_meta, collection_name, partition_tag)
```

2.Save the retrieved data as HDF5 files.

```
data_save.save_yaml(collection_name, partition_tag, collection_parameter, version, save_hdf5_name)
```

### MilvusDM File Structure

The flow chart below shows how MilvusDM performs different tasks according to the YAML file it receives:

![milvusdm blog 2.png](https://assets.zilliz.com/milvusdm_blog_2_7824b16e5e.png)
###### *How does MilvusDM work?*

MilvusDM file structure:

- pymilvusdm

 - core

 - **milvus_client.py**: Performs client operations in Milvus.

 - **read_data.py**: Reads the HDF5 data files on your local drive. (Add your code here to support reading data files in other formats.)

 - **read_faiss_data.py**: Reads the data files in Faiss.

 - **read_milvus_data.py**: Reads the data files in Milvus.

 - **read_milvus_meta.py**: Reads the metadata in Milvus.

 - **data_to_milvus.py**: Creates collections or partitions based on parameters in YAML files and imports the vectors and the corresponding vector IDs into Milvus.

 - **save_data.py**: Saves the data as HDF5 files.

 - **write_logs.py**: Writes logs during runtime.

 - **faiss_to_milvus.py**: Imports data from Faiss into Milvus.

 - **hdf5_to_milvus.py**: Imports data in HDF5 files into Milvus.

 - **milvus_to_milvus.py**: Migrates data from a source Milvus to the target Milvus.

 - **milvus_to_hdf5.p**y: Exports data in Milvus and saves them as HDF5 files.

 - **main.py**: Performs corresponding tasks according to the received YAML file.

 - **setting.py**: Configurations relating to running the MilvusDM code.

- **setup.py**: Creates **pymilvusdm** file packages and uploads them to PyPI (Python Package Index).

 

### Recap

MilvusDM primarily handles migrating data in and out of Milvus, which includes Faiss to Milvus, HDF5 to Milvus, Milvus to Milvus, and Milvus to HDF5. 

The following features are planned for upcoming releases:

- Import binary data from Faiss to Milvus.

- Blocklist and allowlist for data migration between source Milvus and target Milvus.

- Merge and import data from multiple colletions or partitions in source Milvus into a new collection in target Milvus.

- Backup and recovery of the Milvus data.

The MilvusDM project is open sourced on [Github](https://github.com/milvus-io/milvus-tools). Any and all contributions to the project are welcome. Give it a star 🌟, and feel free to file an [issue](https://github.com/milvus-io/milvus-tools/issues) or submit your own code!

large_Generic Tool Announcement.jpg

medium_Generic Tool Announcement.jpg

small_Generic Tool Announcement.jpg

thumbnail_Generic Tool Announcement.jpg

Generic Tool Announcement.jpg

Introducing Milvus Data Migration Tool

iYUNDONG continually seeks to support its users with real-time image retrieval services. When it decided to develop its App, it chose Milvus to build its core image search system. Discover how Milvus, an open-source vector database, can help build an intelligent image retrieval system that can extract sport event highlights.

iYUNDONG is an Internet company aiming to engage more sport lovers and participants of events such as marathon races. It builds [artificial intelligence (AI)](https://en.wikipedia.org/wiki/Artificial_intelligence) tools that can analyze media captured during sporting events to automatically generate highlights. For example, by uploading a selfie, a user of the iYUNDONG sports App who took part in a sport event can instaneously retrieve his or her own photos or video clips from a massive media dataset of the event.

 

One of the key features of iYUNDONG App is called “Find me in motion“. Photographers usually take massive volumes of photos or videos during a sporting event such as a marathon race, and would upload the photos and videos in real time to the iYUNDONG media database. Marathon runners who want to see their highlighted moments can retrieve pictures including themselves simply by uploading one of their selfies. This saves them a lot of time because an image retrieval system in the iYUNDONG App does all the image matching. [Milvus](http://milvus.io/) is adopted by iYUNDONG to power this system as Milvus can greatly accelerate the retrieval process and return highly accurate results.

 

**Jump to:**
- [Difficulties and solutions](#difficulties-and-solutions)
- [What is Milvus](#what-is-milvus)
- [Why Milvus](#why-milvus)
- [System and Workflow](#system-and-workflow)
- [iYUNDONG App Interface](#iyundong-app-interface)
- [Conclusion](#conclusion)

 

### Difficulties and solutions

iYUNDONG faced the following issues and successfully found corresponding solutions when building its image retrieval system. 

- Event photos must be immediately available for search.

iYUNDONG developed a function called InstantUpload to ensure that event photos are available for search immediately after they are uploaded.

- Storage of massive datasets

Massive data such as photos and videos are uploaded to the iYUNDONG backend every millisecond. So iYUNDONG decided to migrate onto cloud storage systems including [AWS](https://aws.amazon.com/), [S3](https://aws.amazon.com/s3/?nc1=h_ls), and [Alibaba Cloud Object Storage Service (OSS)](https://www.alibabacloud.com/product/oss) for handling gargantuan volumes of unstructured data in a secure, fast and reliable way.

- Instant reading

In order to achieve instant reading, iYUNDONG developed its own sharding middleware to achieve horizontal scalability easily and mitigate the impact on the system from disk reading. In addition, [Redis](https://redis.io/) is used to serve as a caching layer to ensure consistent performance in situation of high concurrency.

- Instant extraction of facial features

In order to accurately and efficiently extract facial features from user-uploaded photos, iYUNDONG developed a proprietary image conversion algorithm which converts images into 128-dimensional feature vectors. Another issue encountered was that, oftentimes, many users and photographers uploaded images or videos simultaneously. So system engineers needed to take dynamic scalability into consideration when deploying the system. More specifically, iYUNDONG fully leveraged its elastic compute service (ECS) on the cloud to achieve dynamic scaling.

- Quick and large-scale vector search

iYUNDONG needed a vector database to store its large number of feature vectors extracted by AI models. According to its own unique business application scenario, iYUNDONG expected the vector database to be able to: 
1. Perform blazing fast vector retrieval on ultra-large datasets. 
2. Achieve mass storage at lower costs.

Initially, an average of 1 million images were processed annually, so iYUNDONG stored all its data for search in RAM. However, in the past two years, its business boomed and saw an exponential growth of unstructured data – the number of images in iYUNDONG’s database exceeded 60 milllion in 2019, meaning that there were more than 1 billion feature vectors that needed to be stored. A tremendous amount of data inevitably made the iYUNDONG system heavily-built and resource-consuming. So it had to continuously invest in hardware facilities to ensure high performance. Specifically, iYUNDONG deployed more search servers, larger RAM, and a better-performing CPU to achieve greater efficiency and horizontal scalability. However, one of the defects of this solution was that it drove the operating costs prohibitively high. Therefore, iYUNDONG started to explore a better solution to this issue and pondered on leveraging vector index libraries like Faiss to save costs and better steer its business. Finally iYUNDONG chose open-source vector database Milvus.

 

### What is Milvus

Milvus is an open-source vector database that is easy to use, highly flexible, reliable, and blazing fast. Combined with various deep learning models such as photo and voice recognition, video processing, natural language processing, Milvus can process and analyze unstructured data that are converted into vectors by using various AI algorithms. Below is the workflow of how Milvus processes all unstructured data:

● Unstructured data are converted into embedding vectors by deep learning models or other AI algorithms.

● Then embedding vectors are inserted into Milvus for storage. Milvus also builds indexes for those vectors.

● Milvus performs similarity search and returns accurate search results based on various business needs.

![iYUNDONG Blog 1.png](https://assets.zilliz.com/i_YUNDONG_Blog_1_d8abe065ae.png)
###### *An overview of Milvus.*

 

### Why Milvus
Since the end of 2019, iYUNDONG has carried out a series of testings on using Milvus to power its image retrieval system. The testing results turned out that Milvus outperforms other mainstream vector databases as it supports multiple indexes and can efficiently reduce RAM usage, significantly compressing the timeline for vector similarity search. 

Moreover, new versions of Milvus are released regularly. Over the testing period, Milvus has went through multiple version updates from v0.6.0 to v0.10.1.

Additionally, with its active open-source community and powerful out-of-the-box features, Milvus allows iYUNDONG to operate on a tight development budget.

 

### System and Workflow
iYUNDONG’s system extracts facial features by detecting faces in event photos uploaded by photographers first. Then those facial features are converted into 128-dimensional vectors and stored in the Milvus library. Milvus creates indexes for those vectors and can instantaneously return highly accurate results. 

Other additional information such as photo IDs and coordinates indicating the position of a face in a photo are stored in a third-party database.

Each feature vector has its unique ID in the Milvus library. iYUNDONG adopted the [Leaf algorithm](https://github.com/Meituan-Dianping/Leaf), a distributed ID generation service developed by [Meituan](https://about.meituan.com/en) basic R&D platform, to associate the vector ID in Milvus with its corresponing additional information stored in another database. By combining the feature vector and the additional information, the iYUNDONG system can return similar results upon user search.

 

### iYUNDONG App Interface
A series of latest sports events are listed on the homepage. By tapping one of the events, users can see the full details.

After tapping the button on the top of the photo gallery page, users can then upload a photo of their own to retrieve images of their highlights. 

![iyundong-interface.jpg](https://assets.zilliz.com/iyundong_interface_3da684d206.jpg)
###### *iYUNDONG app interface.*

 

### Conclusion
This article introduces how iYUNDONG App builds an intelligent image retrieval system that can return accurate search results based on user uploaded photos varying in resolution, size, clarity, angle, and other ways that complicate similarity search. With the help of Milvus, iYUNDONG App can successfully run millisecond-level queries on a database of 60+ million images. And accuracy rate of photo retrieval is constantly above 92%. Milvus makes it easier for iYUNDONG to create a powerful, enterprise-grade image retrieval system in a short time with limited resources.


Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

Products

Resources

Open Source

About

Blog