Create a personalized product recommender system using Milvus, an open-source vector database.

With the explosive growth of Internet data scale, the product quantity as well as category in the current mainstream e-commerce platform increase on the one hand, the difficulty for users to find the products they need surges on the other hand.

[Vipshop](https://www.vip.com/) is a leading online discount retailer for brands in China. The Company offers high-quality and popular branded products to consumers throughout China at a significant discount from retail prices. To optimize the shopping experience for their customers, the company decided to build a personalized search recommendation system based on user query keywords and user portraits.

The core function of the e-commerce search recommendation system is to retrieve suitable products from a large number of products and display them to users according to their search intent and preference. In this process, the system needs to calculate the similarity between products and users' search intent & preference, and recommends the TopK products with the highest similarity to users.

Data such as product information, user search intent, and user preferences are all unstructured data. We tried to calculate the similarity of such data using CosineSimilarity(7.x) of the search engine Elasticsearch (ES), but this approach has the following drawbacks.

- Long computational response time - the average latency to retrieve TopK results from millions of items is around 300 ms.

- High maintenance cost of ES indexes - the same set of indexes is used for both commodity feature vectors and other related data, which hardly facilitates the index construction, but produces a massive amount of data.

We tried to develop our own locally sensitive hash plug-in to accelerate the CosineSimilarity calculation of ES. Although the performance and throughput were significantly improved after the acceleration, the latency of 100+ ms was still difficult to meet the actual online product retrieval requirements.

After althorough research, we decided to use Milvus, an open source vector database, which is advantaged with the support for distributed deployment, multi-language SDKs, read/write separation, etc. compared to the commonly used standalone Faiss.

Using various deep learning models, we convert massive unstructured data into feature vectors, and import the vectors into Milvus. With the excellent performance of Milvus, our e-commerce search recommendation system can efficiently query the TopK vectors that are similar to the target vectors.

## Overall Architecture
![Architecture](https://zilliz-cms.s3.us-west-2.amazonaws.com/1_01551e7b2b.jpg)
As shown in the diagram, the system overall architecture consists of two main parts.

- Write process: the item feature vectors (hereinafter referred to as item vectors) generated by the deep learning model are normalized and written into MySQL. MySQL then reads the processed item feature vectors using the data synchronization tool (ETL) and import them into the vector database Milvus.

- Read process: The search service obtains user preference feature vectors (hereinafter referred to as user vectors) based on user query keywords and user portraits, queries similar vectors in Milvus and recalls TopK item vectors.

Milvus supports both incremental data update and entire data update. Each incremental update has to delete the existing item vector and insert a new item vector, meaning that every newly updated collection will be re-indexed. It better suits the scenario with more reads and fewer writes. Therefore, we choose the entire data update method. Moreover, it takes only a few minutes to write the entire data in batches of multiple partitions, which is equivalent to near real-time updates.

Milvus write nodes perform all write operations, including creating data collections, building indexes, inserting vectors, etc., and provide services to the public with write domain names. Milvus read nodes perform all read operations and provide services to the public with read-only domain names.

Whereas the current version of Milvus does not support switching collection aliases, we introduce Redis to seamlessly switch aliases between multiple entire data collections.

The read node only needs to read existing metadata information and vector data or indexes from MySQL, Milvus, and GlusterFS distributed file system, so the read capability can be horizontally extended by deploying multiple instances.

## Implementation Details

### Data Update

The data update service includes not only writing vector data, but also data volume detection of vectors, index construction, index pre-loading, alias control, etc. The overall process is as follows.
![Process](https://zilliz-cms.s3.us-west-2.amazonaws.com/2_6052b01334.jpg)
1. Assume that before building the entire data, CollectionA provides data service to the public, and the entire data being used is directed to CollectionA (`redis key1 = CollectionA`). The purpose of constructing entire data is to create a new collection CollectionB.

2. Commodity data check - check the item number of commodity data in the MySQL table, compare the commodity data with the existing data in CollectionA. Alert can be set in accordance with quantity or percentage. If the set quantity (percentage) is not reached, the entire data will not be built, and it will be regarded as the failure of this building operation, triggering the alert; once the set quantity (percentage) is reached, the entire data building process starts.

3. Start building the entire data - initialize the alias of the entire data being built, and update Redis. After updating, the alias of the entire data being built is directed to CollectionB (`redis key2 = CollectionB`).

4. Create a new entire collection - determine if CollectionB exists. If it does, delete it before creating a new one.

5. Data batch write-in - calculate the partition ID of each commodity data with its own ID using modulo operation, and write the data to multiple partitions to the newly created collection in batches.

6. Build and pre-load index - Create index (`createIndex()`) for the new collection. The index file is stored in distributed storage server GlusterFS. The system automatically simulates query on the new collection and pre-load the index for query warm-up. 

7. Collection data check - check the item number of data in the new collection, compare the data with the existing collection, and set alarms based on the quantity and percentage. If the set number (percentage) is not reached, the collection will not be switched and the building process will be regarded as a failure, triggering the alert.

8. Switching collection - Alias control. After updating Redis, the entire data alias being used is directed to CollectionB (`redis key1 = CollectionB`), the original Redis key2 is deleted, and the building process is completed.

### Data Recall

The Milvus partition data is called several times to calculate the similarity between user vectors, obtained based on user query keywords and user portrait, and item vector, and the TopK item vectors are returned after merging. The overall workflow schematic is as follow:
![workflow](https://zilliz-cms.s3.us-west-2.amazonaws.com/3_93518602b1.jpg)
The following table lists the main services involved in this process. It can be seen that the average latency for recalling TopK vectors is about 30 ms.

| **Service**              | **Role**                                                | **Input Parameters**                                         | **Output parameters** | **Response latency** |
| ------------------------ | ------------------------------------------------------- | ------------------------------------------------------------ | --------------------- | -------------------- |
| User vectors acquisition | Obtain user vector                                      | user info + query                                            | user vector           | 10 ms                |
| Milvus Search            | Calculate the vector similarity and return TopK results | user vector                                                  | item vector           | 10 ms                |
| Scheduling Logic         | Concurrent result recalling and merging                 | Multi-channel recalled item vectors and the similarity score | TopK items            | 10 ms                |

**Implementation process:**

1. Based on the user query keywords and user portrait, the user vector is calculated by the deep learning model.
2. Obtain the collection alias of the entire data being used from Redis currentInUseKeyRef and get Milvus CollectionName. This process is data synchronization service, i.e. switching alias to Redis after entire data update.
3. Milvus is called concurrently and asynchronously with the user vector to obtain data from different partitions of the same collection, and Milvus calculates the similarity between the user vector and the item vector, and returns the TopK similar item vectors in each partition.
4. Merge the TopK item vectors returned from each partition, and rank the results in the reverse order of similarity distance, which are calculated using the IP inner product (the greater the distance between the vectors, the more similar they are). The final TopK item vectors are returned.

## Looking Ahead

At present, Milvus-based vector search can be used steadily in the search of recommendation scenarios, and its high performance gives us more room to play in the dimensionality of the model and algorithm selection.

Milvus will play a crucial role as the middleware for more scenarios, including recall of main site search and all-scenario recommendations.

The three most anticipated features of Milvus in the future are as follows.

- Logic for collection alias switching - coordinate the switching across collections without external conponents.
- Filtering mechanism - Milvus v0.11.0 only supports ES DSL filtering mechanism in standalone version. The newly released Milvus 2.0 supports scalar filtering, and read/write separation.
- Storage support for Hadoop Distributed File System (HDFS) - The Milvus v0.10.6 we are using only supports POSIX file interface, and we have deployed GlusterFS with FUSE support as the storage backend. However, HDFS is a better choice in terms of performance and ease of scaling.

## Lessons Learned and Best Practices

1. For applications where read operations are the primary focus, a read-write separation deployment can significantly increase the processing power and improve performance.
2. The Milvus Java client lacks a reconnection mechanism because the Milvus client used by the recall service is resident in memory. We have to build our own connection pool to ensure the availability of the connection between the Java client and the server through heartbeat test.
3. Slow queries occur occasionally on Milvus. This is due to insufficient warm-up of the new collection. By simulating the query on the new collection, the index file is loaded into the memory to achieve the  index warm-up.
4. nlist is the index building parameter and nprobe is the query parameter. You need to get a reasonable threshold value according to your business scenario through pressure testing experiments to balance the retrieval performance and accuracy.
5. For static data scenario, it is more efficient to import all data into the collection first and build indexes later.

large_blog_shopping.jpg

medium_blog_shopping.jpg

small_blog_shopping.jpg

thumbnail_blog_shopping.jpg

blog_shopping.jpg

Milvus makes it easy to provide the personalized recommendation service to users.

Building a Personalized Product Recommender System with Vipshop and Milvus

Create an audio retrieval system using Milvus, an open-source vector database. Classify and analyze sound data in real time.

Sound is an information dense data type. Although it may feel antiquated in the era of video content, audio remains a primary information source for many people. Despite long-term decline in listeners, 83% of Americans ages 12 or older listened to terrestrial (AM/FM) radio in a given week in 2020 (down from 89% in 2019). Conversely, online audio has seen a steady rise in listeners over the past two decades, with 62% of Americans reportedly listening to some form of it on a weekly basis according to the same [Pew Research Center study](https://www.journalism.org/fact-sheet/audio-and-podcasting/).

As a wave, sound includes four properties: frequency, amplitude, wave form, and duration. In musical terminology, these are called pitch, dynamic, tone, and duration. Sounds also help humans and other animals perceive and understand our environment, providing context clues for the location and movement of objects in our surroundings.

As an information carrier, audio can be classified into three categories:

1. **Speech:** A communication medium composed of words and grammar. With speech recognition algorithms, speech can be converted to text.
2. **Music:** Vocal and/or instrumental sounds combined to produce a composition comprised of melody, harmony, rhythm, and timbre. Music can be represented by a score.
3. **Waveform:** A digital audio signal obtained by digitizing analog sounds. Waveforms can represent speech, music, and natural or synthesized sounds.

Audio retrieval can be used to search and monitor online media in real-time to crack down on infringement of intellectual property rights. It also assumes an important role in the classification and statistical analysis of audio data.

## Processing Technologies

Speech, music, and other generic sounds each have unique characteristics and demand different processing methods. Typically, audio is separated into groups that contain speech and groups that do not:

- Speech audio is processed by automatic speech recognition.
- Non-speech audio, including musical audio, sound effects, and digitized speech signals, are processed using audio retrieval systems. 

This article focuses on how to use an audio retrieval system to process non-speech audio data. Speech recognition is not covered in this article

### Audio feature extraction

Feature extraction is the most important technology in audio retrieval systems as it enables audio similarity search. Methods for extracting audio features are divided into two categories: 

- Traditional audio feature extraction models such as Gaussian mixture models (GMMs) and hidden Markov models (HMMs);
- Deep learning-based audio feature extraction models such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, encoding-decoding frameworks, attention mechanisms, etc.

Deep learning-based models have an error rate that is an order of magnitude lower than traditional models, and therefore are gaining momentum as core technology in the field of audio signal processing.

Audio data is usually represented by the extracted audio features. The retrieval process searches and compares these features and attributes rather than the audio data itself. Therefore, the effectiveness of audio similarity retrieval largely depends on the feature extraction quality.

In this article, [large-scale pre-trained audio neural networks for audio pattern recognition (PANNs)](https://github.com/qiuqiangkong/audioset_tagging_cnn) are used to extract feature vectors for its mean average accuracy (mAP) of 0.439 (Hershey et al., 2017).

After extracting the feature vectors of the audio data, we can implement high-performance feature vector analysis using Milvus.

### Vector similarity search

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following diagram depicts the general similarity search process using Milvus: 
   
![how-does-milvus-work.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/how_does_milvus_work_6926180543.png)
###### *Milvus' vector similarity search process.*

1. Unstructured data are converted to feature vectors by deep learning models and inserted into Milvus.
2. Milvus stores and indexes these feature vectors.
3. Upon request, Milvus searches and returns vectors most similar to the query vector.

## System overview

The audio retrieval system mainly consists of two parts: insert (black line) and search(red line).

![audio-retrieval-system.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/audio_retrieval_system_663a911c95.png)
###### *Audio retrieval system powered by Milvus.*

The sample dataset used in this project contains open-source game sounds, and the code is detailed in the [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search).

### Step 1: Insert data

Below is the example code for generating audio embeddings with the pre-trained PANNs-inference model and inserting them into Milvus, which assigns a unique ID to each vector embedding.

```
1 wav_name, vectors_audio = get_audio_embedding(audio_path)  
2 if vectors_audio:   
3     embeddings.append(vectors_audio)  
4     wav_names.append(wav_name)  
5 ids_milvus = insert_vectors(milvus_client, table_name, embeddings)  
6 
```

Then the returned **ids_milvus** are stored along with other relevant information (e.g. **wav_name**) for the audio data held in a MySQL database for subsequent processing. 

```
1 get_ids_correlation(ids_milvus, wav_name)  
2 load_data_to_mysql(conn, cursor, table_name)   
3 
```

### Step 2: Audio search

Milvus calculates the inner product distance between the pre-stored feature vectors and the input feature vectors, extracted from the query audio data using the PANNs-inference model, and returns the **ids_milvus** of similar feature vectors, which correspond to the audio data searched.

```
1 _, vectors_audio = get_audio_embedding(audio_filename)   
2 results = search_vectors(milvus_client, table_name, [vectors_audio], METRIC_TYPE, TOP_K)  
3 ids_milvus = [x.id for x in results[0]]  
4 audio_name = search_by_milvus_ids(conn, cursor, ids_milvus, table_name)   
5
````

## API reference and demo

### API

This audio retrieval system is built with open-source code. Its main features are audio data insertion and deletion. All APIs can be viewed by typing **127.0.0.1:<port>/docs** in the browser. 





### Demo

We host a [live demo](https://zilliz.com/solutions ) of the Milvus-based audio retrieval system online that you can try out with your own audio data. 

![audio-search-demo.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/audio_search_demo_cae60625db.png)
###### *Audio search demo powered by Milvus.*

## Conclusion

Living in the era of big data, people find their lives abound with all sorts of information. To make better sense of it, traditional text retrieval no long cuts it. Today's information retrieval technology is in urgent need of the retrieval of various unstructured data types, such as videos, images, and audio.

Unstructured data, which is difficult for computers to process, can be converted into feature vectors using deep learning models. This converted data can easily be processed by machines, enabling us to analyze unstructured data in ways our predecessors were never able to. Milvus, an open-source vector database, can efficiently process the feature vectors extracted by AI models and provides a variety of common vector similarity calculations.

## References
Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B. and Slaney, M., 2017, March. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131-135, 2017 

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).

- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).

- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_audio search.jpg

medium_blog_audio search.jpg

small_blog_audio search.jpg

thumbnail_blog_audio search.jpg

blog_audio search.jpg

Audio retrieval with Milvus makes it possible to classify and analyze sound data in real time.

Audio Retrieval Based on Milvus

MilMil helps Milvus users solve problems faster, and was built by and for the Milvus open-source community using Milvus—an advanced vector database built for AI and similarity search. 

Open-source communities are creative and collaborative spaces. In that vein, the Milvus open-source community recently created MilMil—a Milvus FAQ chatbot built by and for Milvus users. MilMil is available 24/7 at [Milvus.io](https://milvus.io/) to answer common questions about Milvus, the world's most advanced open-source vector database. 

This question answering system not only helps solve common problems Milvus users encounter more quickly, but identifies new problems based on user submissions. MilMil's database includes questions users have asked since the project was first released under an open-source license in 2019. Questions are stored in two collections, one for Milvus 1.x and earlier and another for Milvus 2.0.

MilMil is currently only available in English. 

## How does MilMil work?

MilMil relies on the *sentence-transformers/paraphrase-mpnet-base-v2* model to obtain vector representations of the FAQ database, then Milvus is used for vector similarity retrieval to return semantically similar questions. 

First, the FAQ data is converted into semantic vectors using BERT, a natural language processing (NLP) model. The embeddings are then inserted into Milvus and each one assigned a unique ID. Finally, the questions and answers are inserted into PostgreSQL, a relational database, together with their vector IDs.

When users submit a question, the system converts it into a feature vector using BERT. Next it searches Milvus for five vectors that are most similar to the query vector and retrieves their IDs. Finally, the questions and answers that correspond with the retrieved vector IDs are returned to the user.

![system-process.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/system_process_dca67a80a6.png)
###### *The blue line represents the data insertion process, and the black line represents the query process.*

See the [question answering system](https://github.com/milvus-io/bootcamp/tree/master/solutions/question_answering_system) project in the Milvus bootcamp to explore the code used to build AI chatbots.

## Ask MilMil about Milvus

To chat with MilMil, navigate to any page on [Milvus.io](https://milvus.io/) and click the bird icon in the lower-right corner. Type your question into the text input box and hit send. MilMil will get back to you in milliseconds! Additionally, the dropdown list in the upper-left corner can be used to switch between technical documentation for different versions of Milvus. 

![milvus-chatbot-icon.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/milvus_chatbot_icon_f3c25708ca.png)
###### *Click on the Milvus icon in the lower right corner to use Milvus FAQ Chatbot.*

After submitting a question, the bot immediately returns three questions that are semantically similar to the query question. You can click "See answer" to browse potential answers to your question, or click "See more" to view more questions related to your search. If a suitable answer is unavailable,  click "Put in your feedback here" to ask your question along with an email address. Help from the Milvus community will arrive shortly!

![chatbot_UI.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/chatbot_UI_0f4a7655d4.png)
###### *MilMil's search results of "The way to install Milvus."*

Give MilMil a try and let us know what you think. All questions, comments, or any form of feedback are welcome.

## Don't be a stranger
- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_milmil.jpg

medium_milmil.jpg

small_milmil.jpg

thumbnail_milmil.jpg

milmil.jpg

Using open-source vector search tools to build a question answering service. 

MilMil: A Milvus-powered FAQ Chatbot that Answers Questions About Milvus

A new and improved Milvus bootcamp offers updated guides and easier to follow code examples. Discover how to create your own similarity search solutions with the world's most advanced vector database.

With the release of Milvus 2.0, the team has revamped the Milvus [bootcamp](https://github.com/milvus-io/bootcamp). The new and improved bootcamp offers updated guides and easier to follow code examples for a variety of use cases and deployments. Additionally, this new version is updated for [Milvus 2.0](https://zilliz.com/blog/milvus2.0-redefining-vector-database), a reimagined version of the world's most advanced vector databse.
 

Jump to:
- [Stress test your system against 1M and 100M dataset benchmarks](#stress-test-your-system-against-1m-and-100m-dataset-benchmarks)
- [Explore and build popular vector similarity search solutions](#explore-and-build-popular-vector-similarity-search-solutions)
- [Quickly deploy a fully built application on any system](#quickly-deploy-a-fully-built-application-on-any-system)
- [Use scenario specific notebooks to easily deploy pre-configured applications](#use-scenario-specific-notebooks-to-easily-deploy-pre-configured-applications)
- [Learn how to deploy Milvus in different environments](#learn-how-to-deploy-milvus-in-different-environments)

 

### Stress test your system against 1M and 100M dataset benchmarks 
The [benchmark directory](https://github.com/milvus-io/bootcamp/tree/master/benchmark_test) contains 1 million and 100 million vector benchmark tests that indicate how your system will react to differently sized datasets.

 

### Explore and build popular vector similarity search solutions

The [solution directory](https://github.com/milvus-io/bootcamp/tree/master/solutions) includes the most popular vector similarity search use cases. Each use case contains a notebook solution and a docker deployable solution. Use cases include:

- [Image similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search)
- [Video similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/video_similarity_search)
- [Audio similarity search](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search)
- [Recommendation system](https://github.com/milvus-io/bootcamp/tree/master/solutions/recommendation_system)
- [Molecular search](https://github.com/milvus-io/bootcamp/tree/master/solutions/molecular_similarity_search)
- [Question answering system](https://github.com/milvus-io/bootcamp/tree/master/solutions/question_answering_system)

 

### Quickly deploy a fully built application on any system

The quick deploy solutions are dockerized solutions that allow users to deploy fully built applications on any system. These solutions are ideal for brief demos, but require additional work to customize and understand compared to notebooks.

 

### Use scenario specific notebooks to easily deploy pre-configured applications

The notebooks contain a simple example of deploying Milvus to solve the problem in a given use case. Each of the examples are able to be run from start to finish without the need to manage files or configurations. Each notebook is also easy to follow and modifiable, making them ideal base files for other projects.

 

### Image similarity search notebook example
Image similarity search is one of the core ideas behind many different technologies, including autonomous cars recognizing objects. This example explains how to easily build computer vision programs with Milvus.


This [notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/reverse_image_search/reverse_image_search.ipynb) revolves around three things:
- Milvus server 
- Redis server (for metadata storage)
- Pretrained Resnet-18 model. 


#### Step 1: Download required packages
Begin by downloading all the required packages for this project. This notebook includes a table listing the packages to use.

```
pip install -r requirements.txt
```

#### Step 2: Server startup
After the packages are installed, start the servers and ensure both are running properly. Be sure to follow the correct instructions for starting the [Milvus](https://milvus.io/docs/v2.0.0/install_standalone-docker.md) and [Redis](https://hub.docker.com/_/redis) servers.

#### Step 3: Download project data
By default, this notebook pulls a snippet of the VOCImage data for use as an example, but any directory with images should work as long as it follows the file structure that can be seen at the top of the notebook.

```
! gdown "https://drive.google.com/u/1/uc?id=1jdudBiUu41kL-U5lhH3ari_WBRXyedWo&export=download"
! tar -xf 'VOCdevkit.zip'
! rm 'VOCdevkit.zip'
```

#### Step 4: Connect to the servers
In this example, the servers are running on the default ports on the localhost.

```
connections.connect(host="127.0.0.1", port=19537)
red = redis.Redis(host = '127.0.0.1', port=6379, db=0)
```

#### Step 5: Create a collection
After starting the servers, create a collection in Milvus for storing all the vectors. In this example, the dimension size is set to 512, the size of the resnet-18 output, and the similarity metric is set to the Euclidean distance (L2). Milvus supports a variety of different [similarity metrics](https://milvus.io/docs/metric.md#floating).

```
collection_name = "image_similarity_search"
dim = 512
default_fields = [
 schema.FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
 schema.FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
]
default_schema = schema.CollectionSchema(fields=default_fields, description="Image test collection")
collection = Collection(name=collection_name, schema=default_schema)
```

#### Step 6: Build an index for the collection
Once the collection is made, build an index for it. In this case, the IVF_SQ8 index is used. This index requires the 'nlist' parameter, which tells Milvus how many clusters to make within each datafile (segment). Different [indices](https://milvus.io/docs/index.md#CPU) require different parameters.

```
default_index = {"index_type": "IVF_SQ8", "params": {"nlist": 2048}, "metric_type": "L2"}
collection.create_index(field_name="vector", index_params=default_index)
collection.load()
```
#### Step 7: Set up model and data loader
After the IVF_SQ8 index is built, set up the neural network and data loader. The pretrained pytorch resnet-18 used in this example is sans its last layer, which compresses vectors for classification and may lose valuable information.

```
model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet18', pretrained=True)
encoder = torch.nn.Sequential(*(list(model.children())[:-1]))
```

The dataset and data loader needs to be modified so that they are able to preprocess and batch the images while also providing the file paths of the images. This can be done with a slightly modified torchvision dataloader. For preprocessing, the images need to be cropped and normalized due to the resnet-18 model being trained on a specific size and value range. 

```
dataset = ImageFolderWithPaths(data_dir, transform=transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]))

dataloader = torch.utils.data.DataLoader(dataset, num_workers=0, batch_si
```

#### Step 8: Insert vectors into the collection
With the collection setup, the images can be processed and loaded into the created collection. First the images are pulled by the dataloader and run through the resnet-18 model. The resulting vector embeddings are then inserted into Milvus, which returns a unique ID for each vector. The vector IDs and image file paths are then inserted as key-value pairs into the Redis server.

```
steps = len(dataloader)
step = 0
for inputs, labels, paths in dataloader:
 with torch.no_grad():
 output = encoder(inputs).squeeze()
 output = output.numpy()

 mr = collection.insert([output.tolist()])
 ids = mr.primary_keys
 for x in range(len(ids)):
 red.set(str(ids[x]), paths[x])
 if step%5 == 0:
 print("Insert Step: " + str(step) + "/" + str(steps))
 step += 1
```

#### Step 9: Conduct a vector similarity search
Once all of the data is inserted into Milvus and Redis, the actual vector similarity search can be performed. For this example, three randomly selected images are pulled out of the Redis server for a vector similarity search. 

```
random_ids = [int(red.randomkey()) for x in range(3)]
search_images = [x.decode("utf-8") for x in red.mget(random_ids)]
```
These images first go through the same preprocessing that is found in Step 7 and are then pushed through the resnet-18 model. 

```
transform_ops = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
 
embeddings = [transform_ops(Image.open(x)) for x in search_images]
embeddings = torch.stack(embeddings, dim=0)
 
with torch.no_grad():
 embeddings = encoder(embeddings).squeeze().numpy()
```
Then the resulting vector embeddings are used to perform a search. First, set the search parameters, including the name of the collection to search, nprobe (the number of the clusters to search), and top_k (the number of returned vectors). In this example, the search should be very quick. 

```
search_params = {"metric_type": "L2", "params": {"nprobe": 32}}
start = time.time()
results = collection.search(embeddings, "vector", param=search_params, limit=3, expr=None)
end = time.time() - start
```

#### Step 10: Image search results 
The vector IDs returned from the queries are used to find the corresponding images. Matplotlib is then used to display the image search results.
 

![pic1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/pic1_c8652c7fae.png)
![pic2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/pic2_355b054161.png)
![pic3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/pic3_01780c6aac.png)

 

### Learn how to deploy Milvus in different enviroments
The [deployments section](https://github.com/milvus-io/bootcamp/tree/master/deployments) of the new bootcamp contains all the information for using Milvus in different environments and setups. It includes deploying Mishards, using Kubernetes with Milvus, load balancing, and more. Each environment has a detailed step by step guide explaining how to get Milvus working in it.

 

### Don't be a stranger
- Read our our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to Milvus, the world’s most popular vector database, on [Github](http://page/).

thumbnail_cover.png

cover.png

Build, test, and customize vector similarity search solutions with Milvus, an open-source vector database.

Quickly Test and Deploy Vector Search Solutions with the Milvus 2.0 Bootcamp

Mozat is a Singapore-based startup that specializes in social media, lifestyle, and communication applications. Learn how the company uses Milvus to create an AI-powered image search system for its fashion app, Stylepedia.

![stylepedia-1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_1_5f239a8d48.png)

Founded in 2003, [Mozat](http://www.mozat.com/home) is a start-up headquartered in Singapore with offices in China and Saudi Arabia. The company specializes in building social media, communication, and lifestyle applications. [Stylepedia](https://stylepedia.com/) is a wardrobe app built by Mozat that helps users discover new styles and connect with other people that are passionate about fashion. Its key features include the ability to curate a digital closet, personalized style recommendations, social media functionality, and an image search tool for finding similar items to something seen online or in real life.

[Milvus](https://milvus.io) is used to power the image search system within Stylepedia. The app deals with three image types: user images, product images, and fashion photographs. Each image can include one or more items, further complicating each query. To be useful, an image search system must be accurate, fast, and stable, features that lay a solid technical foundation for adding new functionality to the app such as outfit suggestions and fashion content recommendations.

## System overview

![stylepedia-system-process.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_system_process_8e7e2ab3e4.png)
###### *System process diagram.*

The image search system is divided into offline and online components. 

Offline, images are vectorized and inserted into a vector database (Milvus). In the data workflow, relevant product images and fashion photographs are converted into 512-dimensional feature vectors using object detection and feature extraction models. The vector data is then indexed and added to the vector database.

Online, the image database is queried and similar images are returned to the user. Similar to the off-line component, a query image is processed by object detection and feature extraction models to obtain a feature vector. Using the feature vector, Milvus searches for TopK similar vectors and obtains their corresponding image IDs. Finally, after post-processing (filtering, sorting, etc.), a collection of images similar to the query image are returned.

## Implementation

The implementation breaks down into four modules:
1. Garment detection
2. Feature extraction
3. Vector similarity search
4. Post-processing

### Garment detection

In the garment detection module, [YOLOv5](https://pytorch.org/hub/ultralytics_yolov5/), a one-stage, anchor-based target detection framework, is used as the object detection model for its small size and real-time inference. It offers four model sizes (YOLOv5s/m/l/x), and each specific size has pros and cons. The larger models will perform better (higher precision) but require a lot more computing power and run slower. Because the objects in this case are relatively large items and easy to detect, the smallest model, YOLOv5s, suffices.

Clothing items in each image are recognized and cropped out to serve as the feature extraction model inputs used in subsequent processing. Simultaneously, the object detection model also predicts the garment classification according to predefined classes (tops, outerwear, trousers, skirts, dresses, and rompers).

### Feature extraction

The key to similarity search is the feature extraction model. Cropped clothes images are embedded into 512-dimensional floating point vectors that represent their attributes in a machine readable numeric data format. The [deep metric learning (DML)](https://github.com/kdhht2334/Survey_of_Deep_Metric_Learning) methodology is adopted with [EfficientNet](https://arxiv.org/abs/1905.11946) as the backbone model.

Metric learning aims to train a CNN-based nonlinear feature extraction module (or an encoder) to reduce the distance between the feature vectors corresponding to the same class of samples, and increase the distance between the feature vectors corresponding to different classes of samples. In this scenario, the same class of samples refers to the same piece of clothing.

EfficientNet takes into account both speed and precision when uniformly scaling network width, depth, and resolution. EfficientNet-B4 is used as the feature extraction network, and the output of the ultimate fully connected layer is the image features needed to conduct vector similarity search.

### Vector similarity search

Milvus is an open-source vector database that supports create, read, update, and delete (CRUD) operations as well as near real-time search on trillion-byte datasets. In Stylepedia, it is used for large-scale vector similarity search because it is highly elastic, stable, reliable, and lightening fast. Milvus extends the capabilities of widely used vector index libraries (Faiss, NMSLIB, Annoy, etc.), and provides a set of simple and intuitive APIs that allow users to select the ideal index type for a given scenario.

Given the scenario requirements and data scale, Stylepedia's developers used the CPU-only distribution of Milvus paired with the HNSW index. Two indexed collections, one for products and the other for fashion photographs, are built to power different application functionalities. Each collection is further divided into six partitions based on the detection and classification results to narrow the search scope. Milvus performs search on tens of millions of vectors in milliseconds, providing optimal performance while keeping development costs low and minimizing resource consumption.

### Post-processing

To improve the similarity between the image retrieval results and the query image, we use color filtering and key label (sleeve length, clothes length, collar style, etc.) filtering to filter out ineligible images. In addition, an image quality assessment algorithm is used to make sure that higher quality images are presented to users first.

## Application

### User uploads and image search

Users can take pictures of their own clothes and upload them to their Stylepedia digital closet, then retrieve product images most similar to their uploads.

![stylepedia-search-results.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_search_results_0568e20dc0.png)
###### *Search results of a denim jacket image.*

### Outfit suggestions

By conducting similarity search on the Stylepedia database, users can find fashion photographs that contain a specific fashion item. These could be new garments someone is thinking about purchasing, or something from their own collection that could be worn or paired differently. Then, through the clustering of the items it is often paired with, outfit suggestions are generated. For example, a black biker jacket can go with a variety of items, such as a pair of black skinny jeans. Users can then browse relevant fashion photographs where this match occurs in the selected formula.

![stylepedia-jacket-outfit.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_jacket_outfit_e84914da9e.png)
###### *Outfit ideas for a black biker jacket.*
![stylepedia-jacket-snapshot.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_jacket_snapshot_25f53cc09b.png)
###### *A spread of snapshots featuring a black biker jacket + black skinny jeans match.*

### Fashion photograph recommendations

Based on a user's browsing history, likes, and the contents of their digital closet, the system calculates similarity and provides customized fashion photograph recommendations that may be of interest.

![stylepedia-user-wardrobe.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_user_wardrobe_6770c856b9.png)
###### *Items in user's digital closet.*
![stylepedia-streetsnap-rec.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/stylepedia_streetsnap_rec_901601a34d.png)
###### *Fashion photograph recommendations that match user preferences and tastes.*

By combining deep learning and computer vision methodologies, Mozat was able to build a fast, stable, and accurate image similarity search system using Milvus to power various features in the Stylepedia app.

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).


large_mozat-blog.jpg

medium_mozat-blog.jpg

small_mozat-blog.jpg

thumbnail_mozat-blog.jpg

mozat-blog.jpg

Discover how Milvus, an open-source vector database, is used by Mozat to power a fashion app that offers personalized style recommendations and an image search system.

Building a Wardrobe and Outfit Planning App with Milvus

Learn how to build a Milvus cluster based on JuiceFS, a shared file system designed for cloud-native environments.

Collaborations between open-source communities are a magical thing. Not only do passionate, intelligent, and creative volunteers keep open-source solutions innovative, they also work to bring different tools together in interesting and useful ways. [Milvus](https://milvus.io/), the world's most popular vector database, and [JuiceFS](https://github.com/juicedata/juicefs), a shared file system designed for cloud-native environments, were united in this spirit by their respective open-source communities. This article explains what JuiceFS is, how to build a Milvus cluster based on JuiceFS shared file storage, and the performance users can expect using this solution.

## **What is JuiceFS?**

JuiceFS is a high-performance, open-source distributed POSIX file system, which can be built on top of Redis and S3. It was designed for cloud-native environments and supports managing, analyzing, archiving, and backing up data of any type. JuiceFS is commonly used for solving big data challenges, building artificial intelligence (AI) applications, and log collection. The system also supports data sharing across multiple clients and can be used directly as shared storage in Milvus.

After data, and its corresponding metadata, are persisted to object storage and [Redis](https://redis.io/) respectively, JuiceFS serves as a stateless middleware. Data sharing is realized by enabling different applications to dock with each other seamlessly through a standard file system interface. JuiceFS relies on Redis, an open-source in-memory data store, for metadata storage. Redis is used because it guarantees atomicity and provides high performance metadata operations. All data is stored in object storage through the JuiceFS client. The architecture diagram is as follows:

![juicefs-architecture.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/juicefs_architecture_2023b37a4e.png)
###### *Overall architecture of JuiceFS.*

## **Build a Milvus cluster based on JuiceFS**

A Milvus cluster built with JuiceFS (see architecture diagram below) works by splitting upstream requests using Mishards, a cluster sharding middleware, to cascade the requests down to its sub-modules. When inserting data, Mishards allocates upstream requests to the Milvus write node, which stores newly inserted data in JuiceFS. When reading data, Mishards loads the data from JuiceFS through a Milvus read node to memory for processing, then collects and returns results from sub-services upstream.

![milvus-cluster-built-with-juicefs.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/milvus_cluster_built_with_juicefs_3a43cd262c.png)
###### *Architecture of Milvus cluster built with JuiceFS.*

### **Step 1: Launch MySQL service**

Launch the MySQL service on **any** node in the cluster. For details, see [Manage Metadata with MySQL](https://milvus.io/docs/v1.1.0/data_manage.md).

### **Step 2: Create a JuiceFS file system**

For demonstration purposes, the pre-compiled binary JuiceFS program is used. Download the correct [installation package](https://github.com/juicedata/juicefs/releases) for your system and follow the JuiceFS [Quick Start Guide](https://github.com/juicedata/juicefs/blob/main/docs/en/quick_start_guide.md) for detailed install instructions. To create a JuiceFS file system, first set up a Redis database for metadata storage. It is recommended that for public cloud deployments you host the Redis service on the same cloud as the application. Additionally, set up object storage for JuiceFS. In this example, Azure Blob Storage is used; however, JuiceFS supports almost all object services. Select the object storage service that best suits the demands of your scenario.

After configuring the Redis service and object storage, format a new file system and mount JuiceFS to the local directory:

```
1 $ export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=XXX;AccountKey=XXX;EndpointSuffix=core.windows.net"
2 $ ./juicefs format \
3 --storage wasb \
4 --bucket https://<container> \
5 ... \
6 localhost test #format
7 $ ./juicefs mount -d localhost ~/jfs #mount
8
```

> If the Redis server is not running locally, replace the localhost with the following address: `redis://<user:password>@host:6379/1`.

When the installation succeeds, JuiceFS returns the shared storage page **/root/jfs**.

![installation-success.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/installation_success_9d05279ecd.png)

### **Step 3: Start Milvus**

All the nodes in the cluster should have Milvus installed, and each Milvus node should be configured with read or write permission. Only one Milvus node can be configured as write node, and the rest must be read nodes. First, set the parameters of sections `cluster` and `general` in the Milvus system configuration file **server_config.yaml**:

**Section** `cluster` 

| **Parameter** | **Description** | **Configuration** |
| :------------ | :----------------------------- | :---------------- |
| `enable` | Whether to enable cluster mode | `true` |
| `role` | Milvus deployment role | `rw`/`ro` |

**Section** `general` 

```
# meta_uri is the URI for metadata storage, using MySQL (for Milvus Cluster). Format: mysql://<username:password>@host:port/database
general:
 timezone: UTC+8
 meta_uri: mysql://root:milvusroot@host:3306/milvus
```


During installation, the configured JuiceFS shared storage path is set as **/root/jfs/milvus/db**.

```
1 sudo docker run -d --name milvus_gpu_1.0.0 --gpus all \ 
2 -p 19530:19530 \ 
3 -p 19121:19121 \ 
4 -v /root/jfs/milvus/db:/var/lib/milvus/db \ #/root/jfs/milvus/db is the shared storage path 
5 -v /home/$USER/milvus/conf:/var/lib/milvus/conf \ 
6 -v /home/$USER/milvus/logs:/var/lib/milvus/logs \ 
7 -v /home/$USER/milvus/wal:/var/lib/milvus/wal \ 
8 milvusdb/milvus:1.0.0-gpu-d030521-1ea92e 
9
```

After the installation completes, start Milvus and confirm that it is launched properly.
Finally, start the Mishards service on **any** of the nodes in the cluster. The image below shows a successful launch of Mishards. For more information, refer to the GitHub [tutorial](https://github.com/milvus-io/bootcamp/tree/new-bootcamp/deployments/juicefs).

![mishards-launch-success.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/mishards_launch_success_921695d3a8.png)

## **Performance benchmarks**

Shared storage solutions are usually implemented by network-attached storage (NAS) systems. Commonly used NAS systems types include Network File System (NFS) and Server Message Block (SMB). Public cloud platforms generally provide managed storage services compatible with these protocols, such as Amazon Elastic File System (EFS).

Unlike traditional NAS systems, JuiceFS is implemented based on Filesystem in Userspace (FUSE), where all data reading and writing takes place directly on the application side, further reducing access latency. There are also features unique to JuiceFS that cannot be found in other NAS systems, such as data compression and caching.

Benchmark testing reveals that JuiceFS offers major advantages over EFS. In the metadata benchmark (Figure 1), JuiceFS sees I/O operations per second (IOPS) up to ten times higher than EFS. Additionally, the I/O throughput benchmark (Figure 2) shows JuiceFS outperforms EFS in both single- and multi-job scenarios.

![performance-benchmark-1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/performance_benchmark_1_b7fcbb4439.png) 
###### *Figure 1. Metadata benchmark*
![performance-benchmark-2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/performance_benchmark_2_e311098123.png)
###### *Figure 2. Sequential read/write benchmark.*

Additionally, benchmark testing shows first query retrieval time, or time to load newly inserted data from disk to memory, for the JuiceFS-based Milvus cluster is just 0.032 seconds on average, indicating that data is loaded from disk to memory almost instantaneously. For this test, first query retrieval time is measured using one million rows of 128-dimensional vector data inserted in batches of 100k at intervals of 1 to 8 seconds.

JuiceFS is a stable and reliable shared file storage system, and the Milvus cluster built on JuiceFS offers both high performance and flexible storage capacity.

## **Learn more about Milvus**

Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

![writer bio-changjian gao.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/writer_bio_changjian_gao_68018f7716.png)
![writer bio-jingjing jia.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/writer_bio_jingjing_jia_a85d1c2e3b.png)

large_JuiceFS-blog-cover.jpg

medium_JuiceFS-blog-cover.jpg

small_JuiceFS-blog-cover.jpg

thumbnail_JuiceFS-blog-cover.jpg

JuiceFS-blog-cover.jpg

Building a Milvus Cluster Based on JuiceFS

Building a news content recommendation system with semantic vectors is easy with Milvus, an open-source vector database. 

With [71% of Americans](https://www.socialmediatoday.com/news/new-research-shows-that-71-of-americans-now-get-news-content-via-social-pl/593255/) getting their news recommendations from social platforms, personalized content has quickly become how new media is discovered. Whether people are searching for specific topics, or interacting with recommended content, everything users see is optimized by algorithms to improve click-through rates (CTR), engagement, and relevance. Sohu is a NASDAQ-listed Chinese online media, video, search and gaming group. It leveraged [Milvus](https://milvus.io/), an open-source vector database built by [Zilliz](https://zilliz.com/), to build a semantic vector search engine inside its news app. This article explains how the company used user profiles to fine-tune personalized content recommendations over time, improving user experience and engagement.

## Recommending content using semantic vector search

Sohu News user profiles are built from browsing history and adjusted as users search for, and interact with, news content. Sohu’s recommender system uses semantic vector search to find relevant news articles. The system works by identifying a set of tags that are expected to be of interest to each user based on browsing history. It then quickly searches for relevant articles and sorts the results by popularity (measured by average CTR), before serving them to users. 

The New York Times alone publishes [230 pieces of content](https://www.theatlantic.com/technology/archive/2016/05/how-many-stories-do-newspapers-publish-per-day/483845/) a day, which offers a glimpse at the magnitude of new content an effective recommendation system must be capable of processing. Ingesting large volumes of news demands millisecond similarity search and hourly matching of tags to new content. Sohu chose Milvus because it processes massive datasets efficiently and accurately, reduces memory usage during search, and supports high-performance deployments.

## Understanding a news recommendation system workflow

Sohu’s semantic vector search-based content recommendation relies on the Deep Structured Semantic Model (DSSM), which uses two neural networks to represent user queries and news articles as vectors. The model calculates the cosine similarity of the two semantic vectors, then the most similar batch of news is sent to the recommendation candidates pool. Next, news articles are ranked based on their estimated CTR, and those with the highest predicted click-through rate are displayed to users.

### Encoding news articles into semantic vectors with BERT-as-service

To encode news articles into semantic vectors, the system uses the [BERT-as-service](https://github.com/hanxiao/bert-as-service.git) tool. If the word count of any piece of content exceeds 512 while using this model, information loss occurs during the embedding process. To help overcome this, the system first extracts a summary and encodes it into a 768-dimensional semantic vector. Then the two most relevant topics from each news article are extracted, and the corresponding pre-trained topic vectors (200-dimensions) are identified based on topic ID. Next the topic vectors are spliced into the 768-dimensional semantic vector extracted from the article summary, forming a 968-dimensional semantic vector. 

New content continuously comes in through Kafta, and is converted into semantic vectors before being inserted into the Milvus database.

### Extracting semantically similar tags from user profiles with BERT-as-service

The other neural network of the model is user semantic vector. Semantically similar tags (e.g., coronavirus, covid, COVID-19, pandemic, novel strain, pneumonia) are extracted from user profiles based on interests, search queries, and browsing history. The list of acquired tags is sorted by weight, and the top 200 are divided into different semantic groups. Permutations of the tags within each semantic group are used to generate new tag phrases, which are then encoded into semantic vectors through BERT-as-service

For each user profile, sets of tag phrases have a [corresponding set of topics](https://github.com/baidu/Familia) that are marked by a weight indicating a user's interest level. The top two topics out of all relevant topics are selected and encoded by the machine learning (ML) model to be spliced into the corresponding tag semantic vector, forming a 968-dimensional user semantic vector. Even if the system generates the same tags for different users, different weights for tags and their corresponding topics, as well as explicit variance between each user's topic vectors, ensures recommendations are unique

The system is able to make personalized news recommendations by calculating the cosine similarity of the semantic vectors extracted from both user profiles and news articles.

![Sohu01.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Sohu01_1e466fe0c3.jpg)
###### *Sohu News DSSM structure.*

### Computing new semantic user profile vectors and inserting them to Milvus

Semantic user profile vectors are computed daily, with data from the previous 24-hour period processed the following evening. Vectors are inserted into Milvus individually and run through the query process to serve relevant news results to users. News content is inherently topical, requiring computation to be run hourly to generate a current newsfeed that contains content that has a high predicted click-through rate and is relevant to users. News content is also sorted into partitions by date, and old news is purged daily.

### Decreasing semantic vector extraction time from days to hours

Retrieving content using semantic vectors requires converting tens of millions of tag phrases into semantic vectors every day. This is a time-consuming process that would require days to complete even when running on graphics processing units (GPU), which accelerate this type of computation. To overcome this technical issue, semantic vectors from the previous embedding must be optimized so that when similar tag phrases surface corresponding semantic vectors are directly retrieved.

The semantic vector of the existing set of tag phrases is stored, and a new set of tag phrases that is generated daily gets encoded into MinHash vectors. [Jaccard distance](https://milvus.io/docs/metric.md) is used to compute similarity between the MinHash vector of the new tag phrase and the saved tag phrase vector. If the Jaccard distance exceeds a pre-defined threshold, the two sets are considered similar. If the similarity threshold is met, new phrases can leverage the semantic information from previous embeddings. Tests suggest a distance above 0.8 should guarantee enough accuracy for most situations.

Through this process, the daily conversion of the tens of millions of vectors mentioned above is reduced from days to around two hours. Although other methods of storing semantic vectors might be more appropriate depending on specific project requirements, computing similarity between two tag phrases using Jaccard distance in a Milvus database remains an efficient and accurate method in a wide variety of scenarios.

![Sohu02.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Sohu02_d50fccc538.jpg)
###### *Using Jaccard distance to compute similarity between two sets.*

## Overcoming “bad cases” of short text classification

When classifying news text, short news articles have fewer features for extraction than longer ones. Because of this, classification algorithms fail when content of varying lengths is run through the same classifier. Milvus helps solve this issue by searching for multiple pieces of long text classification information with similar semantics and reliable scores, then using a voting mechanism to modify short text classification. 

### Identifying and resolving misclassified short text 

Precise classification of each news article is crucial to providing useful content recommendations. Since short news articles have fewer features, applying the same classifier for news with different lengths results in a higher error rate for short text classification. Human labeling is too slow and inaccurate for this task, so BERT-as-service and Milvus are used to quickly identify misclassified short text in batches, correctly reclassify them, then use batches of data as a corpus for training against this problem.

BERT-as-service is used to encode a total number of five million long news articles with a classifier score greater than 0.9 into semantic vectors. After inserting the long text articles into Milvus, short text news is encoded into semantic vectors. Each short news semantic vector is used to query the Milvus database and obtain the top 20 long news articles with the highest cosine similarity to the target short news. If 18 of the top 20 semantically similar long news appear to be in the same classification and it differs from that of the query short news, then the short news classification is considered incorrect and must be adjusted to align with the 18 long news articles.

This process quickly identifies and corrects inaccurate short text classifications. Random sampling statistics show that after short text classifications are corrected, the overall accuracy of text classification exceeds 95%. By leveraging the classification of high-confidence long text to correct the classification of short text, the majority of bad classification cases are corrected in a short amount of time. This also offers a good corpus for training a short text classifier.

![Sohu03.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Sohu03_a43074cf5f.jpg)
###### *Flowchart of discovery of "bad cases" of short text classification.*

## Milvus can power real-time news content recommendation and more

Milvus greatly improved the real-time performance of Sohu’s news recommendation system, and also bolstered the efficiency of identifying misclassified short text. If you’re interested in learning more about Milvus and its various applications:

- Read our [blog](https://zilliz.com/blog).
- Interact with our open-source community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).

large_blog_SohuNews.jpg

medium_blog_SohuNews.jpg

small_blog_SohuNews.jpg

thumbnail_blog_SohuNews.jpg

blog_SohuNews.jpg

Learn how Milvus was used to build an intelligent news recommendation system inside an app.

Building an Intelligent News Recommendation System Inside Sohu News App

Milvus v1.1.0 is available now. Discover new features, improvements, and bug fixes made to the world' most popular vector database.

[Milvus](https://github.com/milvus-io) is an ongoing open-source software (OSS) project focused on building the world's fastest and most reliable vector database. New features inside Milvus v1.1.0 are the first of many updates to come, thanks to long-term support from the open-source community and sponsorship from Zilliz. This blog article covers the new features, improvements, and bug fixes included with Milvus v1.1.0.

**Jump to:**

- [New features](#new-features)
- [Improvements](#improvements)
- [Bug fixes](#bug-fixes)

 

## New features

Like any OSS project, Milvus is a perpetual work in progress. We strive to listen to our users and the open-source community to prioritize the features that matter most. The latest update, Milvus v1.1.0, offers the following new features:

### Specify partitions with `get_entity_by_id()` method calls

To further accelerate vector similarity search, Milvus 1.1.0 now supports retrieving vectors from a specified partition. Generally, Milvus supports querying vectors through specified vector IDs. In Milvus 1.0, calling the method `get_entity_by_id()` searches the entire collection, which can be time consuming for large datasets. As we can see from the code below, `GetVectorsByIdHelper` uses a `FileHolder` structure to loop through and find a specific vector. 

```
std::vector<meta::CollectionSchema> collection_array; 
 auto status = meta_ptr_->ShowPartitions(collection.collection_id_, collection_array); 
 
 collection_array.push_back(collection); 
 status = meta_ptr_->FilesByTypeEx(collection_array, file_types, files_holder); 
 if (!status.ok()) { 
 std::string err_msg = "Failed to get files for GetVectorByID: " + status.message(); 
 LOG_ENGINE_ERROR_ << err_msg; 
 return status; 
 } 
 
 if (files_holder.HoldFiles().empty()) { 
 LOG_ENGINE_DEBUG_ << "No files to get vector by id from"; 
 return Status(DB_NOT_FOUND, "Collection is empty"); 
 } 
 
 cache::CpuCacheMgr::GetInstance()->PrintInfo(); 
 status = GetVectorsByIdHelper(id_array, vectors, files_holder); 
DBImpl::GetVectorsByIdHelper(const IDNumbers& id_array, std::vector<engine::VectorsData>& vectors, 
 meta::FilesHolder& files_holder) { 
 // attention: this is a copy, not a reference, since the files_holder.UnMarkFile will change the array internal 
 milvus::engine::meta::SegmentsSchema files = files_holder.HoldFiles(); 
 LOG_ENGINE_DEBUG_ << "Getting vector by id in " << files.size() << " files, id count = " << id_array.size(); 
 
 // sometimes not all of id_array can be found, we need to return empty vector for id not found 
 // for example: 
 // id_array = [1, -1, 2, -1, 3] 
 // vectors should return [valid_vector, empty_vector, valid_vector, empty_vector, valid_vector] 
 // the ID2RAW is to ensure returned vector sequence is consist with id_array 
 using ID2VECTOR = std::map<int64_t, VectorsData>; 
 ID2VECTOR map_id2vector; 
 
 vectors.clear(); 
 
 IDNumbers temp_ids = id_array; 
 for (auto& file : files) { 
```

However, this structure is not filtered by any partitions in `FilesByTypeEx()`. In Milvus v1.1.0, it is possible for the system to pass partition names to the `GetVectorsIdHelper` loop so that the `FileHolder` only contains segments from specified partitions. Put differently, if you know exactly which partition the vector for a search belongs to, you can specify the partition name in a `get_entity_by_id()` method call to accelerate the search process.

We not only made modifications to code controlling system queries at the Milvus server level, but also updated all our SDKs(Python, Go, C++, Java, and RESTful) by adding a parameter for specifying partition names. For example, in pymilvus, the definition of `get_entity_by_id` `def get_entity_by_id(self, collection_name, ids, timeout=None)` is changed to `def get_entity_by_id(self, collection_name, partition_tags=None, ids, timeout=None)`.

 

### Specify partitions with `delete_entity_by_id()` method calls

To make vector management more efficient, Milvus v1.1.0 now supports specifying partition names when deleting a vector in a collection. In Milvus 1.0, vectors in a collection can only be deleted by ID. When calling the delete method, Milvus will scan all vectors in the collection. However, it is far more efficient to scan only relevant partitions when working with massive million, billion, or even trillion vector datasets. Similar to the new feature for specifying partitions with `get_entity_by_id()` method calls, modifications were made to the Milvus code using the same logic.

 

### New method `release_collection()`

To free up memory Milvus used to load collections at runtime, a new method `release_collection()` has been added in Milvus v1.1.0 to manually unload specific collections from cache.

 

## Improvements

Although new features are usually all the rage, it's also important to improve what we already have. What follows are upgrades and other general improvements over Milvus v1.0.

 

### Improved performance of `get_entity_by_id()` method call

The chart below is a comparison of vector search performance between Milvus v1.0 and Milvus v1.1.0:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Segment file size = 1024 MB 
> Row count = 1,000,000 
> Dim = 128 

| Query ID Num | v 1.0.0 | v1.1.0 |
| :-----------: | :-----------: | :-----------: |
| 10 | 9 ms | 2 ms |
| 100 | 149 ms | 19 ms |

 

### Hnswlib upgraded to v0.5.0

Milvus adopts multiple widely used index libraries, including Faiss, NMSLIB, Hnswlib, and Annoy to simplify the process of choosing the right index type for a given scenario.

Hnswlib has been upgraded from v0.3.0 to v0.5.0 in Milvus 1.1.0 due to a bug detected in the earlier version. Additionally, upgrading Hnswlib improves `addPoint()` performance in index building.

A Zilliz developer created a pull request (PR) to improve Hnswlib performance while building indexes in Milvus. See [PR #298](https://github.com/nmslib/hnswlib/pull/298) for details.

The chart below is a comparison of `addPoint()` performance between Hnswlib 0.5.0 and the proposed PR:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Dataset: sift_1M (row count = 1000000, dim = 128, space = L2)

| | 0.5.0 | PR-298 |
| :-----------: | :-----------: | :-----------: |
| M = 16, ef_construction = 100 | 274406 ms | 265631 ms |
| M = 16, ef_construction = 200 | 522411 ms | 499639 ms |

 

### Improved IVF index training performance

Creating an index includes training, inserting and writing data to disk. Milvus 1.1.0 improves the training component of index building. The chart below is a comparison of IVF index training performance between Milvus 1.0 and Milvus 1.1.0:

> CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz * 8 
> Dataset: sift_1m (row_count = 1000000, dim = 128, metric_type = L2)

| | v1.0.0 (ms) | v1.1.0 (ms) |
| :-----------: | :-----------: | :-----------: |
| ivf_flat (nlist = 2048) | 90079 | 81544 |
| ivf_pq (nlist = 2048, m=16) | 103535 | 97115 |
| ivf_pq (nlist = 2048, m=32) | 108638 | 104558 |
| ivf_flat (nlist = 4096) | 340643 | 310685 |
| ivf_pq (nlist = 4096, m=16) | 351982 | 323758 |
| ivf_pq (nlist = 4096, m=32) | 357359 | 330887 |


 

## Bug fixes

We also fixed some bugs to make Milvus more stable and efficient when managing vector datasets. See [Fixed Issues](https://milvus.io/docs/v1.1.0/release_notes.md#Fixed-issues) for more details.

large_v1.1 cover.jpeg

medium_v1.1 cover.jpeg

small_v1.1 cover.jpeg

thumbnail_v1.1 cover.jpeg

v1.1 cover.jpeg

Milvus v1.1.0 has arrived! New features, improvements, and bug fixes are available now.

Inside Milvus 1.1.0

The finance industry has long leveraged open-source software to extract insights from large, unstructured datasets. Discover how Milvus, an open-source vector database, can power AI applications including banking chatbots, recommender systems, and semantic text mining. 

Banks and other financial institutions have long been early adopters of open-source software for big data processing and analytics. In 2010, Morgan Stanley [began using](https://www.forbes.com/sites/tomgroenfeldt/2012/05/30/morgan-stanley-takes-on-big-data-with-hadoop/?sh=19f4f8cd16db) the open-source Apache Hadoop framework as part of a small experiment. The company was struggling to successfully scale traditional databases to the massive volumes of data its scientists wanted to leverage, so it decided to explore alternative solutions. Hadoop is now a staple at Morgan Stanley, helping with everything from managing CRM data to portfolio analysis. Other open-source relational database software such as MySQL, MongoDB, and PostgreSQL have been indispensable tools for making sense of big data in the finance industry.

Technology is what gives the financial services industry a competitive edge, and artificial intelligence (AI) is rapidly becoming the standard approach to extracting valuable insights from big data and analyzing activity in real-time across the banking, asset management, and insurance sectors. By using AI algorithms to convert unstructured data such as images, audio, or video to vectors, a machine-readable numeric data format, it is possible to run similarity searches on massive million, billion, or even trillion vector datasets. Vector data is stored in high-dimensional space, and similar vectors are found using similarity search, which requires a dedicated infrastructure called a vector database.

![01 (1).jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/01_1_cb99f15886.jpg)
###### *Development of open source big data technologies.*

[Milvus](https://github.com/milvus-io/milvus) is an open-source vector database built specifically for managing vector data, which means engineers and data scientists can focus on building AI applications or conducting analysis—instead of the underlying data infrastructure. The platform was built around AI application development workflows and is optimized to streamline machine learning operations (MLOps). For more information about Milvus and its underlying technology, check out our [blog](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View).

Common applications of AI in the financial services industry include algorithmic trading, portfolio composition and optimization, model validation, backtesting, Robo-advising, virtual customer assistants, market impact analysis, regulatory compliance, and stress testing. This article covers three specific areas where vector data is leveraged as one of the most valuable assets for banking and financial companies:

1. Enhancing customer experience with banking chatbots
2. Boosting financial services sales and more with recommender systems
3. Analyzing earnings reports and other unstructured financial data with semantic text mining

 

### Enhancing customer experience with banking chatbots

Banking chatbots can improve customer experiences by helping consumers select investments, banking products, and insurance policies. Digital services are rising rapidly in popularity in part due to trends accelerated by the coronavirus pandemic. Chatbots work by using natural language processing (NLP) to convert user-submitted questions into semantic vectors to search for matching answers. Modern banking chatbots offer a personalized natural experience for users and speak in a conversational tone. Milvus provides a data fabric well suited for creating chatbots using real-time vector similarity search.

Learn more in our demo that covers building [chatbots with Milvus](https://zilliz.com/blog/building-intelligent-chatbot-with-nlp-and-milvus).

![02 (1).jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/02_1_8c298c45e5.jpg)

 

#### Boosting financial services sales and more with recommender systems:

The private banking sector uses recommender systems to increase sales of financial products through personalized recommendations based on customer profiles. Recommender systems can also be leveraged in financial research, business news, stock selection, and trading support systems. Thanks to deep learning models, every user and item is described as an embedding vector. A vector database offers an embedding space where similarities between users and items can be calculated.

Learn more from our [demo](https://zilliz.com/blog/graph-based-recommendation-system-with-milvus) covering graph-based recommendation systems with Milvus.

 

#### Analyzing earnings reports and other unstructured financial data with semantic text mining:

Text mining techniques had a substantial impact on the financial industry. As financial data grows exponentially, text mining has emerged as an important field of research in the domain of finance.

Deep learning models are currently applied to represent financial reports through word vectors capable of capturing numerous semantic aspects. A vector database like Milvus is able to store massive semantic word vectors from millions of reports, then conduct similarity searches on them in milliseconds.

Learn more about how to [use deepset's Haystack with Milvus](https://medium.com/deepset-ai/semantic-search-with-milvus-knowledge-graph-qa-web-crawlers-and-more-837451eae9fa).

 

### Don’t be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_03 (1).jpg

medium_03 (1).jpg

small_03 (1).jpg

thumbnail_03 (1).jpg

03 (1).jpg

Milvus can be used to build AI applications for the finance industry including chatbots, recommender systems, and more.

Accelerating AI in Finance with Milvus, an Open-Source Vector Database

VOVA, an emerging e-commerce platform, prioritizes convenient and positive user experiences. Learn how the company used Milvus to build an image search tool to help shoppers find items they encounter in real life.

Jump to:
 
- [How does image search work?](#how-does-image-search-work)
- [Target detection using the YOLO model](#target-detection-using-the-yolo-model)
- [Image feature vector extraction with ResNet](#image-feature-vector-extraction-with-resnet)
- [Vector similarity search powered by Milvus](#vector-similarity-search-powered-by-milvus)
- [VOVA's shop by image tool](#vovas-shop-by-image-tool)

Online shopping surged in 2020, [up 44%](https://www.digitalcommerce360.com/2021/02/15/ecommerce-during-coronavirus-pandemic-in-charts/) in large part due to the coronavirus pandemic. As people sought to socially distance and avoid contact with strangers, no-contact delivery became an incredibly desirable option for many consumers. This popularity has also led to people buying a greater variety of goods online, including niche items that can be hard to describe using a traditional keyword search.

To help users overcome the limitations of keyword-based queries, companies can build image search engines that allow users to use images instead of words for search. Not only does this allow users to find items that are difficult to describe, but it also helps them shop for things they encounter in real life. This functionality helps build a unique user experience and offers general convenience that customers appreciate.

VOVA is an emerging e-commerce platform that focuses on affordability and offering a positive shopping experience to its users, with listings covering millions of products and support for 20 languages and 35 major currencies. To enhance the shopping experience for its users, the company used Milvus to build image search functionality into its e-commerce platform. The article explores how VOVA successfully built an image search engine with Milvus. 

 

### How does image search work?

VOVA's shop by image system searches the company's inventory for product images that are similar to user uploads. The following chart shows the two stages of the system process, the data import stage (blue) and the query stage (orange):

1. Use the YOLO model to detect targets from uploaded photos;
2. Use ResNet to extract feature vectors from the detected targets; 
3. Use Milvus for vector similarity search.

![Vova-1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Vova_1_47ee6f2da9.png)
###### *System process of VOVA's search by image functionality.*

 

### Target detection using the YOLO model

VOVA's mobile apps on Android and iOS currently support image search. The company uses a state-of-the-art, real-time object detection system called YOLO (You only look once) to detect objects in user uploaded images. The YOLO model is currently in its fifth iteration. 

YOLO is a one-stage model, using only one convolutional neural network (CNN) to predict categories and positions of different targets. It is small, compact, and well suited for mobile use. 

YOLO uses convolutional layers to extract features and fully-connected layers to obtain predicted values. Drawing inspiration from the GooLeNet model, YOLO’s CNN includes 24 convolutional layers and two fully-connected layers.

As the following illustration shows, a 448 &times; 448 input image is converted by a number of convolutional layers and pooling layers to a 7 &times; 7 &times; 1024-dimensional tensor (depicted in the third to last cube below), and then converted by two fully-connected layers to a 7 &times; 7 &times; 30-dimensional tensor output.

The predicted output of YOLO P is a two-dimensional tensor, whose shape is [batch,7 &times;7 &times;30]. Using slicing, P[:,0:7&times;7&times;20] is the category probability, P[:,7&times;7&times;20:7&times;7&times;(20+2)] is the confidence, and P[:,7&times;7&times;(20+2)]:] is the predicted result of the bounding box.

![vova-2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/vova_2_1ccf38f721.png)
###### *YOLO network architecture.*

 

### Image feature vector extraction with ResNet

VOVA adopted the residual neural network (ResNet) model to extract feature vectors from an extensive product image library and user uploaded photos. ResNet is limited because as the depth of a learning network increases, the accuracy of the network decreases. The image below depicts ResNet running the VGG19 model (a variant of the VGG model) modified to include a residual unit through the short circuit mechanism. VGG was proposed in 2014 and includes just 14 layers, while ResNet came out a year later and can have up to 152.

The ResNet structure is easy to modify and scale. By changing the number of channels in the block and the number of stacked blocks, the width and depth of the network can be easily adjusted to obtain networks with different expressive capabilities. This effectively solves the network degeneration effect, where accuracy declines as the depth of learning increases. With sufficient training data, a model with improving expressive performance can be obtained while gradually deepening the network. Through model training, features are extracted for each picture and converted to 256-dimensional floating point vectors.

![vova-3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/vova_3_df4b810281.png)
###### *ResNet structure.*

 

### Vector similarity search powered by Milvus
 
VOVA's product image database includes 30 million pictures and is growing rapidly. To quickly retrieve the most similar product images from this massive dataset, Milvus is used to conduct vector similarity search. Thanks to a number of optimizations, Milvus offers a fast and streamlined approach to managing vector data and building machine learning applications. Milvus offers integration with popular index libraries (e.g., Faiss, Annoy), supports multiple index types and distance metrics, has SDKs in multiple languages, and provides rich APIs for managing vector data.

Milvus can conduct similarity search on trillion-vector datasets in milliseconds, with a query time under 1.5 seconds when nq=1 and an average batch query time under 0.08 seconds. To build its image search engine, VOVA referred to the design of Mishards, Milvus' sharding middleware solution (see the chart below for its system design), to implement a highly available server cluster. By leveraging the horizontal scalability of a Milvus cluster, the project requirement for high query performance on massive datasets was met.

![vova-4.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/vova_4_e305f1955c.png)
###### *Mishards architecture in Milvus.*


### VOVA's shop by image tool
 
The screenshots below show the VOVA search by image shopping tool on the company's Android app.

![vova-5.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/vova_5_c4c25a3bae.png)
###### *Screenshots of VOVA's search by image shopping tool.*
 
As more users search for products and upload photos, VOVA will continue to optimize the models that power the system. Additionally, the company will incorporate new Milvus functionality that can further enhance the online shopping experience of its users.

### Reference

**YOLO:**

https://arxiv.org/pdf/1506.02640.pdf

https://arxiv.org/pdf/1612.08242.pdf

**ResNet:**

https://arxiv.org/abs/1512.03385

**Milvus:**

https://milvus.io/docs/overview.md

large_vova-thumbnail.jpg

medium_vova-thumbnail.jpg

small_vova-thumbnail.jpg

thumbnail_vova-thumbnail.jpg

vova-thumbnail.jpg

Discover how Milvus, an open-source vector database, was used by e-commerce platform VOVA to power shopping by image. 

Building a Search by Image Shopping Experience with VOVA and Milvus

We are proud to announce Milvus v1.0, a stable, long-term support version of  the world’s most popular open-source vector database. Discover the highlight features that Milvus v1.0 has to offer.

![Milvus.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Milvus_510cf50aee.jpeg)

Milvus is an open-source vector database designed to manage massive million, billion, or even trillion vector datasets. Milvus has broad applications spanning new drug discovery, computer vision, autonomous driving, recommendation engines, chatbots, and much more. 

In March, 2021 Zilliz, the company behind Milvus, released the platform's first long-term support version—Milvus v1.0. After months of extensive testing, a stable, production ready version of the world's most popular vector database is ready for prime time. This blog article covers some Milvus fundamentals as well as key features of v1.0.

 

### Milvus distributions

Milvus is available in CPU-only and GPU-enabled distributions. The former relies exclusively on CPU for index building and search; the latter enables CPU and GPU hybrid search and index building that further accelerates Milvus. For example, using the hybrid distribution, CPU can be used for search and GPU for index building, further improving query efficiency.

Both Milvus distributions are available in Docker. You can either compile Milvus from Docker (if your operating system supports it) or compile Milvus from source code on Linux (other operating systems are not supported). 

 

### Embedding vectors

Vectors are stored in Milvus as entities. Each entity has one vector ID field and one vector field. Milvus v1.0 supports integer vector IDs only. When creating a collection within Milvus, vector IDs can be automatically generated or manually defined. Milvus ensures auto-generated vector IDs are unique however, manually defined IDs can be duplicated within Milvus. If manually defining IDs, users are responsible for making sure all IDs are unique.

 

### Partitions

Milvus supports creating partitions in a collection. In situations where data is inserted regularly and historical data isn't significant (e.g., streaming data), partitions can be used to accelerate vector similarity search. One collection can have up to 4,096 partitions. Specifying a vector search within a specific partition narrows the search and may significantly reduce query time, particularly for collections that contain more than a trillion vectors.

 

### Index algorithm optimizations
Milvus is built on top of multiple widely-adopted index libraries, including Faiss, NMSLIB, and Annoy. Milvus is far more than a basic wrapper for these index libraries. Here are some of the major enhancements that have been made to the underlying libraries:

- IVF index performance optimizations using the Elkan k-means algorithm.
- FLAT search optimizations.
- IVF_SQ8H hybrid index support, which can reduce index file sizes by up to 75% without sacrificing data accuracy. IVF_SQ8H is built upon IVF_SQ8, with identical recall but much faster query speed. It was designed specifically for Milvus to harnesses the parallel processing capacity of GPUs, and the potential for synergy between CPU/GPU co-processing.
- Dynamic instruction set compatibility.

 

### Search, index building, and other Milvus optimizations

The following optimizations have been made to Milvus to improve search and index building performance. 

- Search performance is optimized in situations when the number of queries (nq) is less than the number of CPU threads. 
- Milvus combines search requests from a client that take the same topK and search parameters. 
- Index building is suspended when search requests come in. 
- Milvus automatically preloads collections to memory at start.
- Multiple GPU devices can be assigned to accelerate vector similarity search.

 

### Distance metrics

Milvus is a vector database built to power vector similarity search. The platform was built with MLOps and production level AI applications in mind. Milvus supports a wide range of distance metrics for calculating similarity, such as Euclidean distance (L2), inner product (IP), Jaccard distance, Tanimoto, Hamming distance, superstructure, and substructure. The last two metrics are commonly used in molecular search and AI-powered new drug discovery.

 

### Logging

Milvus supports log rotation. In the system configuration file, milvus.yaml, you can set the size of a single log file, the number of log files, and log output to stdout.

 

### Distributed solution

Mishards, a Milvus sharding middleware, is the distributed solution for Milvus With one write node and an unlimited number of read nodes, Mishards unleashes the computational potential of server cluster. Its features include request forwarding, read/write splitting, dynamic/horizontal scaling, and more.

 

### Monitoring

Milvus is compatible with Prometheus, an open-source system monitoring and alerts toolkit. Milvus adds support for Pushgateway in Prometheus, making it possible for Prometheus to acquire short-lived batch metrics. The monitoring and alerts system works as follows:

- The Milvus server pushes customized metrics data to Pushgateway.
- Pushgateway ensures ephemeral metric data is safely sent to Prometheus.
- Prometheus continues pulling data from Pushgateway.
- Alertmanager is used to set the alert threshold for different indicators and send alerts via email or message.

 

### Metadata management

Milvus uses SQLite for metadata management by default. SQLite is implemented in Milvus and does not require configuration. In a production environment, it is recommended that you use MySQL for metadata management. 

 

### Engage with our open-source community:

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_Milvus.jpeg

medium_Milvus.jpeg

small_Milvus.jpeg

thumbnail_Milvus.jpeg

Milvus.jpeg

	 Milvus v1.0 is available now. This stable, long-term support version powers image/video search, chatbots, and many more AI applications.

What's Inside Milvus 1.0?

Learn how Milvus is used to mitigate threats to critical data and strengthen cybersecurity with real-time virus detection.

Cybersecurity remains a persistent threat to both individuals and businesses, with data privacy concerns increasing for [86% of companies](https://www.getapp.com/resources/annual-data-security-report/) in 2020 and just [23% of consumers](https://www.firstdata.com/downloads/pdf/digital-commerce-cybersecurity-ebook.pdf) believing their personal data is very secure. As malware becomes steadily more omnipresent and sophisticated, a proactive approach to threat detection has become essential. [Trend Micro](https://www.trendmicro.com/en_us/business.html) is a global leader in hybrid cloud security, network defense, small business security, and endpoint security. To protect Android devices from viruses, the company built Trend Micro Mobile Security—a mobile app that compares APKs (Android Application Package) from the Google Play Store to a database of known malware. The virus detection system works as follows:

- External APKs (Android application package) from the Google Play Store are crawled.
- Known malware is converted into vectors and stored in [Milvus](https://www.milvus.io/docs/v1.0.0/overview.md).
- New APKs are also converted into vectors, then compared to the malware database using similarity search.
- If an APK vector is similar to any of the malware vectors, the app provides users with detailed information about the virus and its threat level.

To work, the system has to perform highly efficient similarity search on massive vector datasets in real time. Initially, Trend Micro used [MySQL](https://www.mysql.com/). However, as its business expanded so did the number of APKs with nefarious code stored in its database. The company’s algorithm team began searching for alternative vector similarity search solutions after quickly outgrowing MySQL.

 

### Comparing vector similarity search solutions

There are a number of vector similarity search solutions available, many of which are open source. Although the circumstances vary from project to project, most users benefit from leveraging a vector database built for unstructured data processing and analytics rather than a simple library that requires extensive configuration. Below we compare some popular vector similarity search solutions and explain why Trend Micro chose Milvus.

#### Faiss
[Faiss](https://ai.facebook.com/tools/faiss/) is a library developed by Facebook AI Research that enables efficient similarity search and clustering of dense vectors. The algorithms it contains search vectors of any size in sets. Faiss is written in C++ with wrappers for Python/numpy, and supports a number of indexes including IndexFlatL2, IndexFlatIP, HNSW, and IVF.

Although Faiss is an incredibly useful tool, it has limitations. It only works as a basic algorithm library, not a database for managing vector datasets. Additionally, it does not offer a distributed version, monitoring services, SDKs, or high availability, which are the key features of most cloud-based services. 

#### Plug-ins based on Faiss & other ANN search libraries
There are several plug-ins built on top of Faiss, NMSLIB, and other ANN search libraries that are designed to enhance the basic functionality of the underlying tool that powers them. Elasticsearch (ES) is a search engine based on the Lucene library with a number of such plugins. Below is an architecture diagram of an ES plug-in:

![1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/1_3ce4e516c3.png)

###### *Architecture diagram of an Elasticsearch plug-in.*

Built in support for distributed systems is a major advantage of an ES solution. This saves developers time and companies money thanks to code that doesn’t have to be written. ES plug-ins are technically advanced and prevalent. Elasticsearch provides a QueryDSL (domain-specific language), which defines queries based on JSON and is easy to grasp. A full set of ES services makes it possible to conduct vector/text search and filter scalar data simultaneously.

Amazon, Alibaba, and Netease are a few large tech companies that currently rely on Elasticsearch plug-ins for vector similarity search. The primary downsides with this solution are high memory consumption and no support for performance tuning. In contrast, [JD.com](http://jd.com/) has developed its own distributed solution based on Faiss called [Vearch](https://github.com/vearch/vearch). However, Vearch is still an incubation-stage project and its open-source community is relatively inactive.

#### Milvus 
[Milvus](https://www.milvus.io/) is an open-source vector database created by [Zilliz](https://zilliz.com). It is highly flexible, reliable, and blazing fast. By encapsulating multiple widely adopted index libraries, such as Faiss, NMSLIB, and Annoy, Milvus provides a comprehensive set of intuitive APIs, allowing developers to choose the ideal index type for their scenario. It also provides distributed solutions and monitoring services. Milvus has a highly active open-source community and over 5.5K stars on [Github](https://github.com/milvus-io/milvus).

#### Milvus bests the competition
We compiled a number of different test results from the various vector similarity search solutions mentioned above. As we can see in the following comparison table, Milvus was significantly faster than the competition despite being tested on a dataset of 1 billion 128-dimensional vectors.

| **Engine** | **Performance (ms)** | **Dataset Size (million)** |
| :---- | :---- | :---- |
| ES | 600 | 1 |
| ES + Alibaba Cloud | 900 | 20 |
| Milvus | 27 | 1000+ |
| SPTAG | Not good | |
| ES + nmslib, faiss | 90 | 150 |

###### *A comparison of vector similarity search solutions.*

After weighing the pros and cons of each solution, Trend Micro settled on Milvus for its vector retrieval model. With exceptional performance on massive, billion-scale datasets, it's obvious why the company chose Milvus for a mobile security service that requires real-time vector similarity search.

 

### Designing a system for real-time virus detection
Trend Micro has more than 10 million malicious APKs stored in its MySQL database, with 100k new APKs added each day. The system works by extracting and calculating Thash values of different components of an APK file, then uses the Sha256 algorithm to transform it into binary files and generate 256-bit Sha256 values that differentiate the APK from others. Since Sha256 values vary with APK files, one APK can have one combined Thash value and one unique Sha256 value.

Sha256 values are only used to differentiate APKs, and Thash values are used for vector similarity retrieval. Similar APKs may have the same Thash values but different Sha256 values.

To detect APKs with nefarious code, Trend Micro developed its own system for retrieving similar Thash values and corresponding Sha256 values. Trend Micro chose Milvus to conduct instantaneous vector similarity search on massive vector datasets converted from Thash values. After similarity search is run, the corresponding Sha256 values are queried in MySQL. A Redis caching layer is also added to the architecture to map Thash values to Sha256 values, significantly reducing query time.

Below is the architecture diagram of Trend Micro’s mobile security system.

![image-20210118-022039.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/image_20210118_022039_ae824b663c.png)

###### *Architecture diagram for Trend Micro Mobile Security.*

 

Choosing an appropriate distance metric helps improve vector classification and clustering performance. The following table shows the [distance metrics](https://www.milvus.io/docs/v1.0.0/metric.md#binary) and the corresponding indexes that work with binary vectors. 

| **Distance Metrics** | **Index Types** |
| ---- | ---- |
| - Jaccard - Tanimoto - Hamming | - FLAT - IVF_FLAT |
| - Superstructure - Substructure | FLAT |

###### *Distance metrics and indexes for binary vectors.*

 

Trend Micro converts Thash values into binary vectors and stores them in Milvus. For this scenario, Trend Micro is using Hamming distance to compare vectors.

Milvus will soon support string vector ID, and integer IDs won’t have to be mapped to the corresponding name in string format. This makes the Redis caching layer unnecessary and the system architecture less bulky.

Trend Micro adopts a cloud-based solution and deploys many tasks on [Kubernetes](https://kubernetes.io/). To achieve high availability, Trend Micro uses [Mishards](https://www.milvus.io/docs/v1.0.0/mishards.md), a Milvus cluster sharding middleware developed in Python. 

![image-20210118-022104.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/image_20210118_022104_3001950ee8.png)

###### *Mishards architecture in Milvus.*

 

Trend Micro separates storage and distance calculation by storing all vectors in the [EFS](https://aws.amazon.com/efs/) (Elastic File System) provided by [AWS](https://aws.amazon.com/). This practice is a popular trend in the industry. Kubernetes is used to start multiple reading nodes, and develops LoadBalancer services on these reading nodes to ensure high availability.

To maintain data consistency Mishards supports just one writing node. However, a distributed version of Milvus with support for multiple writing nodes will be available in the coming months. 

 

### Monitoring and Alert Functions
Milvus is compatible with monitoring systems built on [Prometheus](https://prometheus.io/), and uses [Grafana](https://grafana.com/), an open-source platform for time-series analytics, to visualize various performance metrics.

Prometheus monitors and stores the following metrics:

- Milvus performance metrics including insertion speed, query speed, and Milvus uptime. 
- System performance metrics including CPU/GPU usage, network traffic, and disk access speed.
- Hardware storage metrics including data size and total file number.

The monitoring and alert system works as follows: 

- A Milvus client pushes customized metrics data to Pushgateway.
- The Pushgateway ensures short-lived, ephemeral metric data is safely sent to Prometheus.
- Prometheus keeps pulling data from Pushgateway. 
- Alertmanager sets the alert threshold for different metrics and raises alarms through emails or messages.

 

### System Performance
A couple months have passed since the ThashSearch service built on Milvus was first launched. The graph below shows that end-to-end query latency is less than 95 milliseconds.

![image-20210118-022116.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/image_20210118_022116_a0c735ce20.png)

###### *Query latency for Thash search service built on Milvus.*

 

Insertion is also fast. It takes around 10 seconds to insert 3 million 192-dimensional vectors. With help from Milvus, the system performance was able to meet the performance criteria set by Trend Micro. 

 

### Don’t be a stranger
- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

large_blog_TrendMicro.jpg

medium_blog_TrendMicro.jpg

small_blog_TrendMicro.jpg

thumbnail_blog_TrendMicro.jpg

blog_TrendMicro.jpg

Making with Milvus: Detecting Android Viruses in Real Time for Trend Micro

Learn more about using semantic machine learning methodologies to power more relevant search results across your organization.

[Semantic search](https://lucidworks.com/post/what-is-semantic-search/) is a great tool to help your customers—or your employees—find the right products or information. It can even surface difficult-to-index information for better results. That said, if your semantic methodologies aren’t being deployed to work fast, they won’t do you any good. The customer or employee isn’t just going to sit around while the system takes its time responding to their query—and a thousand others are likely being ingested at the same time.

How can you make semantic search fast? Slow semantic search isn’t going to cut it.

Fortunately, this is the kind of problem Lucidworks loves to solve. We recently tested a modest-sized cluster—read on for more details—that resulted in 1500 RPS (requests per second) against a collection of over one million documents, with an average response time of roughly 40 milliseconds. Now that’s some serious speed.

 

### Implementing Semantic Search
To make lightning-fast, machine learning magic happen, Lucidworks has implemented semantic search using the semantic vector search approach. There are two critical parts.

 

#### Part One: The Machine Learning Model

First, you need a way to encode text into a numerical vector. The text could be a product description, a user search query, a question, or even an answer to a question. A semantic search model is trained to encode text such that text that is semantically similar to other text is encoded into vectors that are numerically “close” to one another. This encoding step needs to be fast in order to support the thousand or more possible customer searches or user queries coming in every second.

 

#### Part Two: The Vector Search Engine
Second, you need a way to quickly find the best matches to the customer search or user query. The model will have encoded that text into a numerical vector. From there, you need to compare that to all the numerical vectors in your catalog or lists of questions and answers to find those best matches—the vectors that are “closest” to the query vector. For that, you will need a vector engine that can handle all of that information effectively and at lightning speed. The engine could contain millions of vectors and you really just want the best twenty or so matches to your query. And of course, it needs to handle a thousand or so such queries every second.

To tackle these challenges, we added the vector search engine [Milvus](https://doc.lucidworks.com/fusion/5.3/8821/milvus) in our [Fusion 5.3 release](https://lucidworks.com/post/enhance-personalization-efforts-with-new-features-in-fusion/). Milvus is open-source software and it is fast. Milvus uses FAISS ([Facebook AI Similarity Search](https://ai.facebook.com/tools/faiss/)), the same technology Facebook uses in production for its own machine learning initiatives. When needed, it can run even faster on [GPU](https://en.wikipedia.org/wiki/Graphics_processing_unit). When Fusion 5.3 (or higher) is installed with the machine learning component, Milvus is automatically installed as part of that component so you can turn on all of these capabilities with ease.

The size of the vectors in a given collection, specified when the collection is created, depends on the model that produces those vectors. For example, a given collection could store the vectors created from encoding (via a model) all of the product descriptions in a product catalog. Without a vector search engine like Milvus, similarity searches would not be feasible across the entire vector space. So, similarity searches would have to be limited to pre-selected candidates from the vector space (for example, 500) and would have both slower performance and lower quality results. Milvus can store hundreds of billions of vectors across multiple collections of vectors to ensure that search is fast and results are relevant.

 

### Using Semantic Search

Let’s get back to the semantic search workflow, now that we’ve learned a little about why Milvus might be so important. Semantic search has three stages. During the first stage, the machine learning model is loaded and/or trained. Afterwards, data is indexed into Milvus and Solr. The final stage is the query stage, when the actual search occurs. We’ll focus on those last two stages below.

 

### Indexing into Milvus
![Lucidworks-1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Lucidworks_1_47a9221723.png)
###### *Architectural diagram for indexing into Milvus.*

As shown in the above diagram, the query stage begins similarly to the indexing stage, just with queries coming in instead of documents. For each query:

1. The query is sent to the [Smart Answers](https://lucidworks.com/products/smart-answers/) index pipeline.
2. The query is then sent to the ML model.
3. The ML model returns a numeric vector (encrypted from the query). Again, the type of model determines the size of the vector.
4. The vector is sent to Milvus, which then determines which vectors, in the specified Milvus collection, best match the provided vector.
5. Milvus returns a list of unique IDs and distances corresponding to the vectors determined in step four.
6. A query containing those IDs and distances is sent to Solr.
7. Solr then returns an ordered list of the documents associated with those IDs.

 

### Scale Testing
In order to prove that our semantic search flows are running at the efficiency we require for our customers, we run scale tests using Gatling scripts on the Google Cloud Platform using a Fusion cluster with eight replicas of the ML model, eight replicas of the query service, and a single instance of Milvus. Tests were run using the Milvus FLAT and HNSW indexes. The FLAT index has 100% recall, but is less efficient – except when the datasets are small. The HNSW (Hierarchical Small World Graph) index still has high quality results and it has improved performance on larger datasets.

Let’s jump into some numbers from a recent example we ran:

![Lucidworks-2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Lucidworks_2_3162113560.png)
###### *Performance of Milvus FLAT and HNSW indexes on a small dataset.*

![Lucidworks-3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Lucidworks_3_3dc17f0ed8.png)
###### *Performance of Milvus FLAT and HNSW indexes on a medium dataset.*

![Lucidworks-4.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Lucidworks_4_8a6edd2f59.png)
###### *Performance of Milvus FLAT and HNSW indexes on a large dataset.*

 

### Getting Started

The [Smart Answers](https://lucidworks.com/products/smart-answers/) pipelines are designed to be easy-to-use. Lucidworks has [pre-trained models that are easy-to-deploy](https://doc.lucidworks.com/how-to/734/set-up-a-pre-trained-cold-start-model-for-smart-answers) and generally have good results—though training your own models, in tandem with pre-trained models, will offer the best results. Contact us today to learn how you can implement these initiatives into your search tools to power more effective and delightful results.


> This blog is reposted from: https://lucidworks.com/post/how-to-build-fast-semantic-search/?utm_campaign=Oktopost-Blog+Posts&utm_medium=organic_social&utm_source=linkedin

medium_lucidworks.png

small_lucidworks.png

thumbnail_lucidworks.png

lucidworks.png

Build Semantic Search at Speed

Milvus accelerates machine learning application development and machine learning operations (MLOps). With Milvus, you can rapidly develop a minimum viable product (MVP) while keeping costs at lower limits.

![blog cover.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/blog_cover_4a9807b9e0.png)

[Milvus](https://milvus.io/) is an open-source vector database. It supports adding, deleting, updating, and near real-time search of massive vector datasets created by extracting feature vectors from unstructured data using AI models. With a comprehensive set of intuitive APIs, and support for multiple widely adopted index libraries (e.g., Faiss, NMSLIB, and Annoy), Milvus accelerates machine learning application development and machine learning operations (MLOps). With Milvus, you can rapidly develop a minimum viable product (MVP) while keeping costs at lower limits.

"What resources are available for developing an AI application with Milvus?” is commonly asked in the Milvus community. Zilliz, the [company](https://zilliz.com/) behind Milvus, developed a number of demos that leverage Milvus to conduct lightening-fast similarity search that powers intelligent applications. Source code of Milvus solutions can be found at [zilliz-bootcamp](https://github.com/zilliz-bootcamp). The following interactive scenarios demonstrate natural language processing (NLP), reverse image search, audio search, and computer vision.


Feel free to try out the solutions to gain some hands-on experience with specific scenarios! Share your own application scenarios via: 
- [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ)
- [GitHub](https://github.com/milvus-io/milvus/discussions)

 

**Jump to:**

- [Natural language processing (chatbots)](#natural-language-processing-chatbots)
- [Reverse image search](#reverse-image-search-systems)
- [Audio search](#audio-search-systems)
- [Video object detection (computer vision)](#video-object-detection-computer-vision)

 

### Natural language processing (chatbots)
Milvus can be used to build chatbots that use natural language processing to simulate a live operator, answer questions, route users to relevant information, and reduce labor costs. To demonstrate this application scenario, Zilliz built an AI-powered chatbot that understands semantic language by combining Milvus with [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)), a machine learning (ML) model developed for NLP pre-training.

👉Source code：[zilliz-bootcamp/intelligent_question_answering_v2](https://github.com/zilliz-bootcamp/intelligent_question_answering_v2)

![1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/1_c301a9e4bd.png)
###### *AI-powered chatbot built with Milvus and BERT.*

#### How to use

1. Upload a dataset that includes question-answer pairs. Format questions and answers in two separate columns. Alternatively, a [sample dataset](https://zilliz.com/solutions/qa) is available for download.

2. After typing in your question, a list of similar questions will be retrieved from the uploaded dataset.

3. Reveal the answer by selecting the question most similar to your own.

👉Video：[[Demo] QA System Powered by Milvus](https://www.youtube.com/watch?v=ANgoyvgAxgU)

#### How it works

Questions are converted into feature vectors using Google’s BERT model, then Milvus is used to manage and query the dataset.

**Data processing:**

1. BERT is used to convert the uploaded question-answer pairs into 768-dimensional feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Question, and corresponding answer, vector IDs are stored in PostgreSQL.

**Searching for similar questions:**

1. BERT is used to extract feature vectors from a user's input question.
2. Milvus retrieves vector IDs for questions that are most similar to the input question.
3. The system looks up the corresponding answers in PostgreSQL.

 

### Reverse image search systems
Reverse image search is transforming e-commerce through personalized product recommendations and similar product lookup tools that can boost sales. In this application scenario, Zilliz built a reverse image search system by combining Milvus with [VGG](https://towardsdatascience.com/how-to-use-a-pre-trained-model-vgg-for-image-classification-8dd7c4a4a517), an ML model that can extract image features.

👉Source code：[zilliz-bootcamp/image_search](https://github.com/zilliz-bootcamp/image_search)

![2.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/2_09000e2e2e.jpeg)
###### *Reverse image search system built with Milvus and VGG.*

#### How to use
1. Upload a zipped image dataset comprised of .jpg images only (other image file types are not accepted). Alternatively, a [sample dataset](https://zilliz.com/solutions/image-search) is available for download.
2. Upload an image to use as the search input for finding similar images.

👉Video: [[Demo] Image Search Powered by Milvus](https://www.youtube.com/watch?v=mTO8YdQObKY)

#### How it works
Images are converted into 512-dimensional feature vectors using the VGG model, then Milvus is used to manage and query the dataset.

**Data processing:**

1. The VGG model is used to convert the uploaded image dataset to feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Image feature vectors, and corresponding image file paths, are stored in CacheDB.

**Searching for similar images:**

1. VGG is used to convert a user’s uploaded image into feature vectors.
2. Vector IDs of images most similar to the input image are retrieved from Milvus.
3. The system looks up the corresponding image file paths in CacheDB.

 

### Audio search systems
Speech, music, sound effects, and other types of audio search makes it possible to quickly query massive volumes of audio data and surface similar sounds. Applications include identifying similar sound effects, minimizing IP infringement, and more. To demonstrate this application scenario, Zilliz built a highly efficient audio similarity search system by combining Milvus with [PANNs](https://arxiv.org/abs/1912.10211)—a large-scale pretrained audio neural networks built for audio pattern recognition.

👉Source code：[zilliz-bootcamp/audio_search](https://github.com/zilliz-bootcamp/audio_search)
![3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/3_419bac3dd2.png)
###### *Audio search system built with Milvus and PANNs.*

#### How to use
1. Upload a zipped audio dataset comprised of .wav files only (other audio file types are not accepted). Alternatively, a [sample dataset](https://zilliz.com/solutions/audio-search) is available for download.
2. Upload a .wav file to use as the search input for finding similar audio.

👉Video: [[Demo] Audio Search Powered by Milvus](https://www.youtube.com/watch?v=0eQHeqriCXw)
#### How it works
Audio is converted into feature vectors using PANNs, large-scale pre-trained audio neural networks built for audio pattern recognition. Then Milvus is used to manage and query the dataset.

**Data processing:**

1. PANNs converts audio from the uploaded dataset to feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Audio feature vector IDs and their corresponding .wav file paths are stored in PostgreSQL.

**Searching for similar audio:**

1. PANNs is used to convert a user’s uploaded audio file into feature vectors.
2. Vector IDs of audio most similar to the uploaded file are retrieved from Milvus by calculating inner product (IP) distance.
3. The system looks up the corresponding audio file paths in MySQL.

 

### Video object detection (computer vision)
Video object detection has applications in computer vision, image retrieval, autonomous driving, and more. To demonstrate this application scenario, Zilliz built a video object detection system by combining Milvus with technologies and algorithms including [OpenCV](https://en.wikipedia.org/wiki/OpenCV), [YOLOv3](https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b), and [ResNet50](https://www.mathworks.com/help/deeplearning/ref/resnet50.html).

👉Source code: [zilliz-bootcamp/video_analysis](https://github.com/zilliz-bootcamp/video_analysis)

![4.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/4_54b4ceb2ad.png)
###### *Video object detection system with Milvus.*

#### How to use
1. Upload a zipped image dataset comprised of .jpg files only (other image file types are not accepted). Ensure that each image file is named by the object it depicts. Alternatively, a [sample dataset](https://zilliz.com/solutions/video-obj-analysis) is available for download. 
2. Upload a video to use for analysis.
3. Click the play button to view the uploaded video with object detection results shown in real time.

👉Video: [[Demo] Video Object Detection System Powered by Milvus](https://www.youtube.com/watch?v=m9rosLClByc)

#### How it works
Object images are converted into 2048-dimensional feature vectors using ResNet50. Then Milvus is used to manage and query the dataset.

**Data processing:**

1. ResNet50 converts object images to 2048-dimensional feature vectors. The vectors are then imported to Milvus and assigned individual IDs.
2. Audio feature vector IDs and their corresponding image file paths are stored in MySQL.

**Detecting objects in video:**

1. OpenCV is used to trim the video.
2. YOLOv3 is used to detect objects in the video.
3. ResNet50 converts detected object images into 2048-dimensional feature vectors.

Milvus searches for the most similar object images in the uploaded dataset. Corresponding object names and image file paths are retrieved from MySQL.

small_blog cover.png

thumbnail_blog cover.png

blog cover.png

How to Make 4 Popular AI Applications with Milvus

Ensuring investments in artificial intelligence are successful requires effectively managing the life cycle of AI models and the applications they are used in. This article explains the concept of machine learning operations, or MLOps, and how the open-source vector database Milvus can be used to advance MLOps programs of varying complexity.

Building machine learning (ML) applications is a complex and iterative process. As more companies realize the untapped potential of unstructured data, the demand for [AI-powered data processing and analytics](https://zilliz.com/blog/Thanks-to-Milvus-Anyone-Can-Build-a-Vector-Database-for-1-Billion-Images) will continue to rise. Without effective machine learning operations, or MLOps, most ML application investments will wither on the vine. Research has found that [as little as 5%](https://www.forbes.com/sites/cognitiveworld/2020/03/31/modelops-is-the-key-to-enterprise-ai/?sh=44c0f5066f5a) of the AI adoptions companies plan to deploy actually reach deployment. Many organizations incur "model debt," where changes in market conditions, and failure to adapt to them, result in unrealized investments in models that linger unrefreshed (or worse, never get deployed at all).

This article explains MLOps, a systemic approach to AI model life cycle management, and how the open-source vector data management platform [Milvus](https://milvus.io/) can be used to operationalize AI at scale.

 

### What is MLOps?

Machine learning operations (MLOps), also known as model operations (ModelOps) or AI model operationalization, is necessary to build, maintain, and deploy AI applications at scale. As companies seek to apply the AI models they develop to hundreds of different scenarios, it is mission critical that models in use, and those under development, are operationalized across the entire organization. MLOps involves monitoring a machine learning model throughout its lifecycle, and governing everything from underlying data to the effectiveness of a production system that relies on a particular model.

![01.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/01_362a07d156.jpg)

###### *MLOps governs the life cycle of a machine learning model.*

Gartner [defines](https://www.gartner.com/en/information-technology/glossary/modelops) ModelOps as the governance and life cycle management of a wide range of operationalized artificial intelligence and decision models. The core functionality of MLOps can be broken down as follows:

- **Continuous integration/continuous delivery (CI/CD):** A set of best practices borrowed from developer operations (DevOps), CI/CD is a method for delivering code changes more frequently and more reliably. [Continuous integration](https://www.gartner.com/en/information-technology/glossary/continuous-integration-ci) promotes implementing code changes in small batches while monitoring them with strict version control. [Continuous delivery](https://www.gartner.com/smarterwithgartner/5-steps-to-master-continuous-delivery/) automates the delivery of applications to various environments (e.g., testing and development environments). 

- **Model development environments (MDE):** A complex process for building, reviewing, documenting, and examining models, MDE help ensure models are created iteratively, documented as they are developed, trusted, and reproducible. Effective MDE ensures models can be explored, researched, and experimented on in a controlled manner.

- **Champion-challenger testing:** Similar to A/B testing methodology used by marketers, [champion-challenger testing](https://medium.com/decision-automation/what-is-champion-challenger-and-how-does-it-enable-choosing-the-right-decision-f57b8b653149) involves experimenting with different solutions to aid the decision-making process that proceeds committing to a single approach. This technique involves monitoring and measuring performance in real time to identify which deviation works best.

- **Model versioning:** As with any complex system, machine learning models are developed in steps by many different people—resulting in data management questions around versions of data and ML models. Model versioning helps manage and govern the iterative process of ML development where data, models, and code may evolve at different rates.

- **Model store and rollback:** When a model is deployed, its corresponding image file must be stored. Rollback and recovery abilities allow MLOps teams to revert to a previous model version if needed.

Using just one model in a production application presents a number of difficult challenges. MLOps is a structured, repeatable method that relies on tools, technology, and best practices to overcome technical or business problems that arise during a machine learning model’s life cycle. Successful MLOps maintains efficiency across teams working to build, deploy, monitor, retrain, and govern AI models and their use in production systems. 

 

### Why is MLOps necessary?

As depicted in the ML model life cycle above, building a machine learning model is an iterative process that involves incorporating new data, retraining models, and dealing with general model decay over time. These are all issues that traditional developer operations, or DevOps, does not address or provide solutions for. MLOps has become necessary as a way to manage investment in AI models and ensure a productive model life cycle. Because machine learning models will be leveraged by a variety of different production systems, MLOps becomes integral to making sure requirements can be met across different environments and amid varying scenarios.

![02.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/02_403e7f2fe2.jpg)

###### *A machine learning model being deployed in a cloud environment that feeds into an application.*

 

The simple illustration above depicts a machine learning model being deployed in a cloud environment that feeds into an application. In this basic scenario, a number of problems could arise that MLOps helps overcome. Because the production application relies on a specific cloud environment, there are latency requirements that the data scientists who developed the ML model don’t have access to. Operationalizing the model life cycle would make it possible for data scientists or engineers with deep knowledge of the model to identify and troubleshoot problems that arise in specific production environments. 

Not only are machine learning models trained in different environments from the production applications they are used in, but they also often rely on historical datasets that differ from the data used in production applications. With MLOps the entire data science team, from those developing the model to people working at the application level, have a means of sharing and requesting information and assistance. The rate at which data and markets change make it imperative that there is as little friction as possible between all key stakeholders and contributors that will come to depend on a given machine learning model. 

### Supporting the transition to Software 2.0

[Software 2.0](https://karpathy.medium.com/software-2-0-a64152b37c35) is the idea that software development will experience a paradigm shift as artificial intelligence increasingly plays a central role in writing AI models that power software applications. Under Software 1.0, development involves programmers writing explicit instructions using a specific programming language (e.g., Python, C++). Software 2.0 is far more abstract. Although people provide input data and set parameters, neural networks are difficult for humans to understand due to their sheer complexity— with typical networks containing millions of weights that impact outcomes (and sometimes billions or trillions). 

DevOps was built around Software 1.0’s reliance on specific instructions dictated by programmers using languages, but never considered the life cycle of a machine learning model that powers a variety of different applications. MLOps addresses the need for the process of managing software development to change alongside the software under development. As Software 2.0 becomes the new standard for computer-based problem solving, having the right tools and processes for managing model life cycles will make or break investments in new technology. Milvus is an open-source vector similarity search engine built to support the transition to Software 2.0 and manage model life cycles with MLOps.

![03.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/03_c63c501995.jpg)

###### *How Milvus supports the transition to Software 2.0.*

 

### Operationalizing AI at scale with Milvus

Milvus is a vector data management platform that was made specifically for storing, querying, updating, and maintaining massive, trillion-scale vector datasets. The platform powers vector similarity search and can integrate with widely adopted index libraries, including Faiss, NMSLIB, and Annoy. By pairing AI models that convert unstructured data to vectors with Milvus, applications spanning new drug development, biometric analysis, recommendation systems and much more can be created. 

[Vector similarity search](https://blog.milvus.io/vector-similarity-search-hides-in-plain-view-654f8152f8ab) is the go-to solution for unstructured data data processing and analytics, and vector data is quickly emerging as a core data type. A comprehensive data management system like Milvus facilitates operationalizing AI in many ways, including:

- Providing an environment for model training that ensures more aspects of development are done in one place, facilitating cross team collaboration, model governance, and more. 

- Offering a comprehensive set of APIs that support popular frameworks such as Python, Java, and Go, making it easy to integrate a common set of ML models. 

- Compatibility with Google Colaboratory, a Jupyter notebook environment that runs in a browser, simplifies the process of compiling Milvus from source code and running basic Python operations.

- Automated machine learning (AutoML) functionality makes it possible to automate the tasks associated with applying machine learning to real world problems. Not only does AutoML lead to efficiency improvements, but it makes it possible for non-experts to take advantage of machine learning models and techniques.


Regardless of the machine learning applications you’re building today, or the plans you have for applications in the future, Milvus is a flexible data management platform created with Software 2.0 and MLOps in mind. To learn more about Milvus or make contributions, find the project on [Github](https://github.com/milvus-io). To get involved with the community or ask questions, join our [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ) channel. Hungry for more content? Check out the following resources:

- [Milvus Is an Open-Source Scalable Vector Database](https://zilliz.com/blog/Milvus-Is-an-Open-Source-Scalable-Vector-Database)
- [Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search](https://zilliz.com/blog/Milvus-Was-Built-for-Massive-Scale-Think-Trillion-Vector-Similarity-Search)
- [Set Up Milvus in Google Colaboratory for Easy ML Application Building](https://zilliz.com/blog/Set-Up-Milvus-in-Google-Colaboratory-for-Easy-ML-Application-Building)

large_milvus.jpg

medium_milvus.jpg

small_milvus.jpg

thumbnail_milvus.jpg

milvus.jpg

MLOps is superseding DevOps as we make the transition to Software 2.0. Find  what model operations is, and how the open-source vector database Milvus supports it. 

Operationalize AI at Scale with Software 2.0, MLOps, and Milvus

Beike wanted to create a personalized home search experience. Discover how they leveraged Milvus to build an intelligent real estate search platform capable of providing tailored recommendations in near real-time. 

Artificial intelligence (AI) has [powerful applications](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#d62b) in real estate that are transforming the home search process. Tech savvy real estate professionals have been taking advantage of AI for years, recognizing its ability to help clients find the right home faster and simplify the process of purchasing property. The coronavirus pandemic has [accelerated](https://www.pwc.com/ca/en/industries/real-estate/emerging-trends-in-real-estate-2021/shifting-landscape-proptech.html) interest, adoption, and investement in property technology (or proptech) worldwide, suggesting it will play an increasingly greater role in the real estate industry moving forward. 

This article explores how Beike used vector similarity search to build a house hunting platform that provides personalized results and recommends listings in near real-time.

**Jump to:**

- [What is vector similarity search?](#what-is-vector-similarity-search)

- [How does Milvus manage vector data?](#how-does-milvus-manage-vector-data)

- [How does Beike use Milvus to make house hunting smarter?](#how-does-beike-use-milvus-to-make-house-hunting-smarter)

 

### What is vector similarity search?

[Vector similarity search](https://medium.com/unstructured-data-service/vector-similarity-search-hides-in-plain-view-654f8152f8ab) has applications spanning a wide variety of artificial intelligence, deep learning, and traditional vector calculation scenarios. The proliferation of AI technology is in part attributed to vector search and its ability to make sense of unstructured data, which includes things like images, video, audio, behavior data, documents, and much more.

Unstructured data makes up an estimated 80-90% of all data, and extracting insights from it is quickly becoming a requrement for businesses that want to remain competitive in an ever-changing world. Increasing demand for unstructured data analytics, rising compute power, and declining compute costs have made AI-enabled vector search more accessible than ever.

![beike-blog-img1.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/beike_blog_img1_2dc95cac08.jpg)
###### *Understanding the difference between structured and unstructured data.*

Traditionally, unstructured data has been a challenge to process and analyze at scale because it doesn't follow a predefined model or organizational structure. Neural networks (e.g., CNN, RNN, and BERT) make it possible to convert unstructured data into feature vectors, a numerical data format that can be easily interpreted by computers. Algorithms are then used to calculate similarity between vectors using metrics like cosine similarity or Euclidean distance.

Ultimately, vector similarity search is a broad term that desribes techniques for identifying similar things in massive datasets. Beike uses this technology to power an intelligent home search engine that automatically recommends listings based on individual user preferences, search history, and property criteria—accelerating the real estate search and purchase process. Milvus is an open-source vector database that connects information with algorithms, enabling Beike to develop and manage its AI real estate platform.

 

### How does Milvus manage vector data?

Milvus was built specifically for large-scale vector data management, and has applications spanning image and video search, chemical similarity analysis, personalized recommendation systems, conversational AI, and much more. Vector datasets stored in Milvus can be efficiently queried, with most implementations following this general process:

![beike-blog-img2.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/beike_blog_img2_d5abb58f95.jpg)
###### *How does Milvus manage vector data?*

 

### How does Beike use Milvus to make house hunting smarter?

Commonly described as China's answer to Zillow, [Beike](https://bj.ke.com/) is an online platform that allows real estate agents to list properties for rent or sale. To help improve the home search experience for house hunters, and to help agents close deals faster, the company built an AI-powered search engine for its listing database. Beike’s real estate listing database was converted into feature vectors then fed into Milvus for indexing and storage. Milvus is then used to conduct similarity search based on an input listing, search criteria, user profile, or other criteria.

For example, when searching for more homes similar to a given listing, features such as floor plan, size, orientation, interior finishings, paint colors, and more are extracted. Since the original database of property listings listing data has been [indexed](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212), searches can be conducted in mere milliseconds. Beike’s final product had an average query time of 113 milliseconds on a dataset containing over 3 million vectors. However, Milvus is capable of maintaining efficient speeds on trillion-scale datasets—making light work of this relatively small real estate database. In general, the system follows the following process:

1. Deep learning models (e.g., CNN, RNN, or BERT) convert unstructured data to feature vectors, which are then imported to Milvus.

2. Milvus stores and indexes the feature vectors.

3. Milvus returns similarity search results based on user queries. 

![milvus-overview-diagram.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/milvus_overview_diagram_d17cda0e47.png)

###### *An overview of Milvus.*

 

Beike’s intelligent real estate search platform is powered by a recommendation algorithm that calculates vector similarity using cosine distance. The system finds similar homes based on favorite listings and search criteria. At a high level, it works as follows:

1. Based on an input listing, characteristics such as floor plan, size, and orientation are used to extract 4 collections of feature vectors.

2. The extracted feature collections are used to perform similarity search in Milvus. Results of the query for each collection of vectors are a measure of similarity between the input listing and other similar listings. 

3. The search results from each of the 4 vector collections are compared then used to recommend similar homes. 

![beike-intelligent-house-platform-diagram.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/beike_intelligent_house_platform_diagram_6e278da118.jpg)

###### *An overview of Beike’s intelligent house hunting platform.*

 

As the figure above shows, the system implements an A/B table switching mechanism for updating data. Milvus stores the data for the first T days in table A, on day T+1 it starts to store data in table B, on day 2T+1, it starts to rewrite table A, and so forth. 

 

### To learn more about making things with Milvus, check out the following resources:

- [Building an AI-Powered Writing Assistant for WPS Office](https://zilliz.com/blog/Building-an-AI-Powered-Writing-Assistant-with-WPS-Office)

- [Making with Milvus: AI-Powered News Recommendation Inside Xiaomi's Mobile Browser](https://zilliz.com/blog/Making-with-Milvus-AI-Powered-News-Recommendation-Inside-Xiaomi-Mobile-Browser)

large_blog_realistate search.jpg

medium_blog_realistate search.jpg

small_blog_realistate search.jpg

thumbnail_blog_realistate search.jpg

blog_realistate search.jpg

AI is transforming the real estate industry, discover how intelligent proptech accelerates the home search and purchase process. 

Making With Milvus: AI-Infused Proptech for Personalized Real Estate Search

Learn how to use Milvus data migration tool to greatly improve efficiency of data management and reduce DevOps costs.

### Overview
[MilvusDM](https://github.com/milvus-io/milvus-tools) (Milvus Data Migration) is an open-source tool designed specifically for importing and exporting data files with Milvus. MilvusDM can greatly improve data mangement efficiency and reduce DevOps costs in the following ways:

- [Faiss to Milvus](#faiss-to-milvus): Import unzipped data from Faiss to Milvus.

- [HDF5 to Milvus](#hdf5-to-milvus): Import HDF5 files to Milvus.

- [Milvus to Milvus](#milvus-to-milvus): Migrate data from a source Milvus to a different target Milvus.

- [Milvus to HDF5](#milvus-to-hdf5): Save data in Milvus as HDF5 files.

![milvusdm blog 1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/milvusdm_blog_1_199cbdebe7.png)
###### *Importing and exporting data files with Milvus.*

MilvusDM is hosted on [Github](https://github.com/milvus-io/milvus-tools) and can be easily installed by running the command line ```pip3 install pymilvusdm```. MilvusDM allows you to migrate data in a specific collection or partition. In the following sections, we will explain how to use each data migration type. 

 


### Faiss to Milvus

#### Steps
1.Download **F2M.yaml**: 

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/F2
```

2.Set the following parameters:

- ``` data_path ```: Data path (vectors and their corresponding IDs) in Faiss.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: Delete data before insertion if the collection or partition already exists.

- ```dest_collection_name```: Name of receiving collection for data import.

- ```dest_partition_name```: Name of receiving partition for data import. 

- ```collection_parameter```: Collection-specific information such as vector dimension, index file size, and distance metric.

```
F2M:
 milvus_version: 1.0.0
 data_path: '/home/data/faiss.index'
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'append' # 'skip/append/overwrite'
 dest_collection_name: 'test'
 dest_partition_name: ''
 collection_parameter:
 dimension: 256
 index_file_size: 1024
 metric_type: 'L2'
```
3.Run **F2M.yaml:**

```
$ milvusdm --yaml F2M.yaml
```

#### Sample Code

1.Read Faiss files to retrieve vectors and their corresponding IDs.

```
ids, vectors = faiss_data.read_faiss_data()
```

2.Insert the retrieved data into Milvus: 

```
insert_milvus.insert_data(vectors, self.dest_collection_name, self.collection_parameter, self.mode, ids, self.dest_partition_name)
```

 

### HDF5 to Milvus
#### Steps
1.Download **H2M.yaml**.

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/H2M.yaml
```

2.Set the following parameters:

- ```data_path```: Path to the HDF5 files. 

- ```data_dir```: Directory holding the HDF5 files.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: Delete data before insertion if the collection or partition already exists.

- ```dest_collection_name```: Name of receiving collection for data import.

- ```dest_partition_name```: Name of receiving partition for data import.

- ```collection_parameter```: Collection-specific information such as vector dimension, index file size, and distance metric. 

> Set either ```data_path``` or ```data_dir```. Do **not** set both. Use ```data_path``` to specify multiple file paths, or ```data_dir``` to specify the directory holding your data file.

```
H2M:
 milvus-version: 1.0.0
 data_path:
 - /Users/zilliz/float_1.h5
 - /Users/zilliz/float_2.h5
 data_dir:
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'overwrite' # 'skip/append/overwrite'
 dest_collection_name: 'test_float'
 dest_partition_name: 'partition_1'
 collection_parameter:
 dimension: 128
 index_file_size: 1024
 metric_type: 'L2'
```
3.Run **H2M.yaml:**

```
$ milvusdm --yaml H2M.yaml
```

#### Sample Code

1.Read the HDF5 files to retrieve vectors and their corresponding IDs:

```
vectors, ids = self.file.read_hdf5_data()
```

2.Insert the retrieved data into Milvus:

```
ids = insert_milvus.insert_data(vectors, self.c_name, self.c_param, self.mode, ids,self.p_name)
```

 

### Milvus to Milvus
#### Steps
1.Download **M2M.yaml**.

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2M.yaml
```

2.Set the following parameters:

- ```source_milvus_path```: Source Milvus work path.

- ```mysql_parameter```: Source Milvus MySQL settings. If MySQL is not used, set mysql_parameter as ''.

- ```source_collection```: Names of the collection and its partitions in the source Milvus.

- ```dest_host```: Milvus server address. 

- ```dest_port```: Milvus server port.

- ```mode```: Data can be imported to Milvus using the following modes:

 - Skip: Ignore data if the collection or partition already exists.

 - Append: Append data if the collection or partition already exists.

 - Overwrite: If the collection or partition already exists, delete the data before inserting it.Delete data before insertion if the collection or partition already exists.

```
M2M:
 milvus_version: 1.0.0
 source_milvus_path: '/home/user/milvus'
 mysql_parameter:
 host: '127.0.0.1'
 user: 'root'
 port: 3306
 password: '123456'
 database: 'milvus'
 source_collection:
 test:
 - 'partition_1'
 - 'partition_2'
 dest_host: '127.0.0.1'
 dest_port: 19530
 mode: 'skip' # 'skip/append/overwrite'
```

3.Run **M2M.yaml.**

```
$ milvusdm --yaml M2M.yaml
```

#### Sample Code

1.According to a specified collection or partition's metadata, read the files under **milvus/db** on your local drive to retrieve vectors and their corresponding IDs from the source Milvus. 

```
collection_parameter, _ = milvus_meta.get_collection_info(collection_name)
r_vectors, r_ids, r_rows = milvusdb.read_milvus_file(self.milvus_meta, collection_name, partition_tag) 
```

2.Insert the retrieved data into the target Milvus.


```
milvus_insert.insert_data(r_vectors, collection_name, collection_parameter, self.mode, r_ids, partition_tag)
```

 

### Milvus to HDF5
#### Steps
1.Download **M2H.yaml**:

```
$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2H.yaml
```


2.Set the following parameters:

- ```source_milvus_path```: Source Milvus work path.

- ```mysql_parameter```: Source Milvus MySQL settings. If MySQL is not used, set mysql_parameter as ''.

- ```source_collection```: Names of the collection and its partitions in the source Milvus.

- ```data_dir```: Directory for holding the saved HDF5 files.

```
M2H:
 milvus_version: 1.0.0
 source_milvus_path: '/home/user/milvus'
 mysql_parameter:
 host: '127.0.0.1'
 user: 'root'
 port: 3306
 password: '123456'
 database: 'milvus'
 source_collection: # specify the 'partition_1' and 'partition_2' partitions of the 'test' collection.
 test:
 - 'partition_1'
 - 'partition_2'
 data_dir: '/home/user/data'
```

3.Run **M2H.yaml**:

```
$ milvusdm --yaml M2H.yaml
```

#### Sample Code
1.According to a specified collection or partition's metadata, read the files under **milvus/db** on your local drive to retrieve vectors and their corresponding IDs.

```
collection_parameter, version = milvus_meta.get_collection_info(collection_name)
r_vectors, r_ids, r_rows = milvusdb.read_milvus_file(self.milvus_meta, collection_name, partition_tag)
```

2.Save the retrieved data as HDF5 files.

```
data_save.save_yaml(collection_name, partition_tag, collection_parameter, version, save_hdf5_name)
```

### MilvusDM File Structure

The flow chart below shows how MilvusDM performs different tasks according to the YAML file it receives:

![milvusdm blog 2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/milvusdm_blog_2_7824b16e5e.png)
###### *How does MilvusDM work?*

MilvusDM file structure:

- pymilvusdm

 - core

 - **milvus_client.py**: Performs client operations in Milvus.

 - **read_data.py**: Reads the HDF5 data files on your local drive. (Add your code here to support reading data files in other formats.)

 - **read_faiss_data.py**: Reads the data files in Faiss.

 - **read_milvus_data.py**: Reads the data files in Milvus.

 - **read_milvus_meta.py**: Reads the metadata in Milvus.

 - **data_to_milvus.py**: Creates collections or partitions based on parameters in YAML files and imports the vectors and the corresponding vector IDs into Milvus.

 - **save_data.py**: Saves the data as HDF5 files.

 - **write_logs.py**: Writes logs during runtime.

 - **faiss_to_milvus.py**: Imports data from Faiss into Milvus.

 - **hdf5_to_milvus.py**: Imports data in HDF5 files into Milvus.

 - **milvus_to_milvus.py**: Migrates data from a source Milvus to the target Milvus.

 - **milvus_to_hdf5.p**y: Exports data in Milvus and saves them as HDF5 files.

 - **main.py**: Performs corresponding tasks according to the received YAML file.

 - **setting.py**: Configurations relating to running the MilvusDM code.

- **setup.py**: Creates **pymilvusdm** file packages and uploads them to PyPI (Python Package Index).

 

### Recap

MilvusDM primarily handles migrating data in and out of Milvus, which includes Faiss to Milvus, HDF5 to Milvus, Milvus to Milvus, and Milvus to HDF5. 

The following features are planned for upcoming releases:

- Import binary data from Faiss to Milvus.

- Blocklist and allowlist for data migration between source Milvus and target Milvus.

- Merge and import data from multiple colletions or partitions in source Milvus into a new collection in target Milvus.

- Backup and recovery of the Milvus data.

The MilvusDM project is open sourced on [Github](https://github.com/milvus-io/milvus-tools). Any and all contributions to the project are welcome. Give it a star 🌟, and feel free to file an [issue](https://github.com/milvus-io/milvus-tools/issues) or submit your own code!

large_Generic Tool Announcement.jpg

medium_Generic Tool Announcement.jpg

small_Generic Tool Announcement.jpg

thumbnail_Generic Tool Announcement.jpg

Generic Tool Announcement.jpg

Introducing Milvus Data Migration Tool

iYUNDONG continually seeks to support its users with real-time image retrieval services. When it decided to develop its App, it chose Milvus to build its core image search system. Discover how Milvus, an open-source vector database, can help build an intelligent image retrieval system that can extract sport event highlights.

iYUNDONG is an Internet company aiming to engage more sport lovers and participants of events such as marathon races. It builds [artificial intelligence (AI)](https://en.wikipedia.org/wiki/Artificial_intelligence) tools that can analyze media captured during sporting events to automatically generate highlights. For example, by uploading a selfie, a user of the iYUNDONG sports App who took part in a sport event can instaneously retrieve his or her own photos or video clips from a massive media dataset of the event.

 

One of the key features of iYUNDONG App is called “Find me in motion“. Photographers usually take massive volumes of photos or videos during a sporting event such as a marathon race, and would upload the photos and videos in real time to the iYUNDONG media database. Marathon runners who want to see their highlighted moments can retrieve pictures including themselves simply by uploading one of their selfies. This saves them a lot of time because an image retrieval system in the iYUNDONG App does all the image matching. [Milvus](http://milvus.io/) is adopted by iYUNDONG to power this system as Milvus can greatly accelerate the retrieval process and return highly accurate results.

 

**Jump to:**
- [Difficulties and solutions](#difficulties-and-solutions)
- [What is Milvus](#what-is-milvus)
- [Why Milvus](#why-milvus)
- [System and Workflow](#system-and-workflow)
- [iYUNDONG App Interface](#iyundong-app-interface)
- [Conclusion](#conclusion)

 

### Difficulties and solutions

iYUNDONG faced the following issues and successfully found corresponding solutions when building its image retrieval system. 

- Event photos must be immediately available for search.

iYUNDONG developed a function called InstantUpload to ensure that event photos are available for search immediately after they are uploaded.

- Storage of massive datasets

Massive data such as photos and videos are uploaded to the iYUNDONG backend every millisecond. So iYUNDONG decided to migrate onto cloud storage systems including [AWS](https://aws.amazon.com/), [S3](https://aws.amazon.com/s3/?nc1=h_ls), and [Alibaba Cloud Object Storage Service (OSS)](https://www.alibabacloud.com/product/oss) for handling gargantuan volumes of unstructured data in a secure, fast and reliable way.

- Instant reading

In order to achieve instant reading, iYUNDONG developed its own sharding middleware to achieve horizontal scalability easily and mitigate the impact on the system from disk reading. In addition, [Redis](https://redis.io/) is used to serve as a caching layer to ensure consistent performance in situation of high concurrency.

- Instant extraction of facial features

In order to accurately and efficiently extract facial features from user-uploaded photos, iYUNDONG developed a proprietary image conversion algorithm which converts images into 128-dimensional feature vectors. Another issue encountered was that, oftentimes, many users and photographers uploaded images or videos simultaneously. So system engineers needed to take dynamic scalability into consideration when deploying the system. More specifically, iYUNDONG fully leveraged its elastic compute service (ECS) on the cloud to achieve dynamic scaling.

- Quick and large-scale vector search

iYUNDONG needed a vector database to store its large number of feature vectors extracted by AI models. According to its own unique business application scenario, iYUNDONG expected the vector database to be able to: 
1. Perform blazing fast vector retrieval on ultra-large datasets. 
2. Achieve mass storage at lower costs.

Initially, an average of 1 million images were processed annually, so iYUNDONG stored all its data for search in RAM. However, in the past two years, its business boomed and saw an exponential growth of unstructured data – the number of images in iYUNDONG’s database exceeded 60 milllion in 2019, meaning that there were more than 1 billion feature vectors that needed to be stored. A tremendous amount of data inevitably made the iYUNDONG system heavily-built and resource-consuming. So it had to continuously invest in hardware facilities to ensure high performance. Specifically, iYUNDONG deployed more search servers, larger RAM, and a better-performing CPU to achieve greater efficiency and horizontal scalability. However, one of the defects of this solution was that it drove the operating costs prohibitively high. Therefore, iYUNDONG started to explore a better solution to this issue and pondered on leveraging vector index libraries like Faiss to save costs and better steer its business. Finally iYUNDONG chose open-source vector database Milvus.

 

### What is Milvus

Milvus is an open-source vector database that is easy to use, highly flexible, reliable, and blazing fast. Combined with various deep learning models such as photo and voice recognition, video processing, natural language processing, Milvus can process and analyze unstructured data that are converted into vectors by using various AI algorithms. Below is the workflow of how Milvus processes all unstructured data:

● Unstructured data are converted into embedding vectors by deep learning models or other AI algorithms.

● Then embedding vectors are inserted into Milvus for storage. Milvus also builds indexes for those vectors.

● Milvus performs similarity search and returns accurate search results based on various business needs.

![iYUNDONG Blog 1.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/i_YUNDONG_Blog_1_d8abe065ae.png)
###### *An overview of Milvus.*

 

### Why Milvus
Since the end of 2019, iYUNDONG has carried out a series of testings on using Milvus to power its image retrieval system. The testing results turned out that Milvus outperforms other mainstream vector databases as it supports multiple indexes and can efficiently reduce RAM usage, significantly compressing the timeline for vector similarity search. 

Moreover, new versions of Milvus are released regularly. Over the testing period, Milvus has went through multiple version updates from v0.6.0 to v0.10.1.

Additionally, with its active open-source community and powerful out-of-the-box features, Milvus allows iYUNDONG to operate on a tight development budget.

 

### System and Workflow
iYUNDONG’s system extracts facial features by detecting faces in event photos uploaded by photographers first. Then those facial features are converted into 128-dimensional vectors and stored in the Milvus library. Milvus creates indexes for those vectors and can instantaneously return highly accurate results. 

Other additional information such as photo IDs and coordinates indicating the position of a face in a photo are stored in a third-party database.

Each feature vector has its unique ID in the Milvus library. iYUNDONG adopted the [Leaf algorithm](https://github.com/Meituan-Dianping/Leaf), a distributed ID generation service developed by [Meituan](https://about.meituan.com/en) basic R&D platform, to associate the vector ID in Milvus with its corresponing additional information stored in another database. By combining the feature vector and the additional information, the iYUNDONG system can return similar results upon user search.

 

### iYUNDONG App Interface
A series of latest sports events are listed on the homepage. By tapping one of the events, users can see the full details.

After tapping the button on the top of the photo gallery page, users can then upload a photo of their own to retrieve images of their highlights. 

![iyundong-interface.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/iyundong_interface_3da684d206.jpg)
###### *iYUNDONG app interface.*

 

### Conclusion
This article introduces how iYUNDONG App builds an intelligent image retrieval system that can return accurate search results based on user uploaded photos varying in resolution, size, clarity, angle, and other ways that complicate similarity search. With the help of Milvus, iYUNDONG App can successfully run millisecond-level queries on a database of 60+ million images. And accuracy rate of photo retrieval is constantly above 92%. Milvus makes it easier for iYUNDONG to create a powerful, enterprise-grade image retrieval system in a short time with limited resources.


Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

large_blog_iyundong.jpg

medium_blog_iyundong.jpg

small_blog_iyundong.jpg

thumbnail_blog_iyundong.jpg

blog_iyundong.jpg

Making with Milvus: Intelligent image retrieval system for sports App iYUNDONG

Extracting Event Highlights Using iYUNDONG Sports App

We are proud to announce Milvus v1.0, a stable, long-term support version of the world’s most popular open-source vector database. Discover what Milvus v1.0 has to offer, and plans for the platform’s future.

Zilliz is proud to announce the release of Milvus v1.0. After months of extensive testing Milvus v1.0, which is based on a stable version of Milvus v0.10.6, is available for use.

Milvus v1.0 offers the following key features:

- Support for mainstream similarity metrics, including Euclidean distance, inner product, Hamming distance, Jaccard coefficient, and more.
- Integration with, and improvements to, SOTA ANNs algorithms, including Faiss, Hnswlib, Annoy, NSG, and more. 
- Scale-out capability through the Mishards sharding proxy.
- Support for processors commonly used in AI scenarios, including X86, Nvidia GPU, Xilinx FPGA, and more.

See the [Release Notes](https://www.milvus.io/docs/v1.0.0/release_notes.md) for additional Milvus v1.0 features. 

Milvus is an ongoing open-source software (OSS) project. Its first major release has the following implications for users:

- Milvus v1.0 will receive long-term support (3+ years).
- The most stable Milvus release to date is well structured and ready for integration with existing AI ecosystems.

### The first version of Milvus with long-term support

Thanks in part to sponsorship from Zilliz, the Milvus community will provide bug fix support for Milvus v1.0 until December 31st, 2024. New features will be available only in releases following v1.0.

See [The Milvus release guideline](https://milvus.io/docs/v1.0.0/milvus_release_guideline.md) for information about release cadences and more.

### Toolchain enhancements and seamless AI ecosystem integration

Beginning with v1.0, Milvus' toolchain will be a primary development focus. We plan to create the necessary tooling and utilities to meet the needs of the Milvus user community. 

Stability makes integrating Milvus with AI ecosystems a breeze. We are seeking further collaboration between the Milvus community and other AI-focused OSS communities. We encourage contributions to the new AI ASICs (application-specific integrated circuits) in Milvus. 

### The future of Milvus

We believe Milvus has a bright future thanks to the following factors:

- Regular contributions from developers in the Milvus community.
- Support for integration with any cloud-native environment.

We have drafted [community charters](https://milvus.io/docs/v1.0.0/milvus_community_charters.md) to help guide, nurture, and advance the Milvus community as our technology and user base grows. The charters include several technical decisions made to attract more participants to the community.

- Golang will now be used to develop the Milvus engine however, the ANNS algorithm component will still be developed in C++.
- The forthcoming distributed version of Milvus will use existing cloud components as much as possible.

We are thrilled to partner with the open-source software community to build the next-generation cloud data fabric made for AI. Let's get to work!

### Don’t be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/)
- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).
- Connect with us on [Twitter](https://twitter.com/milvusio).

Milvus v1.0 is available now. This stable, long-term support version powers image/video search, chatbots, and many more AI applications.

Milvus 1.0: The World's Most Popular Open-Source Vector Database Just Got Better

Discover how Xiaomi leveraged AI and Milvus to build an intelligent news recommendation system capable of finding the most relevant content for users of its mobile web browser. 

From social media feeds to playlist recommendations on Spotify, [artificial intelligence](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View) already plays a major role in the content we see and interact with each day. In an effort to differentiate their mobile web browser, multinational electronics manufacturer Xiaomi built an AI-powered news recommendation engine. [Milvus](https://milvus.io/), an open-source vector database built specifically for similarity search and artificial intelligence, was used as the application’s core data management platform. This article explains how Xiaomi built its AI-powered news recommendation engine, and how Milvus and other AI algorithms were used. 

 


### Using AI to suggest personalized content and cut through news noise

With the New York Times alone publishing over [230 pieces](https://www.theatlantic.com/technology/archive/2016/05/how-many-stories-do-newspapers-publish-per-day/483845/) of content each day, the sheer volume of articles produced makes it impossible for individuals to get a comprehensive view of all the news. To help sift through large volumes of content, and recommend the most relevant or interesting pieces, we increasingly turn to AI. Although recommendations remain far from perfect, machine learning is increasingly necessary to cut through the constant stream of new information pouring out of our increasingly complex and interconnected world.

Xiaomi makes and invests in smartphones, mobile apps, laptops, home appliances, and many more products. In an effort to differentiate a mobile browser that comes preinstalled on many of 40+ million smartphones the company sells each quarter, Xiaomi built a news recommendation system into it. When users launch Xiaomi’s mobile browser, artificial intelligence is used to recommend similar content based on user search history, interests, and more. Milvus is an open-source vector similarity search database used to accelerate retrieval of related articles.

 

### How does AI-powered content recommendation work?

At its core, news recommendation (or any other type of content recommendation system) involves comparing input data to a massive database to find similar information. Successful content recommendation involves balancing relevance with timeliness, and efficiently incorporating huge volumes of new data—often in real time. 

To accommodate massive datasets, recommendation systems are typically divided into two stages:

1. **Retrieval**: During retrieval, content is narrowed down from the broader library based on user interests and behavior. In Xiaomi’s mobile browser, thousands of pieces of content are selected from a massive dataset that contains millions of news articles.
2. **Sorting**: Next, content selected during retrieval is sorted according to certain indicators before being pushed to the user. As users engage with recommended content, the system adapts in real time to provide more relevant suggestions. 

News content recommendations need to be made in real-time based on user behavior and recently published content. Additionally, suggested content must match user interests and search intent as much as possible.

 

### Milvus + BERT = intelligent content suggestions

Milvus is an open-source vector similarity search database that can be integrated with deep learning models to power applications spanning natural language processing, identity verification, and much more. Milvus indexes large vector datasets to make search more efficient, and supports a variety of popular AI frameworks to simplify the process of developing machine learning applications. These characteristics make the platform the ideal for storing and querying vector data, a critical component of many machine learning applications. 

Xiaomi selected Milvus to manage vector data for its intelligent news recommendation system because it is fast, reliable, and requires minimal configuration and maintenance. However, Milvus must be paired with an AI algorithm to build deployable applications. Xiaomi selected BERT, short for Bidirectional Encoder Representation Transformers, as the language representation model in its recommendation engine. BERT can be used as a general NLU (natural language understanding) model that can drive a number of different NLP (natural language processing) tasks. Its key features include:

- BERT’s transformer is used as the main framework of the algorithm and is capable of capturing explicit and implicit relationships within and between sentences.
- Multi-task learning goals, masked language modeling (MLM), and next sentence prediction (NSP).
- BERT performs better with greater amounts of data, and can enhance other natural language processing techniques such as Word2Vec by acting as a conversion matrix.

![Blog_Xiaomi_1.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Xiaomi_1_6301344312.jpeg)
###### *Milvus working with BERT.*

 

BERT’s network architecture uses a multi-layer transformer structure that abandons the traditional RNN and CNN neural networks. It works by converting the distance between two words at any position into one through its attention mechanism, and solves the dependency issue that has persisted in NLP for some time.

![Blog-Xiaomi-2.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Xiaomi_2_fe5cf2e401.jpeg)

###### *A transformer’s network architecture in BERT.*

 

![Blog-Xiaomi-3.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Xiaomi_3_5d10b51440.jpeg)

###### *BERT’s network structure. ‘Trm’ represents the transformer network architecture depicted above.*

 

BERT provides a simple and a complex model. The corresponding hyperparameters are as follows: BERT BASE: L = 12, H = 768, A = 12, total parameter 110M; BERT LARGE: L = 24, H = 1024, A = 16, the total number of parameters is 340M.

In the above hyperparameters, L represents the number of layers in the network (i.e. the number of Transformer blocks), A represents the number of self-Attention in Multi-Head Attention, and the filter size is 4H.

 


### Xiaomi’s content recommendation system 

Xiaomi browser-based news recommender system relies on three key components: vectorization, ID mapping, and approximate nearest neighbor (ANN) service. 

Vectorization is process where article titles are converted into general sentence vectors. The SimBert model, based on BERT, is used in Xiaomi’s recommendation system. SimBert is a 12-layer model with a hidden size of 768. Simbert uses the training model Chinese L-12_H-768_A-12 for continuous training (training task being “metric learning +UniLM”, and has trained 1.17 million steps on a signle TITAN RTX with the Adam optimizer (learning rate 2e-6, batch size 128). Simply put, this is an optimized BERT model.

ANN algorithms compare vectorized article titles to the entire news library stored in Milvus, then return similar content for users. ID mapping is used to obtain relevant information such as page views and clicks for corresponding articles.

![Blog-Xiaomi-N1.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Xiaomi_N1_f4749b3131.jpeg)

###### *How content recommendation works.*

 

The data stored in Milvus that powers Xiaomi’s news recommendation engine is constantly being updated, including additional articles and activity information. As the system incorporates new data, old data must be purged. In this system, full data updates are done for the first T-1 days and incremental updates are done in the subsequent T days.

At defined intervals, old data is deleted and processed data of the T-1 days is inserted into the collection. Here newly generated data is incorporated in real time. Once new data is inserted, similarity search is conducted in Milvus. Retrieved articles are again sorted by click rate and other factors, and the top content is shown to users. In a scenario like this where data is frequently updated and results must be delivered in real time, Milvus' ability to rapidly incorporate and search new data makes it possible to drastically accelerate news content recommendation in Xiaomi’s mobile browser. 

 

### Milvus makes vector similarity search better

Vectorizing data and then calculating similarity between vectors is the most commonly used retrieval technology. The rise of ANN-based vector similarity search engines has greatly improved the efficiency of vector similarity calculations. Compared with similar solutions, Milvus offers optimized data storage, abundant SDKs, and a distributed version that greatly reduces the workload of building a retrieval layer. Additionally, Milvus' active open-source community is a powerful resource that can help answer questions and troubleshoot problems as they arise.

If you would like to learn more about vector similarity search and Milvus, check out the following resources:

- Check out [Milvus](https://github.com/milvus-io/milvus) on Github. 
- [Vector Similarity Search Hides in Plain View](https://zilliz.com/blog/Vector-Similarity-Search-Hides-in-Plain-View)
- [Accelerating Similarity Search on Really Big Data with Vector Indexing](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing)

Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

 Discover how Xiaomi leveraged AI and Milvus to build an intelligent news recommendation system capable of finding the most relevant content for users of its mobile web browser. 

Making with Milvus: AI-Powered News Recommendation Inside Xiaomi's Mobile Browser

How to build a deep learning powered recommender system

## Background Introduction

With the continuous development of network technology and the ever-expanding scale of e-commerce, the number and variety of goods grow rapidly and users need to spend a lot of time to find the goods they want to buy. This is information overload. In order to solve this problem, recommendation system came into being.

The recommendation system is a subset of the Information Filtering System, which can be used in a range of areas such as movies, music, e-commerce, and Feed stream recommendations. The recommendation system discovers the user’s personalized needs and interests by analyzing and mining user behaviors, and recommends information or products that may be of interest to the user. Unlike search engines, recommendation system do not require users to accurately describe their needs, but model their historical behavior to proactively provide information that meets user interests and needs.

In this article we use PaddlePaddle, a deep learning platform from Baidu, to build a model and combine Milvus, a vector similarity search engine, to build a personalized recommendation system that can quickly and accurately provide users with interesting information.

## Data Preparation

We take MovieLens Million Dataset (ml-1m) [1] as an example. The ml-1m dataset contains 1,000,000 reviews of 4,000 movies by 6,000 users, collected by the GroupLens Research lab. The original data includes feature data of the movie, user feature, and user rating of the movie, you can refer to ml-1m-README [2] .

ml-1m dataset includes 3 .dat articles: movies.dat、users.dat and ratings.dat.movies.dat includes movie’s features, see example below:

 MovieID::Title::Genres
 1::ToyStory(1995)::Animation|Children's|Comedy

This means that the movie id is 1, and the title is 《Toy Story》, which is divided into three categories. These three categories are animation, children, and comedy.

users.dat includes user’s features, see example below:

 UserID::Gender::Age::Occupation::Zip-code
 1::F::1::10::48067

This means that the user ID is 1, female, and younger than 18 years old. The occupation ID is 10.

ratings.dat includes the feature of movie rating, see example below:

UserID::MovieID::Rating::Timestamp
1::1193::5::978300760

That is, the user 1 evaluates the movie 1193 as 5 points.

## Fusion Recommendation Model

In the film personalized recommendation system we used the Fusion Recommendation Model [3] which PaddlePaddle has implemented. This model is created from its industrial practice.

First, take user features and movie features as input to the neural network, where:

- The user features incorporate four attribute information: user ID, gender, occupation, and age.
- The movie feature incorporate three attribute information: movie ID, movie type ID, and movie name.

For the user feature, map the user ID to a vector representation with a dimension size of 256, enter the fully connected layer, and do similar processing for the other three attributes. Then the feature representations of the four attributes are fully connected and added separately.

For movie features, the movie ID is processed in a manner similar to the user ID. The movie type ID is directly input into the fully connected layer in the form of a vector, and the movie name is represented by a fixed-length vector using a text convolutional neural network. The feature representations of the three attributes are then fully connected and added separately.

After obtaining the vector representation of the user and the movie, calculate the cosine similarity of them as the score of the personalized recommendation system. Finally, the square of the difference between the similarity score and the user’s true score is used as the loss function of the regression model.

![1-user-film-personalized-recommender-Milvus.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/1_user_film_personalized_recommender_Milvus_9ec39f501d.jpg)
###### *PaddlePaddle's fusion recommendation model.*

## System Overview

Combined with PaddlePaddle’s fusion recommendation model, the movie feature vector generated by the model is stored in the Milvus vector similarity search engine, and the user feature is used as the target vector to be searched. Similarity search is performed in Milvus to obtain the query result as the recommended movies for the user.

![2-system-overview.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/2_system_overview_5652afdca7.jpg)
###### *Fusion recommendation model combined with Milvus.*

> The inner product (IP) method is provided in Milvus to calculate the vector distance. After normalizing the data, the inner product similarity is consistent with the cosine similarity result in the fusion recommendation model.

## Application of Personal Recommender System

There are three steps in building a personalized recommendation system with Milvus, details on how to operate please refer to Mivus Bootcamp [4].

### Step 1：Model Training

 # run train.py
 $ python train.py

Running this command will generate a model recommender_system.inference.model in the directory, which can convert movie data and user data into feature vectors, and generate application data for Milvus to store and retrieve.

### Step 2：Data Preprocessing

 # Data preprocessing, -f followed by the parameter raw movie data file name
 $ python get_movies_data.py -f movies_origin.txt

Running this command will generate test data movies_data.txt in the directory to achieve pre-processing of movie data.

### Step 3：Implementing Personal Recommender System with Milvus

 # Implementing personal recommender system based on user conditions
 $ python infer_milvus.py -a <age>-g <gender>-j <job>[-i]

Running this command will implement personalized recommendations for specified users.

The main process is:

- Through the load_inference_model, the movie data is processed by the model to generate a movie feature vector.
- Load the movie feature vector into Milvus via milvus.insert.
- According to the user’s age / gender / occupation specified by the parameters, it is converted into a user feature vector, milvus.search_vectors is used for similarity retrieval, and the result with the highest similarity between the user and the movie is returned.

Prediction of the top five movies that the user is interested in:

 TopIdsTitleScore
 03030Yojimbo2.9444923996925354
 13871Shane2.8583481907844543
 23467Hud2.849525213241577
 31809Hana-bi2.826111316680908
 43184Montana2.8119677305221558 

## Summary

By inputing user information and movie information to the fusion recommendation model we can get matching scores, and then sort the scores of all movies based on the user to recommend movies that may be of interest to the user. **This article combines Milvus and PaddlePaddle to build a personalized recommendation system. Milvus, a vector search engine, is used to store all movie feature data, and then similarity retrieval is performed on user features in Milvus.** The search result is the movie ranking recommended by the system to the user.

Milvus [5] vector similarity search engine is compatible with various deep learning platforms, searching billions of vectors with only millisecond response. You can explore more possibilities of AI applications with Milvus with ease!

## Reference
1. MovieLens Million Dataset (ml-1m): http://files.grouplens.org/datasets/movielens/ml-1m.zip
2. ml-1m-README: http://files.grouplens.org/datasets/movielens/ml-1m-README.txt
3. Fusion Recommendation Model by PaddlePaddle: https://www.paddlepaddle.org.cn/documentation/docs/zh/beginners_guide/basics/recommender_system/index.html#id7
4. Bootcamp: https://github.com/milvus-io/bootcamp/tree/master/solutions/recommendation_system
5. Milvus: https://milvus.io/en/

small_header.jpg

thumbnail_header.jpg

header.jpg

Building Personalized Recommender Systems with Milvus and PaddlePaddle

Learn how Tokopedia used semantic search to make their search 10x smarter

At Tokopedia, we understand that the value in our product corpus is only unlocked when our buyers can find products that are relevant to them, so we strive to improve the relevance of search results.

To further that effort, we are introducing **similarity search** on Tokopedia. If you go to the search result page on mobile devices, you will find a “…” button that exposes a menu that gives you the option to search for products similar to the product.

## Keyword-based search
Tokopedia Search uses **Elasticsearch** for the search and ranking of products. For each search request, we first query Elasticsearch, which ranks products according to the search query. ElasticSearch stores each word as a sequence of numbers representing [ASCII](https://en.wikipedia.org/wiki/ASCII) (or UTF) codes for each letter. It builds an [inverted-index](https://en.wikipedia.org/wiki/Inverted_index) to quickly find out, which documents contain words from the user query, and then finds the best match among them using various scoring algorithms. These scoring algorithms pay little attention to what the words mean, but rather to how frequently they occur in the document, how close they are to each other, etc. ASCII representation obviously contains enough information to convey the semantics (after all we, humans, can understand it). Unfortunately, there’s no good algorithm for the computer to compare ASCII-encoded words by their meaning.

## Vector representation
One solution to this would be to come up with an alternative representation, which tells us not only about the letters contained in the word but also something about its meaning. For example, we could encode *which other words our word is frequently used together with* (represent by the probable context). We’d then assume that similar contexts represent similar things, and try to compare them using mathematical methods. We could even find a way to encode whole sentences by their meaning.

![Blog_How we used semantic search to make our search 10x smarter_2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_2_776af567a8.png)
###### *Vector representation also reveals the meaning of words compared with ASCII representation.*

## Select an embedding similarity search engine
Now that we have feature vectors, the remaining issue is how to retrieve from the large volume of vectors the ones that are similar to the target vector. When it comes to the embeddings search engine, we tried POC on several engines available on Github some of them are FAISS, Vearch, Milvus.

We prefer Milvus to other engines based on load test results. On the one hand, we have used FAISS before on other teams and hence would like to try something new. Compared to Milvus, FAISS is more of an underlying library, therefore not quite convenient to use. As we learned more about Milvus, we finally decided to adopt Milvus for its two main features:

- Milvus is very easy to use. All you need to do is to pull its Docker image and update the parameters based on your own scenario.

- It supports more indexes and has detailed supporting documentation.

In a nutshell, Milvus is very friendly to users and the documentation is quite detailed. If you come across any problem, you can usually find solutions in the documentation; otherwise, you can always get support from the Milvus community.

## Milvus cluster service
After deciding to use Milvus as the feature vector search engine, we decided to use Milvus for one of our Ads service use-case where we wanted to match [low fill rate](https://www.tradegecko.com/blog/wholesale-management/what-is-fill-rate-and-why-does-it-matter-for-wholesalers) keywords with high fill rate keywords. We configured a standalone node in a development (DEV) environment and started serving, it had been running well for a few days, and giving us improved CTR/CVR metrics. If a standalone node crashed in production, the entire service would become unavailable. Thus, we need to deploy a highly available search service.

Milvus provides both Mishards, a cluster sharding middleware, and Milvus-Helm for configuration. In Tokopedia we use Ansible playbooks for infrastructure setup so we created a playbook for infra orchestration. The diagram below from Milvus’ documentation shows how Mishards works:

![Blog_How we used semantic search to make our search 10x smarter_3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_3_4fa0c8a1a1.png)
###### *How Mishards works.*

Mishards cascade a request from upstream down to its sub-modules splitting the upstream request, and then collects and returns the results of the sub-services to upstream. The overall architecture of the Mishards-based cluster solution is shown below:
![Blog_How we used semantic search to make our search 10x smarter_4.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_4_724618be4e.jpeg)
###### *Overall architecture of Mishards.*

The official documentation provides a clear introduction of Mishards. You can refer to [Mishards](https://milvus.io/cn/docs/v0.10.2/mishards.md) if you are interested.

In our keyword-to-keyword service, we deployed one writable node, two read-only nodes, and one Mishards middleware instance in GCP, using Milvus ansible. It has been stable so far. A huge component of what makes it possible to efficiently query the million-, billion-, or even trillion-vector datasets that similarity search engines rely on is [indexing](https://milvus.io/docs/v0.10.5/index.md), a process of organizing data that drastically accelerates big data search.

## How does vector indexing accelerate similarity search?
Similarity search engines work by comparing input to a database to find objects that are most similar to the input. Indexing is the process of efficiently organizing data, and it plays a major role in making similarity search useful by dramatically accelerating time-consuming queries on large datasets. After a massive vector dataset is indexed, queries can be routed to clusters, or subsets of data, that are most likely to contain vectors similar to an input query. In practice, this means a certain degree of accuracy is sacrificed to speed up queries on really big vector data.

An analogy can be drawn to a dictionary, where words are sorted alphabetically. When looking up a word, it is possible to quickly navigate to a section that only contains words with the same initial — drastically accelerating the search for the input word’s definition.

## What next, you ask?
![Blog_How we used semantic search to make our search 10x smarter_5.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_How_we_used_semantic_search_to_make_our_search_10x_smarter_5_035480c8af.jpeg)

As shown above, there is no solution that fits all, we always want to improve the model’s performance used for getting the embeddings.

Also, from a technical point of view, we want to run multiple learning models at the same time and compare the results from the various experiments. Watch this space for more information on our experiments like image search, video search.

 

## References:
- Mishards Docs：https://milvus.io/docs/v0.10.2/mishards.md
- Mishards: https://github.com/milvus-io/milvus/tree/master/shards
- Milvus-Helm: https://github.com/milvus-io/milvus-helm/tree/master/charts/milvus

 

*This blog article is reposted from: https://medium.com/tokopedia-engineering/how-we-used-semantic-search-to-make-our-search-10x-smarter-bd9c7f601821*

Read other [user stories](https://zilliz.com/user-stories) to learn more about making things with Milvus.

large_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

medium_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

small_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

thumbnail_Blog_How we used semantic search to make our search 10x smarter_1.jpeg

Blog_How we used semantic search to make our search 10x smarter_1.jpeg

How we used semantic search to make our search 10x smarter

Milvus was built from the ground up to power vector similarity search for huge volumes of virtually any data type. Discover how this open-source platform is optimized for fast and efficient massive-scale vector data management. 

Every day, an incalculable number of business-critical insights are squandered because companies can’t make sense of their own data. Unstructured data, such as text, image, video, and audio, is estimated to account for 80% of all data — but just 1% of it is ever analyzed. Fortunately, [artificial intelligence (AI)](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f), open-source software, and Moore’s law are making machine-scale analytics more accessible than ever before. Using vector similarity search, it is possible to extract value from massive unstructured datasets. This technique involves converting unstructured data into feature vectors, a machine-friendly numerical data format that can be processed and analyzed in real time.

Vector similarity search has applications spanning e-commerce, security, new drug development, and more. These solutions rely on dynamic datasets containing millions, billions, or even trillions of vectors, and their usefulness often depends on returning near instantaneous results. [Milvus](https://milvus.io/) is an open-source vector data management solution built from the ground up for efficiently managing and searching large vector datasets. This article covers Milvus’ approach to vector data management, as well as how the platform has been optimized for vector similarity search.

**Jump to:**
- [LSM trees keep dynamic data management efficient at massive scales](#lsm-trees-keep-dynamic-data-management-efficient-at-massive-scales)
- [Data management is optimized for rapid access and limited fragmentation](#data-management-is-optimized-for-rapid-access-and-limited-fragmentation)
- [Similarity searched is accelerated by indexing vector data](#similarity-searched-is-accelerated-by-indexing-vector-data)
- [Learn more about Milvus](#learn-more-about-Milvus)

### LSM trees keep dynamic data management efficient at massive scales

To provide efficient dynamic data management, Milvus uses a log-structured merge-tree (LSM tree) data structure. LSM trees are well suited for accessing data that has a high number of inserts and deletes. For detailed information on specific attributes of LSM trees that help ensure high-performance dynamic data management, see the [original research](http://paperhub.s3.amazonaws.com/18e91eb4db2114a06ea614f0384f2784.pdf) published by its inventors. LSM trees are the underlying data structure used by many popular databases, including [BigTable](https://cloud.google.com/bigtable), [Cassandra](https://cassandra.apache.org/), and [RocksDB](https://rocksdb.org/).

Vectors exist as entities in Milvus and are stored in segments. Each segment contains anywhere from one up to ~8 million entities. Each entity has fields for a unique ID and vector inputs, with the latter representing anywhere from 1 to 32768 dimensions.

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_2_492d31c7a0.png)
###### *A segment of 10-dimensional vectors in Milvus.*

### Data management is optimized for rapid access and limited fragmentation

When receiving an insert request, Milvus writes new data to the [write ahead log (WAL)](https://milvus.io/docs/v0.11.0/write_ahead_log.md). After the request is successfully recorded to the log file, the data is written to a mutable buffer. Finally, one of three triggers results in the buffer becoming immutable and flushing to disk:

1. **Timed intervals:** Data is regularly flushed to disk at defined intervals (1 second by default).
2. **Buffer size:** Accumulated data reaches the upper limit for the mutable buffer (128 MB).
3. **Manual trigger:** Data is manually flushed to disk when the client calls the flush function.

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_3_852dc2c9bb.png)
###### *An illustration of inserting vectors in Milvus.*

Users can add tens or millions of vectors at a time, generating data files of different sizes as new vectors are inserted. This results in fragmentation that can complicate data management and slow down vector similarity search. To prevent excessive data fragmentation, Milvus constantly merges data segments until the combined file size reaches a user configurable limit (e.g., 1 GB). For example, given an upper limit of 1 GB, inserting 100 million 512-dimensional vectors will result in just ~200 data files.

In incremental computation scenarios where vectors are inserted and searched concurrently, Milvus makes newly inserted vector data immediately available for search before merging it with other data. After data merges, the original data files will be removed and the newly created merged file will be used for search instead.

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_4.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_4_6bef3d914c.png)
###### *Queried data files before the merge.*

![Blog_Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search_5.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Milvus_Was_Built_for_Massive_Scale_Think_Trillion_Vector_Similarity_Search_5_3851c2d789.png)
###### *Queried data files after the merge.*

### Similarity searched is accelerated by indexing vector data

By default, Milvus relies on brute-force search when querying vector data. Also known as exhaustive search, this approach checks all vector data each time a query is run. With datasets containing millions or billions of multi-dimensional vectors, this process is too slow to be useful in most similarity search scenarios. To help expedite query time, algorithms are used to build a vector index. The indexed data is clustered such that similar vectors are closer together, allowing the similarity search engine to query just a portion of the total data, drastically reducing query times while sacrificing accuracy.

Most of the vector index types supported by Milvus use approximate nearest neighbor (ANN) search algorithms. There are numerous ANN indexes, and each one comes with tradeoffs between performance, accuracy, and storage requirements. Milvus supports quantization-, graph, and tree-based indexes, all of which serve different application scenarios. See Milvus’ [technical documentation](https://milvus.io/docs/v0.11.0/index.md#CPU) for more information about building indexes and the specific types of vector indexes it supports.

Index building generates a lot of metadata. For example, indexing 100 million 512-dimensional vectors saved in 200 data files will result in an additional 200 index files. In order to efficiently check file statuses and delete or insert new files, an efficient metadata management system is required. Milvus uses online transactional processing (OLTP), a data processing technique that is well-suited for updating and/or deleting small amounts of data in a database. Milvus uses SQLite or MySQL to manage metadata.

### Learn more about Milvus

Milvus is an open-source vector data management platform currently in incubation at [LF AI & Data](https://lfaidata.foundation/), an umbrella organization of the Linux Foundation. Milvus was made open source in 2019 by [Zilliz](https://zilliz.com), the data science software company that initiated the project. More information about Milvus can be found on its [website](https://milvus.io/). If you’re interested in vector similarity search, or using AI to unlock the potential of unstructured data, please join our [open-source community](https://github.com/milvus-io) on GitHub.






large_Milvus Massive scale.jpg

medium_Milvus Massive scale.jpg

small_Milvus Massive scale.jpg

thumbnail_Milvus Massive scale.jpg

Milvus Massive scale.jpg

Explore the power of open-source in your next AI or machine learning project. Manage massive-scale vector data and power similarity search with Milvus. 

Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search

Artificial intelligence makes it possible to extract deep insights from massive unstructured datasets. By converting abstract data like video and audio into vectors, a conceptual task becomes a rigorous mathematical one well suited for machines. Milvus is a vector data management platform that is the ideal backbone for your next AI project.

Searching data using easily defined criteria, for example querying a movie database by actor, director, genre, or release date, is simple. A relational database is well equipped for these types of basic searches using a query language such as SQL. But when searches involve complex objects and more abstract queries, such as searching a video streaming library using natural language or a video clip, simple similarity metrics like matching words in a title or description are no longer sufficient.

[Artificial intelligence (AI)](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#d62b) has made computers significantly more capable of understanding the semantics of language, as well as helping people make sense of massive, difficult to analyze unstructured datasets (e.g., audio, video, documents, and social media data). AI makes it possible for Netflix to create sophisticated content recommendation engines, Google users to search the web by image, and pharmaceutical companies to discover new drugs.

### The challenge of searching large unstructured datasets

These feats of technology are accomplished by using AI algorithms to convert dense unstructured data into vectors, a numerical data format that is easily read by machines. Next, additional algorithms are used to calculate the similarity between vectors for a given search. The large size of unstructured datasets makes searching them in their entirety far too time consuming for most machine learning applications. To overcome this, approximate nearest neighbor (ANN) algorithms are used to cluster similar vectors together, then only search the portion of the dataset most likely to contain similar vectors to the target search vector.

This results in [drastically faster](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212#7a9a) (though slightly less accurate) similarity search, and is key to building useful AI tools. Thanks to vast public resources, it has never been easier or cheaper to build machine learning applications. However, AI-powered vector similarity search often requires interlacing different tools that vary in number and complexity depending on specific project requirements. Milvus is an open-source AI search engine that aims to simplify the process of building machine learning applications by providing robust functionality under a unified platform.

### What is Milvus?

[Milvus](https://milvus.io/) is an open-source data management platform built specifically for massive-scale vector data and streamlining machine learning operations (MLOps). Powered by Facebook AI Similarity Search (Faiss), Non-Metric Space Library (NMSLIB), and Annoy, Milvus brings a variety of powerful tools together in one place while extending their standalone functionality. The system was purpose built for storing, processing, and analyzing large vector datasets, and can be used to build AI applications spanning computer vision, recommendation engines, and much more.

![Blog_Milvus Is an Open-Source Scalable AI Search Engine_1.jpg](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Milvus_Is_an_Open_Source_Scalable_AI_Search_Engine_1_997255eb27.jpg)
###### *The structure of an AI-powered similarity search engine.*

### Milvus was made to power vector similarity search

Milvus was designed for flexibility, allowing developers to optimize the platform for their specific use case. Support for CPU/GPU-only and heterogeneous computing makes it possible to accelerate data processing and optimize resource requirements for any scenario. Data is stored in Milvus on a distributed architecture, making it trivial to scale data volumes. With support for various AI models, programming languages (e.g., C++, Java, and Python), and processor types (e.g., x86, ARM, GPU, TPU, and FPGA), Milvus offers high compatibility with a wide variety of hardware and software.

For more information about Milvus, check out the following resources:

- Explore Milvus' [technical documentation](https://milvus.io/) and learn more about the platform’s inner workings. 
- Learn how to launch Milvus, build applications, and more using [Milvus tutorials](https://tutorials.milvus.io/).
- Make contributions to the project and engage with Milvus' open-source community on [GitHub](https://github.com/milvus-io).





Build powerful machine learning applications and manage massive-scale vector data with Milvus.

Milvus Is an Open-Source Scalable Vector Database

Artificial intelligence influences what we read, the things we buy, who we become friends with, and more everyday human behavior. A huge part of what makes many modern AI applications possible is vector similarity search. Understanding how this underlying technology works can help demystify AI, providing insight into what computer intelligence is really made of. 

[Artificial intelligence (AI)](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#a291) has the potential to change the way even the most obscure things are done. For example, every year (before COVID, anyway) over 73,000 people congregate to compete in the Hong Kong Marathon. In order to properly sense and record finishing times for all race participants, organizers distribute 73,000 RFID chip timers to attach to each runner. Chip timing is a complex undertaking with obvious downsides. Materials (chips and electronic reading devices) must be purchased or rented from timing companies, and a registration area must be staffed for runners to collect chips on race day. Additionally, if sensors are installed only at the start and finish lines, it’s possible for unscrupulous runners to cut the course. 

![blog-1.jpeg](https://zilliz-cms.s3.us-west-2.amazonaws.com/blog_1_e55c133e05.jpeg)

###### Timing marathon runners is a logistical challenge few think about.

Now imagine a [video AI](https://cloud.google.com/video-intelligence) application capable of automatically identifying individual runners from footage captured at the finish line using a single photo. Rather than attach timing chips to each participant, runners simply upload a photo of themselves via an app after they cross the finish line. Instantly, a personalized highlight reel, race stats, and other relevant information is provided. Cameras installed at various points throughout the race can capture additional footage of participants and ensure each runner traverses the entire course. Which solution seems easier and more cost-effective to implement?

Although the Hong Kong Marathon doesn’t leverage machine learning to replace timing chips (yet), this example illustrates the potential AI has to drastically alter everything around us. For race timing, it reduces tens of thousands of chips to a few cameras paired with machine learning algorithms. But video AI is just one of many applications for vector similarity search, a process that uses artificial intelligence to analyze massive, trillion-scale unstructured datasets. This article provides an overview of vector search technology including what it is, how it can be used, as well as the open-source software and resources making it more accessible than ever before.





**Jump to:**

- [What is vector similarity search?](#what-is-vector-similarity-search)

- [What are some applications of vector similarity search?](#what-are-some-applications-of-vector-similarity-search)

- [Open-source vector similarity search software and resources.](#open-source-vector-similarity-search-software-and-resources)

### What is vector similarity search?

Video data is incredibly detailed and increasingly common, so logically it seems like it would be a great unsupervised learning signal for building video AI. In reality, this is not the case. Processing and analyzing video data, especially in large volumes, remains a [challenge for artificial intelligence](https://arxiv.org/pdf/1905.11954.pdf). Recent progress in this field, like much of the progress made in unstructured data analytics, is owed in large part to vector similarity search.

The problem with video, like all unstructured data, is that it doesn’t follow a predefined model or organizational structure, making it difficult to process and analyze at scale. Unstructured data includes things like images, audio, social media behavior, and documents, collectively accounting for an estimated 80-90%+ of all data. Companies are increasingly aware of the business-critical insights buried in massive, enigmatic unstructured datasets, driving demand for AI applications that can tap into this unrealized potential.

Using [neural networks](https://en.wikipedia.org/wiki/Neural_network) such as CNN, RNN, and BERT, unstructured data can be converted into feature vectors (aka embeddings), a machine-readable numerical data format. Algorithms are then used to calculate the similarity between vectors using measures like cosine similarity or Euclidean distance. Vector embedding and similarity search make it possible to analyze and build machine learning applications using previously indiscernible datasets.

Vector similarity is calculated using established algorithms however, unstructured datasets are typically massive. This means efficient and accurate search requires vast storage and compute power. To [accelerate similarity search](https://medium.com/unstructured-data-service/how-to-choose-an-index-in-milvus-4f3d15259212#7a9a) and reduce resource requirements, approximate nearest neighbor (ANN) search algorithms are used. By clustering similar vectors together, ANN algorithms make it possible to send queries to the clusters of vectors most likely to contain similar vectors, rather than searching the entire dataset. Although this approach is faster, it sacrifices some degree of accuracy. Leveraging ANN algorithms allows vector search to comb through billions of deep learning model insights in millisecond.


### What are some applications of vector similarity search?

Vector similarity search has applications spanning a wide variety of artificial intelligence, deep learning, and traditional vector calculation scenarios. The following provides a high-level overview of various vector similarity search applications:

**E-commerce:** Vector similarity search has broad applicability in e-commerce, including reverse image search engines that allow shoppers to search for products using an image captured on their smartphone or found online. Additionally, personalized recommendations based on user behavior, interests, purchase history, and more can be served by specialized recommender systems that rely on vector search.

**Physical & Cyber Security:** Video AI is just one of many applications for vector similarity search in the security field. Other scenarios include facial recognition, behavior tracing, identity authentication, intelligent access control, and more. Additionally, vector similarity search plays an important role in thwarting increasingly common and [sophisticated cyberattacks](https://www.wsj.com/articles/hack-suggests-new-scope-sophistication-for-cyberattacks-11608251360). For example, [code similarity search](https://medium.com/gsi-technology/application-of-ai-to-cybersecurity-part-3-19659bdb3422) can be used to identify security risks by comparing a piece of software to a database of known vulnerabilities or malware.

**Recommendation Engines:** Recommendation engines are systems that use machine learning and data analysis to suggest products, services, content, and information to users. User behavior, the behavior of similar users, and other data is processed using deep learning methods to generate recommendations. With enough data, algorithms can be trained to understand relationships between entities and invent ways to represent them autonomously. Recommendation systems have broad applicability and are something people already interact with every day, including content recommendations on Netflix, shopping recommendations on Amazon, and news feeds on Facebook.

**Chatbots:** Traditionally, chatbots are built using a regular knowledge graph that requires a large training dataset. However, chatbots built using deep learning models don’t need to preprocess data—instead, a map between frequent questions and answers is created. Using a pre-trained natural language processing (NLP) model, feature vectors can be extracted from the questions and then stored and queried using a [vector data management platform](https://medium.com/unstructured-data-service/the-easiest-way-to-search-among-1-billion-image-vectors-d6faf72e361f#92e0).

**Image or Video Search:** Deep learning networks have been used to recognize visual patterns since the late 1970s, and modern technology trends have made image and video search more powerful and accessible than ever before. 

**Chemical Similarity Search:** Chemical similarity is key to predicting the properties of chemical compounds and finding chemicals with specific attributes, making it indispensable to the development of new drugs. Fingerprints represented by feature vectors are created for each molecule, and then the distances between vectors are used to measure similarity. Using AI for new drug discovery is gaining momentum in the tech industry, with ByteDance (TikTok’s Chinese parent company) starting to [hire talent in the field](https://techcrunch.com/2020/12/23/bytedance-ai-drug/).


### Open-source vector similarity search software and resources.

Moore’s law, cloud computing, and declining resource costs are macro trends that have made artificial intelligence more accessible than ever. Thanks to open-source software and other publicly available resources, building AI/ML applications isn’t just for big tech companies. Below we provide a brief overview of Milvus, an open-source vector data management platform, and also highlight some publicly available datasets that help put AI within everyone’s reach.

#### Milvus, an open-source vector data management platform

[Milvus](https://milvus.io/) is an open-source vector data management platform built specifically for massive-scale vector data. Powered by Facebook AI Similarity Search (Faiss), Non-Metric Space Library (NMSLIB), and Annoy, Milvus brings a variety of powerful tools together under a single platform while extending their standalone functionality. The system was purpose built for storing, processing, and analyzing large vector datasets, and can be used to build all the AI applications (and more) mentioned above. 

More information about Milvus can be found on its [website](https://milvus.io/). Tutorials, instructions for setting up Milvus, benchmark testing, and information on building a variety of different applications is available in the [Milvus bootcamp](https://github.com/milvus-io/bootcamp). Developers interested in making contributions to the project can join Milvus' open-source community on [GitHub](https://github.com/milvus-io).

#### Public datasets for artificial intelligence and machine learning

It is no secret that technology giants like Google and Facebook have a data advantage over the little guys, with some pundits even advocating for a “[progressive data-sharing mandate](https://www.technologyreview.com/2019/06/06/135067/making-big-tech-companies-share-data-could-do-more-good-than-breaking-them-up/)” that would force companies that exceed a certain size to share some anonymized data with smaller rivals. Fortunately, there are thousands of publicly available datasets that can be used for AL/ML projects: 

- **The People’s Speech Dataset:** This [dataset from ML Commons](https://mlcommons.org/en/peoples-speech/) offers the largest speech dataset in the world, with over 87,000 hours of transcribed speech in 59 different languages. 

- **UC Irvine Machine Learning Repository:** The University of California at Irvine maintains [hundreds of public datasets](https://archive.ics.uci.edu/ml/index.php) in an effort to help the machine learning community. 

- **Data.gov:** The U.S. government offers [hundreds of thousands of open datasets](https://www.data.gov/) that span education, climate, COVID-19, and more. 

- **Eurostat:** The European Union’s statistical office provides [open datasets](https://ec.europa.eu/eurostat/data/database) spanning a variety of industries from economy and finance to population and social conditions. 

- **Harvard Dataverse:** [The Harvard Dataverse Repository](https://dataverse.harvard.edu/) is a free data repository open to researchers across disciplines. Many datasets are public, while others come with more restricted terms of use.

Although this list is by no means exhaustive, it is a good starting point for discovering the surprisingly wide variety of open datasets. For more information on public datasets as well as choosing the right data for your next ML or data science project, check out this [Medium post](https://altexsoft.medium.com/best-public-datasets-for-machine-learning-and-data-science-sources-and-advice-on-the-choice-636a0e754052).


## To learn more about vector similarity search, check out the following resources:
- [Thanks to Milvus, Anyone Can Build a Search Engine for 1+ Billion Images](https://zilliz.com/blog/Thanks-to-Milvus-Anyone-Can-Build-a-Vector-Database-for-1-Billion-Images)
- [Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity Search](https://zilliz.com/blog/Milvus-Was-Built-for-Massive-Scale-Think-Trillion-Vector-Similarity-Search)
- [Accelerating Similarity Search on Really Big Data with Vector Indexing](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing) 



large_plainview.jpg

medium_plainview.jpg

small_plainview.jpg

thumbnail_plainview.jpg

plainview.jpg

Discover what vector similarity search is, its various applications, and the public resources making artificial intelligence more accessible than ever.

Vector Similarity Search Hides in Plain View

Learn how to setup Milvus, an open-source vector data management platform, in Google Colaboratory. Thanks to Google Colab and Milvus, building machine learning applications has never been simpler.

Technological progress is perpetually making artificial intelligence (AI) and machine-scale analytics more accessible and easier to use. The [proliferation](https://techcrunch.com/2019/01/12/how-open-source-software-took-over-the-world/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAAL_qokucWT-HjbiOznq2RlO5TG78V6UYf332FKnWl3_knuMo6t5HZvZkIHFL3fhYX0nLzM6V1lVVAK4G3BXZHENX3zCXXgggGt39L9HKde3BufW1-iM2oKm0NIav2fcqgxvfpvx_7EPGstI7c_n99stI9oJf9sdsRPTQ6Wnu7DYX) of open-source software, public datasets, and other free tools are primary forces driving this trend. By pairing two free resources, [Milvus](https://milvus.io/) and [Google Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb#scrollTo=5fCEDCU_qrC0) (“Colab” for short), anyone can create powerful, flexible AI and data analytics solutions. This article provides instructions for setting up Milvus in Colab, as well as performing basic operations using the Python software development kit (SDK).

**Jump to:**

- [What is Milvus?](#what-is-milvus)
- [What is Google Colaboratory?](#what-is-google-colaboratory)
- [Getting started with Milvus in Google Colaboratory](#getting-started-with-milvus-in-google-colaboratory)
- [Run basic Milvus operations in Google Colab with Python](#run-basic-milvus-operations-in-google-colab-with-python)
- [Milvus and Google Colaboratory work beautifully together](#milvus-and-google-colaboratory-work-beautifully-together)


### What is Milvus?

[Milvus](https://milvus.io/) is an open-source vector similarity search engine that can integrate with widely adopted index libraries, including Faiss, NMSLIB, and Annoy. The platform also includes a comprehensive set of intuitive APIs. By pairing Milvus with artificial intelligence (AI) models, a wide variety of applications can be built including:

- Image, video, audio, and semantic text search engines.
- Recommendation systems and chatbots.
- New drug development, genetic screening, and other biomedical applications.


### What is Google Colaboratory?

[Google Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb#recent=true) is a product from the Google Research team that allows anyone to write and run python code from a web browser. Colab was built with machine learning and data analysis applications in mind, offers a free Jupyter notebook environment, syncs with Google Drive, and gives users access to powerful cloud computing resources (including GPUs). The platform supports many popular machine learning libraries and can be integrated with PyTorch, TensorFlow, Keras, and OpenCV.

### Getting started with Milvus in Google Colaboratory

Although Milvus recommends [using Docker](https://milvus.io/docs/v0.10.4/milvus_docker-cpu.md) to install and start the service, the current Google Colab cloud environment does not support Docker installation. Additionally, this tutorial aims to be as accessible as possible — and not everyone uses Docker. Install and start the system by [compiling Milvus’ source code](https://github.com/milvus-io/milvus/blob/0.10.4/INSTALL.md) to avoid using Docker.

### Download Milvus’ source code and create a new notebook in Colab

Google Colab comes with all supporting software for Milvus preinstalled, including required compilation tools GCC, CMake, and Git and drivers CUDA and NVIDIA, simplifying the installation and setup process for Milvus. To begin, download Milvus’ source code and create a new notebook in Google Colab:

1. Download Milvus’ source code: Milvus_tutorial.ipynb.

```Wget https://raw.githubusercontent.com/milvus-io/bootcamp/0.10.0/getting_started/basics/milvus_tutorial/Milvus_tutorial.ipynb```

2. Upload Milvus’ source code to [Google Colab](https://colab.research.google.com/notebooks/intro.ipynb#recent=true) and create a new notebook.

![Blog_Set Up Milvus in Google Colaboratory for Easy ML Application Building_2.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Set_Up_Milvus_in_Google_Colaboratory_for_Easy_ML_Application_Building_2_27809b0ce2.png)

### Compile Milvus from source code
#### Download Milvus source code

```git clone -b 0.10.3 https://github.com/milvus-io/milvus.git```

#### Install dependencies

```% cd /content/milvus/core ./ubuntu_build_deps.sh./ubuntu_build_deps.sh```

#### Build Milvus source code

```
% cd /content/milvus/core
!ls
!./build.sh -t Release
# To build GPU version, add -g option, and switch the notebook settings with GPU
#((Edit -> Notebook settings -> select GPU))
# !./build.sh -t Release -g
```

> Note: If the GPU version is correctly compiled, a “GPU resources ENABLED!” notice appears.

### Launch Milvus server

#### Add lib/ directory to LD_LIBRARY_PATH:

```
% cd /content/milvus/core/milvus
! echo $LD_LIBRARY_PATH
import os
os.environ['LD_LIBRARY_PATH'] +=":/content/milvus/core/milvus/lib"
! echo $LD_LIBRARY_PATH
```

#### Start and run Milvus server in the background:

```
% cd scripts
! ls
! nohup ./start_server.sh &
```

#### Show Milvus server status:

```
! ls
! cat nohup.out
```

> Note: If the Milvus server is launched successfully, the following prompt appears:

![Blog_Set Up Milvus in Google Colaboratory for Easy ML Application Building_3.png](https://zilliz-cms.s3.us-west-2.amazonaws.com/Blog_Set_Up_Milvus_in_Google_Colaboratory_for_Easy_ML_Application_Building_3_b15138cd59.png)


###### *Milvus server started successfully!*

### Run basic Milvus operations in Google Colab with Python

After successfully launching in Google Colab, Milvus can provide a variety of API interfaces for Python, Java, Go, Restful, and C++. Below are instructions for using the Python interface to perform basic Milvus operations in Colab.

#### Install pymilvus:

```! pip install pymilvus==0.2.14```

#### Connect to the server:

```
# Connect to Milvus Server
milvus = Milvus(_HOST, _PORT)


# Return the status of the Milvus server.
server_status = milvus.server_status(timeout=10)
```

#### Create a collection/partition/index:

```
# Information needed to create a collection
param={'collection_name':collection_name, 'dimension': _DIM, 'index_file_size': _INDEX_FILE_SIZE, 'metric_type': MetricType.L2}

# Create a collection.
milvus.create_collection(param, timeout=10)

# Create a partition for a collection.
milvus.create_partition(collection_name=collection_name, partition_tag=partition_tag, timeout=10) 
ivf_param = {'nlist': 16384}

# Create index for a collection.
milvus.create_index(collection_name=collection_name, index_type=IndexType.IVF_FLAT, params=ivf_param)
```

#### Insert and flush:

```
# Insert vectors to a collection.
milvus.insert(collection_name=collection_name, records=vectors, ids=ids)

# Flush vector data in one collection or multiple collections to disk.
milvus.flush(collection_name_array=[collection_name], timeout=None)
```

#### Load and search:

```
# Load a collection for caching.
milvus.load_collection(collection_name=collection_name, timeout=None)

# Search vectors in a collection.
search_param = { "nprobe": 16 }
milvus.search(collection_name=collection_name,query_records=[vectors[0]],partition_tags=None,top_k=10,params=search_param)
```

#### Get collection/index information:

```
# Return information of a collection. milvus.get_collection_info(collection_name=collection_name, timeout=10)

# Show index information of a collection. milvus.get_index_info(collection_name=collection_name, timeout=10)
```


#### Get vectors by ID:

```
# List the ids in segment
# you can get the segment_name list by get_collection_stats() function.
milvus.list_id_in_segment(collection_name =collection_name, segment_name='1600328539015368000', timeout=None)

# Return raw vectors according to ids, and you can get the ids list by list_id_in_segment() function.
milvus.get_entity_by_id(collection_name=collection_name, ids=[0], timeout=None)
```


#### Get/set parameters:
```
# Get Milvus configurations. milvus.get_config(parent_key='cache', child_key='cache_size')

# Set Milvus configurations. milvus.set_config(parent_key='cache', child_key='cache_size', value='5G')
```

#### Delete index/vectors/partition/collection:
```
# Remove an index. milvus.drop_index(collection_name=collection_name, timeout=None)

# Delete vectors in a collection by vector ID. 
# id_array (list[int]) -- list of vector id milvus.delete_entity_by_id(collection_name=collection_name, id_array=[0], timeout=None)

# Delete a partition in a collection. milvus.drop_partition(collection_name=collection_name, partition_tag=partition_tag, timeout=None)

# Delete a collection by name. milvus.drop_collection(collection_name=collection_name, timeout=10)
```

 

### Milvus and Google Colaboratory work beautifully together

Google Colaboratory is a free and intuitive cloud service that greatly simplifies compiling Milvus from source code and running basic Python operations. Both resources are available for anyone to use, making AI and machine learning technology more accessible to everyone. For more information about Milvus, check out the following resources:

- For additional tutorials covering a wide variety of applications, visit [Milvus Bootcamp](https://github.com/milvus-io/bootcamp).
- For developers interested in making contributions or leveraging the system, find [Milvus on GitHub](https://github.com/milvus-io/milvus).
- For more information about the company that launched Milvus, visit [Zilliz.com](https://zilliz.com/).

large_3.jpg

medium_3.jpg

small_3.jpg

thumbnail_3.jpg

3.jpg

Google Colab makes developing and testing machine learning applications a breeze. Learn how to setup Milvus in Colab for better massive-scale vector data management.