In this introductory article, we’ll introduce concepts related to vector database, a new type of data infrastructure designed to store, manage, and search embedding vectors. Vector databases are being used by Milvus users in an increasingly large number of applications, including but not limited to reverse image search, text understanding, video summarization, drug discovery, stock market analysis, and much more.


Vector Database 101

*This article is edited by [Claire Yu](https://www.linkedin.com/in/wuchuanzi-yu-claire/).*

<br/>

In this introductory article, we’ll introduce concepts related to the vector database, a new type of technology designed to store, manage, and search embedding vectors. Vector databases are being used in an increasingly large number of applications, including but not limited to image search, recommender system, text understanding, video summarization, drug discovery, stock market analysis, and much more.

## Relational is not Enough

Data is everywhere. In the early days of the internet, data was mostly structured, and could easily be stored and managed in relational databases. Take, for example, a book database:

| ISBN       | Year | Name                                 | Author        |
|------------|------|--------------------------------------|---------------|
| 0767908171 | 2003 | A Short History of Nearly Everything | Bill Bryson   |
| 039516611X | 1962 | Silent Spring                        | Rachel Carson |
| 0374332657 | 1998 | Holes                                | Louis Sachar  |

Storing and searching across table-based data such as the one shown above is exactly what relational databases were designed to do. In the example above, each row within the database represents a particular book, while the columns correspond to a particular category of information. When a user looks up book(s) through an online service, they can do so through any of the column names present within the database. For example, querying overall results where the author name is Bill Bryson returns all of Bryson's books.

As the internet grew and evolved, *unstructured data* (magazine articles, shared photos, short videos, etc.) became increasingly common. Unlike structured data, there is no easy way to store the contents of unstructured data within a relational database. Imagine, for example, trying to search for similar shoes given a collection of shoe pictures from various angles; this would be impossible in a relational database since understanding shoe style, size, color, etc... purely from the image's raw pixel values is impossible.

## X2vec: A New Way to Understand Data

This brings us to vector databases. The increasing ubiquity of unstructured data has led to a steady rise in the use of machine learning models trained to understand such data. [Word2vec](https://jalammar.github.io/illustrated-word2vec/), a natural language processing (NLP) algorithm which uses a neural network to learn word associations, is a well-known early example of this. The word2vec model is capable of turning single words (in a variety of languages, not just English) into a list of floating point values, or vectors. Due to the way the machine learning model is trained, vectors that are close to each other represent words that are similar to each other, hence the term embedding vectors. We'll get into a bit more detail (with code!) in the next section.


The idea of turning a piece of unstructured data into a list of numerical values is nothing new*. As deep learning gained steam in both academic and industry circles, new ways to represent text, audio, and images came to be. A common component of all these representations is their use of embedding vectors generated by trained deep neural networks. Going back to the example of word2vec, we can see that the generated embeddings contain significant semantic information.

###### *Early computer vision and image processing relied on local feature descriptors to turn an image into a “bag” of embedding vectors – one vector for each detected keypoint. [SIFT](https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf), [SURF](https://people.ee.ethz.ch/~surf/eccv06.pdf), and [ORB](http://www.gwylab.com/download/ORB_2012.pdf) are three well-known feature descriptors you may have heard of. These feature descriptors, while useful for matching images with one another, proved to be a fairly poor way to represent audio (via spectrograms) and images.

 ```
from gensim.models import KeyedVectors
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
```

### Example: Apple, the company, the fruit, ... or both?

The word "apple" can refer to both the company as well as the delicious red fruit. In this example, we can see that Word2Vec retains both meanings.

```
print(model.most_similar(positive=['samsung', 'iphone'], negative=['apple'], topn=1))
print(model.most_similar(positive=['fruit'], topn=10)[9:])
```

 [('droid_x', 0.6324754953384399)]

 [('apple', 0.6410146951675415)]

"Droid" refers to Samsung's first 4G LTE smartphone ("Samsung" + "iPhone" - "Apple" = "Droid"), while "apple" is the 10th closest word to "fruit".


![飞书20211224-103850.png](https://assets.zilliz.com/20211224_103850_55dbb00d3c.png)



While there are newer and better deep learning algorithms/models for generating word embeddings (ELMo, GPT-2, and BERT, to name a few), the concept remains the same. 

Vectors generated from multilayer neural networks have enough high-level information to be applicable for a variety of tasks.

Vector embeddings are not just limited to natural language. In the example below, we use the `towhee` [library](https://github.com/towhee-io/towhee) to generate embedding vectors for three different images, two of which have similar content: 

### Generating embeddings

Now let's use towhee to generate embeddings for our images.

```
from towhee import pipeline
p = pipeline('image-embedding')
dog0_vec = p(dog0)
dog1_vec = p(dog1)
car_vec = p(car)
```

```
from sklearn.preprocessing import normalize
dog0_vec = normalize(dog0_vec[0])
dog1_vec = normalize(dog1_vec[0])
car_vec = normalize(car_vec[0])
```

### Now let's compute distances

```
import numpy as np
print('dog0 to dog1 distance:', np.linalg.norm(dog0_vec - dog1_vec))
print('dog0 to car distance:', np.linalg.norm(dog0_vec - car_vec))
```

dog0 to dog1 distance: 0.59794164

dog0 to car distance: 1.1380062

## Searching Across Vectors

Now that we’ve seen the representational power of vector embeddings, let’s take a bit of time to briefly discuss indexing the vectors. Like relational databases, vector databases need to be searchable in order to be truly useful – just storing the vector and its associated metadata is not enough. This is called nearest neighbor search, or NN search for short, and alone can be considered a subfield of machine learning and pattern recognition due to the sheer number of solutions proposed. 

Vector search is generally split into two components - the similarity metric and the index. The similarity metric defines how the distance between two vectors is evaluated, while the index is a data structure that facilitates the search process. Similarity metrics are fairly straightforward - the most common similarity metric is the inverse of the L2 norm (also known as Euclidean distance). On the other hand, a diverse set of indices exist, each of which has its own set of advantages and disadvantages. We won't go into the details of vector indices here (that's a topic for another article), just know that, without them, a single query vector would need to be compared with all other vectors in the database, making the query process excruciatingly long.

## Putting It All Together

Now that we understand the representational power of embedding vectors and have a good general overview of how vector search works, it’s now time to put the two concepts together – welcome to the world of vector databases. Vector databases are purpose-built to store, index, and query across embedding vectors generated by passing unstructured data through machine learning models. 

When scaling to huge numbers of vector embeddings, searching across embedding vectors (even with indices) can be prohibitively expensive. Despite this, the best and most advanced vector databases will allow you to insert and search across millions or even billions of target vectors, in addition to specifying an indexing algorithm and similarity metric of your choosing. 

Like the production-ready relational databases, vector databases should meet a few key performance targets before they can be deployed in actual production environments:

1. **Scalable**: Embedding vectors are fairly small in terms of absolute memory, but to facilitate read and write speeds, they are usually stored in-memory (disk-based NN/ANN search is a topic for another blog post). When scaling to billions of embedding vectors and beyond, storage and compute quickly become unmanageable for a single machine. Sharding can solve this problem, but this requires splitting the indexes across multiple machines as well.

2. **Reliable**: Modern relational databases are fault-tolerant. Replication allows cloud-native enterprise databases to avoid having single points of failure, enabling graceful startup and shutdown. Vector databases are no different, and should be able to handle internal faults without data loss and with minimal operational impact.

3. **Fast**: Yes, query and write speeds are important, even for vector databases. An increasingly common use case is processing and indexing database inputs in real time. For platforms such as Snapchat and Instagram, which can have hundreds or thousands of new photos (a type of unstructured data) uploaded per second, speed becomes an incredibly important factor.

With data being generated at unprecedented rates, making sense of all the data through vector databases will become increasingly important. 

## The World’s Most Advanced Vector Database

[Milvus](https://milvus.io/), an open-source vector database, is a leader in this space. Milvus provides a number of demos that you can use to evaluate the capabilities and uses cases of vector databases. With the release of Milvus 2.0 GA, Milvus is now a cloud-native, fault-tolerant system capable of scaling to billions of vectors and beyond. Setup is done via a simple docker command, while inserts and queries across Milvus can be done via our Python, Go, Node.js, or Java bindings. For more information, we welcome you to visit us at milvus.io.

We hope the information in this post is useful for you. We’ll keep the posts coming on a regular basis, so feel free to come back for more material on vector databases, unstructured data, or AI/ML in general.





large_What are Vector Databases_.png

medium_What are Vector Databases_.png

small_What are Vector Databases_.png

thumbnail_What are Vector Databases_.png

What are Vector Databases_.png

An introduction to the concepts related to vector database.

What is a Vector Database?

What are vector databases? How do they differ from traditional databases? How can vector databases help companies from enterprises to startups? Find all that and more in the podcast interview with Frank Liu at Zilliz.

Traditional databases are great for handling structured data, such as information that can easily be captured and categorized in a table. However, [unstructured data](https://milvus.io/docs/v2.0.x/overview.md#Unstructured-data), accounting for around 80% of the world's data, is not so easily managed. Many companies are struggling with making sense of the massive scale of data they have. In the big data era, how can we [tackle the conundrum of meaningful data analysis](https://zilliz.com/learn/what-is-vector-database)? 

In the new episode of That Digital Show, [Frank Liu](https://twitter.com/frankzliu), AI Tech Lead at Zilliz, explains how AI and machine learning are making it possible for developers to understand and extract more value from unstructured data. This episode is hosted by [Jay Jenkins](https://www.linkedin.cn/incareer/in/jaydjenkins/), Tech Strategist and Evangelist at JAPAC - Google Cloud, and [Theo Davies](https://www.linkedin.cn/incareer/in/theohdavies/), Head of Cloud Sales Enablement at JAPAC - Google Cloud.

Below are some questions discussed in the conversation and their corresponding timestamps. To hear the answers, listen to the whole episode titled "[Using Al to Supercharge Data-Driven Applications with Zilliz](https://thatdigitalshow.withgoogle.com/post/apac-19-ai-and-data-driven-applications-with-zilliz/)".

- Where does the name Zilliz come from and what does Zilliz do? (04:00)

- What are some challenges that make data operational? (07:43)

- How have databases evolved to tackle the challenges that volume and complexity pose in meaningful data analysis? (09:00)

- What are vector databases and how do they differ from traditional databases? (09:50)

- How can vector databases help companies? (11:50)

- Could you share some interesting use cases using Milvus? (13:54)

- Why Milvus is open-source and free? (15:18)

large_What_Is_Vector_DB.png

medium_What_Is_Vector_DB.png

small_What_Is_Vector_DB.png

thumbnail_What_Is_Vector_DB.png

What_Is_Vector_DB.png

Frank Liu shares how vector databases tackle data challenges on That Digital Show.

Podcast: Using AI to Supercharge Data-Driven Applications with Zilliz

Feder allows you to visualize the indexing and search process during a reverse image search.

Applications of Vector Similarity Search

Reverse image search is one the most prevalent applications of vector search or approximate nearest neighbor search. When a user uploads an image to the search engine, a bunch of similar images will be returned. During the process, indexes are built to accelerate the search on large datasets, especially those billion- or even trillion-scale datasets. 

In the [previous blog](https://zilliz.com/blog/Visualize-Your-Approximate-Nearest-Neighbor-Search-with-Feder), we have introduced how to visualize your approximate nearest neighbor search with Feder by using the example of HNSW index visualization. In this article, we will take the example of reverse image search and continue to explain how you can use Feder to visualize the index building and search process. In this article, we use the IVF_FLAT index as it is the most commonly used index in reverse image search applications.


## How to visualize reverse image search with Feder
[Feder](https://github.com/zilliztech/feder) is built with JavaScript. To use Feder for visualization, you need to first build an index and save the index file from [Faiss](https://github.com/facebookresearch/faiss) or [Hnswlib](https://github.com/nmslib/hnswlib). Then Feder analyzes the uploaded file to obtain index information and gets ready for the visualization. During a vector similarity search, you need to provide a target vector and the configuration of search parameters. Then Feder visualizes the whole search process for you.

Learn more about how to use Feder by reading [Feder user guide](https://github.com/zilliztech/feder).

## A use case of visualizing search with IVF_FLAT index 
In this use case, we use [VOC 2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/), the classic ML image dataset that contains more than 17,000 images.

First, we use [Towhee](https://towhee.io/), an open-source ML pipeline to encode the images in the VOC 2012 dataset into vectors. Then we build an IVF_FLAT index with Faiss and save the index file. Finally, use Feder for visualization. 



### Build an IVF_Flat index
Indexes are built to accelerate the search process. An analogy can be drawn to a dictionary. All of the words are organized based on their initials. More specifically, words with the same initials are grouped together. And we all know that the number of entries under each initial is unequal. We have more words starting with the letter "E" than those starting with "Z". When looking up a word, we can quickly navigate to the section that only contains words with the same initial. This helps drastically boost the search speed. 

Similarly, the IVF_FLAT index divides vectors in the vector space into different clusters based on vector distance. Vectors close to each other are more likely to be put in the same cluster. And the vectors are not necessarily evenly distributed in each cluster. Therefore, each cluster contains a different amount of vectors.

In this use case, we used Faiss to build an IVF_FLAT index on the 17,000 images in the VOC 2012 dataset, with an `nlist` of 256. The 17,000 image vectors are divided into 256 clusters based on [K-means](https://en.wikipedia.org/wiki/K-means_clustering) clustering method. 

<iframe src="https://zilliz.com/feder-case?indexType=ivf_flat&source=faiss&viewType=overview&withLayout=false&height=600"  style="width: 100%; height: 600px;" frameborder="0" scrolling="no"> </iframe>



With Feder, you can visualize the clustering of high-dimensional vector space in a 2D view. Feder supports viewing the details of each cluster while providing an interactive user experience. To have a better understanding of the IVF_FLAT index, you can click on one of the clusters in Feder, and then you will see a maximum of nine images represented by vectors within this cluster.

### Coarse search
When you input a target image and convert it into a target vector for reverse image search, the system first calculates the distance between the target vector and the centroid of each cluster to find the nearest clusters.

In this use case, `nlist` equals 256, which means that the whole vector space is divided into 265 cluster units. Therefore, in the coarse search process, the system compares the distance between the target vector and 256 cluster centroids. 

In IVF indexes, the vectors are clustered based on their relative distance to each other. This means that it is highly likely that the nearest neighbors of the target vector is located in its nearest clusters. We can control the number of cluster units to query with the parameter `nprobe`. In this use case, `nprobe` equals 8 meaning that the system will look for the nearest neighbor of the target vector within the top eight closest clusters.

<iframe src="https://zilliz.com/feder-case?indexType=ivf_flat&source=faiss&viewType=search&withLayout=false&targetId=1156&nprobe=8&k=12&height=600"  style="width: 100%; height: 600px;" frameborder="0" scrolling="no"> </iframe>

The screenshot below is a detailed view of the closest clusters. In cluster-186 (the eighth closest cluster to the target vector) we can see it contains some vectors of car images. Though the cars are not at all similar to the airplane in our target image, the images in cluster-186 and the target do share some resemblance as the car tracks in cluster-186 images look very much alike the airport runway in the target image. In a much closer cluster, cluster-96, we can see it contains images of aircrafts in the sky.

![coarse.png](https://assets.zilliz.com/coarse_827557fcc9.png)

The clusters in this use case demonstrate that during embedding, the machine learning model accurately extracts the features including aircraft, runway, and sky in the target image. Then it divides vectors in the vector space based on these features. Cluster-186 shares the feature of "runway" while cluster-96 shares the feature of "aircraft".

### Fine search
After a coarse search, we can secure a number of nprobe clusters for a fine search. In this stage, the system compares the distance between the target vector and all vectors in the nprobe clusters. Then the topK closest vectors are returned as the final results.

In this use case, the system calculates the distance between the target vector and a total of 742 vectors in 8 clusters during the fine search process. 

Feder provides two visualization modes for the fine search process. One mode is visualization based on cluster and vector distance. The other is the projection for dimension reduction mode.

In the screenshot below, different clusters are shown in different colors. The white circle in the center represents the target vector. With the help of Feder, you can see the distance between each vector and the target vector in a clearer and more straight-forward way. You can click on each vector to see more detailed information like its distance to target vector, the image it represents, etc.

![fine-dist.png](https://assets.zilliz.com/fine_dist_ac4f81cec3.png)

The screenshot below is the projection for dimension reduction mode. Still, different clusters are shown in different colors. Currently, we only support [UMAP](https://arxiv.org/abs/1802.03426), one of the most popular method for dimension reduction. More project methods will be supported in future releases of Feder.

![fine-project.png](https://assets.zilliz.com/fine_project_67fe9622ab.png)


### Search performance analysis
When searching without an index, the system needs to calculate the distance between the target vector and all the 17,000 vectors in the database. However, by contrast, if we build an IVF_FLAT index, search efficiency is greatly boosted as the calculation volume is significantly reduced (the system only need to calculate the distance between the target vector and the 256 cluster centroids in coarse search and 742 vectors in fine search).

Also with Feder visualization, we will realize that the value of the index building parameters will influence how the vector space is divided. The `nprobe` parameter can be used to achieve tradeoff between search efficiency and accuracy. The higher the value of `nprobe`, the broader the search scope, and the more accurate the results are. But accordingly, search efficiency will be compromised as the calculation volume increases.

## What's next
- Try [Attu](https://github.com/zilliztech/attu/releases) to manage your vector database with [one-click simplicity](https://zilliz.com/blog/manage-your-milvus-vector-database-with-one-click-simplicity).
- Learn how to [visualize your nearest neighbor search with Feder](https://zilliz.com/blog/Visualize-Your-Approximate-Nearest-Neighbor-Search-with-Feder).

large_Frame 11.png

medium_Frame 11.png

small_Frame 11.png

thumbnail_Frame 11.png

Frame 11.png

See how image vectors are clustered in the vector space.

Visualize Reverse Image Search with Feder

Consensus-based replication is a widely-adopted strategy in many cloud-native, distributed databases. However, it is not a perfect solution. 


> This article is transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

Consensus-based replication is a widely-adopted strategy in many cloud-native, distributed databases. However, it has certain shortcomings and is definitely not the silver bullet.

This post aims to first explain the concepts of replication, consistency, and consensus in a cloud-native and distributed database, then clarify why consensus-based algorithms like Paxos and Raft are not the silver bullet, and finally propose a [solution to consensus-based replication](#a-log-replication-strategy-for-cloud-native-and-distributed-database).

## Understanding replication, consistency, and consensus

Before going deep into the pros and cons of Paxos and Raft, and proposing a best suited log replication strategy, we need to first demystify concepts of replication, consistency, and consensus.

Note that this article mainly focuses on the synchronization of incremental data/log. Therefore, when talking about data/log replication, only replicating incremental data, not historical data, is referred to.

### Replication

Replication is the process of making multiple copies of data and storing them in different disks, processes, machines, clusters, etc, for the purpose of increasing data reliability and accelerating data queries. Since in replication, data are copied and stored at multiple locations, data are more reliable in the face of recovering from disk failures, physical machine failures, or cluster errors. In addition, multiple replicas of data can boost the performance of a distributed database by greatly speeding up queries.

There are various modes of replication, such as synchronous/asynchronous replication, replication with strong/eventual consistency, leader-follower/decentralized replication. The choice of replication mode has an effect on system availability and consistency. Therefore, as proposed in the famous [CAP theorem](https://medium.com/analytics-vidhya/cap-theorem-in-distributed-system-and-its-tradeoffs-d8d981ecf37e), a system architect needs to trade off between consistency and availability when network partition is inevitable.

### Consistency

In short, consistency in a distributed database refers to the property that ensures every node or replica has the same view of data when writing or reading data at a given time. For a full list of consistency levels, read the doc [here](https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels).

To clarify, here we are talking about consistency as in the CAP theorem, not ACID (atomicity, consistency, isolation, durability). Consistency in CAP theorem refers to each node in the system having the same data while consistency in ACID refers to a single node enforcing the same rules on every potential commit.

Generally, OLTP (online transaction processing) databases require strong consistency or linearizability to ensure that:
- Each read can access the latest inserted data.
- If a new value is returned after a read, all following reads, regardless on the same or different clients, must return the new value.

The essence of linearizability is to guarantee the recency of multiple data replicas - once a new value is written or read, all subsequent reads can view the new value until the value is later overwritten. A distributed system providing linearizability can save users the trouble of keeping an eye on multiple replicas, and can guarantee the atomicity and order or each operation.

### Consensus

The concept of consensus is introduced to distributed systems as users are eager to see distributed systems work in the same way as standalone systems.

To put it simple, consensus is a general agreement on value. For instance, Steve and Frank wanted to grab something to eat. Steve suggested having sandwiches. Frank agreed to Steve's suggestion and both of them are had sandwiches. They reached a consensus. More specifically, a value (sandwiches) proposed by one of them is agreed upon by both, and both of them take actions based on the value. Similarly, consensus in a distributed system means when a process propose a value, all the rest processes in the system agree on and act upon this value. 


![Consensus](https://assets.zilliz.com/2bb46e57_9eb5_456e_be7e_e7762aa9eb7e_68dd2e8e65.png "Consensus in a distributed system.")

## Consensus-based replication

The earliest consensus-based algorithms were proposed  along with [viewstamped replication](https://pmg.csail.mit.edu/papers/vr.pdf) in 1988. In 1989, Leslie Lamport proposed [Paxos](https://lamport.azurewebsites.net/pubs/paxos-simple.pdf), a consensus-based algorithm.

In recent years, we witness another prevalent consensus-based algorithm in the industry - [Raft](https://raft.github.io/). It has been adopted by many mainstream NewSQL databases like CockroachDB, TiDB, OceanBase, etc.

Notably, a distributed system does not necessarily support linearizability even if it adopts consensus-based replication. However, linearizability is the prerequisite for building ACID distributed database.

When designing a database system, the commit order of logs and state machines should be taken into consideration. Extra caution is also needed to maintain the leader lease of Paxos or Raft and prevent a split-brain under network partition.


![Raft replication state machine](https://user-images.githubusercontent.com/1500781/165926429-69b5144c-f3ba-4819-87c3-ab7e04a7e22e.png "Structure of Raft replication state machine.")

### Pros and cons

Indeed, Raft, [ZAB](https://marcoserafini.github.io/papers/zab.pdf), and [quorum-based log protocol](https://aws.amazon.com/blogs/database/amazon-aurora-under-the-hood-quorum-and-correlated-failure/) in Aurora are all Paxos variations. Consensus-based replication has the following advantages:

1. Though consensus-based replication focus more on consistency and network partition in the CAP theorem, it provides relatively better availability compared to traditional leader-follower replication. 
2. Raft is a breakthrough that greatly simplified consensus-based algorithms. And there are many open-source Raft libraries on GitHub (Eg. [sofa-jraft](https://github.com/sofastack/sofa-jraft)).
3. The performance of consensus-based replication can satisfy most of the applications and businesses. With the coverage of high-performance SSD and gigabyte NICs (network interface card), the burden of synchronizing multiple replicas is relieved, making Paxos and Raft algorithms the mainstream in the industry.

One misconception is that consensus-based replication is the silver bullet to achieving data consistency in a distributed database. However, this is not the truth. The challenges in availability, complexity, and performance faced by consensus-based algorithm blocks it from being the perfect solution.

1. Compromised availability
Optimized Paxos or Raft algorithm has strong dependency on the leader replica, which comes with a weak ability to fight against grey failure. In consensus-based replication, a new election of leader replica will not take place until the leader node does not respond for a long time. Therefore, consensus-based replication is incapable of handling situations where the leader node is slow or a thrashing occurs.

2. High complexity
Though there are already many extended algorithms based on Paxos and Raft, the emergence of [Multi-Raft](http://www.vldb.org/pvldb/vol13/p3072-huang.pdf) and [Parallel Raft](https://www.vldb.org/pvldb/vol11/p1849-cao.pdf) requires more considerations and tests on synchronization between logs and state machines. 

3. Compromised performance
In a cloud-native era, local storage is replaced by shared storage solutions like EBS and S3 to ensure data reliability and consistency. As a result, consensus-based replication is no longer a must for distributed systems. What's more, consensus-based replication comes with the problem of data redundancy as both the solution and EBS has multiple replicas.

For multi-datacenter and multi-cloud replication, the pursuit for consistency compromises not only availability but also [latency](https://en.wikipedia.org/wiki/PACELC_theorem), resulting in a decline in performance. Therefore, linearizability is not a must for multi-datacenter disaster tolerance in most of the applications. 

## A log replication strategy for cloud-native and distributed database

Undeniably, consensus-based algorithms like Raft and Paxos are still the mainstream algorithms adopted by many OLTP databases. However, by observing the examples of [PacificA](https://www.microsoft.com/en-us/research/publication/pacifica-replication-in-log-based-distributed-storage-systems/) protocol, [Socrates](https://www.microsoft.com/en-us/research/uploads/prod/2019/05/socrates.pdf) and [Rockset](https://rockset.com/), we can see the trend is shifting. 

There are two major principles for a solution that can best serve a cloud-native, distributed database.

### 1. Replication as a service

A separate microservice dedicated to data synchronization is needed. The synchronization module and storage module should no longer be tightly coupled within the same process.

For instance, Socrates decouples storage, log, and computing. There is one dedicated log service (XLog service in the middle of the figure below).


![Socrates architecture](https://assets.zilliz.com/1_0d7822a781.png "The architecture of Socrates.")

XLog service is an individual service. Data persistence is achieved with the help of low-latency storage. The landing zone in Socrates is in charge of keeping three replicas at an accelerated speed.


![Socrates XLog service](https://assets.zilliz.com/2_6d1182b6f1.png "The Socrates XLog service.")

The leader node distributes logs to log broker asynchronously, and flushes data to Xstore. Local SSD cache can accelerate data read. Once data flush is successful, buffers in the landing zone can be cleaned. Obviously, all log data are divided into three layers - landing zone, local SSD, and XStore.

### 2. Russian doll principle

One way to design a system is to follow the Russian doll principle: each layer is complete and perfectly suited to what the layer does so that other layers may be built on-top or around it. 

When designing a cloud-native database, we need to cleverly leverage other third-party services to reduce the complexity of system architecture.

It seems like we cannot get around with Paxos to avoid single point failure. However, we can still greatly simplify log replication by handing leader election over to Raft or Paxos services based on [Chubby](https://research.google.com/archive/chubby-osdi06.pdf), [Zk](https://github.com/bloomreach/zk-replicator), and [etcd](https://etcd.io/).

For instance, [Rockset](https://rockset.com/) architecture follows the Russian doll principle and uses Kafka/Kineses for distributed logs, S3 for storage, and local SSD cache for improving query performance. 


![Rockset architecture](https://user-images.githubusercontent.com/1500781/165926697-c8b380dc-d71a-41a9-a76d-a261b77f0b5d.png "Rockset architecture.")

### The Milvus approach

Tunable consistency in Milvus is in fact similar to follower reads in consensus-based replication. Follower read feature refers to using follower replica to undertake data read tasks under the premise of strong consistency. The purpose is to enhance cluster throughput and reduce the load on leader. The mechanism behind follower read feature is inquiring the commit index of the latest log and providing query service until all data in the commit index are applied to state machines.

However, the design of Milvus did not adopt the follower strategy. In other words, Milvus does not inquire commit index every time it receives a query request. Instead, Milvus adopts a mechanism like the watermark in [Flink](https://flink.apache.org/), which notifies query node the location of commit index at a regular interval. The reason for such a mechanism is that Milvus users usually do not have high demand for data consistency, and they can accept a compromise in data visibility for better system performance. 

In addition, Milvus also adopts multiple microservices and separates storage from computing. In the [Milvus architecture](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#A-bare-bones-skeleton-of-the-Milvus-architecture), S3, MinIo, and Azure Blob are used for storage.


![Milvus architecture](https://assets.zilliz.com/Milvus_architecture_b7743a4a7f.png "Milvus architecture.")

## Summary

Nowadays, an increasing number of cloud-native databases are making log replication an individual service. By doing so, the cost of adding read-only replicas and heterogeneous replication can be reduced. Using multiple microservices enables quick utilization of mature cloud-based infrastructure, which is impossible for traditional databases. An individual log service can rely on consensus-based replication, but it can also follow the Russian doll strategy to adopt various consistency protocols together with Paxos or Raft to achieve linearizability.

## References

- Lamport L. Paxos made simple[J]. ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001), 2001: 51-58.
- Ongaro D, Ousterhout J. In search of an understandable consensus algorithm[C]//2014 USENIX Annual Technical Conference (Usenix ATC 14). 2014: 305-319.
- Oki B M, Liskov B H. Viewstamped replication: A new primary copy method to support highly-available distributed systems[C]//Proceedings of the seventh annual ACM Symposium on Principles of distributed computing. 1988: 8-17.
- Lin W, Yang M, Zhang L, et al. PacificA: Replication in log-based distributed storage systems[J]. 2008.
- Verbitski A, Gupta A, Saha D, et al. Amazon aurora: On avoiding distributed consensus for i/os, commits, and membership changes[C]//Proceedings of the 2018 International Conference on Management of Data. 2018: 789-796.
- Antonopoulos P, Budovski A, Diaconu C, et al. Socrates: The new sql server in the cloud[C]//Proceedings of the 2019 International Conference on Management of Data. 2019: 1743-1756.


large_Tech - Modify (5).png

medium_Tech - Modify (5).png

small_Tech - Modify (5).png

thumbnail_Tech - Modify (5).png

Tech - Modify (5).png

Why consensus-based replication algorithm is not the silver bullet for achieving data consistency in distributed databases?

Raft or not? The Best Solution to Data Consistency in Cloud-native Databases

We aim to supercharge the development of applications that require embeddings and other ML tasks.

## Introduction

Attempts at hand-crafting algorithms for understanding human-generated content have generally been unsuccessful. For example, it is difficult for a computer to “grasp” the semantic content of an image - e.g. a car, cat, coat, etc… - purely by analyzing its low-level pixels. [Color histograms](https://en.wikipedia.org/wiki/Color_histogram) and [feature detectors](https://en.wikipedia.org/wiki/Scale-invariant_feature_transform) worked to a certain extent, but they were rarely accurate enough for most applications.

In the past decade, the combination of big data and deep learning has fundamentally changed the way we approach computer vision, natural language processing, and other machine learning (ML) applications; tasks ranging from spam email detection to realistic text-to-video synthesis have seen incredible strides, with accuracy metrics on certain tasks reaching superhuman levels. A major positive side effect of these improvements is an increase in the use of embedding vectors, i.e. model artifacts generated by taking an intermediate result within a deep neural network. OpenAI’s [docs page](https://beta.openai.com/docs/guides/embeddings) gives a great overview:

> An embedding is a special format of data representation that can be easily utilized by machine learning models and algorithms. The embedding is an information dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. For example, if two texts are similar, then their vector representations should also be similar.

The table below shows three query images along with their corresponding top five images in embedding space (I used the first 1000 images of [Unsplash Lite](https://frankzliu.com/blog/making-machine-learning-more-accessible-for-application-developers) as the dataset):

![2022-04-20-104339.png](https://assets.zilliz.com/2022_04_20_104339_55c4f053dc.png)

These results you see above were generated with an *image embedding model* based on `resnet50`, a well-known pure convolutional image classification model. Embeddings are not limited to images - they can also be generated for wide array of different types of unstructured data, including images, audio, time-series data, and molecular structures. Models which embed multiple different types of data into the same space, also more commonly known as *multi-modal embedding models*, also exist and are being used in an increasingly large number of applications.

As we’ll see in the next two sections, generating these high quality embeddings can be difficult, especially at scale.

## Training a new model for embedding tasks

On paper, training a new ML model and generating embeddings with it sounds easy: take the latest and greatest pre-built model [1], backed by the newest architecture, and train it with some data. Easy, right?

Not so fast. On the surface, it may seem easy to use the latest model architecture to achieve state-of-the-art results [2]. This, however, could not be further from the truth. Let’s go over some common pitfalls related to training embedding models (these also apply to general machine learning models):

1. *Not enough data:* Training a new embedding model from scratch without enough data makes it prone to a phenomenon called *overfitting*. In practice, only the largest global organizations have enough data to make training a new model from scratch worthwhile; others must rely on *[fine-tuning](https://docs.towhee.io/fine-tune/train-operators/quick-start/)*, a process where an already-trained model with large amounts of data is then distilled using a smaller dataset.

2. *Poor hyperparameter selection:* [Hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) are constants which are used to control the training process, such as how quickly the model learns or how much data is used for training in a single batch. Selecting an appropriate set of hyperparameters is extremely important when fine-tuning a model, as small changes to certain values can result in vastly different results. [Recent research](https://arxiv.org/abs/2110.00476) in this area has also shown accuracy improvements on ImageNet-1k of over 5% (that’s a lot) training the same model from scratch with an improved training procedure.

3. *Overestimating self-supervised models:* The term self-supervision refers to a training procedure where “fundamentals” of the input data are learned by leveraging data itself without the need for labels [3]. In general, self-supervised methods are great for pre-training (i.e. training a model in a self-supervised fashion with lots of unlabelled data before fine-tuning it with a smaller labelled dataset) but directly using self-supervised embeddings can result in suboptimal performance.


A common way for tackling all three of the above problems is to first train a self-supervised model using a massive quantity of data before fine-tuning the model on labelled data. This has been shown to [work great for NLP](https://arxiv.org/abs/1810.04805), but [not so much for CV](https://arxiv.org/abs/2003.14323) just yet.

![2022-04-20-104656.png](https://assets.zilliz.com/2022_04_20_104656_1a54362798.png)
###### An illustration of Meta's data2vec training technique, a self-supervised method for training deep neural networks across a variety of unstructured data types. Source: [Meta AI blog](https://ai.facebook.com/blog/the-first-high-performance-self-supervised-algorithm-that-works-for-speech-vision-and-text/)

## Using embedding models has its own pitfalls

These are just some of the many common mistakes associated with training embedding models. As a direct result of this, many developers looking to use embeddings make direct use of pre-trained models on academic datasets such as [ImageNet](https://image-net.org/) (for image classification) and [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) (for question answering). Despite the abundance of pre-trained models available today, there are several pitfalls which should be avoided in order to extract maximum embedding performance:

1. *Training and inference data mismatch:* Using an off-the-shelf model trained by other organizations has become a popular way to develop ML applications without needing thousands of GPU/TPU-hours. Understanding the limitations of a particular embedding model and how that can impact application performance is extremely important; without understanding the model’s training data and methodology, it’s incredibly easy to misinterpret results. For example, a model trained to embed music will work poorly when applied to speech and vice versa.

2. *Improper layer selection:* When using a fully-supervised neural network as an embedding model, features are generally taken from the second-to-last layer (known formally as the penultimate layer) of activations. However, this can result in suboptimal performance, depending on the application. For example, when using a model trained for image classification to embed images of logos and/or brands, using earlier activations may result in improved performance. This is due to better retention of low-level features (edges and corners) which are critical to classifying non-complex images.

3. *Nonidentical inference conditions:* Train and inference conditions must be exactly the same to extract maximum performance out of an embedding model. In practice, this is often times not the case. A standard `resnet50` model from `torchvision`, for example, generates two completely different results when downsampled using bicubic interpolation versus nearest neighbor interpolation (see below).

![photo-1617739680032-8ab04113d6dc.jpeg](https://assets.zilliz.com/photo_1617739680032_8ab04113d6dc_ed6c7ba9dd.jpeg)
###### An Eastern Towhee. Photo by [Patrice Bouchard](https://unsplash.com/photos/CcmxhowdIFc).

![截屏2022-05-09 14.37.09.png](https://assets.zilliz.com/2022_05_09_14_37_09_56d7c061d0.png)

## Deploying an embedding model

Once you’ve jumped through all of the hurdles associated with training and validating a model, scaling and deploying it becomes the next critical step. Again, embedding model deployment is easier said than done. MLOps, a field adjacent to DevOps, exists specifically for this purpose.

1. *Selecting the right hardware:* Embedding models, analogous to most other ML models, can be run on a variety of different types of hardware, ranging from standard everyday CPUs to programmable logic (FPGAs). Entire [research papers](https://www.jaewoong.org/pubs/fpt16-accelerating-bnn.pdf) have been written analyzing the tradeoffs in terms of cost versus efficiency, highlighting the difficulty most organizations face here.

2. *Model deployment platform:* There are numerous MLOps and distributed computing platforms available (including many open-source ones). Figuring out how these will fit into your application can be a challenge in and of itself.

3. *Storage for embedding vectors:* As your application scales, you’ll need to find a scalable and more permanent storage solution for your embedding vectors. This is where [vector databases](https://frankzliu.com/blog/a-gentle-introduction-to-vector-databases) comes in.


## I’ll learn how to do it myself!

I applaud your enthusiasm! A couple of key things to remember:

**ML is very different from software engineering:** Traditional machine learning derives its roots from statistics, a branch of mathematics that is very different from software engineering. Important machine learning concepts such as regularization and feature selection all have strong fundamental roots in mathematics. While modern libraries for training and inference ([PyTorch](https://pytorch.org/) and [Tensorflow](https://www.tensorflow.org/) being two well-known ones) have made it significantly easier to train and productionize embedding models, understanding how different hyperparameters and training methodologies affect embedding model performance is still incredibly important.

**Learning to use PyTorch or Tensorflow can be unintuitive:** As previously mentioned, these libraries have greatly sped up the training, validation, and deployment of modern ML models. Building a new model or implementing an existing one can be very intuitive to seasoned ML developers or programmers who are familiar with HDL, but for most software developers, the underlying concepts can be difficult to grasp. There’s also the question of which framework to choose, as the execution engines used by these two frameworks have quite a few differences (personally, I recommend PyTorch).

**Finding an MLOps platform that fits your codebase will take time:** Here’s a [curated list](https://github.com/kelvins/awesome-mlops) of MLOps platforms and tools. There’s hundreds of different options to choose from, and evaluating the pros and cons of each is a years-long research project in and of itself.

With all this being said, I’d like to amend my statement above to say: I applaud your enthusiasm, but I *don’t recommend learning ML and MLOps*. It’s a fairly long and tedious process that can take time away from what’s most important: developing a solid application that your users will love.

## Supercharging data science with Towhee

[Towhee](https://towhee.io/) is an [open-source project](https://github.com/towhee-io/towhee) that helps software engineers develop and deploy applications which utilize embeddings in just a few lines of code [4]. Towhee affords software developers the freedom and flexibility to develop their own ML applications without having to dive deep into embedding models and machine learning.


### A quick example

A `Pipeline` is a single embedding generation task that is composed of several sub-tasks (also known as `Operators` in Towhee). By abstracting an entire task within a `Pipeline`, Towhee helps users avoid many of the embedding generating pitfalls mentioned above.

```
>>> from towhee import pipeline
>>> embedding_pipeline = pipeline('image-embedding-resnet50')
>>> embedding = embedding_pipeline('https://docs.towhee.io/img/logo.png')
```

In the example above, image decoding, image transformation, feature extraction, and embedding normalization are four substeps compiled into a single pipeline - no need to worry about the model and inference details yourself. Towhee provides pre-built embedding pipelines for a variety of tasks, including audio/music embeddings, image embeddings, face embeddings, and more. For a full list of pipelines, feel free to visit our [Towhee hub](https://towhee.io/pipelines).

### Method-chaining API

Towhee also provides a Pythonic unstructured data processing framework called `DataCollection`. In short, `DataCollection` is a method-chaining API which allows developers to rapidly prototype embedding and other ML models on real-world data. In the example below, we use `DataCollection` compute embeddings using the `resnet50` embedding model.

For this example, we’ll build an “application” that let’s us filter prime numbers with a ones digit of `3`:

```
>>> from towhee.functional import DataCollection
>>> def is_prime(x):
...     if x <= 1:
...         return False
...     for i in range(2, int(x/2)+1):
...         if not x % i:
...             return False
...     return True
...
>>> dc = (
...     DataCollection.range(100)
...         .filter(is_prime)             # stage 1, find prime
...         .filter(lambda x: x%10 == 3)  # stage 2, find primes that ends with '3'
...         .map(str)                     # stage 3, convert to string
... )
...
>>> dc.to_list()
```

`DataCollection` can be used to develop entire applications in just a single line of code. The next section shows how to develop a reverse image search application using `DataCollection` - keep reading ahead to learn more.

### Towhee trainer

As mentioned above, fully- or self-supervised trained models are often good at *generic* tasks. However, you’ll sometimes want to create an embedding model that’s good at something very specific, e.g. differentiating between cats versus dogs. Towhee provides a training/fine-tuning framework specifically for this purpose:

```
>>> from towhee.trainer.training_config import TrainingConfig
>>> training_config = TrainingConfig(
...     batch_size=2,
...     epoch_num=2,
...     output_dir='quick_start_output'
... )
```

You’ll also need to specify a dataset to train on:

```
>>> train_data = dataset('train', size=20, transform=my_data_transformer)
>>> eval_data = dataset('eval', size=10, transform=my_data_transformer)
```

With everything in place, training a new embedding model from an existing operator is a piece of cake:

```
>>> op.train(
...     training_config,
...     train_dataset=train_data,
...     eval_dataset=eval_data
... )
```

Once complete, you can use the same operator in your application with no changes to the rest of the code.

![161373537-3094d15c-3bfd-46b0-95cf-60f2a9c2765c.png](https://assets.zilliz.com/161373537_3094d15c_3bfd_46b0_95cf_60f2a9c2765c_3ba4677780.png)

###### Above: an attention heatmap showing the core areas of an image that an embedding model tries to encode. We'll be integrating attention heatmaps and other visualization tools directly into our fine-tuning framework in a future version of Towhee.

## An example application: reverse image search

To demonstrate how Towhee can be used, let’s quickly build a small [reverse image search](https://en.wikipedia.org/wiki/Reverse_image_search) application. Reverse image search is a well-known. Let’s dive right in:

```
>>> import towhee
>>> from towhee.functional import DataCollection
```

For this example application, we’ll be using a small dataset along with 10 query images (available [here](https://drive.google.com/file/d/1bg1RtUjeZlOfV2BiA2nf7sn5Jec9b-9I/view?usp=sharing)). Using `DataCollection`, we can then load both the dataset and query images:

```
>>> dataset = DataCollection.from_glob('./image_dataset/dataset/*.JPEG').unstream()
>>> query = DataCollection.from_glob('./image_dataset/query/*.JPEG').unstream()
```

The next step is to compute embeddings over the entire dataset collection:

```
>>> dc_data = (
...    dataset.image_decode.cv2()
...        .image_embedding.timm(model_name='resnet50')
... )
...
```

This step creates a local collection of embedding vectors - one for each image in the dataset. With this, we can now query for nearest neighbors:

```
>>> result = (
...     query.image_decode.cv2()                                    # decode all images in the query set
...         .image_embedding.timm(model_name='resnet50')            # compute embeddings using the `resnet50` embedding model
...         .towhee.search_vectors(data=dc_data, cal='L2', topk=5)  # search the dataset
...         .map(lambda x: x.ids)                                   # acquire IDs (file paths) of similar results
...         .select_from(dataset)                                   # get the result image
... )
...
```

We also provide a way to deploy your application using [Ray](https://github.com/ray-project/ray). Simply specify `query.set_engine('ray')` and you’re good to go!

## Closing words

One last note: we do not consider Towhee to be a full-fledged, end-to-end model serving or MLOps platform, nor is that what we set out to achieve. Rather, we aim to supercharge the development of applications that require embeddings and other ML tasks. With Towhee, our hope is to enable rapid prototyping of embedding models and pipelines on your local machine (`Pipeline` + `Trainer`), allow for development an ML-centric application in just a couple of lines of code (`DataCollection`), and allow for easy and rapid deployment to your own cluster (via [Ray](https://github.com/ray-project/ray)).

We’re constantly looking for for folks to join our open-source community - if this is something that interests you, please consider giving us a star on [Github](https://github.com/towhee-io/towhee). You can also get in touch with us on [Slack](https://slack.towhee.io/) and [Twitter](https://twitter.com/towheeio).

That’s all folks - hope this post was informative. If you have any questions, comments, or concerns, feel free to leave a comment below. Stay tuned for more!


## Footnotes

1. Using image classification as an example, we can see the rapid evolution of convolutional neural networks (CNNs): VGG architectures were deeper and had smaller kernel sizes when compared with AlexNet, ResNets introduced skip connections and batch normalization to facilitate backpropogation on ultra-deep models, and more recent Transformer-CNN hybrid architectures add self-attention to achieve state-of-the-art results. 

2. Transformer-based models require more model- and dataset-specific hyperparameter tuning along with a larger dataset size (and/or more data augmentation) to prevent overfitting. More [recent research](https://arxiv.org/abs/2201.03545) has also shown that a pure convolutional model utilizing depth-wise convolutions can outperform transformers for vision tasks, but that’s a story for another day. 

3. This is admittedly a gross oversimplification. I’ll elaborate what I mean by “fundamentals” in a future blog post. 

4. Although our initial focus is on generating embeddings, Towhee can also be used to create and share generic machine learning pipelines such as machine translation, object detection, and [image-to-image](https://towhee.io/towhee/anime-transfer) pipelines. 

large_markus-winkler-f57lx37DCM4-unsplash.jpg

medium_markus-winkler-f57lx37DCM4-unsplash.jpg

small_markus-winkler-f57lx37DCM4-unsplash.jpg

thumbnail_markus-winkler-f57lx37DCM4-unsplash.jpg

markus-winkler-f57lx37DCM4-unsplash.jpg

Making Machine Learning More Accessible for Application Developers

This article is dedicated to going a bit more in-depth into embeddings/embedding vectors, along with how they are used in modern ML algorithms and pipelines.

I’ve broached the subject of embeddings/embedding vectors in prior blog posts on [vector databases](https://frankzliu.com/blog/a-gentle-introduction-to-vector-databases) and [ML application development](https://frankzliu.com/blog/making-machine-learning-more-accessible-for-application-developers), but haven’t yet done a deep dive on embeddings and some of the theory behind how embedding models work. As such, this article will be dedicated towards going a bit more in-depth into embeddings/embedding vectors, along with how they are used in modern ML algorithms and pipelines.

A quick note - this article will require an intermediate knowledge of deep learning and neural networks. If you’re not quite there yet, I recommend first taking a look at [Google’s ML Crash Course](https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture). The course contents are great for understanding the basics of neural networks for CV and NLP.

## A quick recap

Vectorizing data via embeddings [1] is, at its heart, a method for dimensionality reduction. Traditional dimensionality reduction methods - [PCA](https://towardsdatascience.com/the-most-gentle-introduction-to-principal-component-analysis-9ffae371e93b), [LDA](https://www.mygreatlearning.com/blog/understanding-latent-dirichlet-allocation/), etc. - use a combination of linear algebra, kernel tricks, and other statistical methods to “compress” data. On the other hand, modern deep learning models perform dimensionality reduction by mapping the input data into a latent space, i.e. a representation of the input data where nearby points correspond to semantically similar data points. What used to be a one-hot vector representing a single word or phrase, for example, can now be represented as a dense vector with a significantly lower dimension. We can see this in action with the [Towhee library](https://github.com/towhee-io/towhee):

```
% pip install towhee  # pip3
% python              # python3
```

```
>>> import towhee
>>> text_embedding = towhee.dc(['Hello, world!']) \
...     .text_embedding.transformers(model_name='distilbert-base-cased') \
...     .to_list()[0]
...
>>> embedding  # includes punctuation and start & end tokens
```

```
array([[ 0.30296388,  0.19200979,  0.10141158, ..., -0.07752968,  0.28487974, -0.06456392],
       [ 0.03644813,  0.03014304,  0.33564508, ...,  0.11048479,  0.51030815, -0.05664057],
       [ 0.29160976,  0.43050566,  0.46974635, ...,  0.22705288, -0.0923526 , -0.04366254],
       [ 0.14108554, -0.00599108,  0.34098792, ...,  0.16725197,  0.10088076, -0.06183652],
       [ 0.35695776,  0.30499873,  0.400652  , ...,  0.20334958,  0.37474275, -0.19292705],
       [ 0.6206475 ,  0.50192136,  0.602711  , ..., -0.03119299,  1.1860386 , -0.6167787 ]], dtype=float32)
```

Embedding algorithms based on deep neural networks are almost universally considered to be stronger than traditional dimensionality reduction methods. These embeddings are being used more and more frequently in the industry in a variety of applications, e.g. content recommendation, question-answering, chatbots, etc. As we’ll see later, using embeddings to represent images and text within neural networks has also become increasingly popular in recent years.

![截屏2022-05-09 11.54.54.png](https://assets.zilliz.com/2022_05_09_11_54_54_bb76a19677.png)
###### Visualizing text embeddings produced by [DistilBERT](https://towhee.io/text-embedding/transformers). Note how "football" is significantly closer to "soccer" than it is to "footwear" despite "foot" being common both words.

## Supervised embeddings

So far, my previous articles have used embeddings from models trained using supervised learning, i.e. neural network models which are trained from labelled/annotated datasets. The [ImageNet](https://image-net.org/) dataset, for example, contains a curated set of image-to-class mappings, while question-answering datasets such as [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) provide 1:1 sentence mappings in different languages.

Many well-known models trained across labelled data use cross-entropy loss or mean-squared error. Since the end goal of supervised training is to more or less replicate 1:1 mappings between input data and annotations (e.g. output a class token probability given an input image), embeddings generated from supervised models seldom use the output layer. The standard ResNet50 model trained across ImageNet-1k, for example, outputs a 1000-dimension vector corresponding to probabilities that the input image is an instance of the Nth class label.

![lenet-vert.svg](https://assets.zilliz.com/lenet_vert_f78240b6f3.svg)
###### LeNet-5, one of the earliest known neural network architectures for computer vision. Image by [D2L.ai,](https://d2l.ai/) [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).

Instead, most modern applications use the penultimate layer of activations as the embedding. In the image above (LeNet-5), this would correspond to the activations between the layers labelled `FC (10)` (10-dimensional output layer) and `FC (84)`. This layer is close enough to the output to accurately represent the semantics of the input data while also being a reasonably low dimension. I’ve also seen computer vision applications which use pooled activations from a much earlier layer in the model. These activations capture lower-level features of the input image (corners, edges, blogs, etc…), which can result in improved performance for tasks such as logo recognition.

## Encoders and self-supervision

A major downside of using annotated datasets is that they require annotations (this sounds a bit idiotic, but bear with me here). Creating high-quality annotations for a particular set of input data requires hundreds if not thousands of hours of curation by one or many humans. The full ImageNet dataset, for example, contains approximately 22k categories and required an army of 25000 humans to curate. To complicate matters further, many labelled datasets often times contain unintended inaccuracies, flat-out errors, or [NSFW content](https://github.com/vinayprabhu/Dataset_audits/blob/master/Notebooks/ImageNet_2_NSFW_analysis.ipynb) in the curated results. As the number of such instances increases, the quality of embeddings generated by an embedding model trained with supervised learning decreases significantly.

![photo-1511519991062-53acadf7be18.jpeg](https://assets.zilliz.com/photo_1511519991062_53acadf7be18_215beb82dc.jpeg)

###### An image tagged as "Hope" from the [Unsplash dataset](https://unsplash.com/data). To humans, this is a very sensible description, but it may cause a model to learn the wrong types of features during training. Photo by [Lukas L](https://unsplash.com/photos/jSjgQmaQtVQ).

Models trained in an unsupervised fashion, on the other hand, do not require labels. Given the insane amount of text, images, audio, and video generated on a daily basis, models trained using this method essentially have access to an infinite amount of training data. The trick here is developing the right type of models and training methodologies for leveraging this data. An incredibly powerful and increasing popular way of doing this is via autoencoders (or encoder/decoder architectures in general).

Autoencoders generally have two main components. The first component is an encoder: it takes some piece of data as the input and transforms it into a fixed-length vector. The second component is a decoder: it maps the vector back into the original piece of data. This is known as an encoder-decoder architecture [1]:

![encoder-decoder.svg](https://assets.zilliz.com/encoder_decoder_1a8a0477a5.svg)
###### Illustration of encoder-decoder architecture. Image by [D2L.ai](https://d2l.ai/), [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).

The blue boxes corresponding to the `Encoder` and `Decoder` are both feed-forward neural networks, while the `State` is the desired embedding. There’s nothing paricularly special about either of these networks - many image autoencoders will use a standard ResNet50 or ViT for the encoder network and a similarly large network for the decoder.

Since autoencoders are trained to map the input to a latent state and then back to the original data, unsupervised or self-supervised embeddings are taken directly from the output layer of an encoder as opposed to an intermediate layer for models trained with full supervision. As such, autoencoder embeddings are only meant to be used to for *reconstruction* [2]. In other words, they can be used to represent the input data but are generally not powerful enough to represent semantics, such as what differentiates a photo of a cat from a photo of a dog.

In recent years, numerous improvements to self-supervision beyond the traditional autoencoder have come about [3]. For NLP, pre-training models with context, i.e. where a word or character appears relative to others in the same sentence or phrase, is commonplace and is now considered the de-facto technique for training state-of-the-art text embedding models [3]. Self-supervised computer vision embedding models are also roaring to life; contrastive training techniques that rely on data augmentation have shown great results when applied to general computer vision tasks. [SimCLR](https://arxiv.org/abs/2002.05709) and [data2vec](https://arxiv.org/abs/2202.03555) are two examples of neural networks that make use of masking and/or other augmentations for self-supervised training.

![image4.gif](https://assets.zilliz.com/image4_fba49fee9c.gif)
###### An Illustration of [SimCLR](https://arxiv.org/abs/2002.05709). The "representation" layer in the above illustration corresponds to embeddings generated by a convolutional neural network, i.e. the intermediate state. Image by Google Research ([source](https://github.com/google-research/simclr)).

## Embeddings as an input to other models

Embedding models are highly unique; not only are they valuable for generic application development, but their outputs are often used in other machine learning models. [Vision transformers](https://arxiv.org/abs/2010.11929) are an excellent example of this. Their popularity has exploded in the past two years, aided by strong performance and a large receptive field unavailable to traditional convolutional neural networks. The core premise of vision transformers is to divide an image into square patches, generate embeddings for each patch, and use the embeddings as inputs into a standard transformer. This, surprisingly, works very well for image recognition.

Another great example of this is OpenAI’s [CLIP](https://github.com/openai/CLIP), a large neural network model that is trained to match images with natural language. CLIP is trained on what amounts to essentially infinite data from the internet, e.g. Flickr photos and the corresponding photo title.

![CLIP.png](https://assets.zilliz.com/CLIP_c48a7be6e9.png)
###### Encoders and their corresponding embeddings are used to great effect in OpenAI's CLIP model. Image by [OpenAI](https://github.com/openai), [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).

The methodology behind CLIP is actually fairly straightforward. First, CLIP maps images and text into a latent state (i.e. an embedding); these latent states are then trained to map to the *same space*, i.e. a text embedding and image embedding should be very close to each other if the text is able to accurately describe the image. The devil is in the details, when it comes to ML, and CLIP is actually fairly difficult to implement in practice without a sizeable dataset and some hyperparameter tuning.

CLIP is used as a core component in DALLE (and DALLE-2), a text-to-image generation engine from OpenAI. There’s been a lot of buzz around DALLE-2 recently, but none of that would be possible without CLIP’s representational power. While CLIP and DALLE’s results have been impressive, image embeddings still have significant room for growth. Certain higher-level semantics such as object count and mathematical operations are still difficult to represent in an image embedding.

## Generating your own embeddings

I’ve pointed out the [Towhee](https://github.com/towhee-io/towhee) open-source project a couple of times in the past, showing how it can be used to generate embeddings as well as develop applications that require embeddings. Towhee wraps hundreds of image embedding and classification models from a variety of sources (`timm`, `torchvision`, etc…) and that are trained with a variety of different techniques. Towhee also has many NLP models as well, courtesy of 🤗’s [Transformers](https://github.com/huggingface/transformers) library.

Let’s take a step back and peek at how Towhee generates these embeddings under the hood.

```
>>> from towhee import pipeline
>>> p = pipeline('image-embedding')
>>> embedding = p('towhee.jpg')
```

With [Towhee](https://github.com/towhee-io/towhee), the default method for generating embeddings from a supervised model is simply to remove the final classification or regression layer. For PyTorch models, we can do so with the following example code snippet:

```
>>> import torch.nn as nn
>>> import torchvision
>>> resnet50 = torchvision.models.resnet50(pretrained=True)
>>> resnet50_emb = nn.Sequential(*(list(resnet50.children())[:-1]))
```

The last line in the above code snippet recreates a feed-forward network (`nn.Sequential`) composed of all layers in `resnet50` (`resnet50.children()`) with the exception of the final layer (`[:-1]`). Intermediate embeddings can also be generated with the same layer removal method. This step is unnecessary for models trained with contrastive/triplet loss or as an autoencoder.

Models based on `timm` (for CV) and `transformers` (for NLP) also maintain their own methods which make feature extraction easy:

```
>>> import numpy as np
>>> from PIL import Image
>>> import timm
>>> image = numpy.array(Image.open('towhee.jpg'))
>>> model = timm.models.resnet50(pretrained=True)
>>> embedding = model.forward_features(img)
```

```
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
>>> model = AutoModel.from_pretrained('bert-base-uncased')
>>> inputs = tokenizer('Hello, world!', return_tensors='pt')
>>> outputs = model(**inputs)
>>> embedding = outputs.last_hidden_state()
```

Towhee maintains wrappers for both `timm` and `transformers`:

```
>>> import towhee
>>> text_embedding = towhee.dc(['Hello, world!']) \
...     .text_embedding.transformers(model_name='distilbert-base-cased') \
...     .to_list()[0]  # same as intro example
...
>>> img_embedding = towhee.glob('towhee.jpg') \
...    .image_decode() \
...    .image_embedding.timm(model_name='resnet50') \
...    .to_list()[0]
...
```

## Other resources

I’ve compiled a short and very incomplete list of other great resources on embeddings, if you want to read more or approach the topic from another perspective:

1. The [Milvus documentation](https://milvus.io/docs/v1.1.0/vector.md) provides an overview of embedding vectors (as it relates to storage and vector databases).

2. OpenAI maintains page on [text embeddings](https://beta.openai.com/docs/guides/embeddings) that you can check out.

3. [Will Koehrsen](https://willkoehrsen.github.io/) provides a [great overview of embeddings](https://towardsdatascience.com/neural-network-embeddings-explained-4d028e6f0526). It’s a bit dated (2018), but still a great resource.

4. See how embeddings are being used at [Twitter](https://blog.twitter.com/engineering/en_us/topics/insights/2018/embeddingsattwitter) for recommendation.

5. Reverse image search is one of many applications for embedding vectors. [This tutorial](https://docs.towhee.io/Tutorials/reverse-image-search/) demonstrates how to build one in minutes.

If this post was helpful for you, please consider following me on Twitter. In an upcoming post, I’ll provide a brief overview of similarity search and vector indexing algorithms. Stay tuned!


## Footnotes
1. Adapted from [D2L.ai](https://github.com/d2l-ai/d2l-en). [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) license. 

2. Input reconstruction is a fairly complex problem. For example, generating embeddings that preserve face details in an input photo is a desireable feature, but a “generic” autoencoder would have trouble doing so. 

3. Patches or crops of whole images do techically have context as well. Algorithms and models which understand context are crucial to the subfield of computer vision/graphics known as inpainting. Pre-training techniques for computer vision appliations hasn’t been as successful, but it will likely become more viable in the near future. [This 2021 paper](https://arxiv.org/abs/2111.06377) shows how masking patches in an autoencoder can be used for pre-training vision transformers.




large_uriel-sc-11KDtiUWRq4-unsplash.jpg

medium_uriel-sc-11KDtiUWRq4-unsplash.jpg

small_uriel-sc-11KDtiUWRq4-unsplash.jpg

thumbnail_uriel-sc-11KDtiUWRq4-unsplash.jpg

uriel-sc-11KDtiUWRq4-unsplash.jpg

What are embeddings and how are they used in modern ML algorithms and pipelines

Understanding Neural Network Embeddings

Recommending engaging content at lightening speed inside Xiaomi’s mobile news aggregator. 

Xiaomi is a global electronics company that manufactures and invests in smartphones, mobile applications, laptops, home appliances, laptops, and many other products. The company ships its smartphones with a home brewed web browser that has a news aggregator embedded within it. To consistently make content recommendations that appeal to individual users Xiaomi leveraged artificial intelligence (AI).

Milvus, an open-source vector database built by Zilliz, is used to power the core recommendation engine inside Xiaomi's smart news aggregator.


small_xiaomi.png

thumbnail_xiaomi.png

xiaomi.png

large_05.jpg

medium_05.jpg

small_05.jpg

thumbnail_05.jpg

05.jpg

Xiaomi’s Smart News Aggregator

large_beike.png

medium_beike.png

small_beike.png

thumbnail_beike.png

beike.png

Extracting highlights from massive video data for individual marathon runners or participants in other events.

iYUNDONG builds artificial intelligence (AI) tools that analyze media captured during sporting events to automatically extract key highlights. For example, by uploading a single photo, images of individual marathon runners can be automatically retrieved from a massive dataset that includes photos and videos from the event. 

iYUNDONG leverages Milvus, an open-source vector database built by Zilliz, to power its core image retrieval system.

thumbnail_iyundong.png

iyundong.png

large_iyundong.jpg

medium_iyundong.jpg

small_iyundong.jpg

thumbnail_iyundong.jpg

iyundong.jpg

iYUNDONG‘s AI Automatically Extracts Sporting Event Highlights

large_iyundong.png

medium_iyundong.png

small_iyundong.png

Helping people create relevant, high-quality documents at the click of a button inside WPS Office's Writer.

WPS Office is a productivity tool developed by Kingsoft with over 300M monthly active users worldwide. The company’s artificial intelligence (AI) department built a smart writing assistant from scratch using semantic matching algorithms such as intent recognition and text clustering. The tool helps users quickly create outlines, individual paragraphs, and entire documents by simply inputting a title and selecting up to five keywords.

Kingsoft’s writing assistant uses Milvus, an open-source vector database built by Zilliz, to power its core vector processing module.



small_wps.png

thumbnail_wps.png

wps.png

large_wps-blog-cover.png

medium_wps-blog-cover.png

small_wps-blog-cover.png

thumbnail_wps-blog-cover.png

wps-blog-cover.png

Kingsoft’s AI-Powered Writing Assistant

Reducing vector retrieval time from days to hours and minimizing false positive results in short text classification. 



Sohu is a Nasdaq-listed company that offers advertising, search engine, online multiplayer gaming, and many other services. The company built the Sohu News App to provide its users with personalized content. To improve the click-through rate (CTR) and relevance of the app’s recommendations, Sohu leveraged semantic vector similarity search.

Sohu used Milvus, an open-source vector database built by Zilliz, to power the core semantic similarity search engine inside its intelligent news aggregator.



sohu.png

large_sohu-story-cover.jpg

medium_sohu-story-cover.jpg

small_sohu-story-cover.jpg

thumbnail_sohu-story-cover.jpg

sohu-story-cover.jpg

Sohu’s AI News Aggregator

large_sohu-user-image.jpg

medium_sohu-user-image.jpg

small_sohu-user-image.jpg

thumbnail_sohu-user-image.jpg

sohu-user-image.jpg

Recommending outfits and enabling image search in Mozat's Stylepedia lifestyle app.

[Mozat](http://www.mozat.com/home) is a Singaporean start-up focused on building innovative social media, communication, and lifestyle applications. [Stylepedia](https://stylepedia.com/) is a digital closet and outfit planning app that helps people passionate about fashion discover and discuss couture. Stylepedia users can create their own digital closets, receive personalized styling suggestions, and interact with other users to share fashion tips and trends.

Milvus, an open-source vector database, is used to power the image similarity search system that Stylepedia relies on to realize its image search and outfit recommender functionalities.

thumbnail_icon-mozat.jpg

icon-mozat.jpg

large_mozat-story-cover.jpg

medium_mozat-story-cover.jpg

small_mozat-story-cover.jpg

thumbnail_mozat-story-cover.jpg

mozat-story-cover.jpg

Mozat's wardrobe app powered by an image search system built with Milvus

large_mozat-user image.jpg

medium_mozat-user image.jpg

small_mozat-user image.jpg

thumbnail_mozat-user image.jpg

mozat-user image.jpg

Matching what people see in everyday life with artworks from the Cleveland Museum of Art's extensive collection.

Determined to adapt amid the limitations placed on in-person experiences by the coronavirus pandemic, the Cleveland Museum of art sought new, creative, and easy ways for people to engage with art. In this vein, CMA developed a reverse image search tool called "Artlens AI: Share Your View" that is capable of recognizing shapes, patterns, objects, and even architecture in user uploaded photos. After a user submits a photo, ArtLens AI returns the closest match from the CMA's vast art collection. The museum's hope is that by placing artwork next to contemporary views people will be given new perspectives and opportunities for critical thinking.

Milvus, an open-source vector database built by Zilliz, was used to power Artlens AI's core vector management module.

thumbnail_CMA logo.png

CMA logo.png

large_Blog_ArtLens AI_ 1.jpeg

medium_Blog_ArtLens AI_ 1.jpeg

small_Blog_ArtLens AI_ 1.jpeg

thumbnail_Blog_ArtLens AI_ 1.jpeg

Blog_ArtLens AI_ 1.jpeg

The Cleveland Museum of Art Uses AI to Connect Daily Life with Art History

large_artlens AI interface.png

medium_artlens AI interface.png

small_artlens AI interface.png

thumbnail_artlens AI interface.png

artlens AI interface.png

Enriching video content recommendations for Opera.

Opera is a Norwegian software company that creates some of the world's most popular web browsers, Opera News, and other services. The company is a pioneer in leveraging artificial intelligence to make digital content recommendations. To ensure users are served a wide variety of videos, and to avoid recommending the same content multiple times, Opera built a smart video reduplication system inside its video content recommendation platform.

Milvus, an open-source vector database built by Zilliz, is used to power Opera’s smart video deduplication system.

thumbnail_Frame 247.jpg

Frame 247.jpg

large_opera-blog-cover.png

medium_opera-blog-cover.png

small_opera-blog-cover.png

thumbnail_opera-blog-cover.png

opera-blog-cover.png

Opera’s Intelligent Video Deduplication System

large_opera-img.png

medium_opera-img.png

small_opera-img.png

thumbnail_opera-img.png

opera-img.png

Mitigating threats to critical data and strengthening cybersecurity with real-time virus detection.

Trend Micro is a global leader in hybrid cloud security, network defense, small business security, and endpoint security. To protect Android devices from viruses, the company built Trend Micro Mobile Security—a mobile app that compares APKs from the Google Play Store to a database of known malware. If an APK with nefarious code is found, the app provides users with detailed information about the virus and its threat level.


Milvus, an open-source vector database built by Zilliz, is used to power Trend Micro Mobile Security’s virus vector comparison module. 



thumbnail_trend icon.png

trend icon.png

large_trend-story-cover.jpg

medium_trend-story-cover.jpg

small_trend-story-cover.jpg

thumbnail_trend-story-cover.jpg

trend-story-cover.jpg

Trend Micro’s Mobile Security App

large_Trend Micro interface.png

medium_Trend Micro interface.png

small_Trend Micro interface.png

thumbnail_Trend Micro interface.png

Trend Micro interface.png

Energizing a personalized recommender service with unstructured data search on Vipshop’s e-commerce platform.

Vipshop is a leading online discount retailer for brands in China. The company offers high quality and popular branded products to consumers throughout China at a significant discount from retail prices. To optimize the shopping experience for their customers, the company built a personalized search recommendation system based on user query keywords and user portraits.
Milvus, an open-source vector database built by Zilliz, is used to power the personalized recommender system on Vipshop’s mobile apps.

large_vipshop-blog.jpg

medium_vipshop-blog.jpg

small_vipshop-blog.jpg

thumbnail_vipshop-blog.jpg

vipshop-blog.jpg

large_vipshop-story-cover.jpg

medium_vipshop-story-cover.jpg

small_vipshop-story-cover.jpg

thumbnail_vipshop-story-cover.jpg

vipshop-story-cover.jpg

Vipshop's Personalized Recommender System

large_vipshop-user image.jpg

medium_vipshop-user image.jpg

small_vipshop-user image.jpg

thumbnail_vipshop-user image.jpg

vipshop-user image.jpg

Making search results more relevant and boosting online product sales on Tokopedia's e-commerce platform.

Tokopedia is a technology company specializing in e-commerce. Its marketplace is the most popular online shopping platform in Indonesia, with 100+ million monthly active users and 9.7+ million merchants. To make search results more relevant and improve user engagement, Tokopedia built a smart search engine capable of pairing low fill rate keywords with high fill rate keywords.

Milvus, an open-source vector database built by Zilliz, is used to index the underlying product database and optimize Tokopedia's AI-enabled shopping experience.

small_tokopedia.jpeg

thumbnail_tokopedia.jpeg

tokopedia.jpeg

large_beike-blog-cover.jpg

medium_beike-blog-cover.jpg

small_beike-blog-cover.jpg

thumbnail_beike-blog-cover.jpg

beike-blog-cover.jpg

Tokopedia’s AI-enabled E-commerce Site Search

Powering semantic search inside LucidWorks Fusion data discovery platform.

Lucidworks is an enterprise search technology company that offers an application development platform, commercial support, and value-add software for open-source Apache Lucene and Apache Solr. The company built Lucidworks Fusion in a modern, containerized, cloud-native architecture that simplifies the process of deploying AI-powered data discovery and search applications. To ensure users find relevant data fast, Lucidworks implemented semantic search using the vector similarity approach.

Milvus, an open-source vector database built by Zilliz, is used to power the core semantic search module inside Lucidworks Fusion.

thumbnail_icon_Lucidworks.jpg

icon_Lucidworks.jpg

Lucidworks Fusion Data Discovery Platform

Products

Resources

Open Source

About

The world's most advanced open-source vector database

Vector Similarity Search

AI for Everything

Ready for the Cloud

Optimal MLOps