# clip-vit-base-patch32

Source: https://zilliz.com/ai-models/clip-vit-base-patch32
Published: 2024-08-17T21:18:22.725Z

## Model attributes

| Attribute | Value |
| --- | --- |
| Provider | OpenAI |
| Integration | Zilliz Cloud Integrated |
| Task | Embedding |
| Modality | Multimodal |
| Similarity metric | Any (Normalized) |
| License | Apache 2.0 |
| Dimensions | 1536 |
| Max input tokens | 77 |
| Price | Free |

## Introduction to clip-vit-base-patch32

The CLIP model, developed by OpenAI, aims to understand robustness in computer vision tasks and test models' ability to generalize to new image classification tasks without prior training. The `clip-vit-base-patch32` variant utilizes a ViT-B/32 Transformer architecture for image encoding and a masked self-attention Transformer for text encoding. By training these encoders to maximize the similarity of (image, text) pairs through contrastive loss, the model learns to associate images with corresponding textual descriptions.


## How to create multimodal embeddings with clip-vit-base-patch32

There are two primary ways to generate vector embeddings:

1. **[Zilliz Cloud Pipelines](https://zilliz.com/zilliz-cloud-pipelines)**: a built-in feature within [Zilliz Cloud](https://zilliz.com/cloud) (the managed Milvus) that seamlessly integrates the `clip-vit-base-patch32` model. It provides an out-of-box solution that simplifies creating and retrieving text or image vector embeddings.
2. **SentenceTransformers**: the Python library for `sentence_transformers`.

Once the vector embeddings are generated, they can be stored in [Zilliz Cloud](https://zilliz.com/cloud) (a fully managed vector database service powered by Milvus) and used for [semantic similarity search](https://zilliz.com/glossary/semantic-search). Here are four key steps:

1. **[Sign up](https://cloud.zilliz.com/signup)** for a Zilliz Cloud account for free.
2. **[Set up a serverless cluster](https://docs.zilliz.com/docs/create-cluster#set-up-a-free-cluster)** and obtain the [Public Endpoint and API Key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details).
3. **Create a vector collection** and insert your vector embeddings.
4. **Run a semantic search** on the stored embeddings.


### Generate vector embeddings via Zilliz Cloud Pipelines and perform a similarity search 

Refer to the following resources for step-by-step instructions.

- [Zilliz Cloud Pipelines documentation](https://docs.zilliz.com/docs/building-a-rag-app-with-pipelines)
- [Zilliz Cloud Pipelines Demo Video](https://zilliz.com/zilliz-cloud-pipelines)


### Generate vector embeddings via SentenceTransformer and insert them into Zilliz Cloud for similarity search

```
from PIL import Image
from sentence_transformers import SentenceTransformer
from pymilvus import MilvusClient
import requests

#Load CLIP model
model = SentenceTransformer('clip-ViT-B-32')

# Generate image embeddings
image_urls = [
    "https://raw.githubusercontent.com/milvus-io/milvus-docs/v2.4.x/assets/milvus_logo.png",
]
images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]
image_embeddings = model.encode(images)

# Generate text embeddings
queries = ["blue logo"]
query_embeddings = model.encode(queries)

# Connect to Zilliz Cloud with Public Endpoint and API Key
client = MilvusClient(
    uri=ZILLIZ_PUBLIC_ENDPOINT,
    token=ZILLIZ_API_KEY)

COLLECTION = "documents"
if client.has_collection(collection_name=COLLECTION):
    client.drop_collection(collection_name=COLLECTION)
client.create_collection(
    collection_name=COLLECTION,
    dimension=512,
    auto_id=True)

for image_url, embedding in zip(image_urls, image_embeddings):
    client.insert(COLLECTION, {"url": image_url, "vector": embedding})
    
results = client.search(
    collection_name=COLLECTION,
    data=query_embeddings,
    consistency_level="Strong", 
    output_fields=["text"])
```

For more information, refer to [the model page on HuggingFace](https://huggingface.co/sentence-transformers/clip-ViT-B-32).