FiftyOne and Zilliz Cloud Integration
FiftyOne and Zilliz Cloud integrate to power vision searches on computer vision datasets, combining FiftyOne's open-source toolkit for dataset curation and model evaluation with Zilliz Cloud's high-performance vector database for efficient visual similarity search, cross-modal retrieval, and multimodal embedding exploration.
Use this integration for FreeWhat is FiftyOne
FiftyOne is an open-source tool for building high-quality datasets and computer vision models. It enables data science teams to improve computer vision models through dataset curation, model evaluation, error identification, embedding visualization, and production workflows. With native integration for vector search engines, FiftyOne allows users to seamlessly leverage similarity search for efficient visual data exploration with just a single line of code.
By integrating with Zilliz Cloud (fully managed Milvus), FiftyOne gains access to a scalable vector database for storing and searching image embeddings, enabling efficient visual similarity search, cross-modal retrieval (text-to-image searching), object-specific searching within images, and multimodal embedding exploration across large-scale computer vision datasets.
Benefits of the FiftyOne + Zilliz Cloud Integration
- Single-line similarity search: FiftyOne's native Milvus integration lets you compute similarity indices and query datasets with
backend="milvus"— requiring just a single line of code to leverage Zilliz Cloud's vector search capabilities. - Cross-modal retrieval: The integration supports text-to-image searching, enabling users to find images using natural language queries powered by models like CLIP, with embeddings stored and searched in Zilliz Cloud.
- Object-specific searching: Beyond whole-image similarity, users can search for specific objects within images (people, animals, vehicles) using patch-level embeddings stored in Zilliz Cloud.
- Flexible authentication and configuration: The integration supports multiple authentication methods — environment variables, brain config files, or keyword arguments — for connecting to Milvus servers or Zilliz Cloud.
- Visual data exploration: FiftyOne's interactive App provides point-and-click similarity search alongside programmatic Python access, both backed by Zilliz Cloud's fast vector retrieval.
- Single-line similarity search: FiftyOne's native Milvus integration lets you compute similarity indices and query datasets with
How the Integration Works
FiftyOne serves as the computer vision interface and dataset management layer, providing tools for loading datasets, computing embeddings using models like CLIP, managing similarity indices (brain runs), and visualizing search results through its interactive App and Python API.
Zilliz Cloud serves as the vector database backend, storing and indexing the image and patch embeddings computed by FiftyOne. It provides high-performance similarity search using metrics like dot product or Euclidean distance, enabling fast retrieval of the most similar images from large datasets.
Together, FiftyOne and Zilliz Cloud create a complete visual search solution: images are loaded into FiftyOne, embeddings are computed using a chosen model, and a Milvus similarity index is created in Zilliz Cloud via
compute_similarity(). Users can then query their data usingsort_by_similarity()to find similar images by sample ID, text prompts, or embedding vectors — all with Zilliz Cloud handling the vector storage and search at scale.Step-by-Step Guide
1. Install Required Packages
python3 -m pip install pymilvus fiftyone torch torchvision2. Load a Dataset into FiftyOne and Compute Embeddings
Load the sample image set provided by FiftyOne, compute embeddings, and create a Milvus similarity index:
import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz # Step 1: Load your data into FiftyOne dataset = foz.load_zoo_dataset("quickstart") # Steps 2 and 3: Compute embeddings and create a similarity index milvus_index = fob.compute_similarity( dataset, brain_key="milvus_index", backend="milvus", )3. Conduct Vision Similarity Searches
Use the Milvus similarity index to conduct vision similarity searches on your dataset:
# Step 4: Query your data query = dataset.first().id # query by sample ID view = dataset.sort_by_similarity( query, brain_key="milvus_index", k=10, # limit to 10 most similar samples ) # Step 5 (optional): Cleanup # Delete the Milvus collection milvus_index.cleanup() # Delete run record from FiftyOne dataset.delete_brain_run("milvus_index")4. Configure the Milvus Backend
Set the Milvus backend permanently via environment variable:
export FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKEND=milvusOr configure via brain config at
~/.fiftyone/brain_config.json:{ "default_similarity_backend": "milvus" }5. Authentication for Custom Milvus Server or Zilliz Cloud
Configure credentials via environment variables (recommended):
export FIFTYONE_BRAIN_SIMILARITY_MILVUS_URI=XXXXXX export FIFTYONE_BRAIN_SIMILARITY_MILVUS_USER=XXXXXX export FIFTYONE_BRAIN_SIMILARITY_MILVUS_PASSWORD=XXXXXX export FIFTYONE_BRAIN_SIMILARITY_MILVUS_TOKEN=XXXXXXOr pass credentials directly as keyword arguments:
milvus_index = fob.compute_similarity( ... backend="milvus", brain_key="milvus_index", uri="XXXXXX", token="XXXXXX", collection_name="your_collection", metric="dotproduct", )6. Manage Brain Runs
List, inspect, rename, or delete brain runs:
# List all brain runs dataset.list_brain_runs() # Only list similarity runs dataset.list_brain_runs(type=fob.Similarity) # Get brain run info info = dataset.get_brain_info("milvus_index") print(info) # Delete brain run and Milvus collection milvus_index = dataset.load_brain_results("milvus_index") milvus_index.cleanup() dataset.delete_brain_run("milvus_index")Learn More
- Conduct Vision Searches with Milvus and FiftyOne — Official Milvus tutorial for vision searches with FiftyOne
- Exploring Multimodal Embeddings with FiftyOne and Milvus — Zilliz blog on multimodal embeddings with FiftyOne
- Build Better Multimodal RAG Pipelines with FiftyOne, LlamaIndex, and Milvus — Zilliz blog on multimodal RAG pipelines
- Using Vector Search to Better Understand Computer Vision Data — Zilliz blog on vector search for CV data
- FiftyOne Milvus Integration Documentation — Official FiftyOne documentation for Milvus integration