What Is Object Storage? How It Works and Why AI Needs It

Glossary
Object Storage

Key Takeaways

Object storage is a data storage architecture that manages data as discrete objects in flat buckets, accessed by ID over an HTTP API.
It has become the default storage layer for AI infrastructure because AI workloads generate massive amounts of unstructured data (embeddings, training media, conversation logs, model checkpoints), which object storage holds cheaply at near-unlimited scale.
The major services are Amazon S3, Google Cloud Storage, and Azure Blob Storage, plus open-source MinIO and Ceph.
But object storage alone cannot serve the three workloads AI applications need on that data: real-time retrieval, interactive discovery, and batch analytics. Each typically requires its own system, with copies of the data kept in sync.
Zilliz Vector Lakebase fills those gaps by adding a vector index, selective reads, and a tiered cache directly on top of object storage, so all three workloads run on the same data without copies.

What is Object Storage

Object storage is a data storage architecture that manages data as discrete objects in a flat container, accessed by ID over an HTTP API. It is built for unstructured data: photos, video, audio, email, web pages, sensor readings, and the vector embeddings that AI applications produce.

A cloud object store spreads this data across many machines, but lets you reach all of it as a single pool through that HTTP API. Each object holds three things: the data itself, any metadata you add, and a unique ID. You fetch or analyze any object by that ID, whatever its file type.

Amazon S3, the first cloud object store, now holds more than 500 trillion objects. It has become the default storage layer of the cloud.

Today, with AI applications generating embeddings and consuming unstructured data at massive scale, object storage has become the default home for AI data. Most of that data is unstructured: vector embeddings, training media, conversation logs, model checkpoints. A single embedding job can produce billions of vectors, and a retrieval-augmented generation stack often stores hundreds of millions more.

AI teams chose object storage for the same reasons everyone else did: low cost and high durability at near-unlimited scale. The new demand is to do useful work on that data without moving it: retrieval with low latency, analytics across the corpus, and interactive discovery, all running against the same files.

How Object Storage Works

To understand how object storage works, it helps to compare it with the two other storage models software uses. File storage puts data in nested folders. Block storage splits data into fixed-size blocks on a disk. Object storage keeps each item whole in a flat container called a bucket, with no folders.

To store a file, the system takes the data, adds any metadata you give it, and attaches an ID. Apps read, write, and delete objects by that ID over an HTTP API. There is no file path and no mount point.

PUT /my-bucket/embedding-batch-042.parquet     # store an object
GET /my-bucket/embedding-batch-042.parquet     # retrieve it by ID
DELETE /my-bucket/embedding-batch-042.parquet  # remove it

The flat structure is what makes object storage scale. You add capacity by adding machines to the pool, not by reworking a folder tree, so a single bucket can grow almost without limit.

Major Object Storage Services

Service	Provider	API standard
Amazon S3	AWS	S3 (the de facto standard)
Cloud Storage	Google Cloud	S3-interoperable + native
Blob Storage	Microsoft Azure	Native + S3-compatible gateways
MinIO / Ceph	Self-hosted (open source)	S3-compatible

Amazon S3, launched by AWS in 2006, set the object-storage API that the rest of the industry now copies. It carries the widest range of storage classes, from S3 Standard for frequently accessed data through S3 Glacier Deep Archive for long-term cold storage, and its S3 Intelligent-Tiering class moves objects to cheaper tiers automatically as their access patterns change.

Google Cloud Storage reaches all four of its storage classes, Standard, Nearline, Coldline, and Archive, through one bucket and one API. Its Autoclass feature watches each object's access pattern and moves it between those classes on its own, which removes the work of writing lifecycle rules by hand.

Azure Blob Storage is Microsoft's object store, with four access tiers, Hot, Cool, Cold, and Archive, set per blob or per account. Its distinguishing layer is Azure Data Lake Storage Gen2, a hierarchical namespace built on top of Blob that adds real directories and file-level permissions, giving analytics engines file-system semantics at object-storage cost. Blob exposes Azure's own API rather than S3, so S3 tools connect through a compatibility gateway.

MinIO and Ceph are the open-source way to run S3-compatible object storage on your own hardware, in a private cloud, or in an air-gapped environment. MinIO is lightweight and object-only, built for high throughput and common in Kubernetes clusters. Ceph is a unified system that serves object, block, and file storage from one cluster, exposing the S3 API through its RADOS Gateway. Both let applications written for S3 keep running without a rewrite.

Key Features and Limitations of Object Storage

Object storage's design gives it a clear set of strengths and a clear set of limits.

Key Features of Object Storage

Scales almost without limit. A single bucket holds as many objects as you need. You add capacity by writing more data, and you pay only for what you store, with no cluster to size up front.
Durable and resilient. Object stores keep copies of each object on several machines, often across data centers or regions. Amazon S3 is built for eleven nines (99.999999999%) of durability, with every object stored across at least three Availability Zones.
Cheap, with tiers. Per-gigabyte prices sit well below block or file storage, and cold data moves to cheaper tiers automatically.
Holds any data, with rich metadata. One bucket can hold any kind of unstructured data, and each object carries metadata you define. You can find and filter objects by that metadata, whatever the file type.
Open and easy to reach. Objects are read over a plain HTTP API. The S3 API is a de facto standard, so many tools read the same buckets without special connectors.

Limitations of Object Storage

Slower than block storage. Every request goes over HTTP, so object storage is a poor fit for low-latency, random-access work or transactional databases.
No editing in place. You replace a whole object rather than change part of it, which makes object storage a bad fit for data that changes often.
Small reads get pricey. A workload that makes many tiny reads can run up large API charges even when the data itself is small.
No built-in retrieval or analytics layer. Object storage stores and serves bytes. Out of the box, it has no index, no query engine, no similarity search. AWS S3 Vectors (generally available December 2025) added a native vector layer, and S3 Select can run lightweight queries, but anything beyond "fetch this object by ID" still typically needs a query or index layer on top.

Use Cases, and the Bridge to AI

The scale, durability, and low cost of object storage make it the default home for many workloads:

Data lakes (the standard storage foundation)
Backup and disaster recovery
Long-term archives
Media storage and delivery
Logs and analytics data
ML training sets (the large datasets models learn from)

AI search is now asking object storage to do more than hold data. Embeddings are the numerical vectors behind semantic search and retrieval-augmented generation (RAG). There are huge numbers of them, each one is large, and storing them in a specialized database gets expensive.

So teams have started keeping embeddings in object storage directly. Amazon S3 Vectors, generally available since December 2025, adds native vector storage and search to S3. AWS says it cuts the cost of storing and querying vectors by up to 90% against specialized vector databases, holds up to two billion vectors per index, and answers frequent queries in about 100 milliseconds and rare ones in under a second.

The catch is speed and breadth. Object storage can now hold and search vectors cheaply, but doing it at production latency, across retrieval, discovery, and analytics, while the data stays in object storage and is never copied into a separate fast store, is the hard part.

Object Storage in AI Infrastructure

Object storage is where AI data lives. Most of what an AI application touches, such as embeddings, training media, model artifacts, and conversation logs, is large, unstructured, and impractical to keep in any specialized system at scale. Object storage holds it at a fraction of the cost, at any size, and remains reachable by every engine that can speak HTTP.

But three workloads pull on that data, and each one is hard for object storage to serve on its own:

real-time retrieval (find the K most similar embeddings for a query, in milliseconds),
iterative discovery (let humans or agents explore the dataset interactively),
batch analytics (run large jobs across the whole corpus).

Today, teams handle these by putting each workload in its own system: a vector database for retrieval, an OLAP engine for analytics, a dataset-exploration tool for discovery. That means multiple systems to operate, multiple data copies to keep in sync, and ETL pipelines moving the data between them as workloads change.

Object storage on its own does not serve these three workloads well. It has no vector index, so similarity search has to happen in a separate system. It has no selective fetch at sub-file granularity, so even a small query pulls whole files over HTTP. It has no cache layer, so every query pays the full object-storage round trip.

Zilliz Vector Lakebase is Available in Public Preview

Zilliz Vector Lakebase is one platform built to fill those three gaps in object storage directly. It adds a vector index that lives on object storage alongside the data, query reads that fetch only the index pages and data fragments a query touches, and a tiered cache in front of object storage. The result is a single platform that handles real-time retrieval, iterative discovery, and batch analytics against the same data on object storage without copying it between systems.

Vector Lakebase is the evolution of Zilliz Cloud from a managed vector database into a lake-native semantic data platform. Its core capabilities include:

External Collections. Point at lake tables you already have, in Lance, Apache Iceberg, Apache Parquet, or its own Vortex format, and run vector search against them in place, with no data copy.
Tiered Serving. Three serving tiers (Performance-Optimized in memory, Capacity-Optimized in memory plus local NVMe, Tiered-Storage spanning memory, local NVMe, and object storage) match cost to workload latency.
On-Demand Search. Compute scales to zero when idle, so workloads with bursty or infrequent queries pay for what they run rather than for always-on capacity.
Unified Lake-Native Storage. The Vortex format pairs vectors with their metadata in one columnar file the platform can serve and analyze directly, with no stitching across multiple formats.

In Zilliz's benchmark on a one-billion-vector Iceberg table, query latency depends on the cache state: a brute-force Spark scan with no index takes hours; cold, with the index just built in about 20 minutes, queries land in roughly 30 seconds; warm, from local SSD cache, in double-digit milliseconds; hot, from memory, in single-digit milliseconds. In a separate Zilliz benchmark of 3M rows of 128-dimension vectors on S3, with 256 concurrent readers reading 10-row batches, the Vortex format moved about 135x less data per read than Parquet, dropping 9.44 MB per read down to 0.07 MB.

Zilliz Vector Lakebase is now in public preview. If your current stack splits serving and discovery into separate systems, Vector Lakebase might be worth a look. Try it on Zilliz Cloud — new work email signups get $100 free credits — or talk to us about your use case.

This pattern is bigger than any one product. Object storage is turning into a place where AI data is searched directly, on top of being where it is stored.

Frequently Asked Questions

Is object storage good for AI workloads?

Yes, for storing training data, embeddings, and model artifacts at scale, where its durability, near-unlimited capacity, and low cost pay off. The limit is speed: to retrieve, explore, or analyze that data quickly, you need an index and compute layer on top of the raw store.

What are examples of object storage?

The most widely used cloud object stores are Amazon S3, Google Cloud Storage, and Azure Blob Storage. For self-hosted deployments, MinIO and Ceph are the open-source standards. All five expose data through an HTTP API, and S3's API has become a de facto standard, so applications written against S3 often work against the others with only minor adjustments.

What is the difference between object storage and a vector database?

Object storage is a general store for any unstructured data, reached by ID. A vector database is built for similarity search over embeddings, with indexes tuned for fast nearest-neighbor queries. The two are converging: teams increasingly keep the vectors in object storage and add a search layer above them.

Can you run vector search directly on object storage?

Increasingly, yes. AWS S3 Vectors adds native vector search, and platforms that point at lake tables in place let you search object-storage data without copying it out. The queries run slower than an in-memory index, but the cost is far lower, and the data does not need to move.

Why do data lakes use object storage?

A data lake has to hold huge volumes of structured and unstructured data cheaply and durably, in open formats, reachable by many engines at once. Object storage's flat, ID-addressed pool fits that better than file or block storage.

Is object storage slower than block storage?

For a single operation, usually yes. Block storage gives very low latency for transactional and random-access work. Object storage trades that for throughput, durability, and cost on large, mostly static objects read over HTTP. The gap shrinks for big sequential reads but stays wide for small, latency-sensitive ones.