Cốc Cốc Powers Vietnam's National-Scale AI Search with Milvus
700M
Vectors in production, scaling toward 1.5B
19.8 ms
P90 semantic search latency at peak traffic
32.1 ms
P90 hybrid search latency at peak traffic
99–100%
Recall in production with HNSW
We answer in under 20 milliseconds across hundreds of millions of vectors, and every AI product we've built — search, advertising, RAG — runs on the same Milvus deployment. That's what lets us keep shipping without adding infrastructure.
Nam Doan Ngoc Giang
About Cốc Cốc
Cốc Cốc is Vietnam's leading homegrown search engine and browser. More than 30 million people use it — roughly one in three Vietnamese internet users — running over 600 million searches a month, which makes Cốc Cốc the number-two search engine and number-two browser in the country, behind only Google. Its edge is what global engines find hardest to copy: Vietnamese itself, down to the language models Cốc Cốc's own team fine-tuned for it.
Since 2023, Cốc Cốc has been building AI into that experience with AI Chat and AI Search, and extending the same intelligence to Cốc Cốc Ads. That shift changed the problem underneath: results now have to be retrieved by meaning rather than just by keyword, across hundreds of millions of items, in milliseconds. Milvus is the semantic layer Cốc Cốc chose to do it — today holding 700 million vectors across five of their production systems.
The Challenge
Bringing semantic retrieval to a national-scale search engine and ad platform meant solving three problems at once.
- Keyword search can't answer the queries users actually type. New queries and trends appear every day — a phone that launched this morning, a news event from an hour ago, a Vietnamese phrasing no web page uses. Keyword search only matches the user's actual words, so for these it returns little of value.
- Vector libraries can't hold hundreds of millions of vectors. FAISS and USearch worked well on Cốc Cốc's smaller-scale projects, but they give you an index, not a system — nothing that spreads hundreds of millions of vectors across machines, keeps them queryable while new data is being written, and runs reliably in production. Whatever replaced them also had to run on Cốc Cốc's own servers, so search and advertising data stayed inside their infrastructure.
- Ad matching requires two retrieval paths, but the latency budget supports only one. Advertisers bidding on a shopper's exact words must be matched, but a shopper describing the same thing differently should still see them. Running both and fusing them into a single ranked list — at Cốc Cốc's highest-volume vector workload, inside a shared page-load budget — is where it gets hard.
Why Milvus
Cốc Cốc validated Milvus with a proof of concept on roughly 30 million query embeddings before committing. The results exceeded the team's expectations, and Milvus has been their vector database ever since. Five things kept it in place as the workload grew.
- Low latency that held as the dataset grew. This was the primary criterion. Milvus delivered consistently fast vector search as Cốc Cốc's corpus scaled from the 30 million vectors of the proof of concept to 700 million in production, and it kept holding when the team expanded into hybrid search for the advertising platform — a heavier query pattern at higher volume.
- Self-hosted, so data and infrastructure stay with Cốc Cốc. Milvus is open source and runs in Cốc Cốc's own environment. User data never leaves their infrastructure, and the team controls deployment topology, index configuration, resource allocation, and upgrade timing themselves.
- Native hybrid search, rather than a second system to run. Milvus generates sparse vectors with a built-in BM25 function and combines them with dense ANN results inside a single query. Cốc Cốc's ads engine gets lexical and semantic retrieval from one database, with one operational surface — instead of running a separate keyword search cluster and stitching two result sets together in application code.
- A full index toolbox, so each workload gets the right trade-off. Cốc Cốc's workloads don't want the same thing. Online search wants the lowest latency at high recall; offline batch pipelines want exact results and don't care how long they take. Milvus supports HNSW, the IVF family, FLAT, and many more on the same deployment, so the team chose per workload rather than compromising across all of them.
- Straightforward to deploy, operate, and learn. Milvus was simple to stand up, manage, and scale in production, which kept operational overhead low for the engineering team. Milvus Distributed's modular architecture, with detailed dashboard panels for each component, means the team can see exactly which part of the system is under pressure and size hardware to the real bottleneck. And the documentation was comprehensive and easy to follow, which accelerated the initial implementation and simplified integration with Cốc Cốc's existing systems.
The Solution
Milvus sits at the very top of Cốc Cốc's pipelines: it is the retrieval step, ahead of filtering and re-ranking. Cốc Cốc's core web search relevance follows a retrieve-and-rerank architecture, with Milvus handling first-stage recall and downstream models refining the results.
Five production systems run on that one Milvus deployment — each with its own collection, its own definition of what a vector means, and its own index configuration, but no separate infrastructure:
- Core web search relevance — semantic retrieval that surfaces relevant results for unseen and long-tail queries.
- AI-powered Search & Shopping Ads — semantic and hybrid retrieval that matches user intent to relevant advertisements.
- Related search suggestions — vector similarity surfacing contextually relevant follow-up queries.
- Lookalike targeting — browsing history embedded as vectors, used to find users with similar interests for display advertising.
- An internal RAG chatbot — Milvus as the retrieval layer grounding LLM responses in Cốc Cốc's own documents.
What a vector represents depends on the system: a web page, a search query, an advertisement, a user profile, or a text passage. Most embeddings come from bi-encoder models the team fine-tuned from PhoBERT, optimized for Vietnamese language understanding — the domain expertise that makes Cốc Cốc's results Vietnamese rather than merely translated.
To meet production latency and memory requirements, Cốc Cốc applies model optimization to reduce its production embeddings to 128 dimensions, while maintaining retrieval quality. Narrower vectors mean less memory per vector and faster distance computation, which helps keep memory per node manageable at this scale. That gave the team a compact, high-quality vector to build on. The next question was which index to build on it.
Milvus supports multiple index types with a different index for every workload
Cốc Cốc's workloads pull in opposite directions. User-facing search wants the lowest possible latency at high recall; offline batch pipelines want exact results and don't care how long they take. Milvus supports HNSW, the IVF family, FLAT, and more on the same deployment — so instead of picking one index and living with it everywhere, the team could benchmark the options against their own data and then run different indexes side by side.
For online services, they compared the candidates on recall, index size, and performance:
- IVF-SQ8 — a small memory footprint, but lower recall and higher latency.
- IVF-PQ — an even smaller footprint than IVF-SQ8, with lower recall again.
- HNSW — the best recall, at 99%–100%, and the lowest latency, at a higher memory cost.
Because latency is the binding constraint for user-facing services and memory is not currently the limiting factor on their deployment, they chose HNSW and accepted the memory trade. For offline batch workloads — daily inference and data processing pipelines — they use the FLAT index instead, achieving exact nearest-neighbor search with 100% recall, so downstream processing works from ground truth. Neither workload settles for the other's compromise, and neither needs its own database.
Hybrid retrieval for ad matching, built into Milvus
Matching ads on both exact wording and intent usually means running a keyword search cluster alongside a vector database and stitching the two result sets together in application code. Milvus removes that split: lexical and semantic retrieval are native to the same query.
For each user query in Cốc Cốc's AI-powered Search Advertisement system, two retrieval paths run in parallel:
- An ANN semantic search over the dense embedding field, using the HNSW index.
- A full-text search over a sparse vector field that Milvus generates itself, through its built-in BM25 function — so there is no second lexical index for the team to build, run, or keep in sync.
Milvus then fuses the two result sets using WeightedRanker, its built-in fusion primitive, with weights controlled by Cốc Cốc. The team settled on 0.6 toward dense and 0.4 toward sparse — leaning slightly on semantic understanding while keeping real weight on exact keyword matching. That produces higher-quality ad retrieval: the dense side catches intent the advertiser never phrased, and the sparse side protects the exact-match cases that matter commercially. When the balance needs to shift, it's a single tunable number rather than a re-architecture.
Bringing Milvus hybrid search inside the latency budget
When Cốc Cốc first routed all production traffic through hybrid search, at a peak of roughly 166 requests per second, hybrid retrieval latency reached about 120 ms at P90 — above their target.
The team's answer was architectural rather than a retreat on quality. They introduced a 24-hour application-level TTL cache in front of the retrieval layer. At a cache hit rate of roughly 60%, the effective load on Milvus during peak traffic dropped to about 67 RPS, and latency returned to within production requirements. Nothing about the retrieval itself changed: Milvus kept returning the same hybrid results at the same quality, and Cốc Cốc simply changed how much traffic it had to handle.
Keeping latency stable under continuous ingestion
Cốc Cốc's collections are not static; inserts, upserts, index building, and compaction all run against a live system. Milvus handles these as background operations, so collections stay searchable while they are being written to and reindexed. To make latency not just available but predictable, the team schedules that background work into an off-peak window — 2 to 4 A.M. — so maintenance and user traffic never contend for the same resources.
Results & Benefits
- 700 million vectors in production, with headroom planned to ~1.5 billion. Cốc Cốc's deployment currently holds roughly 700 million vectors across multiple production use cases, and the team has designed its infrastructure to scale comfortably beyond 1 billion — with capacity planned for approximately 1.5 billion — without re-architecting the retrieval layer.
- P90 semantic search latency of 19.8 ms at peak. For normal semantic search at peak traffic (34 RPS), Cốc Cốc holds P50 of 11.1 ms, P90 of 19.8 ms, P95 of 26.7 ms, and P99 of 88.3 ms.
- Hybrid search P90 cut to 32.1 ms, bringing a latency-blocked retrieval-quality upgrade into production. At peak traffic (166 RPS, ~67 RPS effective at a 60% cache hit rate), Cốc Cốc holds P50 of 17.1 ms, P90 of 32.1 ms, P95 of 41.6 ms, and P99 of 126 ms.
- 99–100% recall online, 100% recall offline. HNSW provides user-facing services with near-exhaustive recall at the lowest latency among the indexes tested, while FLAT provides offline pipelines with exact nearest-neighbor results, so batch outputs are built on ground truth rather than approximations.
- 30% of search requests are now served by Milvus-powered semantic retrieval - broadening AI Search coverage (up from 63% to 92%) with better results than keyword search alone.
- Vector search over hundreds of millions of vectors at 166 RPS peak is now a workload Cốc Cốc's team can build against — a class of application, under strict low-latency and high-throughput constraints, that was out of reach on their previous stack.
The strategic payoff is that retrieval is no longer the constraint on what Cốc Cốc can ship. Five distinct systems — web search, ads, suggestions, targeting, and internal RAG — now run on a single retrieval foundation that the team controls end-to-end.
Cốc Cốc's advice for teams building large-scale AI search
The Cốc Cốc team has run the full curve from proof of concept to national-scale production. Their advice for peers heading down the same path:
- Start small, but start in production. Build a proof of concept on a few million vectors — say, three months of data — and put that small-scale service in front of real production traffic before scaling up to one or two years of data. Monitor CPU and RAM utilization continuously as you grow, and scale hardware to what you actually observe.
- Benchmark index types and configurations against your own workload. Don't take a default. Evaluate different index types and different configurations within each type, then run load tests and measure P50, P90, and P95 latency. That's how you find the configuration that fits your workload, and the answer is rarely the same for online and offline paths.
- Give the vector database its own hardware. Invest in dedicated server nodes for the distributed vector database, on modern hardware. Sharing infrastructure with other services creates operational problems that worsen as you grow; dedicated nodes provide a more scalable and maintainable foundation.
What's next
Cốc Cốc's Milvus deployment currently runs version 2.5.21, and the team's next step is to move to a newer release to evaluate the improvements — particularly in search latency and hardware efficiency — and to try out new features and configurations. Beyond the upgrade, the team continues to look for opportunities to apply vector search across both existing and upcoming Cốc Cốc products.
Get started with Milvus
Milvus is the world's most widely adopted open-source vector database, purpose-built for vector search at massive scale — with native hybrid search, a full range of index types, and a distributed architecture that runs anywhere from a laptop to a production Kubernetes cluster. It has passed 45,000 GitHub stars and 100 million Docker pulls, and supports more than 10,000 enterprises and AI-native companies worldwide. The latest release, Milvus 3.0, adds a lake-native architecture that indexes and retrieves data directly from object storage.
Whether you're building AI search, ad retrieval, or RAG, Milvus gives you the same retrieval foundation that powers Cốc Cốc. Start on Milvus GitHub, read the Milvus docs, or join the community on Discord.
Zilliz Cloud is a fully managed Vector Lakebase built by the team behind Milvus. Fully compatible with the Milvus API, it delivers up to 10× better price-performance with its proprietary Cardinal indexing engine, plus enterprise-grade security, reliability, scalability, and up to a 99.99% SLA.