Consensus Builds Agentic Literature Search Across 400M+ Scholarly Sources with Zilliz Cloud
400M+
Scholarly sources searchable
~45 ms
P99 dense retrieval latency in production
14% higher
precision in search results after adding semantic search
4× larger
vectors for just 2.5× the cluster size, with no latency penalty
1 day → 1 hour
Full re-index of the entire 400M+ vector collection by daily bulk import
"Our job is to make the best research findable for anyone who uses Consensus. Zilliz Cloud gives our research agent fast, high-quality semantic retrieval, directly widening the evidence it can reach."
Christian Salem
About Consensus
Consensus is building the operating system for scientific research: an agent platform used by more than 10 million researchers, students, and clinicians across 12,500+ universities to screen papers, synthesize evidence, and automate the tedious parts of research, so they can get back to real science. Consensus has handled more than 150 million research questions to date and raised $45M to accelerate the next 100 million researchers worldwide.
The platform's foundation is world-class agentic search across 400+ million scholarly sources. Consensus never answers from the AI model's own knowledge; instead, every claim must trace back to a real, retrieved paper, and the research agent may call retrieval many times to answer a single question. The product is only ever as good as the papers it finds, which makes retrieval the heart of Consensus. Zilliz Cloud powers one of its most important components: the semantic search that understands what a researcher is truly asking and surfaces the research that answers it.
The Challenge
Consensus's retrieval engine previously ran two retrieval methods: sparse and BM25 keyword search. They handled most queries well. But the bar was rising: Consensus was moving its literature review to an agentic architecture, where one question can turn into many retrieval calls, so the retrieval had to find the right papers and return them fast, on every call. Keyword and sparse search miss papers that say the same thing but with different terminology. Researchers hit this constantly because the paper that answers a question is often from a neighboring field that uses its own terms for the same idea. Dense-vector search is key to closing that gap, so the team first tested it in Elasticsearch and ran into three more problems.
- Retrieval quality dropped at hundreds of millions of vectors. To fit that many dense vectors, the team had to compress them with binary quantization, and the compression cost measurable ranking quality. For a research product, that is the wrong trade.
- Re-indexing the full collection took more than 24 hours. New research waited a day or more to become searchable, and every candidate embedding model cost a full day to evaluate in production, which slowed the team's adoption of better models.
- Making vectors bigger meant a much bigger storage bill. At hundreds of millions of vectors, every increase in vector size or coverage is paid across the whole collection, so the team kept its vectors smaller than it wanted.
The team went looking for a purpose-built engine for semantic retrieval: high ranking quality, fast collection refreshes, workable economics at scale, and a managed production experience it would not have to staff.
Why Zilliz Cloud
Consensus ran a real bake-off across four options: dense vector search within Elasticsearch, which it had already run; FAISS, a self-run open-source vector library it had prototyped; Pinecone; and Zilliz Cloud. Zilliz Cloud won on four counts.
- Better results without compressing the vectors. In the team's own tests, Zilliz Cloud delivered more precise retrieval results at hundred-million scale while keeping recall high, without the compression that had cost quality before.
- A full collection rebuild in about an hour, down from more than a day. A daily bulk import of the entire collection turns a full rebuild from a special project into a nightly job. This also unlocks much faster iteration when testing new embedding models.
- Lower storage cost for the same traffic and vector count. The savings enabled vectors 4 times the size and at up to 4x less cost, resulting in noticeably higher relevance.
- A managed service, with a developer experience the team liked. Consensus prototyped on FAISS and found it technically capable, but running it in production would have meant taking on orchestration and operational overhead the team had no interest in owning. The team also found the Zilliz Cloud SDK well documented and pleasant to build against, with a straightforward console for day-to-day collection operations.
The Solution
Consensus answers a research question by retrieving the right papers or portions of papers and synthesizing an answer from them, with every claim cited back to a real publication. Three retrieval paths run in parallel to find those papers, an architecture the team calls tri-brid search, and Zilliz Cloud powers the semantic search layer: matching a question to a paper by what it means rather than the words it uses, and finding the papers the other paths miss. It runs on Google Cloud alongside the rest of the stack.
At query time, the agent plans the search and calls retrieval as a tool. The paths run concurrently, their results fuse and rerank into a single evidence set, and the model writes the answer, tying every claim back to a retrieved paper. That is the machinery under the literature-review agent and the Consensus Meter, which weighs the published evidence for and against a claim.
Three design choices make this work.
A live/cold pair of collections in Zilliz Cloud, rebuilt from scratch every day.
The usual way to keep an index this size current is to update it in place: detect what changed, write the new vectors, delete the old ones, and keep the bookkeeping straight. Consensus skips all of that. It keeps two copies of the collection in Zilliz Cloud, 400M+ vectors in total, one serving queries and one idle, and every day it rebuilds the idle copy from scratch by bulk import and switches it live.
That is only a sensible design if a full rebuild is fast enough to run daily and storage is cheap enough to hold a second copy. On the previous system, neither was true. On Zilliz Cloud, both are: the entire collection is imported, indexed, and ready to go in about an hour, at lower storage cost. So the simpler design wins: nothing is ever written into the serving collection, there is no backlog to reconcile, and a new embedding model is just the same rebuild with different vectors. The corpus is always up to date, ensuring researchers always have access to the very latest publications.
"What the team has found with Zilliz is that we can basically re-ingest the entire collection from scratch, since it's so fast. We have two copies of the index; we ingest the entire thing in one hour, and then flip the switch. We can do it basically daily at this point. This has also greatly enabled experimentation since there is no hesitation to try new ideas." — Heath Hohwald, Tech Leader and Search Manager, Consensus
One 1,024-dimension vector per scholarly source.
Each source is embedded from its title and abstract into a single vector. The team started with 256 dimensions, assuming cost and latency would grow in step with vector size. On Zilliz Cloud, quadrupling the dimension from 256 to 1,024 took roughly 2.5× the cluster size, not the 4× the team had expected, with very little added latency. On the team's internal benchmark, the 1,024-dimension embeddings delivered a 27% increase in the quality of papers found over the smaller model, so Consensus shipped 1,024.
Semantic search as a tool the agent calls directly.
Zilliz Cloud is exposed to the literature-review agent as a callable tool rather than hidden behind a single retrieval step. The agent can reformulate the question, search from several angles, and come back for more after reading what it found, so one task can mean many calls. Tool calling like this only works if every call is fast and precise, because a slow or imprecise search gets multiplied by every extra call the agent makes. At ~45 ms P99 across 400M+ vectors, semantic search is fast and precise enough to hand to the agent as a tool, so it can chase down the last hard-to-find papers a single query would miss.
Results & Benefits
- Up to 4× lower storage cost, and the savings went into better search. Storage costs less at the same traffic and vector count, creating the headroom the team then spent on quality.
- 14% higher precision in search results after adding semantic search, measured on Consensus's own internal benchmark. The gain is strongest on papers that answer a question in words the researcher never used, which the previous stack would have missed.
- P99 retrieval at ~45 ms across 400M+ vectors, a third of the 150 ms P90 target the team set for Zilliz Cloud. Fast enough for the research agent to search, reflect, and iterate multiple times in a single request instead of getting one shot and sometimes missing critical results.
- 4x larger vector dimensions (256 → 1,024) cost only 2.5× the cluster on Zilliz Cloud, about 40% less than linear scaling, with no latency penalty. On Consensus's internal benchmark, the larger embeddings delivered a 27% increase in the quality of papers found.
- A full rebuild of the entire collection: 24+ hours → about 1 hour, as a daily bulk import. Consensus now rebuilds and swaps the entire collection daily, with no interruption to serving. New research is searchable the day it lands.
- The iteration loop gets faster. Larger embeddings, new models, and new retrieval strategies no longer have to be weighed against a day-long rebuild before anyone can see whether they help.
What's Next
Every direction Consensus is heading in puts more weight on Zilliz Cloud. Exposing semantic search more broadly through the agentic framework means more retrieval calls per task, and more traffic against the dense collection. New embedding models will keep shipping on the daily cadence the one-hour rebuild makes possible. Metadata filtering, currently handled in the application layer, is a candidate for moving to Zilliz Cloud.
The biggest opportunity is to further leverage the full-text content. Consensus holds licensed full-text content through its publisher partnerships, and bringing that depth into Zilliz’s dense collection is a natural next step: indexing the body of each paper rather than its title and abstract would multiply the collection several times over and give researchers a far finer-grained match.
"Consensus is doing something genuinely hard: making the world's scientific literature searchable by meaning, for an agent that may ask the same corpus a dozen different ways in one task. We're proud that Zilliz Cloud is the retrieval foundation behind it, and we're excited to keep building together as agentic research scales." — James Luan, CTO of Zilliz
Build Agentic AI with Zilliz Cloud
Agentic systems put new pressure on retrieval: one user request can become several semantic searches, reformulations, and evidence-gathering passes. Whether you're scaling RAG or building agentic search, Zilliz Cloud gives you the same retrieval foundation that powers Consensus.
Get started with Zilliz Cloud for free, or talk to our team directly.
"One of the biggest wins we've seen is better handling of queries that are semantically but not textually relevant. Longer, more conversational queries have also drastically improved."
Heath Hohwald