Enhancing APK Security with Milvus at TrendMicro
<95 milliseconds
end-to-end query latency
<10 seconds
to ingest 3 million vectors
Enable
real-time threat detection
Milvus delivers unparalleled performance and flexibility, integrating seamlessly with leading vector index libraries like Faiss. Its intuitive API and robust solutions for high availability make it an indispensable tool in our APK security efforts.
Wei Huang
About Trend Micro
Trend Micro is a global security software industry leader with a long-standing reputation for delivering cutting-edge solutions to software vendors and individual users. As cybersecurity threats evolve, so does TrendMicro's commitment to innovation.
One of the company's key focus areas is mobile security, which has become increasingly critical in our digitally connected world. Within this context, a specialized team at TrendMicro is responsible for building mobile security apps and developing infrastructure. Their workflow involves crawling external APKs (Android application packages) from platforms like Google Play and applying TrendMicro's proprietary algorithms to detect APKs carrying viruses.
Hitting the Scalability Wall: The Limits of MySQL and Faiss
In the early stages of the project, TrendMicro relied on MySQL for APK similarity search. As a relational database, MySQL was initially sufficient for handling the APK sample size and allowed the team to use SQL queries for similarity searches. However, as the dataset grew into the tens of millions, with daily increments reaching hundreds of thousands, MySQL's performance started to degrade. Query latency increased, and the database needed help to handle the high volume of concurrent searches, leading to bottlenecks in the system.
The team then turned to Faiss, a specialized library for similarity search released by Facebook in 2017. Faiss is known for quickly retrieving similar vectors and offers multiple indexing options like IndexFlatL2, IndexFlatIP, HNSW, and IVF. While Faiss excelled in speed, it was essentially a basic algorithm library, lacking several critical features for a production environment. For instance, Faiss did not offer any data management capabilities, meaning the team would have to build a separate layer for data storage and retrieval. It also lacked high availability features and monitoring tools, making it unsuitable for a mission-critical application like APK security.
Moreover, Faiss was not designed to be a distributed system, which posed a challenge for horizontal scalability. While some industry solutions used Faiss as an underlying library for Elasticsearch plugins, these were not without problems. They were memory-intensive and required significant fine-tuning to optimize performance, which needed to be more practical for TrendMicro's rapidly growing dataset.
The Milvus Breakthrough: A Robust Vector Search Engine for Scalable APK Analysis
After encountering limitations with MySQL and Faiss, TrendMicro's search for a robust and scalable solution led them to Milvus. Developed in C++, Milvus emerged as a comprehensive vector search engine that addressed many shortcomings of previous solutions. One of the standout features was its integration with mainstream vector index libraries like Faiss, NMSLIB, and Annoy. These integrations allowed TrendMicro to leverage the speed of Faiss while benefiting from the additional features Milvus offered.
| Engine | Performance (ms) | Dataset Size (million) | Dimensions |
|---|---|---|---|
| ES | 600 | 1 | 128 |
| ES + Alibaba Cloud | 900 | 20 | 128 |
| Milvus | 27 | 1000+ | 128 |
| SPTAG | Not good | ||
| ES + nmslib, faiss | 90 | 150 | 128 |
Milvus provided a simple and intuitive API, a significant advantage for the development team. The API allowed them to choose different indexing types based on their specific use cases, offering flexibility that was missing in other solutions. This flexibility benefited TrendMicro, as they had to deal with various APK feature vectors and needed a system that could adapt to their diverse requirements.
Another strong point was Milvus's focus on high availability and distributed systems. Unlike Faiss, which was not designed to be a distributed system, Milvus is a mature solution for scaling horizontally. Scalability and performance were crucial for TrendMicro, as their dataset was large and growing rapidly. Milvus's distributed architecture meant they could easily add more nodes to the system to handle increased loads, thereby future-proofing their APK analysis infrastructure.
Monitoring was another area where Milvus excelled. It came with built-in support for Prometheus, a leading open-source monitoring solution that works with Grafana for advanced data visualization. This monitoring capability allowed TrendMicro to closely monitor various performance metrics, including query latency and data import speeds, enabling proactive issue resolution.
Milvus offered a robust, flexible, and scalable solution well-suited for TrendMicro's complex and growing needs. Its integration capabilities, intuitive API, and focus on high availability and monitoring made it a breakthrough choice for TrendMicro's APK security project.
Real-World Impact: Low Latency and High Data Import Speed
Implementing Milvus in TrendMicro's ThashSearch service has yielded tangible results that have significantly improved the efficiency and effectiveness of their APK analysis. One of the most critical metrics for any search service is latency; in this regard, Milvus has been a game-changer. The ThashSearch service has been live for several months and has consistently achieved an average query latency of under 95 milliseconds. This low latency ensures that TrendMicro can provide timely alerts to corporate and individual users about potentially harmful APKs, enhancing overall security posture.
But low latency is just one part of the equation. In a data-intensive environment like APK analysis, the speed at which developers can ingest new data into the system is equally important. Milvus has excelled in this aspect as well. The system has demonstrated an impressive data import speed, capable of ingesting 3 million 192-dimensional vector data in approximately 10 seconds. Given the daily increments of hundreds of thousands of new APK samples, this rapid data import capability is crucial for TrendMicro. It ensures the database is always up-to-date, allowing for the most accurate and current similarity searches.
The combination of low latency and high data import speed has a synergized effect on TrendMicro's operations. It has improved the user experience by providing faster search results and streamlined the backend processes, making it easier to keep the system updated with the latest data. This operational excellence has enabled TrendMicro to meet and exceed the initial design goals for the ThashSearch service, affirming the decision to implement Milvus as the vector search engine of choice.
The Future Plan
As TrendMicro looks to the future, they are keenly interested in the roadmap of Milvus and how its upcoming features can further optimize its APK analysis operations. One such feature is the introduction of string-type IDs in Milvus. TrendMicro plans to utilize this feature to simplify its current architecture by eliminating the need for Redis caching. String-type IDs will streamline the data retrieval process, making it more efficient and reducing complexity.
Moreover, TrendMicro is excited about Milvus' plans to evolve into a fully distributed system. Currently, Milvus supports only one write node, but future versions aim to remove this limitation. TrendMicro sees this as an opportunity to scale its operations horizontally, enhancing its ability to handle even larger datasets and higher query volumes. Given the success they've experienced with Milvus in their ThashSearch service, TrendMicro is also considering expanding the footprint of Milvus within their organization. They are exploring the possibility of integrating Milvus into other projects and workflows that could benefit from efficient vector similarity search, thereby maximizing their return on investment in the technology.