New for Zilliz Cloud: Migration Service, Fivetran Connector, Multi-replica, and More
We're excited to announce new features in Zilliz Cloud, designed to enhance support for running AI workloads in production environments. This update equips developers and technical leaders with advanced tools to efficiently deploy and scale AI-driven applications.
The new features enable technical teams to:
Maintain full ownership of their unstructured data while leveraging best-of-breed technologies
Support the speed and scale requirements of large-scale production GenAI applications
Ensure robust and secure operation of GenAI workloads in production environments
As organizations increasingly deploy AI-driven applications, managing large volumes of unstructured data, ensuring high performance at scale, and maintaining operational robustness become critical challenges. Our new features address these issues, allowing development teams to focus on innovation rather than infrastructure management.
Here’s an overview of our new features–read on for more details:
- Migration Service
- Fivetran source connector
- Multi-replica
- Auto-scaling
- Other Production Readiness updates
Join us in our Launch webinar to see these innovations in action.
An overview of the Zilliz Sep 2024 Launch
Figure 1: An overview of the Zilliz Sep 2024 Launch
Migration Service: Secure and Efficient Vector Data Migration
Zilliz Cloud introduces Migration Services for vector data, a newly released feature that addresses critical challenges developers face when working with vector databases in AI applications. As projects evolve, the need to transfer vector data between databases and other data systems becomes crucial for several reasons:
Avoiding vendor lock-in
Implementing robust data backup and recovery strategies
Ensuring interoperability between different systems
However, this process is often more complex than anticipated, risking loss of information or functionality during transfer.
Migration Services simplifies these challenges by offering the following:
Lossless migration between vector databases (e.g., Milvus, pgvector, Elastic)
Support for both bulk import and incremental data migration
Built-in data validation to ensure consistency and preserve functionality
High-performance operations designed to handle large-scale vector datasets
With Migration Services, developers can maintain complete control over their data across different platforms and scenarios. By streamlining the migration process, Migration Services allows you to focus on building and improving your AI models and applications while ensuring data portability, security, and integrity.
Fivetran Connector: Streamlined Unstructured Data Integration from 500+ Sources
Zilliz Cloud now integrates with Fivetran, significantly expanding the data ingestion capabilities. This new connector enables efficient retrieval of unstructured data from over 500 different systems, broadening the scope of data accessible for your AI applications.
Here's how it works:
Unstructured data from 500+ sources (like Snowflake, MongoDB, Salesforce, etc.) flows into Fivetran.
Fivetran's connectors and transformation logic process the data.
The Milvus source connector (built with Fivetran SDK) channels the processed data into Zilliz/Milvus.
The unstructured data can be vectorized using OpenAI Embedding Services through OpenAI’s Fivetran connector.
Finally, the vector data is ingested into Zilliz Cloud/ Milvus, ready for vector search and AI applications.
How the Fivetran Source Connector Work
Figure 2: How the Fivetran Source Connector Work
This integration streamlines the entire process of gathering, transforming, and vectorizing unstructured data from various sources. It eliminates the need for custom ETL pipelines, saving developers time and effort while ensuring a smooth data flow into Milvus/ Zilliz.
By leveraging this Fivetran connector, you can easily build your AI applications with a vast array of unstructured data, opening up new possibilities for semantic search and AI-driven insights.
Multi-replica: Improving Performance and Availability
Newly available in public preview, the multi-replica functionality bolsters Zilliz Cloud's capabilities for production environments. This feature enables cluster-level replication, significantly improving query performance and system availability.
Many consumer AI applications face query-per-second (QPS) bottlenecks in production due to their high user volume. Multi-replica addresses this by distributing query workloads across replicas, enabling parallel processing that can linearly improve overall QPS as replicas are added. This results in reduced latency and improved efficiency, essential for maintaining responsiveness in high-traffic production environments.
Additionally, multi-replica enhances system reliability by distributing replicas across multiple Availability Zones (AZs). This architecture ensures continuous data access even during an AZ outage, providing the high availability necessary for mission-critical applications. While multi-replica doesn't increase overall cluster capacity, it offers a powerful tool for optimizing query performance and ensuring the system resilience required in production environments, allowing teams to scale their AI applications confidently.
Auto-scaling: Dynamic Capacity Management
Zilliz Cloud is rolling out an auto-scaling (in private preview) feature that addresses a common challenge in production environments: managing cluster capacity in response to fluctuating demands. This feature dynamically adjusts when the cluster's capacity shrinks based on real-time usage, helping to prevent write access restrictions due to insufficient resources and reduce operational burdens.
Auto-scaling is triggered when the cluster's Compute Unit (CU) capacity exceeds a specified threshold (default 70% and adjustable) for two consecutive minutes. Developers can set a maximum CU size for scaling, providing control over resource allocation and costs.
Note: Auto-scaling is currently in private preview and available only for Dedicated (Enterprise) clusters. Organizations interested in trying this feature should contact Zilliz support to enable access.
Other Production Readiness Updates
Zilliz Cloud introduces a few more critical updates to bolster production readiness:
Reliability: 99.95% uptime SLAs for mission-critical AI applications.
Monitoring and Alerting:
39 organizational and project alerts for cost and performance management.
18 key metrics on resource usage, QPS, request results, and data operations.
Customizable charts for in-depth system health analysis.
- Security: Auth0-based SSO (Private Preview) with Google, Github, SSO, and Email login options.
- A New Zilliz Cloud Region Online: Zilliz Cloud has just expanded to a new region—AWS Tokyo (ap-northeast-1)! This means lower latency, better performance, and greater data sovereignty for our users in Asia-Pacific and neighboring regions.
These features provide developers with tools to maintain high-performance, secure, and cost-effective AI applications in production.
Empowering Production-Ready AI Applications with Zilliz Cloud
Zilliz Cloud's latest features are designed to enhance your ability to build and maintain robust AI applications in production environments. From secure data migration to dynamic scaling and comprehensive monitoring, these updates address critical challenges developers and technical leaders face.
Want to learn more? Join our upcoming webinar to learn how these new features can transform your unstructured data management and AI applications. Our product team will discuss the latest developments and best practices for leveraging Zilliz Cloud in production.
Ready to get started? Begin your journey with Zilliz Cloud for free - no installation and no credit card needed. You can also start a 30-day free trial of our dedicated offering, which includes $100 worth of credits upon registration. You'll receive an extra $100 credit if you subscribe Zilliz Cloud through any cloud marketplace!
If you have any questions or need assistance as you explore Zilliz Cloud, our support team is ready to help. Reach out through our support portal, and we'll be glad to help you leverage these new features for your AI applications.
Start building production-ready AI applications with Zilliz Cloud today!
Steffi LiSteffi is the Director of Product Marketing at Zilliz, with prior experience leading the GTM strategies for open-source data technologies such as Apache Kafka and Apache Airflow. Passionate about travel, yoga, and non-fiction reading, she brings a balanced and enriched worldview to her professional pursuits.
- Migration Service: Secure and Efficient Vector Data Migration
- Fivetran Connector: Streamlined Unstructured Data Integration from 500+ Sources
- Multi-replica: Improving Performance and Availability
- Auto-scaling: Dynamic Capacity Management
- Other Production Readiness Updates
- Empowering Production-Ready AI Applications with Zilliz Cloud
Content
Start Free, Scale Easily
Try the fully-managed vector database built for your GenAI applications.
Try Zilliz Cloud for Free