Blog
Introducing Migration Services: Efficiently Move Unstructured Data Across Platforms

Introducing Migration Services: Efficiently Move Unstructured Data Across Platforms

Sep 11, 20245 min read

As a leading vector database service provider, we at Zilliz understand that developing exceptional AI applications relies on the data itself. However, to process unstructured data efficiently for AI apps, we've identified several critical challenges:

Data Fragmentation: User data is scattered across multiple platforms, such as S3, HDFS, Kafka, data warehouses, and data lakes.
Data Format Heterogeneity: Unstructured data exists in various formats, including JSON, CSV, Parquet, JPEG, and more.
Lack of Complete Solutions: No existing product fully addresses the complex requirements for efficient unstructured and vector data transfer across systems.

Among those, efficiently importing and transforming unstructured data from various sources and formats into vector databases presents unique challenges. This process is significantly more complex than handling traditional SQL-based relational data, a fact many companies initially underestimate.

As a result, organizations building custom data pipelines for unstructured data often struggle with performance, scalability, and maintainability. These issues can compromise data quality and accuracy, potentially undermining the insights they seek to gain.

Even worse, many companies overlook crucial factors like vendor lock-in and data disaster recovery when selecting vector databases. This oversight, stemming from a lack of awareness or underestimation, can lead to significant complications. Vendor lock-in, in particular, deserves special attention.

The Impact of Vendor Lock-in

Vendor lock-in occurs when an organization becomes overly dependent on a single vendor's proprietary technology, making it difficult or costly to switch to another solution. This issue is especially pertinent in vector databases because the nature of vector data and the lack of standardized formats can make data migration between systems extremely challenging.

The impact of vendor lock-in can be far-reaching. It limits an organization's flexibility to adapt to changing business needs, potentially increases costs over time, and may constrain innovation by tying the company to a single vendor's ecosystem. Moreover, it can lead to performance limitations if the chosen solution doesn't scale well with the organization's growing needs.

When selecting vector database solutions, organizations should prioritize open standards and interoperability to mitigate these risks. Developing a clear data governance strategy that includes plans for data portability is also crucial. Regularly assessing dependencies on vendor-specific features can help maintain flexibility.

The Challenges of Unstructured Data Migration

However, even with these precautions in place, organizations must be prepared for the unique challenges of vector databases. We've found that data migration between vector databases is far more complex than traditional relational databases. This complexity underscores the importance of choosing the right solution and highlights why avoiding vendor lock-in is critical. The main challenges in vector database migration include:

Lack of vector-oriented ETL tools: Popular tools like Airbyte and Seatunnel, while effective for relational databases, struggle with vector database processes.
Vector Database Capability Gaps:
- Many vector databases lack full data export support
- Poor real-time capability for incremental data
- Data schema mismatches

When addressing these challenges, organizations can build more resilient, flexible, and future-proof AI applications, truly harnessing the power of their unstructured data while maintaining the agility to adapt to future technological advancements.

Introducing Migration Services

Zilliz has developed and open-sourced the Migration Services to address the abovementioned challenges, a service based on Apache Seatunnel for vector data. Several factors drove our decision to build Migration Services:

Meeting the Growing Data Migration Needs: Migration Services evolves from our Milvus Migration Service, which has successfully helped over 100 organizations migrate data between Milvus clusters. User demands have grown to include migrations from different vector databases, traditional search engines like Elasticsearch and Solr, relational databases, data warehouses, document databases, and even S3 and data lakes to Milvus.
Supporting Real-time Data Streaming and Offline Import: As vector database capabilities expand, users require both real-time data streaming and offline batch import options.
Simplifying Unstructured Data Transformation: Unlike traditional ETL, transforming unstructured data requires AI and model capabilities. Migration Services, in conjunction with the Zilliz Cloud Pipelines, enables vector embedding, tagging, and complex transformations, significantly reducing data cleaning costs and operational complexity.
Ensuring End-to-End Data Quality: Data integration and synchronization processes are prone to data loss and inconsistencies. Migration Services addresses these critical data quality concerns with robust monitoring and alerting mechanisms.

Core Capabilities of Migration Services

Built on top of Apache Seatunnel, Migration Services offers:

Rich, extensible connectors
Unified stream and batch processing for real-time synchronization and offline batch imports
Distributed snapshot support for data consistency
High performance, low latency, and scalability
Real-time monitoring and visual management

Figure- How do Migration Services work?

Figure 1: How do Migration Services work?

Additionally, Migration Services introduces vector-specific capabilities such as multiple data source support, schema matching, and basic data validation. Future roadmaps include incremental synchronization, full plus incremental modes, and more advanced data transformation capabilities.

Why Open Source Migration Services?

At Zilliz, we believe in the power of open source to drive innovation and deliver the best solutions to developers. Here's why we've chosen to open source our Migration Services:

Fostering an Open Vector Data Ecosystem: We're building an ecosystem free from vendor lock-in, allowing you to choose and switch between solutions as needed.
Attracting Contributors: We can make our tools more versatile and robust by leveraging the developer community's collective expertise. We invite you to add connectors, sources, and transform codes.
Giving Back to the Open Source Community: As an open-source vector database company, we're committed to sharing knowledge and resources to advance the entire field.
Enhancing Cloud Service Offerings: Your feedback is crucial for faster iteration and improvement of our commercial products. Open sourcing allows us to gain valuable community input.

Our commitment to openness goes beyond just sharing code. In an open ecosystem, we understand that developers have choices. This drives us to strive for excellence, ensuring that choosing Zilliz is always the best decision for your needs. Whether through rapid iteration, comprehensive support, or expanding capabilities, our goal is to earn your trust and business every day by consistently delivering value.

Migration Services Roadmap

Looking ahead, Migration Services will continue to evolve. By open-sourcing this tool, we're not just addressing current challenges in vector data management; we're paving the way for a more innovative future in AI application development.

Figure 2- Migration Services Roadmap

Figure 2: Migration Services Roadmap

Our vision is to create tools that serve developers' needs, not the other way around. We're working towards a future where data and AI technologies are more accessible, adaptable, and aligned with real-world development challenges. We invite the community to join us in this journey, contributing to and benefiting from this powerful tool for unstructured data processing. Together, we can shape the future of vector databases and create a more open, efficient, and innovative ecosystem for AI development.

James Luan
James Luan is the VP of Engineering at Zilliz. With a master's degree in computer engineering from Cornell University, he has extensive experience as a Database Engineer at Oracle, Hedvig, and Alibaba Cloud. James played a crucial role in developing HBase, Alibaba Cloud's open-source database, and Lindorm, a self-developed NoSQL database. He is also a respected member of the Technical Advisory Committee of LF AI & Data Foundation, contributing his expertise to shaping the future of AI and data technologies.