Motivation

Moving large amounts of data between cloud providers or regions is a critical but often time-consuming operation in modern data architectures.

Data engineers frequently face slow transfer speeds and high costs when moving data between cloud storage services, leading to delayed projects and increased expenses.

import boto3
import google.cloud.storage

# Traditional approach - Sequential transfer
s3 = boto3.client('s3')
gcs = google.cloud.storage.Client()

# Download from S3
s3.download_file('source-bucket', 'large_file.parquet', '/tmp/large_file.parquet')

# Upload to GCS
bucket = gcs.bucket('destination-bucket')
blob = bucket.blob('large_file.parquet')
blob.upload_from_filename('/tmp/large_file.parquet')

Introduction to Skyplane

Skyplane is a powerful data transfer tool that accelerates bulk data movement between cloud storage services. It achieves this by provisioning multiple VMs to transfer data in parallel while optimizing for cost and speed.

Installation:

# Install with AWS support
pip install "skyplane[aws]"

# For multiple cloud support
pip install "skyplane[aws,gcp,azure]"

Fast Cloud Transfers

Skyplane provides several advantages:

• 110x faster than AWS DataSync
• 4x more cost-effective than rsync
• Support for major cloud providers (AWS, GCP, Azure, IBM)

Example of a transfer operation:

# Initialize Skyplane
skyplane init

# Transfer data with 2 VMs per region
skyplane cp -n 2 s3://source-bucket/data gs://destination-bucket/data

Conclusion

Skyplane revolutionizes cloud data transfers by leveraging parallel processing and optimized network paths, making it an essential tool for data engineers working with multi-cloud environments.

Link to Skyplane