London Stock Exchange Group (LSEG) recently faced the challenge of migrating their 30 PB of Tick History-PCAP data from Wasabi cloud storage. With an additional 60 TB generated daily, LSEG wanted a solution to optimize storage costs, archive older datasets, and provide easier access to market data for their customers. To address this, LSEG turned to AWS and DataArt for an automated migration strategy that would leverage AWS DataSync.
The migration process involved transferring the data to Amazon Simple Storage Service (S3) using AWS DataSync within a timeframe of three months. LSEG chose to store active datasets in Amazon S3 Intelligent-Tiering for cost savings based on future data access patterns. Older, less accessed data was moved directly to Amazon S3 Glacier Deep Archive, resulting in an 80% reduction in data storage costs for LSEG. This move also allowed LSEG’s customers to access and analyze the data through AWS Data Exchange, enabling them to conduct exploratory analytics more efficiently.
The migration strategy was designed with components like DataSync, AWS Step Functions, AWS Lambda, and Amazon DynamoDB to automate the data transfer and validation process. The architecture of the migration solution was set up in an Amazon Virtual Private Cloud (VPC) across multiple Availability Zones (AZs) with DataSync agents deployed as Amazon EC2 instances to ensure optimal throughput for the data transfer.
To manage the migration workflow effectively, the infrastructure was deployed in a separate account from the destination S3 buckets, and multiple DataSync tasks were pre-generated for each migration session. The migration process was divided into smaller phases, each consisting of metadata collection, task generation, data migration, and post-migration data validation.
Data integrity was crucial throughout the migration process, and a comprehensive post-migration validation process was implemented to ensure the accuracy of every object transferred. Monitoring and alerting mechanisms were in place using Amazon CloudWatch to track the status of the migration components in real time.
Key considerations for a large-scale migration like LSEG’s include planning a migration strategy, efficient scaling, and considering AWS DataSync limitations. To optimize the migration process, it’s essential to address API throttling, data validation strategies, and storage class specifics when moving data to Amazon S3.
In conclusion, LSEG successfully migrated their Tick History-PCAP data to Amazon S3, achieving significant cost savings and improved accessibility for their customers. By leveraging AWS DataSync and other AWS services, LSEG streamlined the migration process, enhanced data availability, and simplified entitlements management for their customers. The implementation of a phased approach and orchestrated validation ensured a successful migration within the specified timeframe, showcasing the benefits of a well-planned and automated data migration strategy.
Article Source
https://aws.amazon.com/blogs/storage/how-london-stock-exchange-group-migrated-30-pb-of-market-data-using-aws-datasync/