Amazon DataZone has released a new feature called data lineage in preview, which allows customers to track and visualize the movement of data from its source to consumption. This feature is available to customers who use OpenLineage-enabled systems or APIs. Amazon DataZone is a data management service that helps customers catalog, discover, share, and govern data at scale while maintaining access and governance controls.
The data lineage feature in Amazon DataZone captures and displays the transformations of data assets and columns, giving users a clear view of how data moves through the system. Customers can use Amazon DataZone’s OpenLineage-compatible API to capture and store lineage events that go beyond what is available within the service. This includes tracking transformations made to data assets in Amazon S3, AWS Glue, and other integrated services.
By visualizing data lineage, data consumers can have confidence in the source and quality of a particular data asset. On the other hand, data producers can better understand how changes to an asset will impact its consumption. Amazon DataZone also versions the lineage with each event, allowing users to track and compare transformations over time. This historical view of data lineage is crucial for troubleshooting, auditing, and ensuring the integrity of data assets.
In summary, Amazon DataZone’s new data lineage feature empowers customers to track and visualize the movement of data assets from their source to consumption. The service allows domain administrators and data producers to capture lineage events using OpenLineage-compatible APIs, providing a comprehensive view of data transformations. This feature is essential for improving data quality, troubleshooting issues, and validating the integrity of data assets within an organization.
Article Source
https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-datazone-openlineage-compatible-data-lineage-visualization-preview