Netflix Bridges AWS IAM and Internal Identity with Workload Attestation

Netflix has developed a workload attestation mechanism that allows Apache Spark jobs running on Amazon EMR to exchange their AWS execution identity for a short-lived internal workload identity. The approach bridges AWS IAM with Netflix’s internal Metatron PKI, allowing managed compute workloads to authenticate to internal services without manually injecting credentials.

Dhruv Pratap, Big Data Platform Engineer at Netflix, described the design in a LinkedIn post as

On infrastructure you build yourself, you can bootstrap your own identity however you like.

Netflix operates two identity systems. AWS provides managed workloads with an IAM execution role, while internal services use Metatron, a private PKI that issues short-lived X.509 certificates for mutual TLS. Netflix also uses Data Projects as the ownership and identity boundary for data access. A Spark job running on EMR therefore needs to establish that its AWS identity corresponds to…