Part 1 of this series showed how to rate-limit calls to Amazon DynamoDB by using Python Boto3 event hooks. In this post, I expand on the concept and show how to rate-limit calls in a distributed environment, where you want a maximum allowed rate across the full set of clients but can’t use direct client-to-client communication.
Rate limiting in a distributed environment can be useful when you have multiple clients and don’t want to give a predetermined fixed capacity to each client because the number of clients might fluctuate or the consumption per client might vary over time. The technique discussed in this post uses an Amazon Simple Storage Service (Amazon S3) bucket for distributed coordination of the shared system state. It’s a proven solution used in Bulk Executor for Amazon DynamoDB and its logic is available encapsulated in the DistributedDynamoDBMonitorWorker and DistributedDynamoDBMonitorAggregator classes on GitHub.
Solution overview
…
