Taming a throttled API with Dynamic Proxies in Java

Jan 20, 2021

Recently, at CleverTap, we’ve begun migrating some of our largest clusters to a new protocol (for starters, think ~115 instances at a time). One of the most fun things I’ve had my hands on during this migration was the AWS Systems Manager API.

When we scaled up our migrations gradually from a 10 node cluster, we were challenged with dealing with API throttling exceptions (because sure, who wouldn’t throttle their APIs?). There were two immediate solutions that hit our mind:

Review every usage of the SSM client and handle the throttling exception gracefully
Wrap the SSM client and handle the throttling exception transparently

Naturally, we settled for option 2. I am a big fan of hidden abstractions. So what did we do? We implemented the AWS interface in question, only to discover that we’d have to handle a ton of methods individually (obviously copy/paste). There had to be a better solution!

And then, Google did it’s thing. We discovered Dynamic Proxies. And viola! We were able to transparently handle and implement an auto retry strategy within just 14 lines!

Here’s what it looked like:

	MyStubbornAPIInterface actualInstance = … // Create it however you'd create your original instance.
	MyStubbornAPIInterface proxiedInstance = (MyStubbornAPIInterface) Proxy.newProxyInstance(actualInstance.getClass().getClassLoader(),
	new Class[]{MyStubbornAPIInterface.class}, (proxy, method, args) -> {
	while (true) {
	try {
	return method.invoke(actualInstance, args);
	} catch (MyThrottlingException e) {
	try {
	Thread.sleep(ThreadLocalRandom.current().nextInt(1, 5) * 1000L);
	} catch (InterruptedException e) {
	Thread.currentThread().interrupt();
	}
	}
	}
	});

view raw TameAThrottledAPI.java hosted with ❤ by GitHub

The code above can be easily adapted to various SDKs (in our case, it was the AWS SDK).

Now, all we had to do was pass around this proxied instance, and viola, the consumers of this API had no clue that the API implemented an auto retry mechanism!

Cheers!