Cloudflare Zero Trust Gateway and Net Neutrality

TL;DR: This post isn’t merely a rant after Cloudflare’s recent outages, but rather meant to serve as an after-thought: is Cloudflare ruining the entire concept of a distributed internet? Is it on a path to violate Net Neutrality?

Let’s analyse it a little bit.

What is it?

Cloudflare’s Zero Trust Gateway routes all internet traffic from your devices such as your laptop, via Clouldflare’s internet backbone. Most likely, your personal device wouldn’t be connected to the zero trust Gateway, since it’s almost always deployed by enterprise companies.

What’s wrong with it?

The internet is de-centralised – no single authority can take it down, nor can control traffic across it. In its literal sense, “internet” means interconnected networks. This means that you’re reading this post through a bunch of networks that eventually connect to the host where this blog is hosted, in Amsterdam via Digital Ocean. The source of your connection could be anything – from a 5G capable device, a wired network connection being shared in a building, your ISP, etc. The list goes on an on.

However, Cloudflare Zero Trust Gateway routes ALL your traffic through a bunch of proxies that Cloudflare exclusively controls:

Yes, Cloudflare actually runs an active MITM attack, decrypting all your TLS data.
Fuck off Cloudflare!

And now we get to the real problem: when Cloudflare deploys a buggy version of their software, to the end user, that is to you, it appears as if the entire internet is down. You can’t get work done, nor can you do anything productive with that €3000 MacBook you’ve just bought. Surprisingly, this happens more often than not, especially in recent times.

In the future, if Cloudflare Zero Trust Gateway captures any significant market share (hopefully unlikely), they can suddenly start to make decisions that violate Net Neutrality.

Net Neutrality

If you’ve never watched John Oliver’s take on Net Neutrality, watch it here. ISPs have done such things in the past, and have managed to get away with it, albeit far fewer of those severely offending ISPs exist today (if somebody finds a valuable source for or against this, please post it in the comments). Of course, the giants still continue to live on.

Final Notes

In order to preserve Net Neutrality, the openness of the internet, no single company should own a significant chunk of the internet traffic. If your company is pushing for Cloudflare Zero Trust Gateway, push your IT administrator to read this post, and help point them in the right direction.

Getting the AWS CLI to accept Cloudflare WARP’s root certificate

When we moved to Cloudflare WARP at CleverTap, everything worked as expected, except for the AWS CLI:

SSL validation failed for [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)

The problem with Cloudflare WARP is that it’s the equivalent of Charle’s Proxy – presenting its own certificate for everything. While this works for almost everything, it doesn’t work for tools that use a well known and publicly trusted CA bundle.

Moreover, AWS’ CLI v2 is built with Python, which apparently doesn’t even have access to macOS’ Keychain. So, although that Cloudflare WARP certificate is installed in Keychain, tools from JetBrains and AWS’ very own CLI will refuse to work.

The solution? Download a .crt from Cloudflare’s documentation, convert it to a .pem (using Keychain), and then add it to your CA bundle (in my case, the default one installed by Homebrew).

Preparing the certificate

Download the certificate from here. Next, open it up in Keychain:

Right click on that entry, to see the Export option:

Export it as a PEM:

Hit Save.

Installing the certificate

And now to the tricky part. The CA bundle that AWS uses for its CLI is a mystery. However, it does allow one to override that, by using an environment variable. So, we want that CLI to trust the usual set of root certificates, along with Cloudflare’s. OpenSSL installs a decent set of trusted root certificates, which we can append to:

$ cat Cloudflare.pem >> /Users/jude/bin/Homebrew/etc/openssl/cert.pem

All that’s remaining is to tell the AWS CLI to use it:

$ export AWS_CA_BUNDLE=/Users/jude/bin/Homebrew/etc/openssl/cert.pem

And voila! It starts working magically!