NGINX rate limiting doesn’t work when using Cloudflare. I can bring down my site with a simple `ab` command

I implemented a pretty simple but super effective rate limiting based on this blog post: https://www.nginx.com/blog/rate-limiting-nginx/

Basically:

limit_req_zone $binary_remote_addr zone=ip:10m rate=10r/s;

limit_req zone=ip burst=20 nodelay;

It works great. However, recently I tried Cloudflare, and this doesn’t protect me anymore. I can bring down the site myself with a simple command of:

ab -k -c 1000 -n 10000 site.com/

What’s happening?

Answer

ab -k -c 1000 -n 10000 site.com/ is running 1000 requests in parallel, until a total of 10 000 requests total have been done.

That’s too brutal. It’s likely that neither the client nor the server are tuned to handle thousands of connections over a few seconds.

Adjust the nginx configuration and do a gentle test ab -k -c 5 -n 500 site.com/

limit_req_zone $http_cf_connecting_ip zone=ip:10m rate=3r/s;
limit_req zone=ip;

limit_conn_status 429;
limit_req_status 429;

429 Too Many Requests

This configures nginx to return the standard status code 429 Too Many Requests when requests are rejected due to rate limiting.

nginx returns a 503 error by default (a bad default) meaning the application is failing, but it is not failing it is rate limited. It’s important to configure status code appropriately to distinguish between server errors and rate limiting.

Cloudflare and client IP

When behind cloudflare, nginx will not see the IP of the client but the IP of the cloudflare server. One might think that it breaks rate limiting by IP but it does not, well, just a bit.

When testing locally with ab, your test computer is only resolving a handful of cloudflare servers, and ab probably only uses the first IP. So no there aren’t numerous clients IP, the rate limiting should work just fine.

When in production, there will be different clients accessing through different cloudflare servers. Still, there aren’t that many cloudflare servers and clients in a geographic area will most likely resolve to the same cloudflare servers. So there will be a bunch of different IPs somewhat defeating the rate limiting, but probably not that many.

> nslookup mycloudflaresite.com

Name:    mycloudflaresite.com
Addresses:  104.28.14.125
            104.28.15.125
            2606:4700:3037::681c:e7d
            2606:4700:3036::681c:f7d

Cloudflare puts the original client IP in the CF-Connecting-IP header. It can also be in the X-Forwarded-For header or X-Real-Ip or True-Client-IP depending on settings and requests. See https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-

Hence the above configuration does rate limiting by client IP using the CF-Connecting-IP header. The nginx variable $binary_remote_addr would be the cloudflare server IP.

Do not use X-Forwarded-For to rate limit

The X-Forwarded-For header can be controlled by the client. It shouldn’t be used for rate limiting because it is trivial to circumvent.

Example with a client having the IP 100.11.22.33:

• On a request without a X-Forwarded-For header => Cloudflare sets X-Forwarded-For: 100.11.22.33 and CF-Connecting-IP: 100.11.22.33 on the request.
• On a request with a X-Forwarded-For: dummyvalue header already set => CloudFlare sets X-Forwarded-For: dummyvalue,100.11.22.33 and CF-Connecting-IP: 100.11.22.33 on the request.

As you can see, it’s trivial for the client to put a random value per request and totally circumvent any rate limiting based on the X-Forwaded-For header.