Surviving a Bot Flood on a $20 VPS: nginx limit_req, fail2ban and Cloudflare Free in the Right Order

It usually starts at an awkward hour. The site is timing out, the CPU graph is flat at 100 per cent, and the WhatsApp group is asking whether you have been hacked. You have not been hacked. You are being scraped, stuffed, or crawled into the ground by something with far more bandwidth than your KES 2,500 per month droplet.

My thesis: on a small VPS you win by applying protection in cost order - free and upstream first, cheap and local second, and application changes last - and by measuring before you touch anything. Most people do this backwards. They add a WAF rule at 2am, block half their real customers, and never find out that the actual problem was one unindexed query.

Step 0: find out what is actually happening

Twenty minutes of diagnosis saves a night of flailing. Three questions: is it a traffic volume problem, a single-endpoint problem, or a database problem that traffic merely exposed?

# Requests per minute over the last hour - is volume actually abnormal?
sudo awk '{print substr($4, 2, 17)}' /var/log/nginx/access.log | uniq -c | tail -60

# Which endpoint is being hammered?
sudo awk '{print $7}' /var/log/nginx/access.log | cut -d? -f1 | sort | uniq -c | sort -rn | head -20

# Who is hammering it?
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# What are they claiming to be?
sudo awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -15

# Slowest responses, if you log $request_time as the last field
sudo awk '{print $NF, $7}' /var/log/nginx/access.log | sort -rn | head -20

If you are not already logging $request_time and $upstream_response_time, fix that now. Without it you cannot tell a flood from a slow query.

log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
                 '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                 'rt=$request_time urt=$upstream_response_time';

access_log /var/log/nginx/access.log timed;

Three patterns and what they mean:

• Thousands of IPs, low rate each, hitting login or search: distributed credential stuffing or scraping. Per-IP rate limiting alone will not save you. You need a challenge upstream.
• A few dozen IPs, high rate each, walking your product pages sequentially: a scraper. Rate limiting works very well here.
• Normal traffic volume, terrible response times: not an attack. Go fix the query or add caching before you touch a firewall.

Layer one: Cloudflare's free tier, because it absorbs the bandwidth

The single highest-leverage move is putting a large network in front of your small server. Cloudflare's free plan gives you unmetered DDoS mitigation, caching, a managed WAF ruleset, Turnstile challenges, and - importantly for this article - one rate limiting rule, with counting periods up to one minute on Free and Pro. Business gets five rules and periods up to about 18 hours.

You get one rule on Free, so spend it well. Spend it on your login or checkout endpoint, not on the whole site.

• Expression: http.request.uri.path contains "/api/method/login" and method is POST
• Characteristics: IP (Free and Pro count by IP)
• Rate: something like 10 requests per 1 minute, tuned to your real traffic
• Action: Block, or Managed Challenge if you have legitimate users behind shared NAT, which is very common in Kenya where offices and estates share a single public IP

That NAT point deserves emphasis. Blocking an IP outright in an environment full of shared connections can take out a whole client's office. Challenge before you block.

Lock the origin so Cloudflare cannot be bypassed

Proxying is worthless if attackers can hit your IP directly, and old DNS records leak it constantly. Restrict your firewall to Cloudflare's published ranges, and restore real client IPs so your logs and rate limits are not all seeing the proxy.

# Only accept web traffic from Cloudflare, plus your own admin access
sudo ufw --force reset
sudo ufw default deny incoming
sudo ufw allow from YOUR.OFFICE.IP to any port 22 proto tcp
for ip in $(curl -s https://www.cloudflare.com/ips-v4); do
  sudo ufw allow from $ip to any port 443 proto tcp
done
sudo ufw enable

# /etc/nginx/conf.d/cloudflare-realip.conf
# Generate the set_real_ip_from lines from cloudflare.com/ips-v4 and ips-v6
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
# ... remaining published ranges ...
real_ip_header CF-Connecting-IP;

Layer two: nginx rate limiting, where the real control lives

nginx's limit_req uses a leaky bucket. Define zones in the http block, apply them per location. Different endpoints deserve different budgets: your product pages are not your login form.

http {
    # 10m of shared memory holds roughly 160,000 IPv4 addresses
    limit_req_zone $binary_remote_addr zone=general:10m rate=30r/s;
    limit_req_zone $binary_remote_addr zone=api:10m     rate=10r/s;
    limit_req_zone $binary_remote_addr zone=login:10m   rate=10r/m;

    limit_conn_zone $binary_remote_addr zone=perip:10m;

    # 429 is honest and cacheable; the default 503 confuses monitoring
    limit_req_status 429;
    limit_conn_status 429;

    server {
        listen 443 ssl;
        server_name shop.example.co.ke;

        client_max_body_size 8m;
        client_body_timeout 10s;
        client_header_timeout 10s;
        send_timeout 10s;

        limit_conn perip 20;

        location / {
            limit_req zone=general burst=60 nodelay;
            proxy_pass http://127.0.0.1:8000;
        }

        location /api/ {
            limit_req zone=api burst=20 nodelay;
            proxy_pass http://127.0.0.1:8000;
        }

        location = /api/method/login {
            limit_req zone=login burst=5 nodelay;
            proxy_pass http://127.0.0.1:8000;
        }
    }
}

Two directives people misuse:

• burst is how many excess requests may queue. Without nodelay they are delayed so the average rate holds. Real browsers fetch a page plus twenty assets in a burst, so a burst of zero will break your own site.
• nodelay serves burst requests immediately and rejects anything beyond the burst with 429. This is what you almost always want for a web app: forgiving of normal bursty browsing, hard on sustained abuse.

Always validate and reload rather than restart, and watch your error log for the specific limiting message.

sudo nginx -t && sudo systemctl reload nginx
sudo tail -f /var/log/nginx/error.log | grep "limiting requests"

Tune from evidence. If genuine users are getting 429s, your rate or burst is wrong, not your users.

Layer three: fail2ban, to turn repeat offenders into firewall drops

Rate limiting still costs you a TCP handshake and an nginx worker for every rejected request. fail2ban watches the logs and pushes persistent abusers down to the kernel, where they cost almost nothing. Usefully, fail2ban ships a filter that reads exactly the log line nginx writes when it limits a request.

# /etc/fail2ban/jail.local
[DEFAULT]
bantime  = 3600
findtime = 600
maxretry = 5
banaction = ufw
ignoreip = 127.0.0.1/8 ::1 YOUR.OFFICE.IP

[sshd]
enabled  = true
backend  = systemd
maxretry = 3
bantime  = 86400

[nginx-limit-req]
enabled  = true
port     = http,https
filter   = nginx-limit-req
logpath  = /var/log/nginx/error.log
findtime = 60
maxretry = 20
bantime  = 3600

[nginx-http-auth]
enabled = true
port    = http,https
logpath = /var/log/nginx/error.log

sudo systemctl restart fail2ban
sudo fail2ban-client status
sudo fail2ban-client status nginx-limit-req

# Test your filter against real logs before trusting it
sudo fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-limit-req.conf

# Unban when you inevitably block yourself
sudo fail2ban-client set nginx-limit-req unbanip 41.90.0.1

One caveat worth knowing: if you are behind Cloudflare and have not configured real_ip_header CF-Connecting-IP, fail2ban will read Cloudflare's proxy addresses from your logs and cheerfully ban Cloudflare. Do the real-IP configuration first.

Layer four: make the expensive requests cheap

Some floods are not malicious at all, just a badly behaved crawler on an endpoint that costs 900 milliseconds of database time. The cheapest defence is to stop that endpoint being expensive.

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=pubcache:20m
                 max_size=2g inactive=10m use_temp_path=off;

location /products/ {
    proxy_cache pubcache;
    proxy_cache_valid 200 5m;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Cache-Status $upstream_cache_status;
    proxy_pass http://127.0.0.1:8000;
}

proxy_cache_use_stale with updating is the quiet hero here: when your backend is struggling, nginx serves the slightly stale copy instead of piling on more load. A five minute cache on public catalogue pages can remove 95 per cent of your origin traffic.

An incident runbook you can follow at 2am

1. Confirm it is traffic. Check requests per minute and $request_time. If volume is normal, this is a performance bug.
2. Identify the target. One endpoint or the whole site.
3. Cheapest lever first. Enable "Under Attack" mode in Cloudflare, or point your single rate limiting rule at the endpoint being hit.
4. Then nginx. Add or tighten a limit_req zone for that specific location. Reload, do not restart.
5. Then fail2ban. Confirm bans are landing with fail2ban-client status.
6. Then cache. If the endpoint is public and mostly static, cache it.
7. Write it down. Which IPs, which user agents, which paths, what you changed, what it cost. Next time this is a fifteen minute job.

What not to do

Do not block by country as a first move. Kenyan businesses have customers, staff, and payment callbacks abroad, and geo-blocking has a way of silently killing an integration nobody remembers. Do not ban an entire ASN because Safaricom or Airtel mobile users share addresses at scale. Do not set an aggressive global rate limit and go back to bed; you will wake up to a customer support fire. And do not upgrade the VPS as your first response. Doubling the server doubles the cost and buys minutes; a five minute cache and one rate limit rule buy orders of magnitude and cost nothing.

Bot floods are survivable on cheap infrastructure. What is not survivable is guessing. Log properly, diagnose first, then apply the cheapest layer that solves the actual problem.