Lately I have been thinking about my Apache APISIX setup in my DMZ. I believe applying the KISS principle to as many IT components as possible inherently improves security.
My API Gateway inside the DMZ is a Debian box with Docker, behind a MikroTik router, facing the internet. Replacing APISIX with something simpler is not that hard. There aren’t many products more complex than APISIX, but there are plenty that are simpler.
The issues I had with APISIX in my infrastructure:
- I ran it using etcd, which made the APIM stateful. I think APIM implementations need to be stateless to enforce the KISS principle.
- Even when you run it in standalone mode (without etcd) it has a lot of moving parts.
- It contains a lot of functionality I was not using.
After some research I landed on KrakenD:
- It runs stateless using a simple YAML file. Too bad it’s YAML, but hey…I will survive.
- It is a simple Go binary and a simple architecture.
- It is a basic APIM solution with plugin support and support for LUA scripting if needed.
- As a bonus: The product has quite a bit of traction in Europe, which is good for my market value as a European IT professional.
- Another bonus: It’s developed by a European company.
In my final setup I came up with a four-layer setup for basic security, of which three layers handle the main subject of this article. Rate limiting.
For an API, and perhaps any web server, rate limiting is a major security measure for the API backends but as well as the APIM infrastructure itself.
This is the setup as it runs now.
graph TD %% Outside the Debian Box internet([internet VLAN]) -->|only 80/443| MikroTik[MikroTik perimeter] InternalNet([some internal VLANs]) -->|80/443 & 22| MikroTik IncomingVPN([some incoming VPN connections]) -->|80/443 & 22| MikroTik %% Network connections hitting the physical host boundary MikroTik -->|ports 80/443| DockerUser MikroTik -->|port 22| INPUT %% The Debian Box Boundary subgraph DebianBox [Debian Host] DockerUser[Kernel Netfilter: DOCKER-USER chain] INPUT[Kernel Netfilter: INPUT chain <br> SSH Protection] SSHD[sshd service] Fail2ban[fail2ban layer 3] Caddy[Caddy layer 1: per-IP throttling] KrakenD[KrakenD layer 2: quota per route] Backends[API backends] AuthLog[(auth.log / journald)] CaddyLog[(Caddy Log File)] %% Internal Host Flows Fail2ban -.->|adds DROP for Web attackers| DockerUser Fail2ban -->|adds DROP for SSH attackers| INPUT INPUT -->|port 22| SSHD DockerUser -->|ports 80/443| Caddy Caddy -->|port 8080. Both Caddy and KrakenD have one leg in the same Docker VNET for secure plain HTTP communication. Not used for anything else. | KrakenD KrakenD -->|ports 80/443| Backends SSHD -.->|writes| AuthLog Caddy -.->|writes| CaddyLog AuthLog -.->|analyzes SSH logs| Fail2ban CaddyLog -.->|analyzes Web logs| Fail2ban end %% Theme Settings & Color Palette (Clean Minimal Dark) style DebianBox fill:#1a1a1a,stroke:#444444,stroke-width:2px,stroke-dasharray: 5 5,color:#ffffff; %% Standard Base Node Styling %% Subtle outer network highlights style internet fill:#111111,stroke:#666666,color:#888888; style InternalNet fill:#111111,stroke:#666666,color:#888888; style IncomingVPN fill:#111111,stroke:#666666,color:#888888; style MikroTik fill:#1a1a1a,stroke:#888888,color:#ffffff,stroke-width:2px;
Layer 1: Caddy throttles everything, 404s included
KrakenD’s rate limiter only runs for endpoints you’ve declared. A request to an unknown path gets a plain 404, and there is no catch-all route to hang middleware on. So the script-kiddie traffic that scans for /wp-admin has to be dealt with at the first layer.
Stock Caddy has no rate-limit directive, but Matt Holt has a module for it: caddy-ratelimit. I put it on top of the official docker:
FROM caddy:2-builder AS builder
RUN xcaddy build --with github.com/mholt/caddy-ratelimit
FROM caddy:2-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddyThe zone is per IP and covers every request to the site, the 404s included:
{
order rate_limit before reverse_proxy
}
api.example.com {
log {
output file /data/access.log {
roll_size 10mb
roll_keep 3
}
format json
}
rate_limit {
zone edge_per_ip {
key {remote_host}
events 10
window 1m
}
}
reverse_proxy krakend:8080 {
header_up X-Real-IP {remote_host}
}
}Two remarks. The access log is needed for layer 3 fail2ban. I write it to Caddy’s /data volume so it lands on the host where fail2ban can read it. And header_up X-Real-IP {remote_host} puts the client’s actual IP in the header, overwriting whatever the client sent along. That is what makes layer 2 possible, and it can’t be spoofed because Caddy is the TLS-terminating edge.
Layer 2: KrakenD quotas per route
KrakenD’s qos/ratelimit/router does per-endpoint quotas: different limits per route, later per token or per tenant if I ever get that far. Two things here cost me an evening.
The middleware only works at endpoint level. A qos/ratelimit/router block in the root of krakend.json is silently ignored. There’s no inheritance at service level, so you repeat the block on every endpoint. In the future I can generate the config with KrakenD’s flexible configuration templates instead.
strategy: "ip" is almost never what you want behind a reverse proxy. It counts on the peer IP, which is the Caddy container, so the whole world is thrown into one single bucket. The key option is ignored under that strategy too; it only applies to "header" or "param". What works is the X-Real-IP header Caddy sends:
{
"endpoint": "/hello",
"method": "GET",
"input_query_strings": ["name"],
"input_headers": ["X-Real-IP"],
"extra_config": {
"qos/ratelimit/router": {
"client_max_rate": 10,
"client_capacity": 10,
"every": "1m",
"strategy": "header",
"key": "X-Real-IP"
}
},
"backend": []
}Don’t forget "input_headers". KrakenD is zero-trust about headers, so without the whitelist the endpoint never sees X-Real-IP and everyone lands in the same anonymous bucket. That one took me a while to spot. It’s completely different from my experiences with Apache APISIX and Azure APIM which are transparent proxies. I like this better as it prevents leaking of information.
Layer 3: fail2ban drops the persistent ones at the kernel filter
A 429 still costs the attacker a TCP handshake, a TLS negotiation and me a Caddy worker eating CPU. Layer 3 takes that away as well: watch the access log for 429s, and once an IP crosses the threshold, add a DROP to the DOCKER-USER chain using IPTables.
The filter, /etc/fail2ban/filter.d/caddy-ratelimit.conf, is one line reading Caddy’s JSON:
[Definition]
failregex = "remote_ip":"<HOST>",".*"status":429
ignoreregex =You can validate it against real log lines before wiring anything up. fail2ban-regex /path/to/access.log /etc/fail2ban/filter.d/caddy-ratelimit.conf tells right away whether your regex sees what you think it sees.
The action is where most people get it wrong. A fail2ban ban in the UFW INPUT chain does nothing for container traffic: packets to Docker-published ports are DNAT’d straight through the FORWARD chain and never touch INPUT. This is exactly why tools like ufw-docker exist. Pls google this if you put Docker on the internet. Maybe I will write a story about it in the future. The ban has to land in DOCKER-USER, the designated user-filtering chain in Docker’s iptables firewall implementation:
# /etc/fail2ban/action.d/docker-block.conf
[Definition]
actionban = iptables -I DOCKER-USER 1 -p tcp -s <ip> -m multiport --dports 80,443 -j DROP
iptables -I DOCKER-USER 1 -p udp -s <ip> --dport 443 -j DROP
actionunban = iptables -D DOCKER-USER -p tcp -s <ip> -m multiport --dports 80,443 -j DROP
iptables -D DOCKER-USER -p udp -s <ip> --dport 443 -j DROPThe rules only cover web traffic (tcp 80/443, plus udp 443 for HTTP/3, or a script kiddie can keep hammering over QUIC) and leave other published ports alone. One subtlety: in DOCKER-USER the destination port is the port after DNAT, so the container port. Same thing with a 1:1 mapping like 443:443; if you remap, the rules have to follow.
The jail:
# /etc/fail2ban/jail.d/caddy-ratelimit.conf
[caddy-ratelimit]
enabled = true
filter = caddy-ratelimit
action = docker-block
logpath = /opt/stacks/proxy/data/access.log
backend = polling
maxretry = 5
findtime = 60
bantime = 1h
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 1w
ignoreip = 127.0.0.1/8 192.168.0.0/16 172.16.0.0/12A few notes on the choices:
- Five throttled requests within a minute escalate to a kernel drop. A scraper that behaves stays inside the limit and never sees a ban; someone hammering is cut off within seconds.
bantime.incrementdoubles the ban for repeat offenders, up to a week. Bans live in fail2ban’s SQLite database and survive restarts.backend = polling: fail2ban re-reads the file in a loop of about a second, instead of trusting inotify, which can silently miss writes on Docker bind mounts. A rate limiter that quietly stops banning is a nasty thing to discover late.ignoreipwhitelists my RFC1918 ranges, so my VLAN clients can never ban themselves. Traffic from the LAN reaches the server with its real private IP, so that just works. I might tighten those ranges in the future.- At startup, a systemd drop-in (
After=docker.service) makes fail2ban start after theDOCKER-USERchain exists.
One remark on banned IPs and dynamic Docker container IPs
The fail2ban bans land in DOCKER-USER; ufw-docker is the tool that keeps UFW meaningful on a Docker host at all, and it manages the same chain. NEW connections to 172.16.0.0/12 are logged and dropped, unless you explicitly allow a specific container:
ufw-docker allow caddy 80/tcp
ufw-docker allow caddy 443/tcp
ufw-docker allow caddy 443/udpNow the part that took my site down twice. Those allow rules pin themselves to the container’s IP, and a container gets a new IP every time (or at least once in a while) it is recreated. The next public visitor falls through to the generic deny. The ugly part is how it looks from the inside: VLAN clients still reach the site just fine, the internet gets a timeout, and no log line tells you your allow rule now points at an address that no longer exists. The only real hint is a line like this in the kernel log:
[UFW DOCKER BLOCK] IN=enp2s0 OUT=br-… SRC=1.2.3.4 DST=172.18.3.1 … DPT=443 SYNA public SYN to the published port, dropped by ufw-docker’s catch-all. Worth knowing what that line does not mean: it is not a fail2ban ban. fail2ban’s rules drop silently at the top of the chain and never reach the logging rule, so a logged block proves the source was not banned and that no allow rule matched.
The fix I settled on is pinning the IP:
networks:
proxy:
name: proxy
ipam:
config:
- subnet: 172.18.3.0/24
services:
caddy:
networks:
proxy:
ipv4_address: 172.18.3.1A static address, and the allow rules can’t age out anymore.
As a safety net for everything else on the host, or when I forget about the static IP, a systemd timer runs ufw-docker update every 5 minutes, which re-pins the allow rules for any container whose IP moved. Before I trusted that timer I wanted to know one thing: does ufw-docker update wipe fail2ban’s manually entered bans? Tested with a deliberate banip and a six-minute wait. The DROP rules at the top of DOCKER-USER survived. Bans first, allows after.
Why layers, and why this order
| Layer | Scope | Stops |
|---|---|---|
| MikroTik | perimeter, only 80/443 | everything else |
| fail2ban → DOCKER-USER | whole host, ports 80/443, per IP | persistent abusers, at the NIC |
Caddy rate_limit | every request to the site, per IP | politely, with a 429 |
KrakenD qos/ratelimit/router | per declared endpoint, per IP | excess traffic per route |
Each layer degrades gracefully, and none of them can reconfigure the layer above it. That is one rule I’m not bending on: the DMZ box never holds the credentials for the firewall that protects it. Yes, the router could ban IPs at the edge too, but a compromised DMZ server must not be able to rewrite the perimeter. I considered making the MikroTik firewall respond to whatever happens on the APIM machine, but decided not to. Each layer only trusts inward.
So that’s the setup. An API on the open internet that answers normal traffic politely, throttles the rude part, and drops the persistent part at the kernel netfilter.
Of course, there’s much more to Caddy and KrakenD, but the idea of this article is only to talk about the rate limiting.
Feel free to ask me any questions on the subject.