Building a Secure High-Availability Reverse Proxy Cluster with NGINX and Keepalived

A single reverse proxy is a single point of failure. It terminates TLS, balances traffic and hides your application servers, which also means that a kernel update, a bad config reload or a dead network card takes every site behind it offline. The usual fix on your own servers is a pair of proxies that share one IP address: one answers, the other waits, and the address moves in a few seconds if the active node stops being healthy.

This guide builds that pair with NGINX and Keepalived. It is a general engineering walkthrough for Linux servers you control. The NGINX configuration was checked with nginx -t on NGINX 1.24 (with the HTTP/2 line adjusted as noted below), the Keepalived files with keepalived -t on Keepalived 2.2.8, and the firewall rules with nft -c.

What you'll build

• Two proxy nodes, lb1 and lb2, each running NGINX and Keepalived.
• One virtual IP (VIP) that clients and DNS point at, owned by whichever node is currently MASTER.
• An NGINX upstream pool of three application servers with passive health checks, connection reuse and sensible retry rules.
• TLS 1.2 and 1.3 termination with identical certificates on both nodes.
• A Keepalived track_script that demotes a node when NGINX stops answering, not only when the machine dies.
• VRRP over unicast between the two peers, with a firewall that accepts VRRP only from the other node.

The addresses below come from the documentation ranges:

How the pieces fit

Keepalived implements VRRP (Virtual Router Redundancy Protocol). The node with the highest priority becomes MASTER, adds the VIP to its interface and sends an advertisement every advert_int seconds. The BACKUP node listens. If advertisements stop for long enough, the backup promotes itself, adds the VIP and sends gratuitous ARP so the switch and router learn the new MAC address for that IP.

NGINX takes no part in this. It runs on both nodes all the time; Keepalived only decides which node holds the address traffic arrives on.

This works cleanly when both nodes share a layer 2 segment, which is normal for physical servers and many private clouds. In most public clouds it does not work as described: the provider's network ignores gratuitous ARP and drops traffic to an IP that has not been assigned through its API. There you keep Keepalived for the election, set VRRP to unicast, and use a notify_master script that calls the provider's API to reassign a floating or elastic IP. Check your provider's documentation before you design around a VIP.

How long failover takes

The backup declares the master dead after the Master Down Interval from the VRRP specification:

Master_Down_Interval = 3 x advert_int + Skew_time
Skew_time            = ((256 - priority) / 256) x advert_int

For a backup with priority 100 and advert_int 1:
Skew_time            = (156 / 256) x 1  = 0.61 s
Master_Down_Interval = 3 x 1 + 0.61     = 3.61 s

That is the worst case for a node that vanishes. When the master is alive but NGINX is broken, the check script decides: with interval 2 and fall 2, the script needs two failures in a row, so detection takes between 2 and 4 seconds, plus one advertisement for the backup to see the lowered priority. Treat these as the arithmetic of your settings, not a measured result; measure your own with the test at the end of this guide.

Prepare both nodes

Install the packages (Debian and Ubuntu shown; on RHEL-family systems use dnf):

sudo apt update
sudo apt install nginx keepalived curl

Allow NGINX to bind the VIP

If NGINX listens on the VIP explicitly, the BACKUP node cannot start NGINX, because the address is not on any of its interfaces. The kernel refuses the bind with "Cannot assign requested address". The fix is a sysctl that permits binding to non-local addresses:

sudo tee /etc/sysctl.d/90-nonlocal-bind.conf >/dev/null <<'EOF'
net.ipv4.ip_nonlocal_bind = 1
net.ipv6.ip_nonlocal_bind = 1
EOF
sudo sysctl --system

The alternative is to listen on all addresses (listen 443 ssl; with no IP). That needs no sysctl, but NGINX then also answers on each node's own address, which you may not want exposed. Binding to the VIP keeps the public surface to one address, and the firewall below handles the rest.

Configure NGINX

Use the same configuration on both nodes, kept in version control and deployed by your usual tooling so the two cannot drift apart.

Upstream pool and passive health checks

Open source NGINX has passive health checks: it marks a server as unavailable after max_fails failed attempts within fail_timeout, then tries it again after fail_timeout has passed. Active checks with the health_check directive, which poll servers on a schedule, are an NGINX Plus feature. If you need active checks with the open source build, run them outside NGINX and remove servers from the pool through your deployment tooling.

Create /etc/nginx/conf.d/upstream.conf:

upstream app_pool {
    zone app_pool 64k;
    least_conn;

    server 192.0.2.21:8080 max_fails=3 fail_timeout=10s;
    server 192.0.2.22:8080 max_fails=3 fail_timeout=10s;
    server 192.0.2.23:8080 max_fails=3 fail_timeout=10s;

    keepalive 32;
    keepalive_timeout 60s;
}

• zone keeps the pool's state in shared memory, so every worker process sees the same failure counts.
• least_conn sends each request to the server with the fewest active connections, which suits requests of uneven length better than plain round robin.
• keepalive 32 holds up to 32 idle connections per worker to the backends, which saves a TCP handshake per request. It only works with proxy_http_version 1.1 and an empty Connection header, set below.

The TLS server block

Create /etc/nginx/conf.d/site.conf:

server {
    listen 198.51.100.10:80;
    server_name www.example.com example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 198.51.100.10:443 ssl;
    http2 on;
    server_name www.example.com example.com;

    ssl_certificate     /etc/nginx/tls/example.com.fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_cache   shared:TLS:10m;
    ssl_session_timeout 1h;
    ssl_session_tickets off;

    add_header Strict-Transport-Security "max-age=31536000" always;

    client_max_body_size 20m;

    location / {
        proxy_pass http://app_pool;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 3s;
        proxy_read_timeout 60s;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
    }
}

server {
    listen 127.0.0.1:8081;
    location = /healthz {
        access_log off;
        default_type text/plain;
        return 200 "ok\n";
    }
}

Points worth knowing:

• The http2 on; directive needs NGINX 1.25.1 or later. On older versions, including the 1.24 package in Ubuntu 24.04, remove that line and write listen 198.51.100.10:443 ssl http2; instead.
• Retries are limited on purpose. proxy_next_upstream retries on connection errors, timeouts, 502 and 503. NGINX does not retry non-idempotent requests such as POST once they have been sent to a server unless you add non_idempotent, and you should not: a retried payment request can run twice.
• Session tickets are off. Each node would otherwise generate its own ticket key, so a failover breaks resumption anyway, and a ticket key that never rotates weakens forward secrecy. If you want tickets, generate a key file with openssl rand 80 > ticket.key, point ssl_session_ticket_key at it on both nodes, and rotate it on a schedule.
• The health endpoint listens on loopback only. Nothing outside the node can reach it.

The application servers see every connection from the proxies' addresses, so they need to read the client address from X-Forwarded-For and trust that header only from 198.51.100.11 and 198.51.100.12. If the backends also run NGINX, that is set_real_ip_from and real_ip_header X-Forwarded-For in the realip module.

Certificates on two nodes

Both nodes must hold the same certificate and key, with the key readable only by root (chmod 600). ACME HTTP-01 validation is awkward here, because the CA's request lands on whichever node holds the VIP. Two clean options: use DNS-01 validation on one node and copy the result to the other, or issue on a separate management host and push the files to both proxies over SSH. Reload with systemctl reload nginx after each renewal, on both nodes.

Check and load the configuration:

sudo nginx -t
sudo systemctl enable --now nginx

Configure Keepalived

The check script

Keepalived needs a test that fails when NGINX cannot serve traffic. Checking that a process exists is not enough: a process can run while its workers are stuck or its config is half-loaded. Create /usr/local/bin/check_nginx.sh:

#!/bin/sh
# Exit 0 if NGINX answers on loopback within one second, 1 otherwise.
exec /usr/bin/curl -fsS --max-time 1 -o /dev/null http://127.0.0.1:8081/healthz

sudo chown root:root /usr/local/bin/check_nginx.sh
sudo chmod 755 /usr/local/bin/check_nginx.sh
sudo useradd --system --no-create-home --shell /usr/sbin/nologin keepalived_script

With enable_script_security set, Keepalived runs scripts as the unprivileged script_user instead of root, and refuses to run any script as root if part of its path is writable by a non-root user. Both matter: the check runs every two seconds, and a script anyone can edit is a way into the machine.

keepalived.conf on lb1

Create /etc/keepalived/keepalived.conf on lb1:

global_defs {
    router_id lb1
    enable_script_security
    script_user keepalived_script
}

vrrp_script chk_nginx {
    script "/usr/local/bin/check_nginx.sh"
    interval 2
    timeout 2
    fall 2
    rise 2
    weight -20
}

vrrp_instance VI_WEB {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 110
    advert_int 1

    unicast_src_ip 198.51.100.11
    unicast_peer {
        198.51.100.12
    }

    virtual_ipaddress {
        198.51.100.10/24 dev eth0
    }

    track_script {
        chk_nginx
    }
}

keepalived.conf on lb2

On lb2, copy the same file and change four lines:

    router_id lb2
    priority 100
    unicast_src_ip 198.51.100.12
        198.51.100.11

The last line is the address inside unicast_peer, which on lb2 points back at lb1.

Why these values

• Priority and weight. lb1 runs at 110 and lb2 at 100. When the check fails, weight -20 lowers the failing node's priority by 20. If lb1 fails, it drops to 90, below lb2's 100, and lb2 takes the VIP. If lb2's check fails, it drops to 80 and nothing changes. The weight must be larger than the gap between the two priorities, or a failed check never moves the address.
• Both nodes start as BACKUP. The election decides who becomes MASTER, so starting both as BACKUP avoids a brief period where a freshly booted node claims the VIP before it has heard from its peer. With preemption on (the default), lb1 takes the VIP back once its check passes again. If you would rather the address stay put until you move it by hand, add nopreempt to both instances.
• Unicast peers. Without unicast_peer, VRRP uses multicast to 224.0.0.18. Unicast works on networks that filter multicast and makes the firewall rule precise.
• No vrrp_strict. That option enforces strict RFC compliance. Keepalived's documentation lists unicast peers among the things strict mode does not allow, and it also installs its own firewall drop rules for the VIP. Leave it off for this design.
• No VRRP authentication block. VRRPv2's auth_type PASS sends the password in clear text and VRRPv3 removed authentication entirely. Restrict VRRP with the firewall instead.
• virtual_router_id must be unique on the network segment. Two unrelated clusters using 51 on the same VLAN will fight over each other's state.

Validate the file and start the service on both nodes:

sudo keepalived -t -f /etc/keepalived/keepalived.conf
sudo systemctl enable --now keepalived

Lock down the firewall

Each node should accept web traffic on the VIP, VRRP (IP protocol 112) only from its peer, and SSH only from the admin network. An nftables ruleset for lb1 (swap the peer address on lb2):

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;

        ct state established,related accept
        iif "lo" accept
        ip protocol icmp accept
        meta l4proto ipv6-icmp accept

        ip saddr 198.51.100.12 ip protocol vrrp accept
        ip saddr 203.0.113.0/24 tcp dport 22 accept
        ip daddr 198.51.100.10 tcp dport { 80, 443 } accept
    }
}

Load it with sudo nft -f /etc/nftables.conf after checking it with sudo nft -c -f /etc/nftables.conf. Keep a console session open the first time, in case a typo locks you out of SSH. SSH itself should accept keys only (PasswordAuthentication no), and the deploy key that pushes certificates should be limited to the files it needs.

Prove that failover works

Run a request loop from a client outside the cluster:

while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' --max-time 2 https://www.example.com/
  sleep 0.5
done

Then, one at a time:

1. Confirm lb1 holds the VIP: ip -brief address show dev eth0 should list 198.51.100.10 on lb1 only.
2. Stop NGINX on lb1 with sudo systemctl stop nginx. Within a few seconds, journalctl -u keepalived -f on lb2 should report entering MASTER state, and the VIP should appear on lb2.
3. Start NGINX on lb1 again. Once rise 2 passes, lb1 takes the VIP back.
4. Stop Keepalived on lb2 while it is MASTER, or power it off, to test the whole-node case.

Count the failed or slow lines for each step: that is your real failover time, and the number to write down.

Failure modes and troubleshooting

Checklist

• Same NGINX configuration on both nodes, deployed from version control, and nginx -t passes on each.
• net.ipv4.ip_nonlocal_bind = 1 if NGINX binds the VIP.
• Certificates identical on both nodes, keys mode 600, renewal reloads both.
• Session tickets off, or a shared ticket key that rotates.
• Check script owned by root, not writable by others, run as an unprivileged script_user.
• weight larger than the priority gap; virtual_router_id unique on the segment.
• Unicast peers set, vrrp_strict not set, VRRP accepted only from the peer.
• SSH reachable only from the admin network, keys only.
• Failover tested in both directions, for NGINX failure and whole-node failure, with the measured time recorded.

Closing

A pair of proxies with a floating address removes one obvious single point of failure, but only if the health check reflects real serving ability and the failover has been exercised. Test it on a quiet day so you are not testing it for the first time during an outage.

If you want to see how quickly your site responds through whatever sits in front of it today, run a speed audit.