haproxy – HA Proxy – Close connections gracefully on servers

We encounter an obstacle in our implementation process. By removing servers from rotation, persistent connections are eliminated. We are currently using cookie-based persistence. We would like the connections to move gracefully to another server, instead of dropping.

This is what we are doing to eliminate a rotation server:

  1. Change the status of the server to DRAIN (through the socat command). This command does not allow new connections on the server, however, persistent connections are still affecting our server.
  2. Change the content from "health.html" to "DOWN". This marks the server as "DOWN", but all connections are cut off and users are returned to another server.

We cannot determine the missing step between # 1 and # 2. We have tried the following:

  • Entering the "MAINT" status
  • Set maxconn value on a server to -1
  • Rename the file "health.html" instead of changing the content. This causes the server to be marked as "NOLB"

Does anybody have any suggestions?

Below is the HA proxy configuration

global
    maxconn 30000
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    nbthread 48

    tune.bufsize 32768
    tune.ssl.cachesize 30000
    tune.ssl.lifetime  600

    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

    stats socket ipv4@127.0.0.1:9999 level admin
    stats socket /var/run/haproxy.sock mode 666 level admin

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 121000
        timeout client  121000
        timeout server  121000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend fe_main
        bind :80
        bind :443 ssl crt /etc/cc-ssl/(redacted).pem crt /etc/cc-ssl/(redacted).pem
        reqadd X-Forwarded-Proto: https

        http-request redirect scheme https unless { ssl_fc }

        default_backend be-https

frontend stats
        bind *:8404
        stats enable
        stats uri /stats

backend be-https
        balance roundrobin
        cookie NUMID insert indirect nocache
        option httpchk GET /health.html HTTP/1.1rnHost: www
        http-check disable-on-404
        http-check expect string UP
        default-server inter 3s fall 2 rise 2 slowstart 5m
        server s1 10.10.10.1:443 ssl verify none check cookie 1
        server s2 10.10.10.2:443 ssl verify none check cookie 2
        server s3 10.10.10.3:443 ssl verify none check cookie 3
        server s4 10.10.10.4:443 ssl verify none check cookie 4