Enabling HTTPS using HAProxy and Certbot with Docker

HAProxy is a free, reliable, and high performant solution offering high availability, load balancing, and proxying for TCP/IP or HTTP-based applications. It has become the de-facto standard open-source load balancer. It is suitable for load-balancing high traffic requests to your cluster of back-end applications.

Certbot is a free and open-source tool for automatically using Let’s Encrypt certificates on manually-administrated websites to enable HTTPS. Certbot is created by Electronic Frontier Foundation (EFF).

Together, HAProxy and Certbot can be used to manage SSL certificates from Let’s Encrypt and enable HTTPS for your web servers. You can also automatically renew the certificate by scheduling the execution of the update script using cron. Running both HAProxy and cron in a Docker container is possible by utilizing the Supervisor process controller.

I write this to document the entire process of how I enabled HTTPS on my project. I hope this can help people that walk the same path as I did.

I used to use xalauc/haproxy-certbot Docker image available on Docker Hub which source is hosted at this BitBucket repository on the development server. HTTPS is not enabled on the dev server, so everything works perfectly. Things are fine until I need to move to the staging and production server.

I was unable to get the SSL certificates from Let’s Encrypt using the provided scripts in the container because the base image (Alpine Linux 3.6) still uses Python 2.7 as the default. So, I need to update the base image.

I used the 3.14 tag for the base image, and things are just as I expected. Docker failed to build the image due to HAProxy source compilation errors. I’m not quite sure about the cause, maybe because of the configuration or even the compiler itself. Anyway, I decided to use the latest version of HAProxy.

I made modifications to the Dockerfile especially the variables that hold the HAProxy version and the MD5 hash of the tarball file. The variables are used by the RUN scripts to download the tarball and verify its checksum.

I also changed the line that sets the option for the HAProxy compilation process.

FROM alpine:3.14
ENV HAPROXY_MAJOR 2.4
ENV HAPROXY_VERSION 2.4.4
ENV HAPROXY_MD5 19cd1f31f0e45b72f19d070fa25517b0
...
&& makeOpts=' \
TARGET=linux-musl \
USE_OPENSSL=1 \
USE_PCRE=1 PCREDIR= \
USE_ZLIB=1 \
' \

I changed the TARGET value to linux-musl from linux2628 because apparently linux2628 is no longer supported by the HAProxy version 2.4.4.

Docker was able to build the image after the previous modifications. But, the haproxy process keeps crashing, and there’s no clue from the container log.

INFO exited: haproxy (exit status 1; not expected)
INFO spawned: 'haproxy' with pid 502
INFO success: haproxy entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)

So I run HAProxy directly instead of using Supervisor and got the following error:

[ALERT]    (646) : parsing [/config/haproxy.cfg:62] : The 'reqadd' directive is not supported anymore since HAProxy 2.1. Use 'http-request add-header' instead.

This error is caused because of these lines:

reqadd X-Forwarded-Proto:\ http
...
reqadd X-Forwarded-Porot:\ https

I need to change it to this:

http-request add-header X-Forwarded-Proto http
...
http-request add-header X-Forwarded-Proto https

Also, I changed the supervisord.conf file to add the user configuration to remove the warning from Supervisor.

[supervisord]
user=root
nodaemon = true

You might want to use another user instead of using root.

I rebuild the image because I changed the supervisord.conf file and now the HAProxy runs smoothly. I tested the certbot script using the following command but it returned an error

docker exec haproxy-certbot certbot-certonly --domain example.com --email my-email@redacted.com --dry run

Here’s the error:

usage: 
  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...
Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. 
certbot: error: unrecognized arguments: --tls-sni-01-port=8443 --standalone-supported-challenges=http-01

Apparently, the --tls-sni-01-port flag and the is no longer supported, and the --standalone-supported-challenges is changed to --preferred-challenges. Both of these flags are provided in the cli.ini file. So, I changed that file to remove the unsupported flags.

Here’s the content of the cli.ini file:

authenticator = standalone
agree-tos = True
http-01-port = 8080
non-interactive = True

I run the same docker exec command and it worked! I removed the --dry-run flag to actually get the real certificate from Let’s Encrypt.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for REDACTED_DOMAIN.com
The dry run was successful.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

So, Both HAProxy and Certbot work now, but the HTTPS is not yet enabled. I need to merge the full chain certificate and private key file from Let’s Encrypt and store the result in a file with the domain as the file name in the /usr/local/etc/haproxy/certs.d directory according to the haproxy.cfg configuration file.

frontend https_in
  bind *:${HTTPS_PORT} ssl crt /usr/local/etc/haproxy/default.pem crt /usr/local/etc/haproxy/certs.d ciphers ECDHE-RSA-AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM
  option forwardfor
  http-request add-header X-Forwarded-Proto https
  default_backend website-backend

I run the following commands in the container’s shell to achieve that goal

cd /etc/letsencrypt/live/REDACTED_DOMAIN
cat fullchain.pem privkey.pem > /usr/local/etc/haproxy/certs.d/REDACTED_DOMAIN

The full chain certificate is stored in the fullchain.pem file, meanwhile the private key is stored in the privkey.pem file.

This can also be achieved by running the haproxy-refresh.sh script file in the container manually, or wait until that script is run by cron at 12 AM or PM.

I restarted the container, and now the HTTPS works!