How To Use HAProxy to Set Up HTTP Load Balancing on an Ubuntu Server

HAProxy (High Availability Proxy) is an open source load balancer through which you can load balance any TCP service. It is particularly suited for HTTP load balancing as it supports session persistence and layer 7 processing.

Prerequisites:

For this particular guide, we will be using three Servers in our lab:


Server1 - Load Balancer
Hostname: haproxy
OS: Ubuntu
Public IP: 192.168.1.1
Private IP: 172.22.10.100

Server2 - Lamp1
Hostname: lamp1
OS: LAMP on Ubuntu
Private IP: 172.22.10.101

Server3 - Lamp2
Hostname: lamp2
OS: LAMP on Ubuntu
Private IP: 172.22.10.102

Installing HAProxy

Use the apt-get command to install HAProxy.

apt-get install haproxy

We need to enable HAProxy to be started by the init script.

nano /etc/default/haproxy
Set the ENABLED option to 1
ENABLED=1

To check if this change is done properly execute the init script of HAProxy without any parameters. You should see the following.

root@haproxy:~# service haproxy
Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}

Configuring HAProxy

We'll move the default configuration file and create our own one.

mv /etc/haproxy/haproxy.cfg{,.original}

Create and edit a new configuration file:

nano /etc/haproxy/haproxy.cfg

Let us begin by adding configuration block by block to this file:

global
    log 127.0.0.1 local0 notice
    maxconn 2000
    user haproxy
    group haproxy

The log directive mentions a syslog server to which log messages will be sent. On Ubuntu rsyslog is already installed and running but it doesn't listen on any IP address. We'll modify the config files of rsyslog later.

The maxconn directive specifies the number of concurrent connections on the frontend. The default value is 2000 and should be tuned according to your VPS' configuration.

The user and group directives changes the HAProxy process to the specified user/group. These shouldn't be changed.

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    timeout connect  5000
    timeout client  10000
    timeout server  10000

We're specifying default values in this section. The values to be modified are the various timeout directives. The connect option specifies the maximum time to wait for a connection attempt to a VPS to succeed.

The client and server timeouts apply when the client or server is expected to acknowledge or send data during the TCP process. HAProxy recommends setting the client and server timeouts to the same value.

The retries directive sets the number of retries to perform on a VPS after a connection failure.

The option redispatch enables session redistribution in case of connection failures. So session stickness is overriden if a VPS goes down.

listen appname 0.0.0.0:80
    mode http
    stats enable
    stats uri /haproxy?stats
    stats realm Strictly\ Private
    stats auth A_Username:YourPassword
    stats auth Another_User:passwd
    balance roundrobin
    option httpclose
    option forwardfor
    server lamp1 172.22.10.101:80 check
    server lamp2 172.22.10.102:80 check

This contains configuration for both the frontend and backend. We are configuring HAProxy to listen on port 80 for appname which is just a name for identifying an application. The stats directives enable the connection statistics page and protects it with HTTP Basic authentication using the credentials specified by the stats auth directive.

This page can viewed with the URL mentioned in stats uri so in this case, it is http://192.168.1.1/haproxy?stats; a demo of this page can be viewed here.

The balance directive specifies the load balancing algorithm to use. Options available are Round Robin (roundrobin), Static Round Robin (static-rr), Least Connections (leastconn), Source (source), URI (uri) and URL parameter (url_param).

Information about each algorithm can be obtained from the official documentation.

The server directive declares a backend server, the syntax is:

server

[:port] [param*]

The name we mention here will appear in logs and alerts. There are many paratmeters supported by this directive and we'll be using the check and cookie parameters in this article. The check option enables health checks on the VPS otherwise, the VPS is
always considered available.

Once you're done configuring start the HAProxy service:

service haproxy start

Testing Load Balancing and Failover

To test this setup, create a PHP script on all your web servers (backend servers - LAMP1 and LAMP2 here).

/var/www/file.php

header('Content-Type: text/plain');
echo "Server IP: ".$_SERVER['SERVER_ADDR'];
echo "\nClient IP: ".$_SERVER['REMOTE_ADDR'];
echo "\nX-Forwarded-for: ".$_SERVER['HTTP_X_FORWARDED_FOR'];
?>

Now we will use curl and request this file multiple times.

> curl http://192.168.1.1/file.php

Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X

> curl http://192.168.1.1/file.php

Server IP: 172.22.10.102
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X

> curl http://192.168.1.1/file.php

Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X

Notice here how HAProxy alternatively toggled the connection between LAMP1 and LAMP2, this is how Round Robin works. The client IP we see here is the Private IP address of the load balancer and the X-Forwarded-For header is your IP.

To see how failover works, go to a web server and stop the service:

lamp1@haproxy:~#service apache2 stop

Send requests with curl again to see how things work.

Session Stickiness

If your web application serves dynamic content based on users' login sessions (which application doesn't), visitors will experience odd things due to continuous switching between VPS. Session stickiness ensures that a visitor sticks on to the VPS which served their first request. This is possible by tagging each backend server with a cookie.

We'll use the following PHP code to demonstrate how session stickiness works.

/var/www/session.php

header('Content-Type: text/plain');
session_start();
if(!isset($_SESSION['visit']))
{
        echo "This is the first time you're visiting this server";
        $_SESSION['visit'] = 0;
}
else
        echo "Your number of visits: ".$_SESSION['visit'];

$_SESSION['visit']++;

echo "\nServer IP: ".$_SERVER['SERVER_ADDR'];
echo "\nClient IP: ".$_SERVER['REMOTE_ADDR'];
echo "\nX-Forwarded-for: ".$_SERVER['HTTP_X_FORWARDED_FOR']."\n";
print_r($_COOKIE);
?>

This code creates a PHP sesssion and displays the number of page views in a single session.

Cookie insert method

In this method, all responses from HAProxy to the client will contain a Set-Cookie: header with the name of a backend server as its cookie value. So going forward the client (web browser) will include this cookie with all its requests and HAProxy will forward the request to the right backend server based on the cookie value.






For this method, you will need to add the cookie directive and modify the server directives under listen

   cookie SRVNAME insert
   server lamp1 172.22.10.101:80 cookie S1 check
   server lamp2 172.22.10.102:80 cookie S2 check

This causes HAProxy to add a Set-Cookie: header with a cookie named SRVNAME having its value as S1 or S2 based on which backend answered the request. Once this is added restart the service:

service haproxy restart

and use curl to check how this works.

> curl -i http://192.168.1.1/session.php
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 13:11:22 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.8
Set-Cookie: PHPSESSID=l9haakejnvnat7jtju64hmuab5; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 143
Connection: close
Content-Type: text/plain
Set-Cookie: SRVNAME=S1; path=/

This is the first time you're visiting this server
Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X
Array
(
)

This is the first request we made and it was answered by LAMP1 as we can see from Set-Cookie: SRVNAME=S1; path=/. Now, to emulate what a web browser would do for the next request, we add these cookies to the request using the --cookie parameter of curl.

> curl -i http://192.168.1.1/session.php --cookie "PHPSESSID=l9haakejnvnat7jtju64hmuab5;SRVNAME=S1;"
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 13:11:45 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 183
Connection: close
Content-Type: text/plain

Your number of visits: 1
Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.87.127
Array
(
    [PHPSESSID] => l9haakejnvnat7jtju64hmuab5
    [SRVNAME] => S1
)

> curl -i http://192.168.1.1/session.php --cookie "PHPSESSID=l9haakejnvnat7jtju64hmuab5;SRVNAME=S1;"
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 13:11:45 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 183
Connection: close
Content-Type: text/plain

Your number of visits: 2
Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.87.127
Array
(
    [PHPSESSID] => l9haakejnvnat7jtju64hmuab5
    [SRVNAME] => S1
)

Both of these requests were served by LAMP1 and the session was properly maintained. This method is useful if you want stickiness for all files on the web server.
Cookie Prefix Method

On the other hand, if you want stickiness only for specific cookies or don't want to have a separate cookie for session stickiness, the prefix option is for you.

To use this method, use the following cookie directive:

Cookie PHPSESSID prefix

The PHPSESSID can be replaced with your own cookie name. The server directive remains the same as the previous configuration.

Now let's see how this works.

> curl -i http://192.168.1.1/session.php
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 13:36:27 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.8
Set-Cookie: PHPSESSID=S1~6l2pou1iqea4mnhenhkm787o56; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 143
Content-Type: text/plain

This is the first time you're visiting this server
Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X
Array
(
)

Notice how the server cookie S1 is prefixed to the session cookie. Now, let's send two more requests with this cookie.

> curl -i http://192.168.1.1/session.php --cookie "PHPSESSID=S1~6l2pou1iqea4mnhenhkm787o56;"
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 13:36:45 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 163
Content-Type: text/plain

Your number of visits: 1
Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X
Array
(
    [PHPSESSID] => 6l2pou1iqea4mnhenhkm787o56
)

> curl -i http://192.168.1.1/session.php --cookie "PHPSESSID=S1~6l2pou1iqea4mnhenhkm787o56;"
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 13:36:54 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 163
Content-Type: text/plain

Your number of visits: 2
Server IP: 172.22.10.101
Client IP: 172.22.10.100
X-Forwarded-for: 117.213.X.X
Array
(
    [PHPSESSID] => 6l2pou1iqea4mnhenhkm787o56
)

We can clearly see that both the requests were served by LAMP1 and the session is perfectly working.

Configure Logging for HAProxy

When we began configuring HAProxy, we added a line: log 127.0.0.1 local0 notice which sends syslog messages to the localhost IP address. But by default, rsyslog on Ubuntu doesn't listen on any address. So we have to make it do so.

Edit the config file of rsyslog.

nano /etc/rsyslog.conf

Add/Edit/Uncomment the following lines:

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

Now rsyslog will work on UDP port 514 on address 127.0.0.1 but all HAProxy messages will go to /var/log/syslog so we have to separate them.

Create a rule for HAProxy logs.

nano /etc/rsyslog.d/haproxy.conf

Add the following line to it.

if ($programname == 'haproxy') then -/var/log/haproxy.log

Now restart the rsyslog service:

service rsyslog restart

This writes all HAProxy messages and access logs to /var/log/haproxy.log.

Keepalives in HAProxy

Under the listen directive, we used option httpclose which adds a Connection: close header. This tells the client (web browser) to close a connection after a response is received.

If you want to enable keep-alives on HAProxy, replace the option httpclose line with:

option http-server-close
timeout http-keep-alive 3000

Set the keep-alive timeout wisely so that a few connections don't drain all the resources of the load balancer.

Testing Keepalives

Keepalives can be tested using curl by sending multiple requests at the same time. Unnecessary output will be omitted in the following example:

curl -v http://192.168.1.1/index.html http://192.168.1.1/index.html
About to connect() to 192.168.1.1 port 80 (#0)
Trying 192.168.1.1... connected
GET /index.html HTTP/1.1
User-Agent: curl/7.23.1 (x86_64-pc-win32) libcurl/7.23.1 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.1.1
Accept: */*
;
......[Output omitted].........
* Connection #0 to host 192.168.1.1 left intact
* Re-using existing connection! (#0) with host 192.168.1.1
* Connected to 192.168.1.1 (192.168.1.1) port 80 (#0)
GET /index.html HTTP/1.1
User-Agent: curl/7.23.1 (x86_64-pc-win32) libcurl/7.23.1 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.1.1
Accept: */*
.......[Output Omitted].........
* Connection #0 to host 192.168.1.1 left intact
* Closing connection #0





In this output, we have to look for the line: Re-using existing connection! (#0) with host 192.168.1.1, which indicates that curl used the same connection to make subsequent requests.