| 1 |
# Serf Demo: Web Servers + Load Balancer |
| 2 |
|
| 3 |
In this demo, Serf is used to coordinate web servers entering a load |
| 4 |
balancer. The nodes in this example can come up in any order and they'll |
| 5 |
eventually converge to a running cluster. When visiting the load balancer, |
| 6 |
you'll see each of the nodes visited in round-robin. |
| 7 |
|
| 8 |
After telling the cluster to launch, no further human interaction is |
| 9 |
required. The cluster will launch and orchestrate itself into a load |
| 10 |
balanced web server setup. |
| 11 |
|
| 12 |
The nodes are launched on AWS using CloudFormation and therefore will |
| 13 |
incur a real cost. You can control this cost based on how many nodes |
| 14 |
you decide to launch. |
| 15 |
|
| 16 |
## Running the Demo |
| 17 |
|
| 18 |
To run the demo, log into your AWS console, go to the CloudFormation |
| 19 |
tab, and create a new stack using the "cloudformation.json" file in this |
| 20 |
directory. |
| 21 |
|
| 22 |
Configure how many web nodes you'd like to launch then create the stack. |
| 23 |
When the stack is finished creating, the load balancer IP will be in |
| 24 |
the "outputs" tab of that stack. |
| 25 |
|
| 26 |
Visit `http://IP:9999/` to see the HAProxy stats page and list of |
| 27 |
configured backends. Or visit the IP directly in your browser to see |
| 28 |
it round robin between the web nodes. |
| 29 |
|
| 30 |
It will take some time for the load balancer and web servers to install |
| 31 |
their software and configure themselves. If the load balancer stats page |
| 32 |
is not up yet, give it a few seconds and try again. Once it is up, it |
| 33 |
will automatically refresh so you can watch as nodes eventually come |
| 34 |
online. |
| 35 |
|
| 36 |
## How it Works |
| 37 |
|
| 38 |
### Node Setup |
| 39 |
|
| 40 |
All the nodes (webs and load balancer) come up at the same time, with no |
| 41 |
explicit ordering defined. If there is an order, it doesn't actually matter |
| 42 |
for the function of the demo. The web servers can all come up before the load |
| 43 |
balancer, the load balancer can come up in the middle of some web servers, etc. |
| 44 |
|
| 45 |
Each of the nodes does an initial setup using a shell script. In a real |
| 46 |
environment, this initial setup would probably be done with a real |
| 47 |
configuration management system. For the purposes of this demo, we use |
| 48 |
shell scripts for ease of understanding. |
| 49 |
|
| 50 |
The load balancer runs `setup_load_balancer.sh` which installs HAProxy. |
| 51 |
HAProxy is configured with no backend servers. Serf will handle this later. |
| 52 |
|
| 53 |
The web server runs `setup_web_server.sh` which installs Apache and |
| 54 |
sets up an index.html to just say what host it is. |
| 55 |
|
| 56 |
At this stage, the load balancer is running HAProxy, and the web servers |
| 57 |
are running Apache, but they're not aware of each other in any way. |
| 58 |
|
| 59 |
Next, all nodes run `setup_serf.sh` which installs Serf and configures it |
| 60 |
to run. This script also starts the Serf agent. For web servers, it also |
| 61 |
starts a task that continuosly tries to join the Serf cluster at address |
| 62 |
`10.0.0.5`, which happens to be the static IP assigned to the load balancer. |
| 63 |
Note that Serf can join _any_ node in the cluster. We just set a static IP |
| 64 |
on the load balancer for ease of automation, but in a real production |
| 65 |
environment you can use any existing node, potentially just using DNS. |
| 66 |
|
| 67 |
Web server roles also make use of a "serf-query" upstart task to dynamically |
| 68 |
update the index page they serve. By using a "load" query, each node asks |
| 69 |
the rest of the cluster for their current load. This information is used |
| 70 |
to dynamically generate the index page. |
| 71 |
|
| 72 |
### Serf Configuration |
| 73 |
|
| 74 |
The Serf configuration is very simple. Each node runs a Serf agent. The |
| 75 |
web servers run the agent with a role of "web" and the load balancer runs |
| 76 |
the agent with a role of "lb". |
| 77 |
|
| 78 |
#### Event: member-join |
| 79 |
|
| 80 |
On member-join, the following shell script is run. We'll talk about the |
| 81 |
function of the shell script after the code sample. |
| 82 |
|
| 83 |
```sh |
| 84 |
if [ "x${SERF_TAG_ROLE}" != "xlb" ]; then |
| 85 |
echo "Not an lb. Ignoring member join." |
| 86 |
exit 0 |
| 87 |
fi |
| 88 |
|
| 89 |
while read line; do |
| 90 |
ROLE=`echo $line | awk '{print \$3 }'` |
| 91 |
if [ "x${ROLE}" != "xweb" ]; then |
| 92 |
continue |
| 93 |
fi |
| 94 |
|
| 95 |
echo $line | \ |
| 96 |
awk '{ printf " server %s %s check\n", $1, $2 }' >>/etc/haproxy/haproxy.cfg |
| 97 |
done |
| 98 |
|
| 99 |
/etc/init.d/haproxy reload |
| 100 |
``` |
| 101 |
|
| 102 |
The script first checks if this node is a load balancer. If it isn't a load |
| 103 |
balancer, we don't do anything on the "member-join" event. Web servers simply |
| 104 |
don't care about other members. |
| 105 |
|
| 106 |
Otherwise, if this is a load balancer, then it handles the "member-join" event |
| 107 |
and reads the event data and updates the configuration of HAProxy. In a |
| 108 |
production environment, you might use something like a template processor to do |
| 109 |
this rather than appending to the configuration. For the purposes of this demo, |
| 110 |
this works as well. |
| 111 |
|
| 112 |
Finally, HAProxy is reloaded so the configuration is read. |
| 113 |
|
| 114 |
The result of this is that as members join, they're automatically added into |
| 115 |
rotation on the load balancer. |
| 116 |
|
| 117 |
#### Event: member-leave, member-failed |
| 118 |
|
| 119 |
On member-leave and member-failed evets, the following shell script is run. |
| 120 |
This demo doesn't differentiate between the two events, treating both |
| 121 |
the same way and simply removing the node from the HAProxy configuration. |
| 122 |
|
| 123 |
```sh |
| 124 |
if [ "x${SERF_TAG_ROLE}" != "xlb" ]; then |
| 125 |
echo "Not an lb. Ignoring member leave" |
| 126 |
exit 0 |
| 127 |
fi |
| 128 |
|
| 129 |
while read line; do |
| 130 |
NAME=`echo $line | awk '{print \$1 }'` |
| 131 |
sed -i'' "/${NAME} /d" /etc/haproxy/haproxy.cfg |
| 132 |
done |
| 133 |
|
| 134 |
/etc/init.d/haproxy reload |
| 135 |
``` |
| 136 |
|
| 137 |
This script also does nothing if this node is not a load balanacer. Otherwise, |
| 138 |
it uses `sed` to remove each of the nodes from the HAProxy configuration and |
| 139 |
reloads it. |