| 1 |
The Booth Cluster Ticket Manager |
| 2 |
============= |
| 3 |
|
| 4 |
Booth manages tickets which authorize cluster sites located in |
| 5 |
geographically dispersed locations to run resources. It |
| 6 |
facilitates support of geographically distributed clustering in |
| 7 |
Pacemaker. |
| 8 |
|
| 9 |
Booth is based on the Raft consensus algorithm. Though the |
| 10 |
implementation is not complete (there is no log) and there are a |
| 11 |
few additions and modifications, booth guarantees that a ticket |
| 12 |
is always available at just one site as long as it has exclusive |
| 13 |
control of the tickets. |
| 14 |
|
| 15 |
The git repository is available at github: |
| 16 |
|
| 17 |
<https://github.com/ClusterLabs/booth> |
| 18 |
|
| 19 |
github can also track issues or bug reports. |
| 20 |
|
| 21 |
Description of a booth cluster |
| 22 |
============================== |
| 23 |
|
| 24 |
Booth cluster is a collection of cooperative servers |
| 25 |
communicating using the booth protocol. The purpose of the booth |
| 26 |
cluster is to manage cluster tickets. The booth cluster consists |
| 27 |
of at least three servers. |
| 28 |
|
| 29 |
A booth server can be either a site or an arbitrator. Arbitrators |
| 30 |
take part in elections and so help resolve ties, but cannot hold |
| 31 |
tickets. |
| 32 |
|
| 33 |
The basic unit in the booth cluster is a ticket. Every |
| 34 |
non-granted ticket is in the initial state on all servers. For |
| 35 |
granted tickets, the server holding the ticket is the leader and |
| 36 |
other servers are followers. The leader issues heartbeats and |
| 37 |
ticket updates to the followers. The followers are required to |
| 38 |
obey the leader. |
| 39 |
|
| 40 |
Booth startup |
| 41 |
------------ |
| 42 |
|
| 43 |
On startup, the booth process first loads tickets, if available, |
| 44 |
from the CIB. Afterwards, it broadcasts a query to get tickets' |
| 45 |
status from other servers. In-memory copies are updated from |
| 46 |
the replies if they contain newer ticket data. |
| 47 |
|
| 48 |
If the server discovers that itself is the ticket leader, it |
| 49 |
tries to establish its authority again by broadcasting heartbeat. |
| 50 |
If it succeeds, it continues as the leader for this ticket. The |
| 51 |
other booth servers become followers. This procedure is possible |
| 52 |
only immediately after the booth startup. It also serves as a |
| 53 |
configuration reload. |
| 54 |
|
| 55 |
Grant and revoke operations |
| 56 |
------------ |
| 57 |
|
| 58 |
A ticket first has to be granted using the 'booth client grant' |
| 59 |
command. |
| 60 |
|
| 61 |
Obviously, it is not possible to grant a ticket which is |
| 62 |
currently granted. |
| 63 |
|
| 64 |
Ticket revoke is the operation which is the opposite of grant. |
| 65 |
An administrative revoke may be started at any server, but the |
| 66 |
operation itself happens only at the leader. If the leader is |
| 67 |
unreachable, the ticket cannot be revoked. The user will need to |
| 68 |
wait until the ticket expires. |
| 69 |
|
| 70 |
A ticket grant may be delayed if not all sites are reachable. |
| 71 |
The delay is the ticket expiry time extended by acquire-after, if |
| 72 |
set. This is to ensure that the unreachable site relinquished the |
| 73 |
ticket it may have been holding and stopped the corresponding |
| 74 |
cluster resources. |
| 75 |
|
| 76 |
If the user is absolutely sure that the unreachable site does not |
| 77 |
hold the ticket, the delay may be skipped by using the '-F' |
| 78 |
option of the 'booth grant' command. |
| 79 |
|
| 80 |
If in effect, the grant delay time is shown in the 'booth list' |
| 81 |
command output. |
| 82 |
|
| 83 |
Ticket management and server operation |
| 84 |
------------ |
| 85 |
|
| 86 |
A granted ticket is managed by the booth servers so that its |
| 87 |
availability is maximized without breaking the basic guarantee |
| 88 |
that the ticket is granted to one site only. |
| 89 |
|
| 90 |
The server where the ticket is granted is the leader, the other |
| 91 |
servers are followers. The leader occasionally sends heartbeats, |
| 92 |
once every half ticket expiry under normal circumstances. |
| 93 |
|
| 94 |
If a follower doesn't hear from the leader longer than the ticket |
| 95 |
expiry time, it will consider the ticket lost, and try to acquire |
| 96 |
it by starting new elections. |
| 97 |
|
| 98 |
A server starts elections by broadcasting the REQ_VOTE RPC. |
| 99 |
Other servers reply with the VOTE_FOR RPC, in which they record |
| 100 |
its vote. Normally, the sender of the first REQ_VOTE gets the |
| 101 |
vote of the receiver. Whichever server gets a majority of votes |
| 102 |
wins the elections. On ties, elections are restarted. To |
| 103 |
decrease chance of elections ending in a tie, a server waits for a |
| 104 |
short random period before sending out the REQ_VOTE packets. |
| 105 |
Everything else being equal, the server which sends REQ_VOTE |
| 106 |
first gets elected. |
| 107 |
|
| 108 |
Elections are described in more detail in the raft paper at |
| 109 |
<https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf>. |
| 110 |
|
| 111 |
Ticket renewal (or update) is a two-step process. Before actually |
| 112 |
writing the ticket to the CIB, the server holding the ticket |
| 113 |
first tries to establish that it still has the majority for that |
| 114 |
ticket. That is done by broadcasting a heartbeat. If the server |
| 115 |
receives enough acknowledgements, it then stores the ticket to |
| 116 |
the CIB and broadcasts the UPDATE RPC with updated ticket expiry |
| 117 |
time so that the followers can update local ticket copies. Ticket |
| 118 |
renewals are configurable and by default set to half ticket |
| 119 |
expire time. |
| 120 |
|
| 121 |
Before ticket renewal, the leader runs one or more external |
| 122 |
programs if such are set in 'before-acquire-handler'. This can |
| 123 |
point either to a file or a directory. In the former case, that |
| 124 |
file is the program, but in the latter there could be a number of |
| 125 |
programs in the specified directory. All files which have the |
| 126 |
executable bit set and whose names don't start with a '.' are |
| 127 |
run sequentially. This program or programs should ensure that the |
| 128 |
cluster managed service which is protected by this ticket can run |
| 129 |
at this site. If any of them fails, the leader relinquishes the |
| 130 |
ticket. It announces its intention to step down by broadcasting |
| 131 |
an unsolicited VOTE_FOR with an empty vote. On receiving such RPC |
| 132 |
other servers start new elections to elect a new leader. |
| 133 |
|
| 134 |
Split brain |
| 135 |
------------ |
| 136 |
|
| 137 |
On split brains two possible issues arise: leader in minority and |
| 138 |
follower disconnected from the leader. |
| 139 |
|
| 140 |
Let's take a look at the first one. The leader in minority |
| 141 |
eventually expires the ticket because it cannot receieve majority |
| 142 |
of acknowledgements in reply to its heartbeats. The other |
| 143 |
partition runs elections (at about the same time, as they find |
| 144 |
the ticket lost after its expiry) and, if it can get the |
| 145 |
majority, the elections winner becomes a new leader for the |
| 146 |
ticket. After split brain gets resolved, the old leader will |
| 147 |
become follower as soon as it receives heartbeat from the new |
| 148 |
leader. Note the timing: the old leader releases the ticket at |
| 149 |
around the same time as when new elections in the other partition |
| 150 |
are held. This is because the leader ensures that the ticket |
| 151 |
expire time is always the same on all servers in the booth |
| 152 |
cluster. |
| 153 |
|
| 154 |
The second situation, where a follower is disconnected from the |
| 155 |
leader, is a bit more difficult to handle. After the ticket |
| 156 |
expiry time, the follower will consider the ticket lost and start |
| 157 |
new elections. The elections repeatedly get restarted until the |
| 158 |
split brain is resolved. Then, the rest of the cluster send |
| 159 |
rejects in reply to REQ_VOTE RPC because the ticket is still |
| 160 |
valid and therefore couldn't have been lost. They know that |
| 161 |
because the reason for elections is included with every REQ_VOTE. |
| 162 |
|
| 163 |
Short intermittent split brains are handled well because the |
| 164 |
leader keeps resending heartbeats until it gets replies from all |
| 165 |
servers serving sites. |
| 166 |
|
| 167 |
Authentication |
| 168 |
============== |
| 169 |
|
| 170 |
In order to prevent malicious parties from affecting booth |
| 171 |
operation, booth server can authenticate both clients (connecting |
| 172 |
over TCP) and other booth servers (connecting over UDP). The |
| 173 |
authentication is based on SHA1 HMAC (Keyed-Hashing Message |
| 174 |
Authentication) and shared key. The HMAC implementation is |
| 175 |
provided by the libgcrypt or mhash library. |
| 176 |
|
| 177 |
Message encryption is not included as the information exchanged |
| 178 |
between various booth parties does not seem to justify that. |
| 179 |
|
| 180 |
Every message (packet) contains a hash code computed from the |
| 181 |
combination of payload and the secret key. Whoever has the secret |
| 182 |
key can then verify that the message is authentic. |
| 183 |
|
| 184 |
The shared key is used by both the booth client and the booth |
| 185 |
server, hence it needs to be copied to all nodes at each site and |
| 186 |
all arbitrators. Of course, a secure channel is required for key |
| 187 |
transfer. It is recommended to use csync2 or ssh. |
| 188 |
|
| 189 |
Timestamps are included and verified to fend against replay |
| 190 |
attacks. Certain time skew, 10 minutes by default, is tolerated. |
| 191 |
Packets either not older than that or with a timestamp more |
| 192 |
recent than the previous one from the same peer are accepted. The |
| 193 |
time skew can be configured in the booth configuration file. |
| 194 |
|
| 195 |
# vim: set ft=asciidoc : |