Debian Codegraph @

README

raw
1 The Booth Cluster Ticket Manager
2 =============
3
4 Booth manages tickets which authorize cluster sites located in
5 geographically dispersed locations to run resources. It
6 facilitates support of geographically distributed clustering in
7 Pacemaker.
8
9 Booth is based on the Raft consensus algorithm. Though the
10 implementation is not complete (there is no log) and there are a
11 few additions and modifications, booth guarantees that a ticket
12 is always available at just one site as long as it has exclusive
13 control of the tickets.
14
15 The git repository is available at github:
16
17 <https://github.com/ClusterLabs/booth>
18
19 github can also track issues or bug reports.
20
21 Description of a booth cluster
22 ==============================
23
24 Booth cluster is a collection of cooperative servers
25 communicating using the booth protocol. The purpose of the booth
26 cluster is to manage cluster tickets. The booth cluster consists
27 of at least three servers.
28
29 A booth server can be either a site or an arbitrator. Arbitrators
30 take part in elections and so help resolve ties, but cannot hold
31 tickets.
32
33 The basic unit in the booth cluster is a ticket. Every
34 non-granted ticket is in the initial state on all servers. For
35 granted tickets, the server holding the ticket is the leader and
36 other servers are followers. The leader issues heartbeats and
37 ticket updates to the followers. The followers are required to
38 obey the leader.
39
40 Booth startup
41 ------------
42
43 On startup, the booth process first loads tickets, if available,
44 from the CIB. Afterwards, it broadcasts a query to get tickets'
45 status from other servers. In-memory copies are updated from
46 the replies if they contain newer ticket data.
47
48 If the server discovers that itself is the ticket leader, it
49 tries to establish its authority again by broadcasting heartbeat.
50 If it succeeds, it continues as the leader for this ticket. The
51 other booth servers become followers. This procedure is possible
52 only immediately after the booth startup. It also serves as a
53 configuration reload.
54
55 Grant and revoke operations
56 ------------
57
58 A ticket first has to be granted using the 'booth client grant'
59 command.
60
61 Obviously, it is not possible to grant a ticket which is
62 currently granted.
63
64 Ticket revoke is the operation which is the opposite of grant.
65 An administrative revoke may be started at any server, but the
66 operation itself happens only at the leader. If the leader is
67 unreachable, the ticket cannot be revoked. The user will need to
68 wait until the ticket expires.
69
70 A ticket grant may be delayed if not all sites are reachable.
71 The delay is the ticket expiry time extended by acquire-after, if
72 set. This is to ensure that the unreachable site relinquished the
73 ticket it may have been holding and stopped the corresponding
74 cluster resources.
75
76 If the user is absolutely sure that the unreachable site does not
77 hold the ticket, the delay may be skipped by using the '-F'
78 option of the 'booth grant' command.
79
80 If in effect, the grant delay time is shown in the 'booth list'
81 command output.
82
83 Ticket management and server operation
84 ------------
85
86 A granted ticket is managed by the booth servers so that its
87 availability is maximized without breaking the basic guarantee
88 that the ticket is granted to one site only.
89
90 The server where the ticket is granted is the leader, the other
91 servers are followers. The leader occasionally sends heartbeats,
92 once every half ticket expiry under normal circumstances.
93
94 If a follower doesn't hear from the leader longer than the ticket
95 expiry time, it will consider the ticket lost, and try to acquire
96 it by starting new elections.
97
98 A server starts elections by broadcasting the REQ_VOTE RPC.
99 Other servers reply with the VOTE_FOR RPC, in which they record
100 its vote. Normally, the sender of the first REQ_VOTE gets the
101 vote of the receiver. Whichever server gets a majority of votes
102 wins the elections. On ties, elections are restarted. To
103 decrease chance of elections ending in a tie, a server waits for a
104 short random period before sending out the REQ_VOTE packets.
105 Everything else being equal, the server which sends REQ_VOTE
106 first gets elected.
107
108 Elections are described in more detail in the raft paper at
109 <https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf>.
110
111 Ticket renewal (or update) is a two-step process. Before actually
112 writing the ticket to the CIB, the server holding the ticket
113 first tries to establish that it still has the majority for that
114 ticket. That is done by broadcasting a heartbeat. If the server
115 receives enough acknowledgements, it then stores the ticket to
116 the CIB and broadcasts the UPDATE RPC with updated ticket expiry
117 time so that the followers can update local ticket copies. Ticket
118 renewals are configurable and by default set to half ticket
119 expire time.
120
121 Before ticket renewal, the leader runs one or more external
122 programs if such are set in 'before-acquire-handler'. This can
123 point either to a file or a directory. In the former case, that
124 file is the program, but in the latter there could be a number of
125 programs in the specified directory. All files which have the
126 executable bit set and whose names don't start with a '.' are
127 run sequentially. This program or programs should ensure that the
128 cluster managed service which is protected by this ticket can run
129 at this site. If any of them fails, the leader relinquishes the
130 ticket. It announces its intention to step down by broadcasting
131 an unsolicited VOTE_FOR with an empty vote. On receiving such RPC
132 other servers start new elections to elect a new leader.
133
134 Split brain
135 ------------
136
137 On split brains two possible issues arise: leader in minority and
138 follower disconnected from the leader.
139
140 Let's take a look at the first one. The leader in minority
141 eventually expires the ticket because it cannot receieve majority
142 of acknowledgements in reply to its heartbeats. The other
143 partition runs elections (at about the same time, as they find
144 the ticket lost after its expiry) and, if it can get the
145 majority, the elections winner becomes a new leader for the
146 ticket. After split brain gets resolved, the old leader will
147 become follower as soon as it receives heartbeat from the new
148 leader. Note the timing: the old leader releases the ticket at
149 around the same time as when new elections in the other partition
150 are held. This is because the leader ensures that the ticket
151 expire time is always the same on all servers in the booth
152 cluster.
153
154 The second situation, where a follower is disconnected from the
155 leader, is a bit more difficult to handle. After the ticket
156 expiry time, the follower will consider the ticket lost and start
157 new elections. The elections repeatedly get restarted until the
158 split brain is resolved. Then, the rest of the cluster send
159 rejects in reply to REQ_VOTE RPC because the ticket is still
160 valid and therefore couldn't have been lost. They know that
161 because the reason for elections is included with every REQ_VOTE.
162
163 Short intermittent split brains are handled well because the
164 leader keeps resending heartbeats until it gets replies from all
165 servers serving sites.
166
167 Authentication
168 ==============
169
170 In order to prevent malicious parties from affecting booth
171 operation, booth server can authenticate both clients (connecting
172 over TCP) and other booth servers (connecting over UDP). The
173 authentication is based on SHA1 HMAC (Keyed-Hashing Message
174 Authentication) and shared key. The HMAC implementation is
175 provided by the libgcrypt or mhash library.
176
177 Message encryption is not included as the information exchanged
178 between various booth parties does not seem to justify that.
179
180 Every message (packet) contains a hash code computed from the
181 combination of payload and the secret key. Whoever has the secret
182 key can then verify that the message is authentic.
183
184 The shared key is used by both the booth client and the booth
185 server, hence it needs to be copied to all nodes at each site and
186 all arbitrators. Of course, a secure channel is required for key
187 transfer. It is recommended to use csync2 or ssh.
188
189 Timestamps are included and verified to fend against replay
190 attacks. Certain time skew, 10 minutes by default, is tolerated.
191 Packets either not older than that or with a timestamp more
192 recent than the previous one from the same peer are accepted. The
193 time skew can be configured in the booth configuration file.
194
195 # vim: set ft=asciidoc :