Debian Codegraph @

UG9/en/troubleshooting.adoc

raw
1 [[ch-troubleshooting]]
2 == Troubleshooting and error recovery
3
4 This chapter describes tasks to be performed in the event of hardware
5 or system failures.
6
7 [[s-hard-drive-failure]]
8
9 === Dealing with hard drive failure
10
11 indexterm:[drive failure]indexterm:[disk failure]How to deal with hard
12 drive failure depends on the way DRBD is configured to handle disk I/O
13 errors (see <<s-handling-disk-errors>>), and on the type of meta data
14 configured (see <<s-metadata>>).
15
16 [NOTE]
17 ==================
18 For the most part, the steps described here apply only if you
19 run DRBD directly on top of physical hard drives. They generally do
20 not apply in case you are running DRBD layered on top of
21
22 * an MD software RAID set (in this case, use `mdadm` to manage drive
23 replacement),
24 * device-mapper RAID (use `dmraid`),
25 * a hardware RAID appliance (follow the vendor's instructions on how
26 to deal with failed drives),
27 * some non-standard device-mapper virtual block devices (see the
28 device mapper documentation).
29 ==================
30
31 [[s-detach-hard-drive-manual]]
32 ==== Manually detaching DRBD from your hard drive
33
34 indexterm:[drbdadm, detach]If DRBD is <<fp-io-error-pass-on,configured to pass
35 on I/O errors>> (not recommended), you must first detach the DRBD
36 resource, that is, disassociate it from its backing storage:
37
38 ----------------------------
39 # drbdadm detach <resource>
40 ----------------------------
41
42 By running the `drbdadm status` or the `drbdadm dstate` command, you will now be able to verify
43 that the resource is now in indexterm:[diskless
44 mode]indexterm:[diskless (disk state)]indexterm:[disk state, Diskless]_diskless
45 mode_:
46
47 ----------------------------
48 # drbdadm status <resource>
49 <resource> role:Primary
50 volume:0 disk:Diskless
51 <peer> role:Secondary
52 volume:0 peer-disk:UpToDate
53 # drbdadm dstate <resource>
54 Diskless/UpToDate
55 ----------------------------
56
57 If the disk failure has occurred on your primary node, you may combine
58 this step with a switch-over operation.
59
60 [[s-detach-hard-drive-auto]]
61 ==== Automatic detach on I/O error
62
63 If DRBD is <<fp-io-error-detach,configured to automatically detach
64 upon I/O error>> (the recommended option), DRBD should have
65 automatically detached the resource from its backing storage already,
66 without manual intervention. You may still use the `drbdadm status`
67 command to verify that the resource is in fact running in diskless
68 mode.
69
70 [[s-replace-disk-internal-metadata]]
71 ==== Replacing a failed disk when using internal meta data
72
73 If using <<s-internal-meta-data,internal meta data>>, it is sufficient
74 to bind the DRBD device to the new hard disk. If the new hard disk has
75 to be addressed by another Linux device name than the defective disk,
76 the DRBD configuration file has to be modified accordingly.
77
78 This process involves creating a new meta data set, then re-attaching
79 the resource: indexterm:[drbdadm, create-md]
80
81 ----------------------------
82 # drbdadm create-md <resource>
83 v08 Magic number not found
84 Writing meta data...
85 initialising activity log
86 NOT initializing bitmap
87 New drbd meta data block successfully created.
88
89 # drbdadm attach <resource>
90 ----------------------------
91
92 Full synchronization of the new hard disk starts instantaneously and
93 automatically. You will be able to monitor the synchronization's
94 progress via `drbdadm status --verbose`, as with any background synchronization.
95
96 [[s-replace-disk-external-metadata]]
97 ==== Replacing a failed disk when using external meta data
98
99 When using <<s-external-meta-data,external meta data>>, the procedure
100 is basically the same. However, DRBD is not able to recognize
101 independently that the hard drive was swapped, thus an additional step
102 is required.
103
104 ----------------------------
105 # drbdadm create-md <resource>
106 v08 Magic number not found
107 Writing meta data...
108 initialising activity log
109 NOT initializing bitmap
110 New drbd meta data block successfully created.
111
112 # drbdadm attach <resource>
113 # drbdadm invalidate <resource>
114 ----------------------------
115
116 WARNING: Make sure to run `drbdadm invalidate` on the node __*without*__ good data;
117 this command will cause the local contents to be overwritten with data from the
118 peers, so running this command on the wrong node might lose data!
119
120 Here, the `drbdadm invalidate` command triggers synchronization. Again,
121 sync progress may be observed via `drbdadm status --verbose`.
122
123 [[s-node-failure]]
124 === Dealing with node failure
125
126 indexterm:[node failure]When DRBD detects that its peer node is down
127 (either by true hardware failure or manual intervention), DRBD changes
128 its connection state from _Connected_ to _Connecting_ and waits for
129 the peer node to re-appear. The DRBD resource is then said to operate
130 in _disconnected mode_. In disconnected mode, the resource and its
131 associated block device are fully usable, and may be promoted and
132 demoted as necessary, but no block modifications are being replicated
133 to the peer node. Instead, DRBD stores which
134 blocks are being modified while disconnected, on a per-peer basis.
135
136 [[s-temp-node-failure-secondary]]
137 ==== Dealing with temporary secondary node failure
138
139 indexterm:[node failure]If a node that currently has a resource in the
140 secondary role fails temporarily (due to, for example, a memory
141 problem that is subsequently rectified by replacing RAM), no further
142 intervention is necessary — besides the obvious necessity to repair
143 the failed node and bring it back online. When that happens, the two
144 nodes will simply re-establish connectivity upon system
145 start-up. After this, DRBD synchronizes all modifications made on the
146 primary node in the meantime to the secondary node.
147
148 IMPORTANT: At this point, due to the nature of DRBD's
149 re-synchronization algorithm, the resource is briefly inconsistent on
150 the secondary node. During that short time window, the secondary node
151 can not switch to the Primary role if the peer is unavailable. Thus,
152 the period in which your cluster is not redundant consists of the
153 actual secondary node down time, plus the subsequent
154 re-synchronization.
155
156 Please note that with DRBD 9 more than two nodes can be connected for each
157 resource, so for e.g. 4 nodes a single failing secondary still leaves two other
158 secondaries available for failover.
159
160 [[s-temp-node-failure-primary]]
161 ==== Dealing with temporary primary node failure
162
163 indexterm:[node failure]From DRBD's standpoint, failure of the primary
164 node is almost identical to a failure of the secondary node. The
165 surviving node detects the peer node's failure, and switches to
166 disconnected mode. DRBD does _not_ promote the surviving node to the
167 primary role; it is the cluster management application's
168 responsibility to do so.
169
170 When the failed node is repaired and returns to the cluster, it does
171 so in the secondary role, thus, as outlined in the previous section,
172 no further manual intervention is necessary. Again, DRBD does not
173 change the resource role back, it is up to the cluster manager to do
174 so (if so configured).
175
176 DRBD ensures block device consistency in case of a primary node
177 failure by way of a special mechanism. For a detailed discussion,
178 refer to <<s-activity-log>>.
179
180 [[s-perm-node-failure]]
181 ==== Dealing with permanent node failure
182
183 indexterm:[node failure]If a node suffers an unrecoverable problem or
184 permanent destruction, you must follow the following steps:
185
186 * Replace the failed hardware with one with similar performance and
187 disk capacity.
188 +
189 NOTE: Replacing a failed node with one with worse performance
190 characteristics is possible, but not recommended. Replacing a failed
191 node with one with less disk capacity is not supported, and will cause
192 DRBD to refuse to connect to the replaced nodefootnote:[It couldn't replicate
193 the data, anyway!].
194
195 * Install the base system and applications.
196 * Install DRBD and copy `/etc/drbd.conf` and all of `/etc/drbd.d/`
197 from one of the surviving nodes.
198 * Follow the steps outlined in <<ch-configure>>, but stop short
199 of <<s-initial-full-sync>>.
200
201 Manually starting a full device synchronization is not necessary at
202 this point, it will commence automatically upon connection to the
203 surviving primary and/or secondary node(s).
204
205 [[s-resolve-split-brain]]
206 === Manual split brain recovery
207
208 indexterm:[split brain]DRBD detects split brain at the time
209 connectivity becomes available again and the peer nodes exchange the
210 initial DRBD protocol handshake. If DRBD detects that both nodes are
211 (or were at some point, while disconnected) in the primary role, it
212 immediately tears down the replication connection. The tell-tale sign
213 of this is a message like the following appearing in the system log:
214
215 ----------------------------
216 Split-Brain detected, dropping connection!
217 ----------------------------
218
219 After split brain has been detected, one node will always have the
220 resource in a indexterm:[StandAlone (connection
221 state)]indexterm:[connection state]_StandAlone_ connection state. The
222 other might either also be in the _StandAlone_ state (if both nodes
223 detected the split brain simultaneously), or in
224 indexterm:[Connecting (connection state)] indexterm:[connection
225 state]_Connecting_ (if the peer tore down the connection before the
226 other node had a chance to detect split brain).
227
228 At this point, unless you configured DRBD to automatically recover
229 from split brain, you must manually intervene by selecting one node
230 whose modifications will be discarded (this node is referred to as the
231 indexterm:[split brain]_split brain victim_). This intervention is
232 made with the following commands:
233
234 [WARNING]
235 ================
236 This is still _work in progress_.
237
238 Expect rough edges and changes.
239
240 ================
241
242 /////////////////
243 [NOTE]
244 ===========================
245 The split brain victim needs to be in the connection state of
246 _StandAlone_ or the following commands will return an error.
247 You can ensure it is standalone by issuing:
248
249 ----------
250 # drbdadm disconnect <resource>
251 ----------
252 ===========================
253 /////////////////
254
255
256 ----------------------------
257 # drbdadm disconnect <resource>
258 # drbdadm secondary <resource>
259 # drbdadm connect --discard-my-data <resource>
260 ----------------------------
261
262 On the other node (the indexterm:[split brain]_split brain survivor_),
263 if its connection state is also _StandAlone_, you would enter:
264
265 ----------------------------
266 # drbdadm disconnect <resource>
267 # drbdadm connect <resource>
268 ----------------------------
269
270 You may omit this step if the node is already in the
271 _Connecting_ state; it will then reconnect automatically.
272
273 Upon connection, your split brain victim immediately changes its
274 connection state to _SyncTarget_, and gets its modifications
275 overwritten by the other node(s).
276
277 NOTE: The split brain victim is not subjected to a full device
278 synchronization. Instead, it has its local modifications rolled back,
279 and any modifications made on the split brain survivor(s) propagate to
280 the victim.
281
282 After re-synchronization has completed, the split brain is considered
283 resolved and the nodes form a fully consistent, redundant
284 replicated storage system again.