Debian Codegraph @

src/backend/storage/aio/README.md

raw
1 # Asynchronous & Direct IO
2
3 ## Motivation
4
5 ### Why Asynchronous IO
6
7 Until the introduction of asynchronous IO postgres relied on the operating
8 system to hide the cost of synchronous IO from postgres. While this worked
9 surprisingly well in a lot of workloads, it does not do as good a job on
10 prefetching and controlled writeback as we would like.
11
12 There are important expensive operations like `fdatasync()` where the operating
13 system cannot hide the storage latency. This is particularly important for WAL
14 writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
15 writes can yield significantly higher throughput.
16
17
18 ### Why Direct / unbuffered IO
19
20 The main reasons to want to use Direct IO are:
21
22 - Lower CPU usage / higher throughput. Particularly on modern storage buffered
23 writes are bottlenecked by the operating system having to copy data from the
24 kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
25 can often move the data directly between the storage devices and postgres'
26 buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
27 perform other work.
28 - Reduced latency - Direct IO can have substantially lower latency than
29 buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
30 write latency.
31 - Avoiding double buffering between operating system cache and postgres'
32 shared_buffers.
33 - Better control over the timing and pace of dirty data writeback.
34
35
36 The main reasons *not* to use Direct IO are:
37
38 - Without AIO, Direct IO is unusably slow for most purposes.
39 - Even with AIO, many parts of postgres need to be modified to perform
40 explicit prefetching.
41 - In situations where shared_buffers cannot be set appropriately large,
42 e.g. because there are many different postgres instances hosted on shared
43 hardware, performance will often be worse than when using buffered IO.
44
45
46 ## AIO Usage Example
47
48 In many cases code that can benefit from AIO does not directly have to
49 interact with the AIO interface, but can use AIO via higher-level
50 abstractions. See [Helpers](#helpers).
51
52 In this example, a buffer will be read into shared buffers.
53
54 ```C
55 /*
56 * Result of the operation, only to be accessed in this backend.
57 */
58 PgAioReturn ioret;
59
60 /*
61 * Acquire an AIO Handle, ioret will get result upon completion.
62 *
63 * Note that ioret needs to stay alive until the IO completes or
64 * CurrentResourceOwner is released (i.e. an error is thrown).
65 */
66 PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
67
68 /*
69 * Reference that can be used to wait for the IO we initiate below. This
70 * reference can reside in local or shared memory and waited upon by any
71 * process. An arbitrary number of references can be made for each IO.
72 */
73 PgAioWaitRef iow;
74
75 pgaio_io_get_wref(ioh, &iow);
76
77 /*
78 * Arrange for shared buffer completion callbacks to be called upon completion
79 * of the IO. This callback will update the buffer descriptors associated with
80 * the AioHandle, which e.g. allows other backends to access the buffer.
81 *
82 * A callback can be passed a small bit of data, e.g. to indicate whether to
83 * zero a buffer if it is invalid.
84 *
85 * Multiple completion callbacks can be registered for each handle.
86 */
87 pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
88
89 /*
90 * The completion callback needs to know which buffers to update when the IO
91 * completes. As the AIO subsystem does not know about buffers, we have to
92 * associate this information with the AioHandle, for use by the completion
93 * callback registered above.
94 *
95 * In this example we're reading only a single buffer, hence the 1.
96 */
97 pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
98
99 /*
100 * Pass the AIO handle to lower-level function. When operating on the level of
101 * buffers, we don't know how exactly the IO is performed, that is the
102 * responsibility of the storage manager implementation.
103 *
104 * E.g. md.c needs to translate block numbers into offsets in segments.
105 *
106 * Once the IO handle has been handed off to smgrstartreadv(), it may not
107 * further be used, as the IO may immediately get executed below
108 * smgrstartreadv() and the handle reused for another IO.
109 *
110 * To issue multiple IOs in an efficient way, a caller can call
111 * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
112 * with pgaio_exit_batchmode(). Note that one needs to be careful while there
113 * may be unsubmitted IOs, as another backend may need to wait for one of the
114 * unsubmitted IOs. If this backend then had to wait for the other backend,
115 * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
116 * details.
117 *
118 * Note that even while in batchmode an IO might get submitted immediately,
119 * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
120 * complete before smgrstartreadv() returns.
121 */
122 void *page = BufferGetBlock(buffer);
123 smgrstartreadv(ioh, operation->smgr, forknum, blkno,
124 &page, 1);
125
126 /*
127 * To benefit from AIO, it is beneficial to perform other work, including
128 * submitting other IOs, before waiting for the IO to complete. Otherwise
129 * we could just have used synchronous, blocking IO.
130 */
131 perform_other_work();
132
133 /*
134 * We did some other work and now need the IO operation to have completed to
135 * continue.
136 */
137 pgaio_wref_wait(&iow);
138
139 /*
140 * At this point the IO has completed. We do not yet know whether it succeeded
141 * or failed, however. The buffer's state has been updated, which allows other
142 * backends to use the buffer (if the IO succeeded), or retry the IO (if it
143 * failed).
144 *
145 * Note that in case the IO has failed, a LOG message may have been emitted,
146 * but no ERROR has been raised. This is crucial, as another backend waiting
147 * for this IO should not see an ERROR.
148 *
149 * To check whether the operation succeeded, and to raise an ERROR, or if more
150 * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
151 */
152 if (ioret.result.status == PGAIO_RS_ERROR)
153 pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
154
155 /*
156 * Besides having succeeded completely, the IO could also have a) partially
157 * completed or b) succeeded with a warning (e.g. due to zero_damaged_pages).
158 * If we e.g. tried to read many blocks at once, the read might have
159 * only succeeded for the first few blocks.
160 *
161 * If the IO partially succeeded and this backend needs all blocks to have
162 * completed, this backend needs to reissue the IO for the remaining buffers.
163 * The AIO subsystem cannot handle this retry transparently.
164 *
165 * As this example is already long, and we only read a single block, we'll just
166 * error out if there's a partial read or a warning.
167 */
168 if (ioret.result.status != PGAIO_RS_OK)
169 pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
170
171 /*
172 * The IO succeeded, so we can use the buffer now.
173 */
174 ```
175
176
177 ## Design Criteria & Motivation
178
179 ### Deadlock and Starvation Dangers due to AIO
180
181 Using AIO in a naive way can easily lead to deadlocks in an environment where
182 the source/target of AIO are shared resources, like pages in postgres'
183 shared_buffers.
184
185 Consider one backend performing readahead on a table, initiating IO for a
186 number of buffers ahead of the current "scan position". If that backend then
187 performs some operation that blocks, or even just is slow, the IO completion
188 for the asynchronously initiated read may not be processed.
189
190 This AIO implementation solves this problem by requiring that AIO methods
191 either allow AIO completions to be processed by any backend in the system
192 (e.g. io_uring), or to guarantee that AIO processing will happen even when the
193 issuing backend is blocked (e.g. worker mode, which offloads completion
194 processing to the AIO workers).
195
196
197 ### IO can be started in critical sections
198
199 Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
200
201 - AIO allows to start WAL writes eagerly, so they complete before needing to
202 wait
203 - AIO allows to have multiple WAL flushes in progress at the same time
204 - AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
205 the number of roundtrips to storage on some OSs and storage HW (buffered IO
206 and direct IO without O_DSYNC needs to issue a write and after the write's
207 completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
208 Force Unit Access (FUA) write).
209
210 The need to be able to execute IO in critical sections has substantial design
211 implication on the AIO subsystem. Mainly because completing IOs (see prior
212 section) needs to be possible within a critical section, even if the
213 to-be-completed IO itself was not issued in a critical section. Consider
214 e.g. the case of a backend first starting a number of writes from shared
215 buffers and then starting to flush the WAL. Because only a limited amount of
216 IO can be in-progress at the same time, initiating IO for flushing the WAL may
217 require to first complete IO that was started earlier.
218
219
220 ### State for AIO needs to live in shared memory
221
222 Because postgres uses a process model and because AIOs need to be
223 complete-able by any backend much of the state of the AIO subsystem needs to
224 live in shared memory.
225
226 In an `EXEC_BACKEND` build, a backend's executable code and other process
227 local state is not necessarily mapped to the same addresses in each process
228 due to ASLR. This means that the shared memory cannot contain pointers to
229 callbacks.
230
231
232 ## Design of the AIO Subsystem
233
234
235 ### AIO Methods
236
237 To achieve portability and performance, multiple methods of performing AIO are
238 implemented and others are likely worth adding in the future.
239
240
241 #### Synchronous Mode
242
243 `io_method=sync` does not actually perform AIO but allows to use the AIO API
244 while performing synchronous IO. This can be useful for debugging. The code
245 for the synchronous mode is also used as a fallback by e.g. the [worker
246 mode](#worker) uses it to execute IO that cannot be executed by workers.
247
248
249 #### Worker
250
251 `io_method=worker` is available on every platform postgres runs on, and
252 implements asynchronous IO - from the view of the issuing process - by
253 dispatching the IO to one of several worker processes performing the IO in a
254 synchronous manner.
255
256
257 #### io_uring
258
259 `io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
260 dispatches all IO from within the process, lowering context switch rate /
261 latency.
262
263
264 ### AIO Handles
265
266 The central API piece for postgres' AIO abstraction are AIO handles. To
267 execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
268 then "define" it, i.e. associate an IO operation with the handle.
269
270 Often AIO handles are acquired on a higher level and then passed to a lower
271 level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
272 routines acquire the handle, which is then passed through smgr.c, md.c to be
273 finally fully defined in fd.c.
274
275 The functions used at the lowest level to define the operation are
276 `pgaio_io_start_*()`.
277
278 Because acquisition of an IO handle
279 [must always succeed](#io-can-be-started-in-critical-sections)
280 and the number of AIO Handles
281 [has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
282 AIO handles can be reused as soon as they have completed. Obviously code needs
283 to be able to react to IO completion. State can be updated using
284 [AIO Completion callbacks](#aio-callbacks)
285 and the issuing backend can provide a backend local variable to receive the
286 result of the IO, as described in
287 [AIO Result](#aio-results).
288 An IO can be waited for, by both the issuing and any other backend, using
289 [AIO References](#aio-wait-references).
290
291
292 Because an AIO Handle is not executable just after calling
293 `pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
294 (absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
295 `pgaio_io_acquire()`) without causing the IO to have been defined (by,
296 potentially indirectly, causing `pgaio_io_start_*()` to have been
297 called). Otherwise a backend could trivially self-deadlock by using up all AIO
298 Handles without the ability to wait for some of the IOs to complete.
299
300 If it turns out that an AIO Handle is not needed, e.g., because the handle was
301 acquired before holding a contended lock, it can be released without being
302 defined using `pgaio_io_release()`.
303
304
305 ### AIO Callbacks
306
307 Commonly several layers need to react to completion of an IO. E.g. for a read
308 md.c needs to check if the IO outright failed or was shorter than needed,
309 bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
310 BufferDesc to update the buffer's state.
311
312 The fact that several layers / subsystems need to react to IO completion poses
313 a few challenges:
314
315 - Upper layers should not need to know details of lower layers. E.g. bufmgr.c
316 should not assume the IO will pass through md.c. Therefore upper levels
317 cannot know what lower layers would consider an error.
318
319 - Lower layers should not need to know about upper layers. E.g. smgr APIs are
320 used going through shared buffers but are also used bypassing shared
321 buffers. This means that e.g. md.c is not in a position to validate
322 checksums.
323
324 - Having code in the AIO subsystem for every possible combination of layers
325 would lead to a lot of duplication.
326
327 The "solution" to this is the ability to associate multiple completion
328 callbacks with a handle. E.g. bufmgr.c can have a callback to update the
329 BufferDesc state and to verify the page and md.c can have another callback to
330 check if the IO operation was successful.
331
332 As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
333 currently cannot contain function pointers. Because of that completion
334 callbacks are not directly identified by function pointers but by IDs
335 (`PgAioHandleCallbackID`). A substantial added benefit is that that
336 allows callbacks to be identified by much smaller amount of memory (a single
337 byte currently).
338
339 In addition to completion, AIO callbacks also are called to "stage" an
340 IO. This is, e.g., used to increase buffer reference counts to account for the
341 AIO subsystem referencing the buffer, which is required to handle the case
342 where the issuing backend errors out and releases its own pins while the IO is
343 still ongoing.
344
345 As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
346 need to be safe to execute in critical sections. To allow the backend that
347 issued the IO to error out in case of failure [AIO Result](#aio-results) can
348 be used.
349
350
351 ### AIO Targets
352
353 In addition to the completion callbacks describe above, each AIO Handle has
354 exactly one "target". Each target has some space inside an AIO Handle with
355 information specific to the target and can provide callbacks to allow to
356 reopen the underlying file (required for worker mode) and to describe the IO
357 operation (used for debug logging and error messages).
358
359 I.e., if two different uses of AIO can describe the identity of the file being
360 operated on the same way, it likely makes sense to use the same
361 target. E.g. different smgr implementations can describe IO with
362 RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
363 contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
364 and it would not make sense to use the same target for smgr and WAL.
365
366
367 ### AIO Wait References
368
369 As [described above](#aio-handles), AIO Handles can be reused immediately
370 after completion and therefore cannot be used to wait for completion of the
371 IO. Waiting is enabled using AIO wait references, which do not just identify
372 an AIO Handle but also include the handles "generation".
373
374 A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
375 then waited upon using `pgaio_wref_wait()`.
376
377
378 ### AIO Results
379
380 As AIO completion callbacks
381 [are executed in critical sections](#io-can-be-started-in-critical-sections)
382 and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
383 completion callbacks cannot be used to, e.g., make the query that triggered an
384 IO ERROR out.
385
386 To allow to react to failing IOs the issuing backend can pass a pointer to a
387 `PgAioReturn` in backend local memory. Before an AIO Handle is reused the
388 `PgAioReturn` is filled with information about the IO. This includes
389 information about whether the IO was successful (as a value of
390 `PgAioResultStatus`) and enough information to raise an error in case of a
391 failure (via `pgaio_result_report()`, with the error details encoded in
392 `PgAioResult`).
393
394
395 ### AIO Errors
396
397 It would be very convenient to have shared completion callbacks encode the
398 details of errors as an `ErrorData` that could be raised at a later
399 time. Unfortunately doing so would require allocating memory. While elog.c can
400 guarantee (well, kinda) that logging a message will not run out of memory,
401 that only works because a very limited number of messages are in the process
402 of being logged. With AIO a large number of concurrently issued AIOs might
403 fail.
404
405 To avoid the need for preallocating a potentially large amount of memory (in
406 shared memory no less!), completion callbacks instead have to encode errors in
407 a more compact format that can be converted into an error message.
408
409
410 ## Helpers
411
412 Using the low-level AIO API introduces too much complexity to do so all over
413 the tree. Most uses of AIO should be done via reusable, higher-level,
414 helpers.
415
416
417 ### Read Stream
418
419 A common and very beneficial use of AIO are reads where a substantial number
420 of to-be-read locations are known ahead of time. E.g., for a sequential scan
421 the set of blocks that need to be read can be determined solely by knowing the
422 current position and checking the buffer mapping table.
423
424 The [Read Stream](../../../include/storage/read_stream.h) interface makes it
425 comparatively easy to use AIO for such use cases.