| 1 |
# Asynchronous & Direct IO |
| 2 |
|
| 3 |
## Motivation |
| 4 |
|
| 5 |
### Why Asynchronous IO |
| 6 |
|
| 7 |
Until the introduction of asynchronous IO postgres relied on the operating |
| 8 |
system to hide the cost of synchronous IO from postgres. While this worked |
| 9 |
surprisingly well in a lot of workloads, it does not do as good a job on |
| 10 |
prefetching and controlled writeback as we would like. |
| 11 |
|
| 12 |
There are important expensive operations like `fdatasync()` where the operating |
| 13 |
system cannot hide the storage latency. This is particularly important for WAL |
| 14 |
writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC |
| 15 |
writes can yield significantly higher throughput. |
| 16 |
|
| 17 |
|
| 18 |
### Why Direct / unbuffered IO |
| 19 |
|
| 20 |
The main reasons to want to use Direct IO are: |
| 21 |
|
| 22 |
- Lower CPU usage / higher throughput. Particularly on modern storage buffered |
| 23 |
writes are bottlenecked by the operating system having to copy data from the |
| 24 |
kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO |
| 25 |
can often move the data directly between the storage devices and postgres' |
| 26 |
buffer cache, using DMA. While that transfer is ongoing, the CPU is free to |
| 27 |
perform other work. |
| 28 |
- Reduced latency - Direct IO can have substantially lower latency than |
| 29 |
buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL |
| 30 |
write latency. |
| 31 |
- Avoiding double buffering between operating system cache and postgres' |
| 32 |
shared_buffers. |
| 33 |
- Better control over the timing and pace of dirty data writeback. |
| 34 |
|
| 35 |
|
| 36 |
The main reasons *not* to use Direct IO are: |
| 37 |
|
| 38 |
- Without AIO, Direct IO is unusably slow for most purposes. |
| 39 |
- Even with AIO, many parts of postgres need to be modified to perform |
| 40 |
explicit prefetching. |
| 41 |
- In situations where shared_buffers cannot be set appropriately large, |
| 42 |
e.g. because there are many different postgres instances hosted on shared |
| 43 |
hardware, performance will often be worse than when using buffered IO. |
| 44 |
|
| 45 |
|
| 46 |
## AIO Usage Example |
| 47 |
|
| 48 |
In many cases code that can benefit from AIO does not directly have to |
| 49 |
interact with the AIO interface, but can use AIO via higher-level |
| 50 |
abstractions. See [Helpers](#helpers). |
| 51 |
|
| 52 |
In this example, a buffer will be read into shared buffers. |
| 53 |
|
| 54 |
```C |
| 55 |
/* |
| 56 |
* Result of the operation, only to be accessed in this backend. |
| 57 |
*/ |
| 58 |
PgAioReturn ioret; |
| 59 |
|
| 60 |
/* |
| 61 |
* Acquire an AIO Handle, ioret will get result upon completion. |
| 62 |
* |
| 63 |
* Note that ioret needs to stay alive until the IO completes or |
| 64 |
* CurrentResourceOwner is released (i.e. an error is thrown). |
| 65 |
*/ |
| 66 |
PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret); |
| 67 |
|
| 68 |
/* |
| 69 |
* Reference that can be used to wait for the IO we initiate below. This |
| 70 |
* reference can reside in local or shared memory and waited upon by any |
| 71 |
* process. An arbitrary number of references can be made for each IO. |
| 72 |
*/ |
| 73 |
PgAioWaitRef iow; |
| 74 |
|
| 75 |
pgaio_io_get_wref(ioh, &iow); |
| 76 |
|
| 77 |
/* |
| 78 |
* Arrange for shared buffer completion callbacks to be called upon completion |
| 79 |
* of the IO. This callback will update the buffer descriptors associated with |
| 80 |
* the AioHandle, which e.g. allows other backends to access the buffer. |
| 81 |
* |
| 82 |
* A callback can be passed a small bit of data, e.g. to indicate whether to |
| 83 |
* zero a buffer if it is invalid. |
| 84 |
* |
| 85 |
* Multiple completion callbacks can be registered for each handle. |
| 86 |
*/ |
| 87 |
pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0); |
| 88 |
|
| 89 |
/* |
| 90 |
* The completion callback needs to know which buffers to update when the IO |
| 91 |
* completes. As the AIO subsystem does not know about buffers, we have to |
| 92 |
* associate this information with the AioHandle, for use by the completion |
| 93 |
* callback registered above. |
| 94 |
* |
| 95 |
* In this example we're reading only a single buffer, hence the 1. |
| 96 |
*/ |
| 97 |
pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1); |
| 98 |
|
| 99 |
/* |
| 100 |
* Pass the AIO handle to lower-level function. When operating on the level of |
| 101 |
* buffers, we don't know how exactly the IO is performed, that is the |
| 102 |
* responsibility of the storage manager implementation. |
| 103 |
* |
| 104 |
* E.g. md.c needs to translate block numbers into offsets in segments. |
| 105 |
* |
| 106 |
* Once the IO handle has been handed off to smgrstartreadv(), it may not |
| 107 |
* further be used, as the IO may immediately get executed below |
| 108 |
* smgrstartreadv() and the handle reused for another IO. |
| 109 |
* |
| 110 |
* To issue multiple IOs in an efficient way, a caller can call |
| 111 |
* pgaio_enter_batchmode() before starting multiple IOs, and end that batch |
| 112 |
* with pgaio_exit_batchmode(). Note that one needs to be careful while there |
| 113 |
* may be unsubmitted IOs, as another backend may need to wait for one of the |
| 114 |
* unsubmitted IOs. If this backend then had to wait for the other backend, |
| 115 |
* it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more |
| 116 |
* details. |
| 117 |
* |
| 118 |
* Note that even while in batchmode an IO might get submitted immediately, |
| 119 |
* e.g. due to reaching a limit on the number of unsubmitted IOs, and even |
| 120 |
* complete before smgrstartreadv() returns. |
| 121 |
*/ |
| 122 |
void *page = BufferGetBlock(buffer); |
| 123 |
smgrstartreadv(ioh, operation->smgr, forknum, blkno, |
| 124 |
&page, 1); |
| 125 |
|
| 126 |
/* |
| 127 |
* To benefit from AIO, it is beneficial to perform other work, including |
| 128 |
* submitting other IOs, before waiting for the IO to complete. Otherwise |
| 129 |
* we could just have used synchronous, blocking IO. |
| 130 |
*/ |
| 131 |
perform_other_work(); |
| 132 |
|
| 133 |
/* |
| 134 |
* We did some other work and now need the IO operation to have completed to |
| 135 |
* continue. |
| 136 |
*/ |
| 137 |
pgaio_wref_wait(&iow); |
| 138 |
|
| 139 |
/* |
| 140 |
* At this point the IO has completed. We do not yet know whether it succeeded |
| 141 |
* or failed, however. The buffer's state has been updated, which allows other |
| 142 |
* backends to use the buffer (if the IO succeeded), or retry the IO (if it |
| 143 |
* failed). |
| 144 |
* |
| 145 |
* Note that in case the IO has failed, a LOG message may have been emitted, |
| 146 |
* but no ERROR has been raised. This is crucial, as another backend waiting |
| 147 |
* for this IO should not see an ERROR. |
| 148 |
* |
| 149 |
* To check whether the operation succeeded, and to raise an ERROR, or if more |
| 150 |
* appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used. |
| 151 |
*/ |
| 152 |
if (ioret.result.status == PGAIO_RS_ERROR) |
| 153 |
pgaio_result_report(ioret.result, &ioret.target_data, ERROR); |
| 154 |
|
| 155 |
/* |
| 156 |
* Besides having succeeded completely, the IO could also have a) partially |
| 157 |
* completed or b) succeeded with a warning (e.g. due to zero_damaged_pages). |
| 158 |
* If we e.g. tried to read many blocks at once, the read might have |
| 159 |
* only succeeded for the first few blocks. |
| 160 |
* |
| 161 |
* If the IO partially succeeded and this backend needs all blocks to have |
| 162 |
* completed, this backend needs to reissue the IO for the remaining buffers. |
| 163 |
* The AIO subsystem cannot handle this retry transparently. |
| 164 |
* |
| 165 |
* As this example is already long, and we only read a single block, we'll just |
| 166 |
* error out if there's a partial read or a warning. |
| 167 |
*/ |
| 168 |
if (ioret.result.status != PGAIO_RS_OK) |
| 169 |
pgaio_result_report(ioret.result, &ioret.target_data, ERROR); |
| 170 |
|
| 171 |
/* |
| 172 |
* The IO succeeded, so we can use the buffer now. |
| 173 |
*/ |
| 174 |
``` |
| 175 |
|
| 176 |
|
| 177 |
## Design Criteria & Motivation |
| 178 |
|
| 179 |
### Deadlock and Starvation Dangers due to AIO |
| 180 |
|
| 181 |
Using AIO in a naive way can easily lead to deadlocks in an environment where |
| 182 |
the source/target of AIO are shared resources, like pages in postgres' |
| 183 |
shared_buffers. |
| 184 |
|
| 185 |
Consider one backend performing readahead on a table, initiating IO for a |
| 186 |
number of buffers ahead of the current "scan position". If that backend then |
| 187 |
performs some operation that blocks, or even just is slow, the IO completion |
| 188 |
for the asynchronously initiated read may not be processed. |
| 189 |
|
| 190 |
This AIO implementation solves this problem by requiring that AIO methods |
| 191 |
either allow AIO completions to be processed by any backend in the system |
| 192 |
(e.g. io_uring), or to guarantee that AIO processing will happen even when the |
| 193 |
issuing backend is blocked (e.g. worker mode, which offloads completion |
| 194 |
processing to the AIO workers). |
| 195 |
|
| 196 |
|
| 197 |
### IO can be started in critical sections |
| 198 |
|
| 199 |
Using AIO for WAL writes can reduce the overhead of WAL logging substantially: |
| 200 |
|
| 201 |
- AIO allows to start WAL writes eagerly, so they complete before needing to |
| 202 |
wait |
| 203 |
- AIO allows to have multiple WAL flushes in progress at the same time |
| 204 |
- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce |
| 205 |
the number of roundtrips to storage on some OSs and storage HW (buffered IO |
| 206 |
and direct IO without O_DSYNC needs to issue a write and after the write's |
| 207 |
completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single |
| 208 |
Force Unit Access (FUA) write). |
| 209 |
|
| 210 |
The need to be able to execute IO in critical sections has substantial design |
| 211 |
implication on the AIO subsystem. Mainly because completing IOs (see prior |
| 212 |
section) needs to be possible within a critical section, even if the |
| 213 |
to-be-completed IO itself was not issued in a critical section. Consider |
| 214 |
e.g. the case of a backend first starting a number of writes from shared |
| 215 |
buffers and then starting to flush the WAL. Because only a limited amount of |
| 216 |
IO can be in-progress at the same time, initiating IO for flushing the WAL may |
| 217 |
require to first complete IO that was started earlier. |
| 218 |
|
| 219 |
|
| 220 |
### State for AIO needs to live in shared memory |
| 221 |
|
| 222 |
Because postgres uses a process model and because AIOs need to be |
| 223 |
complete-able by any backend much of the state of the AIO subsystem needs to |
| 224 |
live in shared memory. |
| 225 |
|
| 226 |
In an `EXEC_BACKEND` build, a backend's executable code and other process |
| 227 |
local state is not necessarily mapped to the same addresses in each process |
| 228 |
due to ASLR. This means that the shared memory cannot contain pointers to |
| 229 |
callbacks. |
| 230 |
|
| 231 |
|
| 232 |
## Design of the AIO Subsystem |
| 233 |
|
| 234 |
|
| 235 |
### AIO Methods |
| 236 |
|
| 237 |
To achieve portability and performance, multiple methods of performing AIO are |
| 238 |
implemented and others are likely worth adding in the future. |
| 239 |
|
| 240 |
|
| 241 |
#### Synchronous Mode |
| 242 |
|
| 243 |
`io_method=sync` does not actually perform AIO but allows to use the AIO API |
| 244 |
while performing synchronous IO. This can be useful for debugging. The code |
| 245 |
for the synchronous mode is also used as a fallback by e.g. the [worker |
| 246 |
mode](#worker) uses it to execute IO that cannot be executed by workers. |
| 247 |
|
| 248 |
|
| 249 |
#### Worker |
| 250 |
|
| 251 |
`io_method=worker` is available on every platform postgres runs on, and |
| 252 |
implements asynchronous IO - from the view of the issuing process - by |
| 253 |
dispatching the IO to one of several worker processes performing the IO in a |
| 254 |
synchronous manner. |
| 255 |
|
| 256 |
|
| 257 |
#### io_uring |
| 258 |
|
| 259 |
`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it |
| 260 |
dispatches all IO from within the process, lowering context switch rate / |
| 261 |
latency. |
| 262 |
|
| 263 |
|
| 264 |
### AIO Handles |
| 265 |
|
| 266 |
The central API piece for postgres' AIO abstraction are AIO handles. To |
| 267 |
execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and |
| 268 |
then "define" it, i.e. associate an IO operation with the handle. |
| 269 |
|
| 270 |
Often AIO handles are acquired on a higher level and then passed to a lower |
| 271 |
level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c |
| 272 |
routines acquire the handle, which is then passed through smgr.c, md.c to be |
| 273 |
finally fully defined in fd.c. |
| 274 |
|
| 275 |
The functions used at the lowest level to define the operation are |
| 276 |
`pgaio_io_start_*()`. |
| 277 |
|
| 278 |
Because acquisition of an IO handle |
| 279 |
[must always succeed](#io-can-be-started-in-critical-sections) |
| 280 |
and the number of AIO Handles |
| 281 |
[has to be limited](#state-for-aio-needs-to-live-in-shared-memory) |
| 282 |
AIO handles can be reused as soon as they have completed. Obviously code needs |
| 283 |
to be able to react to IO completion. State can be updated using |
| 284 |
[AIO Completion callbacks](#aio-callbacks) |
| 285 |
and the issuing backend can provide a backend local variable to receive the |
| 286 |
result of the IO, as described in |
| 287 |
[AIO Result](#aio-results). |
| 288 |
An IO can be waited for, by both the issuing and any other backend, using |
| 289 |
[AIO References](#aio-wait-references). |
| 290 |
|
| 291 |
|
| 292 |
Because an AIO Handle is not executable just after calling |
| 293 |
`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed |
| 294 |
(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by |
| 295 |
`pgaio_io_acquire()`) without causing the IO to have been defined (by, |
| 296 |
potentially indirectly, causing `pgaio_io_start_*()` to have been |
| 297 |
called). Otherwise a backend could trivially self-deadlock by using up all AIO |
| 298 |
Handles without the ability to wait for some of the IOs to complete. |
| 299 |
|
| 300 |
If it turns out that an AIO Handle is not needed, e.g., because the handle was |
| 301 |
acquired before holding a contended lock, it can be released without being |
| 302 |
defined using `pgaio_io_release()`. |
| 303 |
|
| 304 |
|
| 305 |
### AIO Callbacks |
| 306 |
|
| 307 |
Commonly several layers need to react to completion of an IO. E.g. for a read |
| 308 |
md.c needs to check if the IO outright failed or was shorter than needed, |
| 309 |
bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the |
| 310 |
BufferDesc to update the buffer's state. |
| 311 |
|
| 312 |
The fact that several layers / subsystems need to react to IO completion poses |
| 313 |
a few challenges: |
| 314 |
|
| 315 |
- Upper layers should not need to know details of lower layers. E.g. bufmgr.c |
| 316 |
should not assume the IO will pass through md.c. Therefore upper levels |
| 317 |
cannot know what lower layers would consider an error. |
| 318 |
|
| 319 |
- Lower layers should not need to know about upper layers. E.g. smgr APIs are |
| 320 |
used going through shared buffers but are also used bypassing shared |
| 321 |
buffers. This means that e.g. md.c is not in a position to validate |
| 322 |
checksums. |
| 323 |
|
| 324 |
- Having code in the AIO subsystem for every possible combination of layers |
| 325 |
would lead to a lot of duplication. |
| 326 |
|
| 327 |
The "solution" to this is the ability to associate multiple completion |
| 328 |
callbacks with a handle. E.g. bufmgr.c can have a callback to update the |
| 329 |
BufferDesc state and to verify the page and md.c can have another callback to |
| 330 |
check if the IO operation was successful. |
| 331 |
|
| 332 |
As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory |
| 333 |
currently cannot contain function pointers. Because of that completion |
| 334 |
callbacks are not directly identified by function pointers but by IDs |
| 335 |
(`PgAioHandleCallbackID`). A substantial added benefit is that that |
| 336 |
allows callbacks to be identified by much smaller amount of memory (a single |
| 337 |
byte currently). |
| 338 |
|
| 339 |
In addition to completion, AIO callbacks also are called to "stage" an |
| 340 |
IO. This is, e.g., used to increase buffer reference counts to account for the |
| 341 |
AIO subsystem referencing the buffer, which is required to handle the case |
| 342 |
where the issuing backend errors out and releases its own pins while the IO is |
| 343 |
still ongoing. |
| 344 |
|
| 345 |
As [explained earlier](#io-can-be-started-in-critical-sections) IO completions |
| 346 |
need to be safe to execute in critical sections. To allow the backend that |
| 347 |
issued the IO to error out in case of failure [AIO Result](#aio-results) can |
| 348 |
be used. |
| 349 |
|
| 350 |
|
| 351 |
### AIO Targets |
| 352 |
|
| 353 |
In addition to the completion callbacks describe above, each AIO Handle has |
| 354 |
exactly one "target". Each target has some space inside an AIO Handle with |
| 355 |
information specific to the target and can provide callbacks to allow to |
| 356 |
reopen the underlying file (required for worker mode) and to describe the IO |
| 357 |
operation (used for debug logging and error messages). |
| 358 |
|
| 359 |
I.e., if two different uses of AIO can describe the identity of the file being |
| 360 |
operated on the same way, it likely makes sense to use the same |
| 361 |
target. E.g. different smgr implementations can describe IO with |
| 362 |
RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In |
| 363 |
contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr |
| 364 |
and it would not make sense to use the same target for smgr and WAL. |
| 365 |
|
| 366 |
|
| 367 |
### AIO Wait References |
| 368 |
|
| 369 |
As [described above](#aio-handles), AIO Handles can be reused immediately |
| 370 |
after completion and therefore cannot be used to wait for completion of the |
| 371 |
IO. Waiting is enabled using AIO wait references, which do not just identify |
| 372 |
an AIO Handle but also include the handles "generation". |
| 373 |
|
| 374 |
A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and |
| 375 |
then waited upon using `pgaio_wref_wait()`. |
| 376 |
|
| 377 |
|
| 378 |
### AIO Results |
| 379 |
|
| 380 |
As AIO completion callbacks |
| 381 |
[are executed in critical sections](#io-can-be-started-in-critical-sections) |
| 382 |
and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio) |
| 383 |
completion callbacks cannot be used to, e.g., make the query that triggered an |
| 384 |
IO ERROR out. |
| 385 |
|
| 386 |
To allow to react to failing IOs the issuing backend can pass a pointer to a |
| 387 |
`PgAioReturn` in backend local memory. Before an AIO Handle is reused the |
| 388 |
`PgAioReturn` is filled with information about the IO. This includes |
| 389 |
information about whether the IO was successful (as a value of |
| 390 |
`PgAioResultStatus`) and enough information to raise an error in case of a |
| 391 |
failure (via `pgaio_result_report()`, with the error details encoded in |
| 392 |
`PgAioResult`). |
| 393 |
|
| 394 |
|
| 395 |
### AIO Errors |
| 396 |
|
| 397 |
It would be very convenient to have shared completion callbacks encode the |
| 398 |
details of errors as an `ErrorData` that could be raised at a later |
| 399 |
time. Unfortunately doing so would require allocating memory. While elog.c can |
| 400 |
guarantee (well, kinda) that logging a message will not run out of memory, |
| 401 |
that only works because a very limited number of messages are in the process |
| 402 |
of being logged. With AIO a large number of concurrently issued AIOs might |
| 403 |
fail. |
| 404 |
|
| 405 |
To avoid the need for preallocating a potentially large amount of memory (in |
| 406 |
shared memory no less!), completion callbacks instead have to encode errors in |
| 407 |
a more compact format that can be converted into an error message. |
| 408 |
|
| 409 |
|
| 410 |
## Helpers |
| 411 |
|
| 412 |
Using the low-level AIO API introduces too much complexity to do so all over |
| 413 |
the tree. Most uses of AIO should be done via reusable, higher-level, |
| 414 |
helpers. |
| 415 |
|
| 416 |
|
| 417 |
### Read Stream |
| 418 |
|
| 419 |
A common and very beneficial use of AIO are reads where a substantial number |
| 420 |
of to-be-read locations are known ahead of time. E.g., for a sequential scan |
| 421 |
the set of blocks that need to be read can be determined solely by knowing the |
| 422 |
current position and checking the buffer mapping table. |
| 423 |
|
| 424 |
The [Read Stream](../../../include/storage/read_stream.h) interface makes it |
| 425 |
comparatively easy to use AIO for such use cases. |