| 1 |
src/backend/storage/page/README |
| 2 |
|
| 3 |
Checksums |
| 4 |
--------- |
| 5 |
|
| 6 |
Checksums on data pages are designed to detect corruption by the I/O system. |
| 7 |
We do not protect buffers against uncorrectable memory errors, since these |
| 8 |
have a very low measured incidence according to research on large server farms, |
| 9 |
http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf, discussed |
| 10 |
2010/12/22 on -hackers list. |
| 11 |
|
| 12 |
Current implementation requires this be enabled system-wide at initdb time, or |
| 13 |
by using the pg_checksums tool on an offline cluster. |
| 14 |
|
| 15 |
The checksum is not valid at all times on a data page!! |
| 16 |
The checksum is valid when the page leaves the shared pool and is checked |
| 17 |
when it later re-enters the shared pool as a result of I/O. |
| 18 |
We set the checksum on a buffer in the shared pool immediately before we |
| 19 |
flush the buffer. As a result we implicitly invalidate the page's checksum |
| 20 |
when we modify the page for a data change or even a hint. This means that |
| 21 |
many or even most pages in shared buffers have invalid page checksums, |
| 22 |
so be careful how you interpret the pd_checksum field. |
| 23 |
|
| 24 |
That means that WAL-logged changes to a page do NOT update the page checksum, |
| 25 |
so full page images may not have a valid checksum. But those page images have |
| 26 |
the WAL CRC covering them and so are verified separately from this |
| 27 |
mechanism. WAL replay should not test the checksum of a full-page image. |
| 28 |
|
| 29 |
The best way to understand this is that WAL CRCs protect records entering the |
| 30 |
WAL stream, and data page verification protects blocks entering the shared |
| 31 |
buffer pool. They are similar in purpose, yet completely separate. Together |
| 32 |
they ensure we are able to detect errors in data re-entering |
| 33 |
PostgreSQL-controlled memory. Note also that the WAL checksum is a 32-bit CRC, |
| 34 |
whereas the page checksum is only 16-bits. |
| 35 |
|
| 36 |
Any write of a data block can cause a torn page if the write is unsuccessful. |
| 37 |
Full page writes protect us from that, which are stored in WAL. Setting hint |
| 38 |
bits when a page is already dirty is OK because a full page write must already |
| 39 |
have been written for it since the last checkpoint. Setting hint bits on an |
| 40 |
otherwise clean page can allow torn pages; this doesn't normally matter since |
| 41 |
they are just hints, but when the page has checksums, then losing a few bits |
| 42 |
would cause the checksum to be invalid. So if we have full_page_writes = on |
| 43 |
and checksums enabled then we must write a WAL record specifically so that we |
| 44 |
record a full page image in WAL. Hint bits updates should be protected using |
| 45 |
MarkBufferDirtyHint(), which is responsible for writing the full-page image |
| 46 |
when necessary. |
| 47 |
|
| 48 |
Note that when we write a page checksum we include the hopefully zeroed bytes |
| 49 |
that form the hole in the centre of a standard page. Thus, when we read the |
| 50 |
block back from storage we implicitly check that the hole is still all zeroes. |
| 51 |
We do this to ensure that we spot errors that could have destroyed data even |
| 52 |
if they haven't actually done so. Full page images stored in WAL do *not* |
| 53 |
check that the hole is all zero; the data in the hole is simply skipped and |
| 54 |
re-zeroed if the backup block is reapplied. We do this because a failure in |
| 55 |
WAL is a fatal error and prevents further recovery, whereas a checksum failure |
| 56 |
on a normal data block is a hard error but not a critical one for the server, |
| 57 |
even if it is a very bad thing for the user. |
| 58 |
|
| 59 |
New WAL records cannot be written during recovery, so hint bits set during |
| 60 |
recovery must not dirty the page if the buffer is not already dirty, when |
| 61 |
checksums are enabled. Systems in Hot-Standby mode may benefit from hint bits |
| 62 |
being set, but with checksums enabled, a page cannot be dirtied after setting a |
| 63 |
hint bit (due to the torn page risk). So, it must wait for full-page images |
| 64 |
containing the hint bit updates to arrive from the primary. |