| 1 |
<!DOCTYPE html> |
| 2 |
<html><head> |
| 3 |
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| 4 |
<meta http-equiv="content-type" content="text/html; charset=UTF-8"> |
| 5 |
<link href="sqlite.css" rel="stylesheet"> |
| 6 |
<title>Memory-Mapped I/O</title> |
| 7 |
<!-- path= --> |
| 8 |
</head> |
| 9 |
<body> |
| 10 |
<div class=nosearch> |
| 11 |
<a href="index.html"> |
| 12 |
<img class="logo" src="images/sqlite370_banner.svg" alt="SQLite" border="0"> |
| 13 |
</a> |
| 14 |
<div><!-- IE hack to prevent disappearing logo --></div> |
| 15 |
<div class="tagline desktoponly"> |
| 16 |
Small. Fast. Reliable.<br>Choose any three. |
| 17 |
</div> |
| 18 |
<div class="menu mainmenu"> |
| 19 |
<ul> |
| 20 |
<li><a href="index.html">Home</a> |
| 21 |
<li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a> |
| 22 |
<li class='wideonly'><a href='about.html'>About</a> |
| 23 |
<li class='desktoponly'><a href="docs.html">Documentation</a> |
| 24 |
<li class='desktoponly'><a href="download.html">Download</a> |
| 25 |
<li class='wideonly'><a href='copyright.html'>License</a> |
| 26 |
<li class='desktoponly'><a href="support.html">Support</a> |
| 27 |
<li class='desktoponly'><a href="prosupport.html">Purchase</a> |
| 28 |
<li class='search' id='search_menubutton'> |
| 29 |
<a href="javascript:void(0)" onclick='toggle_search()'>Search</a> |
| 30 |
</ul> |
| 31 |
</div> |
| 32 |
<div class="menu submenu" id="submenu"> |
| 33 |
<ul> |
| 34 |
<li><a href='about.html'>About</a> |
| 35 |
<li><a href='docs.html'>Documentation</a> |
| 36 |
<li><a href='download.html'>Download</a> |
| 37 |
<li><a href='support.html'>Support</a> |
| 38 |
<li><a href='prosupport.html'>Purchase</a> |
| 39 |
</ul> |
| 40 |
</div> |
| 41 |
<div class="searchmenu" id="searchmenu"> |
| 42 |
<form method="GET" action="search"> |
| 43 |
<select name="s" id="searchtype"> |
| 44 |
<option value="d">Search Documentation</option> |
| 45 |
<option value="c">Search Changelog</option> |
| 46 |
</select> |
| 47 |
<input type="text" name="q" id="searchbox" value=""> |
| 48 |
<input type="submit" value="Go"> |
| 49 |
</form> |
| 50 |
</div> |
| 51 |
</div> |
| 52 |
<script> |
| 53 |
function toggle_div(nm) { |
| 54 |
var w = document.getElementById(nm); |
| 55 |
if( w.style.display=="block" ){ |
| 56 |
w.style.display = "none"; |
| 57 |
}else{ |
| 58 |
w.style.display = "block"; |
| 59 |
} |
| 60 |
} |
| 61 |
function toggle_search() { |
| 62 |
var w = document.getElementById("searchmenu"); |
| 63 |
if( w.style.display=="block" ){ |
| 64 |
w.style.display = "none"; |
| 65 |
} else { |
| 66 |
w.style.display = "block"; |
| 67 |
setTimeout(function(){ |
| 68 |
document.getElementById("searchbox").focus() |
| 69 |
}, 30); |
| 70 |
} |
| 71 |
} |
| 72 |
function div_off(nm){document.getElementById(nm).style.display="none";} |
| 73 |
window.onbeforeunload = function(e){div_off("submenu");} |
| 74 |
/* Disable the Search feature if we are not operating from CGI, since */ |
| 75 |
/* Search is accomplished using CGI and will not work without it. */ |
| 76 |
if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){ |
| 77 |
document.getElementById("search_menubutton").style.display = "none"; |
| 78 |
} |
| 79 |
/* Used by the Hide/Show button beside syntax diagrams, to toggle the */ |
| 80 |
function hideorshow(btn,obj){ |
| 81 |
var x = document.getElementById(obj); |
| 82 |
var b = document.getElementById(btn); |
| 83 |
if( x.style.display!='none' ){ |
| 84 |
x.style.display = 'none'; |
| 85 |
b.innerHTML='show'; |
| 86 |
}else{ |
| 87 |
x.style.display = ''; |
| 88 |
b.innerHTML='hide'; |
| 89 |
} |
| 90 |
return false; |
| 91 |
} |
| 92 |
var antiRobot = 0; |
| 93 |
function antiRobotGo(){ |
| 94 |
if( antiRobot!=3 ) return; |
| 95 |
antiRobot = 7; |
| 96 |
var j = document.getElementById("mtimelink"); |
| 97 |
if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href"); |
| 98 |
} |
| 99 |
function antiRobotDefense(){ |
| 100 |
document.body.onmousedown=function(){ |
| 101 |
antiRobot |= 2; |
| 102 |
antiRobotGo(); |
| 103 |
document.body.onmousedown=null; |
| 104 |
} |
| 105 |
document.body.onmousemove=function(){ |
| 106 |
antiRobot |= 2; |
| 107 |
antiRobotGo(); |
| 108 |
document.body.onmousemove=null; |
| 109 |
} |
| 110 |
setTimeout(function(){ |
| 111 |
antiRobot |= 1; |
| 112 |
antiRobotGo(); |
| 113 |
}, 100) |
| 114 |
antiRobotGo(); |
| 115 |
} |
| 116 |
antiRobotDefense(); |
| 117 |
</script> |
| 118 |
|
| 119 |
|
| 120 |
|
| 121 |
<h1 align="center">Memory-Mapped I/O</h1> |
| 122 |
|
| 123 |
<p>The default mechanism by which SQLite accesses and updates database disk |
| 124 |
files is the xRead() and xWrite() methods of the |
| 125 |
<a href="c3ref/io_methods.html">sqlite3_io_methods</a> VFS object. These methods are typically implemented as |
| 126 |
"read()" and "write()" system calls which cause the operating system |
| 127 |
to copy disk content between the kernel buffer cache and user space.</p> |
| 128 |
|
| 129 |
<p>Beginning with <a href="releaselog/3_7_17.html">version 3.7.17</a> (2013-05-20), SQLite has the option of |
| 130 |
accessing disk content directly using memory-mapped I/O and the new |
| 131 |
xFetch() and xUnfetch() methods on <a href="c3ref/io_methods.html">sqlite3_io_methods</a>.</p> |
| 132 |
|
| 133 |
<p>There are advantages and disadvantages to using memory-mapped I/O. |
| 134 |
Advantages include:</p> |
| 135 |
|
| 136 |
<ol> |
| 137 |
<li><p>Many operations, especially I/O intensive operations, can be |
| 138 |
faster since content need not be copied between kernel space |
| 139 |
and user space.</p> |
| 140 |
|
| 141 |
<li><p>The SQLite library may need less RAM since it shares pages with |
| 142 |
the operating-system page cache and does not always need its own copy of |
| 143 |
working pages.</p> |
| 144 |
</ol> |
| 145 |
|
| 146 |
<p>But there are also disadvantages:</p> |
| 147 |
|
| 148 |
<ol> |
| 149 |
<li><p>An I/O error on a memory-mapped file cannot be caught and dealt with by |
| 150 |
SQLite. Instead, the I/O error causes a signal which, if not caught |
| 151 |
by the application, results in a program crash.</p> |
| 152 |
|
| 153 |
<li><p>The operating system must have a unified buffer cache in order for |
| 154 |
the memory-mapped I/O extension to work correctly, especially in |
| 155 |
situations where two processes are accessing the same database |
| 156 |
file and one process is using memory-mapped I/O while the other |
| 157 |
is not. Not all operating systems have a unified buffer cache. |
| 158 |
In some operating systems that claim to have a unified buffer cache, |
| 159 |
the implementation is buggy and can lead to corrupt databases.</p> |
| 160 |
|
| 161 |
<li><p>Performance does not always increase with memory-mapped I/O. In fact, |
| 162 |
it is possible to construct test cases where performance is reduced |
| 163 |
by the use of memory-mapped I/O.</p> |
| 164 |
|
| 165 |
<li><p>Windows is unable to truncate a memory-mapped file. Hence, on Windows, |
| 166 |
if an operation such as <a href="lang_vacuum.html">VACUUM</a> or <a href="pragma.html#pragma_auto_vacuum">auto_vacuum</a> tries to reduce the |
| 167 |
size of a memory-mapped database file, the size reduction attempt will |
| 168 |
silently fail, leaving unused space at the end of the database file. |
| 169 |
No data is lost due to this problem, and the unused space will be |
| 170 |
reused again the next time the database grows. However if a version |
| 171 |
of SQLite prior to 3.7.0 runs <a href="pragma.html#pragma_integrity_check">PRAGMA integrity_check</a> on such a |
| 172 |
database, it will (incorrectly) report database corruption due to |
| 173 |
the unused space at the end. Or if a version of SQLite prior to 3.7.0 |
| 174 |
writes to the database while it still has unused space at the end, it |
| 175 |
may make that unused space inaccessible and unavailable for reuse until |
| 176 |
after the next <a href="lang_vacuum.html">VACUUM</a>. |
| 177 |
</ol> |
| 178 |
|
| 179 |
<p>Because of the potential disadvantages, memory-mapped I/O is disabled |
| 180 |
by default. To activate memory-mapped I/O, use the <a href="pragma.html#pragma_mmap_size">mmap_size pragma</a> |
| 181 |
and set the mmap_size to some large number, usually 256MB or larger, depending |
| 182 |
on how much address space your application can spare. The rest is |
| 183 |
automatic. The <a href="pragma.html#pragma_mmap_size">PRAGMA mmap_size</a> statement will be a silent no-op on |
| 184 |
systems that do not support memory-mapped I/O.</p> |
| 185 |
|
| 186 |
<h2>How Memory-Mapped I/O Works</h2> |
| 187 |
|
| 188 |
<p>To read a page of database content using the legacy xRead() method, |
| 189 |
SQLite first allocates a page-size chunk of heap memory then invokes |
| 190 |
the xRead() method which causes the database page content to be copied |
| 191 |
into the newly allocated heap memory. This involves (at a minimum) |
| 192 |
a copy of the entire page.</p> |
| 193 |
|
| 194 |
<p>But if SQLite wants to access a page of the database file and |
| 195 |
memory mapped I/O is enabled, it first calls the xFetch() method. |
| 196 |
The xFetch() method asks the operating system to return a pointer to |
| 197 |
the requested page, if possible. If the requested page has been or |
| 198 |
can be mapped into the application address space, then xFetch returns |
| 199 |
a pointer to that page for SQLite to use without having to copy anything. |
| 200 |
Skipping the copy step is what makes memory mapped I/O faster.</p> |
| 201 |
|
| 202 |
<p>SQLite does not assume that the xFetch() method will work. If |
| 203 |
a call to xFetch() returns a NULL pointer (indicating that the requested |
| 204 |
page is not currently mapped into the applications address space) then |
| 205 |
SQLite silently falls back to using xRead(). An error is only reported |
| 206 |
if xRead() also fails.</p> |
| 207 |
|
| 208 |
<p>When updating the database file, SQLite always makes a copy of the |
| 209 |
page content into heap memory before modifying the page. This is necessary |
| 210 |
for two reasons. First, changes to the database |
| 211 |
are not supposed to be visible to other processes until |
| 212 |
after the transaction commits and so the changes must occur in private memory. |
| 213 |
Second, SQLite uses a read-only memory map to prevent stray pointers in the |
| 214 |
application from overwriting and corrupting the database file. |
| 215 |
|
| 216 |
<p> |
| 217 |
After all needed changes are completed, xWrite() is used to move the content |
| 218 |
back into the database file. |
| 219 |
Hence the use of memory mapped I/O does not significantly change the |
| 220 |
performance of database changes. |
| 221 |
Memory mapped I/O is mostly a benefit for queries.</p> |
| 222 |
|
| 223 |
<h2>Configuring Memory-Mapped I/O</h2> |
| 224 |
|
| 225 |
<p>The "mmap_size" is the maximum number of bytes of the database file that |
| 226 |
SQLite will try to map into the process address space at one time. The |
| 227 |
mmap_size applies separately to each database file, so the total amount |
| 228 |
of process address space that could potentially be used is the mmap_size |
| 229 |
times the number of open database files.</p> |
| 230 |
|
| 231 |
<p>To activate memory-mapped I/O, an application can set the mmap_size to some |
| 232 |
large value. For example:</p> |
| 233 |
|
| 234 |
<blockquote><pre> |
| 235 |
PRAGMA mmap_size=268435456; |
| 236 |
</pre></blockquote> |
| 237 |
|
| 238 |
<p>To disable memory-mapped I/O, simply set the mmap_size to zero:</p> |
| 239 |
|
| 240 |
<blockquote><pre> |
| 241 |
PRAGMA mmap_size=0; |
| 242 |
</pre></blockquote> |
| 243 |
|
| 244 |
<p>If mmap_size is set to N then all current implementations map the first |
| 245 |
N bytes of the database file and use legacy xRead() calls for any content |
| 246 |
beyond N bytes. If the database file is smaller than N bytes, then the entire |
| 247 |
file is mapped. In the future, new OS interfaces could, in theory, map |
| 248 |
regions of the file other than the first N bytes, but no such |
| 249 |
implementation currently exists.</p> |
| 250 |
|
| 251 |
<p>The mmap_size is set separately for each database file using the |
| 252 |
"<a href="pragma.html#pragma_mmap_size">PRAGMA mmap_size</a>" statement. The usual default mmap_size is zero, |
| 253 |
meaning that memory mapped I/O is disabled by default. However, the |
| 254 |
default mmap_size can be increased either at compile-time using |
| 255 |
the <a href="compile.html#default_mmap_size">SQLITE_DEFAULT_MMAP_SIZE</a> macro or at start-time using the |
| 256 |
<a href="c3ref/config.html">sqlite3_config</a>(<a href="c3ref/c_config_covering_index_scan.html#sqliteconfigmmapsize">SQLITE_CONFIG_MMAP_SIZE</a>,...) interface.</p> |
| 257 |
|
| 258 |
<p>SQLite also maintains a hard upper bound on the mmap_size. Attempts |
| 259 |
to increase the mmap_size above this hard upper bound (using |
| 260 |
<a href="pragma.html#pragma_mmap_size">PRAGMA mmap_size</a>) will automatically cap the mmap_size at the hard |
| 261 |
upper bound. If the hard upper bound is zero, then memory mapped I/O |
| 262 |
is impossible. The hard upper bound can be set at compile-time using |
| 263 |
the <a href="compile.html#max_mmap_size">SQLITE_MAX_MMAP_SIZE</a> macro. If <a href="compile.html#max_mmap_size">SQLITE_MAX_MMAP_SIZE</a> is set to |
| 264 |
zero, then the code used to implement memory mapped I/O is omitted from |
| 265 |
the build. The hard upper bound is automatically set to zero on certain |
| 266 |
platforms (ex: OpenBSD) where memory mapped I/O does not work due to the |
| 267 |
lack of a unified buffer cache.</p> |
| 268 |
|
| 269 |
<p>If the hard upper bound on mmap_size is non-zero at compilation time, |
| 270 |
it may still be reduced or zeroed at start-time using the |
| 271 |
<a href="c3ref/config.html">sqlite3_config</a>(<a href="c3ref/c_config_covering_index_scan.html#sqliteconfigmmapsize">SQLITE_CONFIG_MMAP_SIZE</a>,X,Y) interface. The X and |
| 272 |
Y parameters must both be 64-bit signed integers. The X parameter |
| 273 |
is the default mmap_size of the process and the Y is the new hard upper bound. |
| 274 |
The hard upper bound cannot be increased above its compile-time setting |
| 275 |
using <a href="c3ref/c_config_covering_index_scan.html#sqliteconfigmmapsize">SQLITE_CONFIG_MMAP_SIZE</a> but it can be reduced or zeroed.</p> |
| 276 |
<p align="center"><small><i>This page was last updated on 2022-04-18 02:55:50Z </small></i></p> |
| 277 |
|