Debian Codegraph @

www/vfs.html

raw
1 <!DOCTYPE html>
2 <html><head>
3 <meta name="viewport" content="width=device-width, initial-scale=1.0">
4 <meta http-equiv="content-type" content="text/html; charset=UTF-8">
5 <link href="sqlite.css" rel="stylesheet">
6 <title>The SQLite OS Interface or "VFS"</title>
7 <!-- path= -->
8 </head>
9 <body>
10 <div class=nosearch>
11 <a href="index.html">
12 <img class="logo" src="images/sqlite370_banner.svg" alt="SQLite" border="0">
13 </a>
14 <div><!-- IE hack to prevent disappearing logo --></div>
15 <div class="tagline desktoponly">
16 Small. Fast. Reliable.<br>Choose any three.
17 </div>
18 <div class="menu mainmenu">
19 <ul>
20 <li><a href="index.html">Home</a>
21 <li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a>
22 <li class='wideonly'><a href='about.html'>About</a>
23 <li class='desktoponly'><a href="docs.html">Documentation</a>
24 <li class='desktoponly'><a href="download.html">Download</a>
25 <li class='wideonly'><a href='copyright.html'>License</a>
26 <li class='desktoponly'><a href="support.html">Support</a>
27 <li class='desktoponly'><a href="prosupport.html">Purchase</a>
28 <li class='search' id='search_menubutton'>
29 <a href="javascript:void(0)" onclick='toggle_search()'>Search</a>
30 </ul>
31 </div>
32 <div class="menu submenu" id="submenu">
33 <ul>
34 <li><a href='about.html'>About</a>
35 <li><a href='docs.html'>Documentation</a>
36 <li><a href='download.html'>Download</a>
37 <li><a href='support.html'>Support</a>
38 <li><a href='prosupport.html'>Purchase</a>
39 </ul>
40 </div>
41 <div class="searchmenu" id="searchmenu">
42 <form method="GET" action="search">
43 <select name="s" id="searchtype">
44 <option value="d">Search Documentation</option>
45 <option value="c">Search Changelog</option>
46 </select>
47 <input type="text" name="q" id="searchbox" value="">
48 <input type="submit" value="Go">
49 </form>
50 </div>
51 </div>
52 <script>
53 function toggle_div(nm) {
54 var w = document.getElementById(nm);
55 if( w.style.display=="block" ){
56 w.style.display = "none";
57 }else{
58 w.style.display = "block";
59 }
60 }
61 function toggle_search() {
62 var w = document.getElementById("searchmenu");
63 if( w.style.display=="block" ){
64 w.style.display = "none";
65 } else {
66 w.style.display = "block";
67 setTimeout(function(){
68 document.getElementById("searchbox").focus()
69 }, 30);
70 }
71 }
72 function div_off(nm){document.getElementById(nm).style.display="none";}
73 window.onbeforeunload = function(e){div_off("submenu");}
74 /* Disable the Search feature if we are not operating from CGI, since */
75 /* Search is accomplished using CGI and will not work without it. */
76 if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){
77 document.getElementById("search_menubutton").style.display = "none";
78 }
79 /* Used by the Hide/Show button beside syntax diagrams, to toggle the */
80 function hideorshow(btn,obj){
81 var x = document.getElementById(obj);
82 var b = document.getElementById(btn);
83 if( x.style.display!='none' ){
84 x.style.display = 'none';
85 b.innerHTML='show';
86 }else{
87 x.style.display = '';
88 b.innerHTML='hide';
89 }
90 return false;
91 }
92 var antiRobot = 0;
93 function antiRobotGo(){
94 if( antiRobot!=3 ) return;
95 antiRobot = 7;
96 var j = document.getElementById("mtimelink");
97 if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href");
98 }
99 function antiRobotDefense(){
100 document.body.onmousedown=function(){
101 antiRobot |= 2;
102 antiRobotGo();
103 document.body.onmousedown=null;
104 }
105 document.body.onmousemove=function(){
106 antiRobot |= 2;
107 antiRobotGo();
108 document.body.onmousemove=null;
109 }
110 setTimeout(function(){
111 antiRobot |= 1;
112 antiRobotGo();
113 }, 100)
114 antiRobotGo();
115 }
116 antiRobotDefense();
117 </script>
118 <div class=fancy>
119 <div class=nosearch>
120 <div class="fancy_title">
121 The SQLite OS Interface or "VFS"
122 </div>
123 <details class="fancy_toc">
124 <summary>Table Of Contents</summary>
125 <div id="toc_sub"><div class="fancy-toc1"><a href="#introduction">1. Introduction</a></div>
126 <div class="fancy-toc1"><a href="#the_vfs_in_relation_to_the_rest_of_sqlite">2. The VFS In Relation To The Rest Of SQLite</a></div>
127 <div class="fancy-toc1"><a href="#multiple_vfses">3. Multiple VFSes</a></div>
128 <div class="fancy-toc2"><a href="#standard_unix_vfses">3.1. Standard Unix VFSes</a></div>
129 <div class="fancy-toc2"><a href="#standard_windows_vfses">3.2. Standard Windows VFSes</a></div>
130 <div class="fancy-toc2"><a href="#specifying_which_vfs_to_use">3.3. Specifying Which VFS To Use</a></div>
131 <div class="fancy-toc2"><a href="#vfs_shims">3.4. VFS Shims</a></div>
132 <div class="fancy-toc2"><a href="#other_example_vfses">3.5. Other Example VFSes</a></div>
133 <div class="fancy-toc1"><a href="#vfs_implementations">4. VFS Implementations</a></div>
134 </div>
135 </details>
136 </div>
137
138
139
140
141
142 <h1 id="introduction"><span>1. </span>Introduction</h1>
143
144 <p>
145 This article describes the SQLite OS portability layer or "VFS" - the
146 module at the bottom of the SQLite implementation stack
147 that provides portability across operating systems.
148 </p>
149
150 <h1 id="the_vfs_in_relation_to_the_rest_of_sqlite"><span>2. </span>The VFS In Relation To The Rest Of SQLite</h1>
151
152 <div>
153 <img src="images/vfs1.gif" align="right" hspace="10">
154 </div>
155
156 <p>
157 The internal organization of the SQLite library can be viewed as the
158 stack of modules shown to the right.
159 The Tokenizer, Parser, and Code Generator components are used to
160 process SQL statements and convert them into executable programs
161 in a virtual machine language or byte code.
162 Roughly speaking, these top three layers implement
163 <a href="c3ref/prepare.html">sqlite3_prepare_v2()</a>. The byte code generated by the top three
164 layers is a <a href="c3ref/stmt.html">prepared statement</a>.
165 The Virtual Machine module is responsible for running the SQL statement
166 byte code. The B-Tree module organizes a database file into multiple
167 key/value stores with ordered keys and logarithmic performance.
168 The Pager module is responsible for loading pages of the database
169 file into memory, for implementing and controlling transactions, and
170 for creating and maintaining the journal files that prevent database
171 corruption following a crash or power failure.
172 The OS Interface is a thin abstraction that provides a common set of
173 routines for adapting SQLite to run on different operating systems.
174 Roughly speaking, the bottom four layers implement
175 <a href="c3ref/step.html">sqlite3_step()</a>.
176 </p>
177
178 <p>
179 This article is about the bottom layer.
180 </p>
181
182 <p>The OS Interface - also called the "VFS" - is what makes SQLite
183 portable across operating systems. Whenever any of the other modules
184 in SQLite needs to communicate with the operating
185 system, they invoke methods in the VFS. The VFS then invokes the
186 operating-specific code needed to satisfy the request.
187 Hence, porting SQLite to a new
188 operating system is simply a matter of writing a new OS interface layer
189 or "VFS".</p>
190
191 <h1 id="multiple_vfses"><span>3. </span>Multiple VFSes</h1>
192
193 <p>
194 The standard SQLite source tree contains built-in VFSes for unix
195 and windows. Alternative VFSes can be
196 added at start-time or run-time using the
197 <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface.
198 </p>
199
200 <p>
201 Multiple VFSes can be registered at the same time.
202 Each VFS has a unique name.
203 Separate <a href="c3ref/sqlite3.html">database connections</a> within the same process can be using
204 different VFSes at the same time. For that matter, if a single
205 database connection has multiple database files open using
206 the <a href="lang_attach.html">ATTACH</a> command, then each attached database might be using a
207 different VFS.
208 </p>
209
210 <h2 id="standard_unix_vfses"><span>3.1. </span>Standard Unix VFSes</h2>
211
212 <p>
213 Unix builds come with multiple built-in VFSes. The default VFS
214 for unix is called "unix" and is used in most applications.
215 Other VFSes that might be found in unix (depending on compile-time
216 options) include:
217 </p>
218
219 <ol>
220 <li><p><b>unix-dotfile</b> - uses dot-file locking rather than
221 POSIX advisory locks.
222 </p></li><li><p><b>unix-excl</b> - obtains and holds an exclusive lock on
223 database files, preventing other processes from accessing the
224 database. Also keeps the <a href="walformat.html#shm">wal-index</a> in heap rather than in
225 shared memory.
226 </p></li><li><p><b>unix-none</b> - all file locking operations are no-ops.
227 </p></li><li><p><b>unix-namedsem</b> - uses named semaphores for file locking.
228 VXWorks only.
229 </p></li></ol>
230
231 <p>
232 The various unix VFSes differ only in the way they handle file locking -
233 they share most of their implementation in common with one another and
234 are all located in the same SQLite source file:
235 <a href="https://sqlite.org/src/doc/trunk/src/os_unix.c">os_unix.c</a>.
236 Note that except for "unix" and "unix-excl", the various unix VFSes all
237 use incompatible locking implementations. If two processes are accessing
238 the same SQLite database using different unix VFSes, they may
239 not see each others locks and may end up interfering with one another,
240 resulting in database corruption. The "unix-none" VFS in particular
241 does no locking at all and will easily result in database corruption if
242 used by two or more database connections at the same time.
243 Programmers are encouraged to use only "unix" or "unix-excl" unless
244 there is a compelling reason to do otherwise.
245 </p>
246
247 <h2 id="standard_windows_vfses"><span>3.2. </span>Standard Windows VFSes</h2>
248
249
250 <p>
251 Windows builds also come with multiple built-in VFSes. The default
252 Windows VFS is called "win32" and is used in most applications.
253 Other VFSes that might be found on windows builds include:
254 </p>
255
256 <ol>
257 <li><p><b>win32-longpath</b> - like "win32" except that pathnames can
258 be up to 65534 bytes in length, whereas pathnames max out at
259 1040 bytes in "win32".
260 </p></li><li><p><b>win32-none</b> - all file locking operations are no-ops.
261 </p></li><li><p><b>win32-longpath-none</b> - combination of "win32-longpath"
262 and "win32-none" - long pathnames are supported and all lock
263 operations are no-ops.
264 </p></li></ol>
265
266 <p>As with unix, most of the code for the various Windows VFSes is shared.
267
268 </p><h2 id="specifying_which_vfs_to_use"><span>3.3. </span>Specifying Which VFS To Use</h2>
269
270 <p>
271 There is always one VFS which is the default VFS. On unix systems,
272 the "unix" VFS comes up as the default and on windows it is "win32".
273 If no other actions are taken, new database connections will make use
274 of the default VFS.
275 </p>
276
277 <p>
278 The default VFS can be changed by registering or re-registering the
279 VFS using the <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface with a second parameter
280 of 1. Hence, if a (unix) process wants to always use the "unix-nolock" VFS
281 in place of "unix", the following code would work:
282 </p>
283
284 <blockquote><pre>
285 sqlite3_vfs_register(sqlite3_vfs_find("unix-nolock"), 1);
286 </pre></blockquote>
287
288 <p>
289 An alternate VFS can also be specified as the 4th parameter to the
290 <a href="c3ref/open.html">sqlite3_open_v2()</a> function. For example:
291 </p>
292
293 <blockquote><pre>
294 int rc = sqlite3_open_v2("demo.db", &db, SQLITE_OPEN_READWRITE, "unix-nolock");
295 </pre></blockquote>
296
297 <p>
298 Finally, if <a href="uri.html">URI filenames</a> have been enabled, then the alternative
299 VFS can be specified using the "vfs=" parameter on the URI. This technique
300 works with <a href="c3ref/open.html">sqlite3_open()</a>, <a href="c3ref/open.html">sqlite3_open16()</a>, <a href="c3ref/open.html">sqlite3_open_v2()</a>, and
301 when a new database is <a href="lang_attach.html">ATTACH</a>-ed to an existing database connection.
302 For example:
303 </p>
304
305 <blockquote><pre>
306 ATTACH 'file:demo2.db?vfs=unix-none' AS demo2;
307 </pre></blockquote>
308
309 <p>
310 The VFS specified by a URI has the highest priority. After that comes
311 a VFS specified as the fourth argument to <a href="c3ref/open.html">sqlite3_open_v2()</a>. The
312 default VFS is used if no VFS is specified otherwise.
313 </p>
314
315 <a name="shim"></a>
316
317 <h2 id="vfs_shims"><span>3.4. </span>VFS Shims</h2>
318
319 <p>
320 From the point of view of the upper layers of the SQLite stack, each
321 open database file uses exactly one VFS.
322 But in practice, a particular VFS might
323 just be a thin wrapper around another VFS that does the real work.
324 We call a wrapper VFS a "shim".
325 </p>
326
327 <p>
328 A simple example of a shim is the "vfstrace" VFS. This is a VFS
329 (implemented in the
330 <a href="https://sqlite.org/src/doc/trunk/ext/misc/vfstrace.c">vfstrace.c</a>
331 source file) that writes a message associated with each VFS method call
332 into a log file, then passes control off to another VFS to do the actual
333 work.
334 </p>
335
336 <h2 id="other_example_vfses"><span>3.5. </span>Other Example VFSes</h2>
337
338 <p>
339 The following are other VFS implementations available in the public
340 SQLite source tree:
341 </p>
342
343 <ul>
344 <li><p>
345 <a href="https://sqlite.org/src/file/ext/misc/appendvfs.c">appendvfs.c</a> -
346 This VFS allows an SQLite database to be appended to the end of some
347 other file. This can be used, for example, to append an SQLite database
348 onto the end of an executable such that, when run, it can easily
349 locate the appended database. The <a href="cli.html">command-line shell</a> will use this
350 VFS if launched with the --append option, and its .archive command
351 will use it given the --append flag.
352
353 </p></li><li><p>
354 <a href="https://sqlite.org/src/doc/trunk/src/test_demovfs.c">test_demovfs.c</a> -
355 This file implements a very simple VFS named "demo" that uses POSIX
356 functions such as
357 open(), read(), write(), fsync(), close(), fsync(), sleep(), time(),
358 and so forth. This VFS only works on unix systems. But it is not
359 intended as a replacement for the standard "unix" VFS used by default
360 on unix platforms. The "demo" VFS is deliberately kept very simple
361 so that it can be used as a learning aid or as template for building
362 other VFSes or for porting SQLite to new operating systems.
363
364 </p></li><li><p>
365 <a href="https://sqlite.org/src/doc/trunk/src/test_quota.c">test_quota.c</a> -
366 This file implements a shim called "quota" that enforces cumulative
367 file size limits on a collection of database files. An auxiliary
368 interface is used to define "quota groups". A quota group is a
369 set of files (database files, journals, and temporary files) whose
370 names all match a <a href="lang_expr.html#glob">GLOB</a> pattern. The sum of the sizes of all files
371 in each quota group is tracked, and if that sum exceeds a threshold
372 defined for the quota group, a callback function is invoked. That
373 callback can either increase the threshold or cause the operation
374 that would have exceeded the quota to fail with an
375 <a href="rescode.html#full">SQLITE_FULL</a> error. One of the uses of this shim is used to enforce
376 resource limits on application databases in Firefox.
377
378 </p></li><li><p>
379 <a href="https://sqlite.org/src/doc/trunk/src/test_multiplex.c">test_multiplex.c</a> -
380 This file implements a shim that allows database files to exceed the
381 maximum file size of the underlying filesystem. This shim presents
382 an interface to the upper six layers of SQLite that makes it look like
383 very large files are being used, when in reality each such large file
384 is split up into many smaller files on the underlying system.
385 This shim has been used, for example, to allow databases to grow
386 larger than 2 gibibytes on FAT16 filesystems.
387
388 </p></li><li><p>
389 <a href="https://sqlite.org/src/doc/trunk/src/test_onefile.c">test_onefile.c</a> -
390 This file implements a demonstration VFS named "fs" that shows how SQLite
391 can be used on an embedded device that lacks a filesystem. Content is
392 written directly to the underlying media. A VFS derived from this
393 demonstration code could be used by a gadget with a limited amount of
394 flash memory to make SQLite behave as the filesystem for the flash memory
395 on the device.
396
397 </p></li><li><p>
398 <a href="https://sqlite.org/src/doc/trunk/src/test_journal.c">test_journal.c</a> -
399 This file implements a shim used during SQLite testing that verifies that
400 the database and rollback journal are written in the correct order and
401 are "synced" at appropriate times in order to guarantee that the database
402 can recover from a power loss or hard reset at any time. The shim
403 checks several invariants on the operation of databases and rollback
404 journals and raises exceptions if any of those invariants are violated.
405 These invariants, in turn, assure that the database is always recoverable.
406 Running a large suite of test cases using this shim provides added
407 assurance that SQLite databases will not be damaged by unexpected
408 power failures or device resets.
409
410 </p></li><li><p>
411 <a href="https://sqlite.org/src/doc/trunk/src/test_vfs.c">test_vfs.c</a> -
412 This file implements a shim that can be used to simulate filesystem faults.
413 This shim is used during testing to verify that SQLite responds sanely
414 to hardware malfunctions or to other error conditions such as running out
415 of filesystem space that are difficult to test on a real system.
416 </p></li></ul>
417
418 <p>
419 There are other VFS implementations both in the core SQLite source code
420 library and in available extensions. The list above is not meant to be
421 exhaustive but merely representative of the kinds of features that can
422 be realized using the VFS interface.
423 </p>
424
425 <h1 id="vfs_implementations"><span>4. </span>VFS Implementations</h1>
426
427 <p>
428 A new VFS is implemented by subclassing three objects:
429 </p>
430
431 <ul>
432 <li><a href="c3ref/vfs.html">sqlite3_vfs</a>
433 </li><li><a href="c3ref/io_methods.html">sqlite3_io_methods</a>
434 </li><li><a href="c3ref/file.html">sqlite3_file</a>
435 </li></ul>
436
437 <p>
438 An <a href="c3ref/vfs.html">sqlite3_vfs</a> object defines the name of the VFS and the core
439 methods that implement the interface to the operating system, such
440 as checking for existence of files, deleting files, creating files
441 and opening and for reading and/or writing, converting filenames
442 into their canonical form. The <a href="c3ref/vfs.html">sqlite3_vfs</a> object also contains
443 methods for obtaining randomness from the operating system, for
444 suspending a process (sleeping) and for finding the current date and
445 time.
446 </p>
447
448 <p>
449 The <a href="c3ref/file.html">sqlite3_file</a> object represents an open file.
450 The xOpen method of <a href="c3ref/vfs.html">sqlite3_vfs</a> constructs an <a href="c3ref/file.html">sqlite3_file</a>
451 object when the file is opened. The <a href="c3ref/file.html">sqlite3_file</a> keeps track
452 of the state of the file while it is opened.
453 </p>
454
455 <p>
456 The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object holds the methods used to interact
457 with an open file. Each <a href="c3ref/file.html">sqlite3_file</a> contains a pointer to
458 an <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object that is appropriate for the file it
459 represents. The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object contains methods to do
460 things such as read and write from the file, to truncate the file,
461 to flush any changes to persistent storage, to find the size of the
462 file, to lock and unlock the file, and to close file and destroy
463 the <a href="c3ref/file.html">sqlite3_file</a> object.
464 </p>
465
466 <p>
467 Writing the code for a new VFS involves constructing a subclass for
468 the <a href="c3ref/vfs.html">sqlite3_vfs</a> object and then registering that VFS object using
469 a call to <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a>. The VFS implementation also
470 provides subclasses for <a href="c3ref/file.html">sqlite3_file</a> and <a href="c3ref/io_methods.html">sqlite3_io_methods</a> but
471 those objects are not registered directly with SQLite. Instead, the
472 <a href="c3ref/file.html">sqlite3_file</a> object is returned from the xOpen method of
473 <a href="c3ref/vfs.html">sqlite3_vfs</a> and the <a href="c3ref/file.html">sqlite3_file</a> object points to an instance
474 of the <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object.
475 </p>
476 <p align="center"><small><i>This page was last updated on 2025-05-31 13:08:22Z </small></i></p>
477