Debian Codegraph @

www/invalidutf.html

raw
1 <!DOCTYPE html>
2 <html><head>
3 <meta name="viewport" content="width=device-width, initial-scale=1.0">
4 <meta http-equiv="content-type" content="text/html; charset=UTF-8">
5 <link href="sqlite.css" rel="stylesheet">
6 <title>Invalid UTF Policy</title>
7 <!-- path= -->
8 </head>
9 <body>
10 <div class=nosearch>
11 <a href="index.html">
12 <img class="logo" src="images/sqlite370_banner.svg" alt="SQLite" border="0">
13 </a>
14 <div><!-- IE hack to prevent disappearing logo --></div>
15 <div class="tagline desktoponly">
16 Small. Fast. Reliable.<br>Choose any three.
17 </div>
18 <div class="menu mainmenu">
19 <ul>
20 <li><a href="index.html">Home</a>
21 <li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a>
22 <li class='wideonly'><a href='about.html'>About</a>
23 <li class='desktoponly'><a href="docs.html">Documentation</a>
24 <li class='desktoponly'><a href="download.html">Download</a>
25 <li class='wideonly'><a href='copyright.html'>License</a>
26 <li class='desktoponly'><a href="support.html">Support</a>
27 <li class='desktoponly'><a href="prosupport.html">Purchase</a>
28 <li class='search' id='search_menubutton'>
29 <a href="javascript:void(0)" onclick='toggle_search()'>Search</a>
30 </ul>
31 </div>
32 <div class="menu submenu" id="submenu">
33 <ul>
34 <li><a href='about.html'>About</a>
35 <li><a href='docs.html'>Documentation</a>
36 <li><a href='download.html'>Download</a>
37 <li><a href='support.html'>Support</a>
38 <li><a href='prosupport.html'>Purchase</a>
39 </ul>
40 </div>
41 <div class="searchmenu" id="searchmenu">
42 <form method="GET" action="search">
43 <select name="s" id="searchtype">
44 <option value="d">Search Documentation</option>
45 <option value="c">Search Changelog</option>
46 </select>
47 <input type="text" name="q" id="searchbox" value="">
48 <input type="submit" value="Go">
49 </form>
50 </div>
51 </div>
52 <script>
53 function toggle_div(nm) {
54 var w = document.getElementById(nm);
55 if( w.style.display=="block" ){
56 w.style.display = "none";
57 }else{
58 w.style.display = "block";
59 }
60 }
61 function toggle_search() {
62 var w = document.getElementById("searchmenu");
63 if( w.style.display=="block" ){
64 w.style.display = "none";
65 } else {
66 w.style.display = "block";
67 setTimeout(function(){
68 document.getElementById("searchbox").focus()
69 }, 30);
70 }
71 }
72 function div_off(nm){document.getElementById(nm).style.display="none";}
73 window.onbeforeunload = function(e){div_off("submenu");}
74 /* Disable the Search feature if we are not operating from CGI, since */
75 /* Search is accomplished using CGI and will not work without it. */
76 if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){
77 document.getElementById("search_menubutton").style.display = "none";
78 }
79 /* Used by the Hide/Show button beside syntax diagrams, to toggle the */
80 function hideorshow(btn,obj){
81 var x = document.getElementById(obj);
82 var b = document.getElementById(btn);
83 if( x.style.display!='none' ){
84 x.style.display = 'none';
85 b.innerHTML='show';
86 }else{
87 x.style.display = '';
88 b.innerHTML='hide';
89 }
90 return false;
91 }
92 var antiRobot = 0;
93 function antiRobotGo(){
94 if( antiRobot!=3 ) return;
95 antiRobot = 7;
96 var j = document.getElementById("mtimelink");
97 if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href");
98 }
99 function antiRobotDefense(){
100 document.body.onmousedown=function(){
101 antiRobot |= 2;
102 antiRobotGo();
103 document.body.onmousedown=null;
104 }
105 document.body.onmousemove=function(){
106 antiRobot |= 2;
107 antiRobotGo();
108 document.body.onmousemove=null;
109 }
110 setTimeout(function(){
111 antiRobot |= 1;
112 antiRobotGo();
113 }, 100)
114 antiRobotGo();
115 }
116 antiRobotDefense();
117 </script>
118 <div class=fancy>
119 <div class=nosearch>
120 <div class="fancy_title">
121 Invalid UTF Policy
122 </div>
123 </div>
124
125
126
127
128
129 <h1 id="garbage_in_garbage_out"><span>1. </span>Garbage In, Garbage Out</h1>
130
131 <p>
132 With regard to invalid UTF, SQLite follows a policy of
133 Garbage-In, Garbage-Out (GIGO). If you insert invalid UTF
134 into an SQLite database, then try to query that data, what you get back out
135 might not be exactly what you put in. If you put garbage in, then you
136 may not complain if you get different garbage back out.
137
138 </p><p>
139 For the purposes of this discussion, "invalid UTF" can mean any of
140 the following circumstances:
141
142 </p><ul>
143 <li><p>
144 Invalid surrogate pairs in UTF-16.
145
146 </p></li><li><p>
147 Invalid multi-byte sequences in UTF-8.
148
149 </p></li><li><p>
150 Using more bytes of UTF-8 than necessary to represent a single
151 code point. (Example: encoding 'A' as the two-byte sequence
152 0xc1, 0x01 instead of just a single 0x41 byte.)
153
154 </p></li><li><p>
155 NUL characters (U+0000) embedded in strings.
156
157 </p></li><li><p>
158 Invalid sequences of combining characters.
159
160 </p></li><li><p>
161 UTF-8 or UTF-16 bytes sequences that encode numbers that are not
162 defined Unicode characters.
163
164 </p></li></ul>
165
166 <h2 id="invalid_utf_will_never_cause_memory_errors"><span>1.1. </span>Invalid UTF will never cause memory errors</h2>
167
168 <p>
169 If you insert invalid UTF into an SQLite database, then SQLite makes
170 no guarantees about what text you might get back out. But it does
171 promise that invalid UTF will never cause memory errors
172 (array overruns, reads or writes of uninitialized memory, etc), at
173 least for the built-in processing of SQLite.
174 In other words, invalid UTF will not cause SQLite to crash.
175
176 </p><p>
177 This promise only applies to the core SQLite components, not
178 application-provided extensions, of course.
179 If an application adds new application-defined SQL functions or
180 virtual tables or collating sequences or other extensions, and a
181 database contains invalid UTF, then invalid UTF might get passed
182 into those extensions. If the invalid UTF causes one of those
183 extensions to crash, then that is a problem with the extension,
184 not with SQLite.
185
186 </p><h1 id="no_enforcement_of_text_formatting_rules"><span>2. </span>No enforcement of text formatting rules</h1>
187
188 <p>
189 SQLite does not try to enforce UTF formatting rules. You can
190 insert invalid UTF into a TEXT field and SQLite will not complain
191 about this. It stores the invalid TEXT as best it can. SQLite
192 sees its role in the world as a storage engine, not a text format
193 validation engine.
194
195 </p><h1 id="best_effort_to_preserve_text"><span>3. </span>Best effort to preserve text</h1>
196
197 <p>
198 SQLite does not promise to always preserve invalid UTF, but it does
199 make an effort. Generally speaking, if you insert invalid UTF into
200 SQLite, you will get the exact same byte sequence back out, as long
201 as you do not ask SQLite to transform the text in any way.
202
203 </p><p>
204 For example, if you insert some UTF-16LE with invalid surrogates into
205 a TEXT column of a table of a database that has <a href="pragma.html#pragma_encoding">PRAGMA encoding=UTF16LE</a>,
206 then later query that column using <a href="c3ref/column_blob.html">sqlite3_column_text16()</a>, you will
207 probably get back the same exact invalid UTF-16. But if you insert the
208 same invalid UTF-16LE content in a <a href="pragma.html#pragma_encoding">PRAGMA encoding=UTF8</a> database,
209 the content must be converted into UTF8 when it is stored, which could
210 cause irreversible changes to the content. Or if you insert that
211 same invalid UTF-16LE content into a <a href="pragma.html#pragma_encoding">PRAGMA encoding=UTF16LE</a> database
212 but then read it out using <a href="c3ref/column_blob.html">sqlite3_column_text()</a>, then a UTF16 to
213 UTF8 conversion must occur during the read-out and that conversion might
214 introduce irreversible changes.
215
216 </p><p>
217 Or, suppose you are doing everything using UTF-8 (the most common case).
218 Invalid UTF-8 will normally pass through the database without any change
219 in its byte sequence. However, if you try to transform the invalid
220 UTF-8 with SQL functions like <a href="lang_corefunc.html#substr">substr()</a> or <a href="lang_corefunc.html#replace">replace()</a>
221 or if you try to do string matching with the <a href="lang_expr.html#like">LIKE</a> operator, then
222 you might get unexpected results.
223
224 </p><p>
225 So, in other words, SQLite does not actively try to subvert your
226 invalid text. But when you ask SQLite to make transformations of invalid
227 UTF, there are no guarantees that those transformations will be reversible
228 or even sensible.
229
230 </p><h1 id="invalid_utf_in_the_database_schema"><span>4. </span>Invalid UTF in the database schema</h1>
231
232 <p>
233 If a database schema contains names (table names, column names, index
234 names, and so forth) that are invalid UTF, SQLite will continue to
235 operate normally. As far as SQLite is concerned, those names are just
236 byte sequences. SQLite does not care whether they are valid UTF or not.
237
238 </p><p>
239 When generating error messages (using, for example, <a href="c3ref/errcode.html">sqlite3_errmsg()</a>),
240 sometimes SQLite will embed parts of the database schema into the
241 error message. If those embedded schema elements
242 are invalid UTF, then the resulting error message might also be
243 invalid UTF.
244 Similarly, the output from the <a href="pragma.html#pragma_integrity_check">PRAGMA integrity_check</a> and similar
245 statements will sometimes embed the names of schema elements. If those
246 schema element names are invalid UTF, then the output of the command
247 will also be invalid UTF.
248 </p><p align="center"><small><i>This page was last updated on 2025-05-31 13:08:22Z </small></i></p>
249