| 1 |
<!DOCTYPE html> |
| 2 |
<html><head> |
| 3 |
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| 4 |
<meta http-equiv="content-type" content="text/html; charset=UTF-8"> |
| 5 |
<link href="sqlite.css" rel="stylesheet"> |
| 6 |
<title>Invalid UTF Policy</title> |
| 7 |
<!-- path= --> |
| 8 |
</head> |
| 9 |
<body> |
| 10 |
<div class=nosearch> |
| 11 |
<a href="index.html"> |
| 12 |
<img class="logo" src="images/sqlite370_banner.svg" alt="SQLite" border="0"> |
| 13 |
</a> |
| 14 |
<div><!-- IE hack to prevent disappearing logo --></div> |
| 15 |
<div class="tagline desktoponly"> |
| 16 |
Small. Fast. Reliable.<br>Choose any three. |
| 17 |
</div> |
| 18 |
<div class="menu mainmenu"> |
| 19 |
<ul> |
| 20 |
<li><a href="index.html">Home</a> |
| 21 |
<li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a> |
| 22 |
<li class='wideonly'><a href='about.html'>About</a> |
| 23 |
<li class='desktoponly'><a href="docs.html">Documentation</a> |
| 24 |
<li class='desktoponly'><a href="download.html">Download</a> |
| 25 |
<li class='wideonly'><a href='copyright.html'>License</a> |
| 26 |
<li class='desktoponly'><a href="support.html">Support</a> |
| 27 |
<li class='desktoponly'><a href="prosupport.html">Purchase</a> |
| 28 |
<li class='search' id='search_menubutton'> |
| 29 |
<a href="javascript:void(0)" onclick='toggle_search()'>Search</a> |
| 30 |
</ul> |
| 31 |
</div> |
| 32 |
<div class="menu submenu" id="submenu"> |
| 33 |
<ul> |
| 34 |
<li><a href='about.html'>About</a> |
| 35 |
<li><a href='docs.html'>Documentation</a> |
| 36 |
<li><a href='download.html'>Download</a> |
| 37 |
<li><a href='support.html'>Support</a> |
| 38 |
<li><a href='prosupport.html'>Purchase</a> |
| 39 |
</ul> |
| 40 |
</div> |
| 41 |
<div class="searchmenu" id="searchmenu"> |
| 42 |
<form method="GET" action="search"> |
| 43 |
<select name="s" id="searchtype"> |
| 44 |
<option value="d">Search Documentation</option> |
| 45 |
<option value="c">Search Changelog</option> |
| 46 |
</select> |
| 47 |
<input type="text" name="q" id="searchbox" value=""> |
| 48 |
<input type="submit" value="Go"> |
| 49 |
</form> |
| 50 |
</div> |
| 51 |
</div> |
| 52 |
<script> |
| 53 |
function toggle_div(nm) { |
| 54 |
var w = document.getElementById(nm); |
| 55 |
if( w.style.display=="block" ){ |
| 56 |
w.style.display = "none"; |
| 57 |
}else{ |
| 58 |
w.style.display = "block"; |
| 59 |
} |
| 60 |
} |
| 61 |
function toggle_search() { |
| 62 |
var w = document.getElementById("searchmenu"); |
| 63 |
if( w.style.display=="block" ){ |
| 64 |
w.style.display = "none"; |
| 65 |
} else { |
| 66 |
w.style.display = "block"; |
| 67 |
setTimeout(function(){ |
| 68 |
document.getElementById("searchbox").focus() |
| 69 |
}, 30); |
| 70 |
} |
| 71 |
} |
| 72 |
function div_off(nm){document.getElementById(nm).style.display="none";} |
| 73 |
window.onbeforeunload = function(e){div_off("submenu");} |
| 74 |
/* Disable the Search feature if we are not operating from CGI, since */ |
| 75 |
/* Search is accomplished using CGI and will not work without it. */ |
| 76 |
if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){ |
| 77 |
document.getElementById("search_menubutton").style.display = "none"; |
| 78 |
} |
| 79 |
/* Used by the Hide/Show button beside syntax diagrams, to toggle the */ |
| 80 |
function hideorshow(btn,obj){ |
| 81 |
var x = document.getElementById(obj); |
| 82 |
var b = document.getElementById(btn); |
| 83 |
if( x.style.display!='none' ){ |
| 84 |
x.style.display = 'none'; |
| 85 |
b.innerHTML='show'; |
| 86 |
}else{ |
| 87 |
x.style.display = ''; |
| 88 |
b.innerHTML='hide'; |
| 89 |
} |
| 90 |
return false; |
| 91 |
} |
| 92 |
var antiRobot = 0; |
| 93 |
function antiRobotGo(){ |
| 94 |
if( antiRobot!=3 ) return; |
| 95 |
antiRobot = 7; |
| 96 |
var j = document.getElementById("mtimelink"); |
| 97 |
if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href"); |
| 98 |
} |
| 99 |
function antiRobotDefense(){ |
| 100 |
document.body.onmousedown=function(){ |
| 101 |
antiRobot |= 2; |
| 102 |
antiRobotGo(); |
| 103 |
document.body.onmousedown=null; |
| 104 |
} |
| 105 |
document.body.onmousemove=function(){ |
| 106 |
antiRobot |= 2; |
| 107 |
antiRobotGo(); |
| 108 |
document.body.onmousemove=null; |
| 109 |
} |
| 110 |
setTimeout(function(){ |
| 111 |
antiRobot |= 1; |
| 112 |
antiRobotGo(); |
| 113 |
}, 100) |
| 114 |
antiRobotGo(); |
| 115 |
} |
| 116 |
antiRobotDefense(); |
| 117 |
</script> |
| 118 |
<div class=fancy> |
| 119 |
<div class=nosearch> |
| 120 |
<div class="fancy_title"> |
| 121 |
Invalid UTF Policy |
| 122 |
</div> |
| 123 |
</div> |
| 124 |
|
| 125 |
|
| 126 |
|
| 127 |
|
| 128 |
|
| 129 |
<h1 id="garbage_in_garbage_out"><span>1. </span>Garbage In, Garbage Out</h1> |
| 130 |
|
| 131 |
<p> |
| 132 |
With regard to invalid UTF, SQLite follows a policy of |
| 133 |
Garbage-In, Garbage-Out (GIGO). If you insert invalid UTF |
| 134 |
into an SQLite database, then try to query that data, what you get back out |
| 135 |
might not be exactly what you put in. If you put garbage in, then you |
| 136 |
may not complain if you get different garbage back out. |
| 137 |
|
| 138 |
</p><p> |
| 139 |
For the purposes of this discussion, "invalid UTF" can mean any of |
| 140 |
the following circumstances: |
| 141 |
|
| 142 |
</p><ul> |
| 143 |
<li><p> |
| 144 |
Invalid surrogate pairs in UTF-16. |
| 145 |
|
| 146 |
</p></li><li><p> |
| 147 |
Invalid multi-byte sequences in UTF-8. |
| 148 |
|
| 149 |
</p></li><li><p> |
| 150 |
Using more bytes of UTF-8 than necessary to represent a single |
| 151 |
code point. (Example: encoding 'A' as the two-byte sequence |
| 152 |
0xc1, 0x01 instead of just a single 0x41 byte.) |
| 153 |
|
| 154 |
</p></li><li><p> |
| 155 |
NUL characters (U+0000) embedded in strings. |
| 156 |
|
| 157 |
</p></li><li><p> |
| 158 |
Invalid sequences of combining characters. |
| 159 |
|
| 160 |
</p></li><li><p> |
| 161 |
UTF-8 or UTF-16 bytes sequences that encode numbers that are not |
| 162 |
defined Unicode characters. |
| 163 |
|
| 164 |
</p></li></ul> |
| 165 |
|
| 166 |
<h2 id="invalid_utf_will_never_cause_memory_errors"><span>1.1. </span>Invalid UTF will never cause memory errors</h2> |
| 167 |
|
| 168 |
<p> |
| 169 |
If you insert invalid UTF into an SQLite database, then SQLite makes |
| 170 |
no guarantees about what text you might get back out. But it does |
| 171 |
promise that invalid UTF will never cause memory errors |
| 172 |
(array overruns, reads or writes of uninitialized memory, etc), at |
| 173 |
least for the built-in processing of SQLite. |
| 174 |
In other words, invalid UTF will not cause SQLite to crash. |
| 175 |
|
| 176 |
</p><p> |
| 177 |
This promise only applies to the core SQLite components, not |
| 178 |
application-provided extensions, of course. |
| 179 |
If an application adds new application-defined SQL functions or |
| 180 |
virtual tables or collating sequences or other extensions, and a |
| 181 |
database contains invalid UTF, then invalid UTF might get passed |
| 182 |
into those extensions. If the invalid UTF causes one of those |
| 183 |
extensions to crash, then that is a problem with the extension, |
| 184 |
not with SQLite. |
| 185 |
|
| 186 |
</p><h1 id="no_enforcement_of_text_formatting_rules"><span>2. </span>No enforcement of text formatting rules</h1> |
| 187 |
|
| 188 |
<p> |
| 189 |
SQLite does not try to enforce UTF formatting rules. You can |
| 190 |
insert invalid UTF into a TEXT field and SQLite will not complain |
| 191 |
about this. It stores the invalid TEXT as best it can. SQLite |
| 192 |
sees its role in the world as a storage engine, not a text format |
| 193 |
validation engine. |
| 194 |
|
| 195 |
</p><h1 id="best_effort_to_preserve_text"><span>3. </span>Best effort to preserve text</h1> |
| 196 |
|
| 197 |
<p> |
| 198 |
SQLite does not promise to always preserve invalid UTF, but it does |
| 199 |
make an effort. Generally speaking, if you insert invalid UTF into |
| 200 |
SQLite, you will get the exact same byte sequence back out, as long |
| 201 |
as you do not ask SQLite to transform the text in any way. |
| 202 |
|
| 203 |
</p><p> |
| 204 |
For example, if you insert some UTF-16LE with invalid surrogates into |
| 205 |
a TEXT column of a table of a database that has <a href="pragma.html#pragma_encoding">PRAGMA encoding=UTF16LE</a>, |
| 206 |
then later query that column using <a href="c3ref/column_blob.html">sqlite3_column_text16()</a>, you will |
| 207 |
probably get back the same exact invalid UTF-16. But if you insert the |
| 208 |
same invalid UTF-16LE content in a <a href="pragma.html#pragma_encoding">PRAGMA encoding=UTF8</a> database, |
| 209 |
the content must be converted into UTF8 when it is stored, which could |
| 210 |
cause irreversible changes to the content. Or if you insert that |
| 211 |
same invalid UTF-16LE content into a <a href="pragma.html#pragma_encoding">PRAGMA encoding=UTF16LE</a> database |
| 212 |
but then read it out using <a href="c3ref/column_blob.html">sqlite3_column_text()</a>, then a UTF16 to |
| 213 |
UTF8 conversion must occur during the read-out and that conversion might |
| 214 |
introduce irreversible changes. |
| 215 |
|
| 216 |
</p><p> |
| 217 |
Or, suppose you are doing everything using UTF-8 (the most common case). |
| 218 |
Invalid UTF-8 will normally pass through the database without any change |
| 219 |
in its byte sequence. However, if you try to transform the invalid |
| 220 |
UTF-8 with SQL functions like <a href="lang_corefunc.html#substr">substr()</a> or <a href="lang_corefunc.html#replace">replace()</a> |
| 221 |
or if you try to do string matching with the <a href="lang_expr.html#like">LIKE</a> operator, then |
| 222 |
you might get unexpected results. |
| 223 |
|
| 224 |
</p><p> |
| 225 |
So, in other words, SQLite does not actively try to subvert your |
| 226 |
invalid text. But when you ask SQLite to make transformations of invalid |
| 227 |
UTF, there are no guarantees that those transformations will be reversible |
| 228 |
or even sensible. |
| 229 |
|
| 230 |
</p><h1 id="invalid_utf_in_the_database_schema"><span>4. </span>Invalid UTF in the database schema</h1> |
| 231 |
|
| 232 |
<p> |
| 233 |
If a database schema contains names (table names, column names, index |
| 234 |
names, and so forth) that are invalid UTF, SQLite will continue to |
| 235 |
operate normally. As far as SQLite is concerned, those names are just |
| 236 |
byte sequences. SQLite does not care whether they are valid UTF or not. |
| 237 |
|
| 238 |
</p><p> |
| 239 |
When generating error messages (using, for example, <a href="c3ref/errcode.html">sqlite3_errmsg()</a>), |
| 240 |
sometimes SQLite will embed parts of the database schema into the |
| 241 |
error message. If those embedded schema elements |
| 242 |
are invalid UTF, then the resulting error message might also be |
| 243 |
invalid UTF. |
| 244 |
Similarly, the output from the <a href="pragma.html#pragma_integrity_check">PRAGMA integrity_check</a> and similar |
| 245 |
statements will sometimes embed the names of schema elements. If those |
| 246 |
schema element names are invalid UTF, then the output of the command |
| 247 |
will also be invalid UTF. |
| 248 |
</p><p align="center"><small><i>This page was last updated on 2025-05-31 13:08:22Z </small></i></p> |
| 249 |
|