| 1 |
The Unicode Consortium provide a range of data files detailing |
| 2 |
the nature of code points in Unicode. These data files are |
| 3 |
machine-readable but large. Here, a set of loaders are provided |
| 4 |
to parse these files during a TeX run and set appropriate |
| 5 |
parameters in an automated fashion. |
| 6 |
|
| 7 |
File: `load-unicode-data.tex` |
| 8 |
============================= |
| 9 |
|
| 10 |
The loader parses a number of data files provided by the Unicode |
| 11 |
Consortium and when used with used Unicode-capable engine sets |
| 12 |
up a range of TeX-related parameters based on the extracted |
| 13 |
information. |
| 14 |
|
| 15 |
From the file `UnicodeData.txt` the following properties are set: |
| 16 |
- `\catcode` 11 for all letters (Unicode class "L") |
| 17 |
- `\catcode` 11 for all combining marks (Unicode class "M") |
| 18 |
- `\sfcode` 999 for all code points of class "Lu" (upper case |
| 19 |
letters) |
| 20 |
- `\lccode` for all of class "Ll" (lower case letters) to the code |
| 21 |
point itself, and `\uccode` to the upper case mapping (or if |
| 22 |
not given to the code point itself) |
| 23 |
- `\uccode` for all of class "Lu" (upper case letters) to the code |
| 24 |
point itself, and `\lccode` to the lower case mapping (or if |
| 25 |
not given to the code point itself) |
| 26 |
- `\lccode` and `\uccode` for all of class "Lt" (title case |
| 27 |
letters) to the lower and upper case mappings (or if not given |
| 28 |
to the code point itself) |
| 29 |
- `\lccode` and `\uccode` for all other letter code points are |
| 30 |
set to the code point itself |
| 31 |
- `\lccode` and/or `\uccode` for non-letter code points for |
| 32 |
which an upper or lower case mapping is given |
| 33 |
- `\sfcode` 0 (ignored) for code points of Unicode classes "Pe" |
| 34 |
(closing punctuation marks) and "Pf" (final quotation marks) |
| 35 |
- `\Umathcode` for all letters (Unicode class "L") and combining |
| 36 |
marks (Unicode class "M") as TeX class 7 (var) |
| 37 |
|
| 38 |
In the above, 'letters' refers to code points with Unicode class |
| 39 |
"L", irrespective of their TeX category code. |
| 40 |
|
| 41 |
File: `load-unicode-math-classes.tex` |
| 42 |
====================================== |
| 43 |
|
| 44 |
From the file `MathClass-15.txt` the following mapping between |
| 45 |
Unicode math classes and TeX math types are set up |
| 46 |
- "L" (large) `\mathop` |
| 47 |
- "B" (binary) `\mathbin` |
| 48 |
- "V" (vary) `\mathbin` |
| 49 |
- "R" (relation) `\mathrel` |
| 50 |
- "O" (opening) `\mathopen` |
| 51 |
- "C" (closing) `\mathclose` |
| 52 |
- "P" (punctuation) `\mathpunct` |
| 53 |
- "A" (alphabetic) `\mathalpha` |
| 54 |
|
| 55 |
For each code point processed, the result is of the form |
| 56 |
|
| 57 |
\Umathcode <codepoint> = <type> 1 <codepoint> |
| 58 |
|
| 59 |
File: `load-unicode-xetex-classes.tex` |
| 60 |
====================================== |
| 61 |
|
| 62 |
From the files `EastAsianWidth.txt` and `LineBreak.txt` inter |
| 63 |
character classes for XeTeX (`\XeTeXcharclass`) are set for the |
| 64 |
following classes |
| 65 |
- "ID" (ideographic) |
| 66 |
- "CJ" (conditional Japanese starter) |
| 67 |
- "OP" (opener) |
| 68 |
- "CL" (closer) |
| 69 |
- "NS" (non-starter) |
| 70 |
- "EX" (exclamation) |
| 71 |
- "IS" (infix separator) |
| 72 |
- "CM" (combining marks) |
| 73 |
|
| 74 |
All code points of classes "ID" and "CJ" are assigned to a |
| 75 |
`\XeTeXcharclass`, but for other classes this only occurs when |
| 76 |
they fall into east Asian width type "F", "H" or "W" (full-, |
| 77 |
half- and wide-width). |
| 78 |
|
| 79 |
The following mappings between Unicode and XeTeX classes occur |
| 80 |
- "ID" and "CJ" are class 1 |
| 81 |
- "OP" is class 2 |
| 82 |
- "CL", "NS", "EX", "IS" are class 3 |
| 83 |
- "CM" is class 256 (ignored) |
| 84 |
|
| 85 |
as standard: these may be over-ridden by defining |
| 86 |
`\XeTeXcharclass<class>` as required. (If classes "ID" or "CL" |
| 87 |
are explicitly set, the other members of the same groups above |
| 88 |
will inherit these values.) |
| 89 |
|
| 90 |
This file does _not_ activate XeTeX's inter-character token |
| 91 |
mechanism (`\XeTeXinterchartokenstate` is not set) nor does it |
| 92 |
install any material in the inter-character token registers. |
| 93 |
|
| 94 |
Note that this file is separate from the main loader as the data |
| 95 |
structure here may need more refinement at the macro level. |
| 96 |
|
| 97 |
Data files |
| 98 |
========== |
| 99 |
|
| 100 |
The following data files are included here with permission of the |
| 101 |
Unicode Consortium (see below): |
| 102 |
- `UnicodeData.txt` |
| 103 |
- `BidiBrackets.txt` |
| 104 |
- `BidiMirroring.txt` |
| 105 |
- `BidiMirroring-5-1-0.txt` |
| 106 |
- `CaseFolding.txt` |
| 107 |
- `EastAsianWidth.txt` |
| 108 |
- `emoji-data.txt` |
| 109 |
- `GraphemeBreakProperty.txt` |
| 110 |
- `LineBreak.txt` |
| 111 |
- `MathClass.txt` |
| 112 |
- `PropList.txt` |
| 113 |
- `SpecialCasing.txt` |
| 114 |
- `Scripts.txt` |
| 115 |
- `ScriptExtension.txt` |
| 116 |
- `WordBreakProperty.txt` |
| 117 |
|
| 118 |
With the exception of `MathClass.txt`, which is currently not |
| 119 |
formally part of the Unicode Character Database, and |
| 120 |
`BidiMirroring-5-1-0.txt`, which is a data file for Unicode 5.1.0, |
| 121 |
all of the data files here are for Unicode 15.1.0. |
| 122 |
|
| 123 |
`BidiMirroring-5-1-0.txt` will never be updated and is included for |
| 124 |
OpenType font support: While `BidiMirroring.txt` lists all mirror |
| 125 |
pairs in the current Unicode version, only the pairs from Unicode 5.1.0 |
| 126 |
are handled directly by the OpenType rendering system. All newer pairs |
| 127 |
have to be handled through font features. |
| 128 |
|
| 129 |
Requests to include additional Unicode data files in this bundle are |
| 130 |
welcome. Ideally this bundle will act as a single location for all |
| 131 |
Unicode data for use by TeX systems. |
| 132 |
|
| 133 |
Issues and improvements |
| 134 |
======================= |
| 135 |
|
| 136 |
The home page for this bundle is |
| 137 |
https://github.com/latex3/unicode-data, and issues may be |
| 138 |
reported there. |
| 139 |
|
| 140 |
Changes |
| 141 |
======= |
| 142 |
|
| 143 |
- v1.0 (2015-12-01) |
| 144 |
- First public release |
| 145 |
- v1.1 (2016-01-04) |
| 146 |
- Allow setting of XeTeX character class slot numbers |
| 147 |
- v1.1a (2016-01-06) |
| 148 |
- Correctly assign XeTeX character class for code points in |
| 149 |
ranges |
| 150 |
- Fix issue which meant `EastAsianWidth.txt` was not read |
| 151 |
- v1.1b (2016-01-08) |
| 152 |
- Avoid save stack issue when loading `EastAsianWidth.txt` |
| 153 |
- Fix application of settings to ranges of Unicode characters |
| 154 |
- Correct setting of space factor for ending punctuation |
| 155 |
- v1.2 (2016-02-02) |
| 156 |
- Drop use of Unicode math data at present: undesired effect on |
| 157 |
output if Unicode fonts are not loaded (classical OT1 |
| 158 |
situation) |
| 159 |
- v1.2a (2016-02-05) |
| 160 |
- Correct split between base and supplementary plane when |
| 161 |
setting math codes |
| 162 |
- v1.3 (2016-02-06) |
| 163 |
- New `load-unicode-math-classes.tex` loader |
| 164 |
- Set all letters as math type 7 (var) in |
| 165 |
`load-unicode-data.tex` |
| 166 |
- v1.3a (2016-02-06) |
| 167 |
- Add definition of `\loop` to `load-unicode-math-classes.tex` |
| 168 |
- v1.4 (2016-02-16) |
| 169 |
- Treat "CJ" chars like "ID" chars for setting XeTeX character |
| 170 |
class |
| 171 |
- v1.4a (2016-02-21) |
| 172 |
- Inherit XeTeX classes for "CJ" from "ID" and for "NS", "EX" |
| 173 |
and "IS" from "CL" if appropriate |
| 174 |
- v1.5 (2016-06-22) |
| 175 |
- Update data files to Unicode release 9.0.0 |
| 176 |
- v1.6 (2017-06-27) |
| 177 |
- Update data files to Unicode release 10.0.0 |
| 178 |
- v1.7 (2018-08-09) |
| 179 |
- Update data files to Unicode release 11.0.0 |
| 180 |
- v1.8 (2019-03-09) |
| 181 |
- Update data files to Unicode release 12.0.0 |
| 182 |
- v1.9 (2019-08-13) |
| 183 |
- Update data files to Unicode release 12.1.0 |
| 184 |
- Add `BidiBrackets.txt`, `Scripts.txt` and `ScriptExtension.txt` |
| 185 |
- v1.10 (2019-08-21) |
| 186 |
- Add `PropList.txt` |
| 187 |
- v1.11 (2019-11-28) |
| 188 |
- Add `BidiMirroring.txt` and `BidiMirroring-5-1-0.txt` |
| 189 |
- v1.12 (2020-03-11) |
| 190 |
- Update data files to Unicode release 13.0.0 |
| 191 |
- v1.13 (2020-04-15) |
| 192 |
- Add `emoji-data.txt` and `GraphemeBreakProperty.txt` |
| 193 |
- v1.14 (2020-10-24) |
| 194 |
- Add `WordBreakProperty.txt` |
| 195 |
- v1.15 (2021-09-15) |
| 196 |
- Update data files to Unicode release 14.0.0 |
| 197 |
- v1.16 (2022-09-17) |
| 198 |
- Update data files to Unicode release 15.0.0 |
| 199 |
- v1.17 (2023-09-18) |
| 200 |
- Update data files to Unicode release 15.1.0 |
| 201 |
- Adjust `load-unicode-xetex-classes.tex` for modified data file format |
| 202 |
- v1.18 (2025-01-07) |
| 203 |
- Update data files to Unicode release 16.0.0 |
| 204 |
- v1.19 (2025-09-26) |
| 205 |
- Update data files to Unicode release 17.0.0 |
| 206 |
|
| 207 |
License and permission |
| 208 |
====================== |
| 209 |
|
| 210 |
This bundle is copyright (C) 2015-2025 The LaTeX Project |
| 211 |
|
| 212 |
It may be distributed and/or modified under the conditions of |
| 213 |
the LaTeX Project Public License (LPPL), either version 1.3c of |
| 214 |
this license or (at your option) any later version. The latest |
| 215 |
version of this license is in the file |
| 216 |
http://www.latex-project.org/lppl.txt. |
| 217 |
|
| 218 |
The source data (`.txt` files) are supplied by the Unicode |
| 219 |
Consortium and the following notice applies. |
| 220 |
|
| 221 |
COPYRIGHT AND PERMISSION NOTICE |
| 222 |
|
| 223 |
|
| 224 |
Copyright © 1991-2025 Unicode, Inc. All rights reserved. |
| 225 |
Distributed under the Terms of Use in https://www.unicode.org/copyright.html. |
| 226 |
|
| 227 |
Permission is hereby granted, free of charge, to any person obtaining |
| 228 |
a copy of the Unicode data files and any associated documentation |
| 229 |
(the "Data Files") or Unicode software and any associated documentation |
| 230 |
(the "Software") to deal in the Data Files or Software |
| 231 |
without restriction, including without limitation the rights to use, |
| 232 |
copy, modify, merge, publish, distribute, and/or sell copies of |
| 233 |
the Data Files or Software, and to permit persons to whom the Data Files |
| 234 |
or Software are furnished to do so, provided that either |
| 235 |
(a) this copyright and permission notice appear with all copies |
| 236 |
of the Data Files or Software, or |
| 237 |
(b) this copyright and permission notice appear in associated |
| 238 |
Documentation. |
| 239 |
|
| 240 |
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF |
| 241 |
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE |
| 242 |
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND |
| 243 |
NONINFRINGEMENT OF THIRD PARTY RIGHTS. |
| 244 |
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS |
| 245 |
NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL |
| 246 |
DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, |
| 247 |
DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER |
| 248 |
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR |
| 249 |
PERFORMANCE OF THE DATA FILES OR SOFTWARE. |
| 250 |
|
| 251 |
Except as contained in this notice, the name of a copyright holder |
| 252 |
shall not be used in advertising or otherwise to promote the sale, |
| 253 |
use or other dealings in these Data Files or Software without prior |
| 254 |
written authorization of the copyright holder. |