Debian Codegraph @

texmf-dist/doc/generic/unicode-data/README.md

raw
1 The Unicode Consortium provide a range of data files detailing
2 the nature of code points in Unicode. These data files are
3 machine-readable but large. Here, a set of loaders are provided
4 to parse these files during a TeX run and set appropriate
5 parameters in an automated fashion.
6
7 File: `load-unicode-data.tex`
8 =============================
9
10 The loader parses a number of data files provided by the Unicode
11 Consortium and when used with used Unicode-capable engine sets
12 up a range of TeX-related parameters based on the extracted
13 information.
14
15 From the file `UnicodeData.txt` the following properties are set:
16 - `\catcode` 11 for all letters (Unicode class "L")
17 - `\catcode` 11 for all combining marks (Unicode class "M")
18 - `\sfcode` 999 for all code points of class "Lu" (upper case
19 letters)
20 - `\lccode` for all of class "Ll" (lower case letters) to the code
21 point itself, and `\uccode` to the upper case mapping (or if
22 not given to the code point itself)
23 - `\uccode` for all of class "Lu" (upper case letters) to the code
24 point itself, and `\lccode` to the lower case mapping (or if
25 not given to the code point itself)
26 - `\lccode` and `\uccode` for all of class "Lt" (title case
27 letters) to the lower and upper case mappings (or if not given
28 to the code point itself)
29 - `\lccode` and `\uccode` for all other letter code points are
30 set to the code point itself
31 - `\lccode` and/or `\uccode` for non-letter code points for
32 which an upper or lower case mapping is given
33 - `\sfcode` 0 (ignored) for code points of Unicode classes "Pe"
34 (closing punctuation marks) and "Pf" (final quotation marks)
35 - `\Umathcode` for all letters (Unicode class "L") and combining
36 marks (Unicode class "M") as TeX class 7 (var)
37
38 In the above, 'letters' refers to code points with Unicode class
39 "L", irrespective of their TeX category code.
40
41 File: `load-unicode-math-classes.tex`
42 ======================================
43
44 From the file `MathClass-15.txt` the following mapping between
45 Unicode math classes and TeX math types are set up
46 - "L" (large) `\mathop`
47 - "B" (binary) `\mathbin`
48 - "V" (vary) `\mathbin`
49 - "R" (relation) `\mathrel`
50 - "O" (opening) `\mathopen`
51 - "C" (closing) `\mathclose`
52 - "P" (punctuation) `\mathpunct`
53 - "A" (alphabetic) `\mathalpha`
54
55 For each code point processed, the result is of the form
56
57 \Umathcode <codepoint> = <type> 1 <codepoint>
58
59 File: `load-unicode-xetex-classes.tex`
60 ======================================
61
62 From the files `EastAsianWidth.txt` and `LineBreak.txt` inter
63 character classes for XeTeX (`\XeTeXcharclass`) are set for the
64 following classes
65 - "ID" (ideographic)
66 - "CJ" (conditional Japanese starter)
67 - "OP" (opener)
68 - "CL" (closer)
69 - "NS" (non-starter)
70 - "EX" (exclamation)
71 - "IS" (infix separator)
72 - "CM" (combining marks)
73
74 All code points of classes "ID" and "CJ" are assigned to a
75 `\XeTeXcharclass`, but for other classes this only occurs when
76 they fall into east Asian width type "F", "H" or "W" (full-,
77 half- and wide-width).
78
79 The following mappings between Unicode and XeTeX classes occur
80 - "ID" and "CJ" are class 1
81 - "OP" is class 2
82 - "CL", "NS", "EX", "IS" are class 3
83 - "CM" is class 256 (ignored)
84
85 as standard: these may be over-ridden by defining
86 `\XeTeXcharclass<class>` as required. (If classes "ID" or "CL"
87 are explicitly set, the other members of the same groups above
88 will inherit these values.)
89
90 This file does _not_ activate XeTeX's inter-character token
91 mechanism (`\XeTeXinterchartokenstate` is not set) nor does it
92 install any material in the inter-character token registers.
93
94 Note that this file is separate from the main loader as the data
95 structure here may need more refinement at the macro level.
96
97 Data files
98 ==========
99
100 The following data files are included here with permission of the
101 Unicode Consortium (see below):
102 - `UnicodeData.txt`
103 - `BidiBrackets.txt`
104 - `BidiMirroring.txt`
105 - `BidiMirroring-5-1-0.txt`
106 - `CaseFolding.txt`
107 - `EastAsianWidth.txt`
108 - `emoji-data.txt`
109 - `GraphemeBreakProperty.txt`
110 - `LineBreak.txt`
111 - `MathClass.txt`
112 - `PropList.txt`
113 - `SpecialCasing.txt`
114 - `Scripts.txt`
115 - `ScriptExtension.txt`
116 - `WordBreakProperty.txt`
117
118 With the exception of `MathClass.txt`, which is currently not
119 formally part of the Unicode Character Database, and
120 `BidiMirroring-5-1-0.txt`, which is a data file for Unicode 5.1.0,
121 all of the data files here are for Unicode 15.1.0.
122
123 `BidiMirroring-5-1-0.txt` will never be updated and is included for
124 OpenType font support: While `BidiMirroring.txt` lists all mirror
125 pairs in the current Unicode version, only the pairs from Unicode 5.1.0
126 are handled directly by the OpenType rendering system. All newer pairs
127 have to be handled through font features.
128
129 Requests to include additional Unicode data files in this bundle are
130 welcome. Ideally this bundle will act as a single location for all
131 Unicode data for use by TeX systems.
132
133 Issues and improvements
134 =======================
135
136 The home page for this bundle is
137 https://github.com/latex3/unicode-data, and issues may be
138 reported there.
139
140 Changes
141 =======
142
143 - v1.0 (2015-12-01)
144 - First public release
145 - v1.1 (2016-01-04)
146 - Allow setting of XeTeX character class slot numbers
147 - v1.1a (2016-01-06)
148 - Correctly assign XeTeX character class for code points in
149 ranges
150 - Fix issue which meant `EastAsianWidth.txt` was not read
151 - v1.1b (2016-01-08)
152 - Avoid save stack issue when loading `EastAsianWidth.txt`
153 - Fix application of settings to ranges of Unicode characters
154 - Correct setting of space factor for ending punctuation
155 - v1.2 (2016-02-02)
156 - Drop use of Unicode math data at present: undesired effect on
157 output if Unicode fonts are not loaded (classical OT1
158 situation)
159 - v1.2a (2016-02-05)
160 - Correct split between base and supplementary plane when
161 setting math codes
162 - v1.3 (2016-02-06)
163 - New `load-unicode-math-classes.tex` loader
164 - Set all letters as math type 7 (var) in
165 `load-unicode-data.tex`
166 - v1.3a (2016-02-06)
167 - Add definition of `\loop` to `load-unicode-math-classes.tex`
168 - v1.4 (2016-02-16)
169 - Treat "CJ" chars like "ID" chars for setting XeTeX character
170 class
171 - v1.4a (2016-02-21)
172 - Inherit XeTeX classes for "CJ" from "ID" and for "NS", "EX"
173 and "IS" from "CL" if appropriate
174 - v1.5 (2016-06-22)
175 - Update data files to Unicode release 9.0.0
176 - v1.6 (2017-06-27)
177 - Update data files to Unicode release 10.0.0
178 - v1.7 (2018-08-09)
179 - Update data files to Unicode release 11.0.0
180 - v1.8 (2019-03-09)
181 - Update data files to Unicode release 12.0.0
182 - v1.9 (2019-08-13)
183 - Update data files to Unicode release 12.1.0
184 - Add `BidiBrackets.txt`, `Scripts.txt` and `ScriptExtension.txt`
185 - v1.10 (2019-08-21)
186 - Add `PropList.txt`
187 - v1.11 (2019-11-28)
188 - Add `BidiMirroring.txt` and `BidiMirroring-5-1-0.txt`
189 - v1.12 (2020-03-11)
190 - Update data files to Unicode release 13.0.0
191 - v1.13 (2020-04-15)
192 - Add `emoji-data.txt` and `GraphemeBreakProperty.txt`
193 - v1.14 (2020-10-24)
194 - Add `WordBreakProperty.txt`
195 - v1.15 (2021-09-15)
196 - Update data files to Unicode release 14.0.0
197 - v1.16 (2022-09-17)
198 - Update data files to Unicode release 15.0.0
199 - v1.17 (2023-09-18)
200 - Update data files to Unicode release 15.1.0
201 - Adjust `load-unicode-xetex-classes.tex` for modified data file format
202 - v1.18 (2025-01-07)
203 - Update data files to Unicode release 16.0.0
204 - v1.19 (2025-09-26)
205 - Update data files to Unicode release 17.0.0
206
207 License and permission
208 ======================
209
210 This bundle is copyright (C) 2015-2025 The LaTeX Project
211
212 It may be distributed and/or modified under the conditions of
213 the LaTeX Project Public License (LPPL), either version 1.3c of
214 this license or (at your option) any later version. The latest
215 version of this license is in the file
216 http://www.latex-project.org/lppl.txt.
217
218 The source data (`.txt` files) are supplied by the Unicode
219 Consortium and the following notice applies.
220
221 COPYRIGHT AND PERMISSION NOTICE
222
223
224 Copyright © 1991-2025 Unicode, Inc. All rights reserved.
225 Distributed under the Terms of Use in https://www.unicode.org/copyright.html.
226
227 Permission is hereby granted, free of charge, to any person obtaining
228 a copy of the Unicode data files and any associated documentation
229 (the "Data Files") or Unicode software and any associated documentation
230 (the "Software") to deal in the Data Files or Software
231 without restriction, including without limitation the rights to use,
232 copy, modify, merge, publish, distribute, and/or sell copies of
233 the Data Files or Software, and to permit persons to whom the Data Files
234 or Software are furnished to do so, provided that either
235 (a) this copyright and permission notice appear with all copies
236 of the Data Files or Software, or
237 (b) this copyright and permission notice appear in associated
238 Documentation.
239
240 THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF
241 ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
242 WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
243 NONINFRINGEMENT OF THIRD PARTY RIGHTS.
244 IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS
245 NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL
246 DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
247 DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
248 TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
249 PERFORMANCE OF THE DATA FILES OR SOFTWARE.
250
251 Except as contained in this notice, the name of a copyright holder
252 shall not be used in advertising or otherwise to promote the sale,
253 use or other dealings in these Data Files or Software without prior
254 written authorization of the copyright holder.