| 1 |
<?xml version="1.0" encoding="utf-8"?> |
| 2 |
<!DOCTYPE chapter SYSTEM "docbook-dtd-45/docbookx.dtd"> |
| 3 |
|
| 4 |
<chapter id="tags"> |
| 5 |
<title>Tags</title> |
| 6 |
|
| 7 |
<para> |
| 8 |
First some example tags as we know them from CG-2 and VISLCG: |
| 9 |
<screen> |
| 10 |
"<wordform>" |
| 11 |
"baseform" |
| 12 |
<W-max> |
| 13 |
ADV |
| 14 |
@error |
| 15 |
(<civ> N) |
| 16 |
</screen> |
| 17 |
</para> |
| 18 |
<para> |
| 19 |
Now some example tags as they may look in VISL CG-3: |
| 20 |
<screen> |
| 21 |
"<Wordform>"i |
| 22 |
"^[Bb]ase.*"r |
| 23 |
/^@<?ADV>?$/r |
| 24 |
<W>65> |
| 25 |
(<F>=15> <F<=30>) |
| 26 |
!ADV |
| 27 |
^<dem> |
| 28 |
(N <civ>) |
| 29 |
</screen> |
| 30 |
</para> |
| 31 |
|
| 32 |
<para> |
| 33 |
The tag '>>>' is added to the 0th (invisible) cohort in the window, and the tag '<<<' is added to the last cohort in the window. They can be used as markers to see if scans have reached those positions. |
| 34 |
</para> |
| 35 |
|
| 36 |
<section id="tag-order"> |
| 37 |
<title>Tag Order</title> |
| 38 |
<para> |
| 39 |
Starting with the latter, (N <civ>), as this merely signifies that tags |
| 40 |
with multiple parts do not have to match in-order; (N <civ>) is the same |
| 41 |
as (<civ> N). This is different from previous versions of CG, but I deemed |
| 42 |
it unncecessary to spend extra time checking the tag order when hash lookups can |
| 43 |
verify the existence so fast. |
| 44 |
</para> |
| 45 |
</section> |
| 46 |
|
| 47 |
<section id="literal-string-modifiers"> |
| 48 |
<title>Literal String Modifiers</title> |
| 49 |
<para> |
| 50 |
The first two additions to the feature sheet all display what I refer to as |
| 51 |
literal string modifiers, and there are two of such: 'i' for case-insensitive, |
| 52 |
and 'r' for a regular expression match. |
| 53 |
Using these modifiers will significantly slow down the matching as a hash lookup will no |
| 54 |
longer be enough. You can combine 'ir' for case-insensitive regular expressions. |
| 55 |
Regular expressions are evaluated via ICU, so |
| 56 |
<ulink url="http://icu-project.org/userguide/regexp.html">their documentation</ulink> is a good source. |
| 57 |
Regular expressions may also contain groupings that can later be used in variable string tags (see below). |
| 58 |
</para> |
| 59 |
<para> |
| 60 |
Due to tags themselves needing the occasional escaping, regular expressions need double-escaping of |
| 61 |
symbols that have special meaning to CG-3. E.g. literal non-grouping () |
| 62 |
need to be written as "a\\(b\\)c"r. Metacharacters also need double-escaping, so \w needs to be written as \\w. |
| 63 |
</para> |
| 64 |
<para> |
| 65 |
This will not work for wordforms used as the first qualifier of a rule, e.g: |
| 66 |
<screen> |
| 67 |
"<wordform>"i SELECT (tag) ; |
| 68 |
</screen> |
| 69 |
but those can be rewritten in a form such as |
| 70 |
<screen> |
| 71 |
SELECT ("<wordform>"i) + (tag) ; |
| 72 |
</screen> |
| 73 |
which will work, but be slightly slower. |
| 74 |
</para> |
| 75 |
</section> |
| 76 |
|
| 77 |
<section id="regex-icase"> |
| 78 |
<title>Regular Expressions</title> |
| 79 |
<para> |
| 80 |
Tags in the form //r and //i and //ri are general purpose regular expression and case insensitive matches |
| 81 |
that may act on any tag type, and unlike Literal String Modifiers they can do partial matches. Thus a tag like |
| 82 |
/^@<?ADV>?$/r will match any of @<ADV, @<ADV>, @ADV>, and plain @ADV. |
| 83 |
A tag like /word/ri will match any tag containing a substring with any case-variation of the text 'word'. |
| 84 |
Asides from that, the rules and gotchas are the same as for Literal String Modifiers. |
| 85 |
</para> |
| 86 |
</section> |
| 87 |
|
| 88 |
<section id="line-match"> |
| 89 |
<title>Line Matching</title> |
| 90 |
<para> |
| 91 |
Tags in the form <code>//l</code> are regular expressions matched on the whole literal reading. Used to match exact tag sequences. Special helper <code>__</code> will expand to <code>(^|$| | .+? )</code> and can be used to ensure there are only whole tags before/after/between something. |
| 92 |
</para> |
| 93 |
</section> |
| 94 |
|
| 95 |
<section id="variable-strings"> |
| 96 |
<title>Variable Strings</title> |
| 97 |
<para> |
| 98 |
Variable string tags contain markers that are replaced with matches from the previously run grouping regular expression tag. |
| 99 |
Regular expression tags with no groupings will not have any effect on this behavior. |
| 100 |
Time also has no effect, so one could theoretically perform a group match in a previous rule and use the results later, |
| 101 |
though that would be highly unpredictable in practice. |
| 102 |
</para> |
| 103 |
<para> |
| 104 |
Variable string tags are in the form of "string"v, "<string>"v, and <string>v, where variables matching |
| 105 |
$1 through $9 will be replaced with the corresponding group from the regular expression match. |
| 106 |
Multiple occurances of a single variable is allowed, so e.g. "$1$2$1"v would contain group 1 twice. |
| 107 |
</para> |
| 108 |
<para> |
| 109 |
Alternative syntax is prefixing with <code>VSTR:</code>. This is used to build tags that are not textual, or tags |
| 110 |
that need secondary processing such as regex or case-insensitive matching. E.g., <code>VSTR:@m$1</code> would |
| 111 |
create a mapping tag or <code>VSTR:"$1.*"r</code> to create a regex tag. To include spaces in such tags, escape them |
| 112 |
with a backslash, e.g. <code>VSTR:"$1\ $2"r</code> (otherwise it is treated as two tags). |
| 113 |
</para> |
| 114 |
<para> |
| 115 |
One can also manipulate the case of the resulting tag via %U, %u, %L, and %l. |
| 116 |
%U upper-cases the entire following string. |
| 117 |
%u upper-cases the following single letter. |
| 118 |
%L lower-cases the entire following string. |
| 119 |
%l lower-cases the following single letter. |
| 120 |
The case folding is performed right-to-left one-by-one. |
| 121 |
</para> |
| 122 |
<para> |
| 123 |
It is also possible to include references to unified $$sets or &&sets in {} where they will be replaced |
| 124 |
with the tags that the unification resulted in. If there are multiple tags, they will be delimited by an underscore _. |
| 125 |
</para> |
| 126 |
<para> |
| 127 |
It should be noted that you can use varstring tags anywhere, not just when manipulating tags. |
| 128 |
When used in a contextual test they are fleshed out with the information available at the time and then attempted matched. |
| 129 |
</para> |
| 130 |
<para> |
| 131 |
<screen> |
| 132 |
# Adds a lower-case <wordform> to all readings. |
| 133 |
ADD (<%L$1>v) TARGET ("<(.*)>"r) ; |
| 134 |
|
| 135 |
# Adds a reading with a normalized baseform for all suspicious wordforms ending in 'ies' |
| 136 |
APPEND ("$1y"v N P NOM) TARGET N + ("<(.*)ies>"r) IF (1 VFIN) ; |
| 137 |
|
| 138 |
# Merge results from multiple unified $$sets into a single tag |
| 139 |
LIST ROLE = human anim inanim (bench table) ; |
| 140 |
LIST OTHER = crispy waffles butter ; |
| 141 |
MAP (<{$$ROLE}/{$$OTHER}>v) (target tags) (-1 $$OTHER) (-2C $$ROLE) ; |
| 142 |
</screen> |
| 143 |
</para> |
| 144 |
</section> |
| 145 |
|
| 146 |
<section id="numerical-matches"> |
| 147 |
<title>Numerical Matches</title> |
| 148 |
<para> |
| 149 |
Then there are the numerical matches, e.g. <W>65>. |
| 150 |
This will match tags such as <W:204> and <W=156> but not <W:32>. |
| 151 |
The second tag, (<F>15> <F<30>), matches values 15>F>30. |
| 152 |
These constructs are also slower than simple hash lookups. |
| 153 |
</para> |
| 154 |
<para> |
| 155 |
The two special values MIN and MAX (both case-sensitive) will scan the cohort for their respective minimum |
| 156 |
or maximum value, and use that for the comparison. |
| 157 |
Internally the value is stored in a double, and the range is capped between -281474976710656.0 to +281474976710655.0, |
| 158 |
and using values beyond that range will also act as those limits. |
| 159 |
<screen> |
| 160 |
# Select the maximum value of W. Readings with no W will also be removed. |
| 161 |
SELECT (<W=MAX>) ; |
| 162 |
|
| 163 |
# Remove the minimum F. Readings with no F will not be removed. |
| 164 |
REMOVE (<N=MIN>) ; |
| 165 |
</screen> |
| 166 |
</para> |
| 167 |
<para> |
| 168 |
<table id="numeric-operators"> |
| 169 |
<title>Valid Operators</title> |
| 170 |
<tgroup cols="2"> |
| 171 |
<thead> |
| 172 |
<row> |
| 173 |
<entry>Operator</entry> |
| 174 |
<entry>Meaning</entry> |
| 175 |
</row> |
| 176 |
</thead> |
| 177 |
<tbody> |
| 178 |
<row> |
| 179 |
<entry>=</entry> |
| 180 |
<entry>Equal to</entry> |
| 181 |
</row> |
| 182 |
<row> |
| 183 |
<entry>!=</entry> |
| 184 |
<entry>Not equal to</entry> |
| 185 |
</row> |
| 186 |
<row> |
| 187 |
<entry><</entry> |
| 188 |
<entry>Less than</entry> |
| 189 |
</row> |
| 190 |
<row> |
| 191 |
<entry>></entry> |
| 192 |
<entry>Greater than</entry> |
| 193 |
</row> |
| 194 |
<row> |
| 195 |
<entry><=</entry> |
| 196 |
<entry>Less than or equal to</entry> |
| 197 |
</row> |
| 198 |
<row> |
| 199 |
<entry>>=</entry> |
| 200 |
<entry>Greater than or equal to</entry> |
| 201 |
</row> |
| 202 |
<row> |
| 203 |
<entry><></entry> |
| 204 |
<entry>Not equal to</entry> |
| 205 |
</row> |
| 206 |
</tbody> |
| 207 |
</tgroup> |
| 208 |
</table> |
| 209 |
Anywhere that an = is valid you can also use : for backwards compatibility. |
| 210 |
</para> |
| 211 |
<para> |
| 212 |
<table id="numeric-truths"> |
| 213 |
<title>Comparison Truth Table</title> |
| 214 |
<tgroup cols="3"> |
| 215 |
<thead> |
| 216 |
<row> |
| 217 |
<entry>A</entry> |
| 218 |
<entry>B</entry> |
| 219 |
<entry>Result</entry> |
| 220 |
</row> |
| 221 |
</thead> |
| 222 |
<tbody> |
| 223 |
<row> |
| 224 |
<entry>= x</entry> |
| 225 |
<entry>= y</entry> |
| 226 |
<entry>True if x = y</entry> |
| 227 |
</row> |
| 228 |
<row> |
| 229 |
<entry>= x</entry> |
| 230 |
<entry>!= y</entry> |
| 231 |
<entry>True if x != y</entry> |
| 232 |
</row> |
| 233 |
<row> |
| 234 |
<entry>= x</entry> |
| 235 |
<entry>< y</entry> |
| 236 |
<entry>True if x < y</entry> |
| 237 |
</row> |
| 238 |
<row> |
| 239 |
<entry>= x</entry> |
| 240 |
<entry>> y</entry> |
| 241 |
<entry>True if x > y</entry> |
| 242 |
</row> |
| 243 |
<row> |
| 244 |
<entry>= x</entry> |
| 245 |
<entry><= y</entry> |
| 246 |
<entry>True if x <= y</entry> |
| 247 |
</row> |
| 248 |
<row> |
| 249 |
<entry>= x</entry> |
| 250 |
<entry>>= y</entry> |
| 251 |
<entry>True if x >= y</entry> |
| 252 |
</row> |
| 253 |
|
| 254 |
<row> |
| 255 |
<entry>< x</entry> |
| 256 |
<entry>!= y</entry> |
| 257 |
<entry>Always true</entry> |
| 258 |
</row> |
| 259 |
<row> |
| 260 |
<entry>< x</entry> |
| 261 |
<entry>< y</entry> |
| 262 |
<entry>Always true</entry> |
| 263 |
</row> |
| 264 |
<row> |
| 265 |
<entry>< x</entry> |
| 266 |
<entry>> y</entry> |
| 267 |
<entry>True if x > y</entry> |
| 268 |
</row> |
| 269 |
<row> |
| 270 |
<entry>< x</entry> |
| 271 |
<entry><= y</entry> |
| 272 |
<entry>Always true</entry> |
| 273 |
</row> |
| 274 |
<row> |
| 275 |
<entry>< x</entry> |
| 276 |
<entry>>= y</entry> |
| 277 |
<entry>True if x > y</entry> |
| 278 |
</row> |
| 279 |
|
| 280 |
<row> |
| 281 |
<entry>> x</entry> |
| 282 |
<entry>!= y</entry> |
| 283 |
<entry>Always true</entry> |
| 284 |
</row> |
| 285 |
<row> |
| 286 |
<entry>> x</entry> |
| 287 |
<entry>> y</entry> |
| 288 |
<entry>Always true</entry> |
| 289 |
</row> |
| 290 |
<row> |
| 291 |
<entry>> x</entry> |
| 292 |
<entry><= y</entry> |
| 293 |
<entry>True if x < y</entry> |
| 294 |
</row> |
| 295 |
<row> |
| 296 |
<entry>> x</entry> |
| 297 |
<entry>>= y</entry> |
| 298 |
<entry>Always true</entry> |
| 299 |
</row> |
| 300 |
|
| 301 |
<row> |
| 302 |
<entry><= x</entry> |
| 303 |
<entry>!= y</entry> |
| 304 |
<entry>Always true</entry> |
| 305 |
</row> |
| 306 |
<row> |
| 307 |
<entry><= x</entry> |
| 308 |
<entry><= y</entry> |
| 309 |
<entry>Always true</entry> |
| 310 |
</row> |
| 311 |
<row> |
| 312 |
<entry><= x</entry> |
| 313 |
<entry>>= y</entry> |
| 314 |
<entry>True if x >= y</entry> |
| 315 |
</row> |
| 316 |
|
| 317 |
<row> |
| 318 |
<entry>>= x</entry> |
| 319 |
<entry>!= y</entry> |
| 320 |
<entry>Always true</entry> |
| 321 |
</row> |
| 322 |
<row> |
| 323 |
<entry>>= x</entry> |
| 324 |
<entry>>= y</entry> |
| 325 |
<entry>Always true</entry> |
| 326 |
</row> |
| 327 |
|
| 328 |
<row> |
| 329 |
<entry>!= x</entry> |
| 330 |
<entry>!= y</entry> |
| 331 |
<entry>True if x = y</entry> |
| 332 |
</row> |
| 333 |
</tbody> |
| 334 |
</tgroup> |
| 335 |
</table> |
| 336 |
</para> |
| 337 |
</section> |
| 338 |
|
| 339 |
<section id="stream-metadata"> |
| 340 |
<title>Stream Metadata</title> |
| 341 |
<para> |
| 342 |
CG-3 will <link linkend="streamformats">store and forward</link> any data between cohorts, |
| 343 |
attached to the preceding cohort. |
| 344 |
<code>META:/.../r</code> and <code>META:/.../ri</code> lets you query and capture from this data with regular expressions. |
| 345 |
Data before the first cohort is not accessible. |
| 346 |
</para> |
| 347 |
<screen> |
| 348 |
ADD (@header) (*) IF (-1 (META:/<h\d+>$/ri)) ; |
| 349 |
</screen> |
| 350 |
<para> |
| 351 |
I recommend keeping META tags in the contextual tests, since they cannot currently be cached and will be checked |
| 352 |
every time. |
| 353 |
</para> |
| 354 |
</section> |
| 355 |
|
| 356 |
<section id="stream-static-tags"> |
| 357 |
<title>Stream Static Tags</title> |
| 358 |
<para> |
| 359 |
In the CG and Apertium stream formats, it is allowed to have tags after the word form / surface form. |
| 360 |
These tags behave as if they contextually exist in every reading of the cohort - they will not be seen by rule targets. |
| 361 |
</para> |
| 362 |
</section> |
| 363 |
|
| 364 |
<section id="global-variables"> |
| 365 |
<title>Global Variables</title> |
| 366 |
<para> |
| 367 |
Global variables are manipulated with rule types <link linkend="setvariable">SETVARIABLE</link> and |
| 368 |
<link linkend="remvariable">REMVARIABLE</link>, plus the stream commands <link linkend="cmd-setvar">SETVAR</link> |
| 369 |
and <link linkend="cmd-remvar">REMVAR</link>. |
| 370 |
Global variables persist until unset and are not bound to any window, cohort, or reading. |
| 371 |
</para> |
| 372 |
<para> |
| 373 |
You can query a global variable with the form <code>VAR:name</code> or query whether a variable has a |
| 374 |
specific value with <code>VAR:name=value</code>. Both the name and value test can be in the form of |
| 375 |
<link linkend="regex-icase">// regular expressions</link>, a'la <code>VAR:/ame/r=value</code> or <code>VAR:name=/val.*/r</code>, |
| 376 |
including capturing parts. |
| 377 |
</para> |
| 378 |
<para> |
| 379 |
The runtime value of mapping prefix is stored in the special variable named <code>_MPREFIX</code>. |
| 380 |
</para> |
| 381 |
<screen> |
| 382 |
REMOVE (@poetry) IF (0 (VAR:news)) ; |
| 383 |
SELECT (<historical>) IF (0 (VAR:year=1764)) ; |
| 384 |
</screen> |
| 385 |
<para> |
| 386 |
I recommend keeping VAR tags in the contextual tests, since they cannot currently be cached and will be checked |
| 387 |
every time. |
| 388 |
</para> |
| 389 |
</section> |
| 390 |
|
| 391 |
<section id="local-variables"> |
| 392 |
<title>Local Variables</title> |
| 393 |
<para> |
| 394 |
Almost identical to <link linkend="global-variables">global variables</link>, but uses <code>LVAR</code> instead of <code>VAR</code>, and variables are bound to windows. |
| 395 |
</para> |
| 396 |
<para> |
| 397 |
Global variables are remembered on a per-window basis. When the current window has no more possible rules, the current variable state is recorded in the window. Later windows looking back with <link linkend="test-spanning">W</link> can then query what a given variable's value was at that time. It is also possible to query future windows' variable values if the stream contains <link linkend="cmd-setvar">SETVAR</link> and the window is in the lookahead buffer. |
| 398 |
</para> |
| 399 |
<para> |
| 400 |
When LVAR queries the current window, it is the same as VAR. |
| 401 |
</para> |
| 402 |
</section> |
| 403 |
|
| 404 |
<section id="fail-fast-tag"> |
| 405 |
<title>Fail-Fast Tag</title> |
| 406 |
<para> |
| 407 |
A Fail-Fast tag is the ^ prefix, such as ^<dem>. |
| 408 |
This will be checked first of a set and if found will block the set from matching, |
| 409 |
regardless of whether later independent tags could match. |
| 410 |
It is mostly useful for sets such as LIST SetWithFail = (N <bib>) (V TR) ^<dem>. |
| 411 |
This set will never match a reading with a <dem> tag, even if the reading matches (V TR). |
| 412 |
</para> |
| 413 |
</section> |
| 414 |
|
| 415 |
<section id="strict-tags"> |
| 416 |
<title>STRICT-TAGS</title> |
| 417 |
<para> |
| 418 |
If you are worried about typos or need to otherwise enforce a strict tagset, <code>STRICT-TAGS</code> is your friend. |
| 419 |
You can add tags to the list of allowed tags with <code>STRICT-TAGS += ... ;</code> where <code>...</code> is a list |
| 420 |
of tags to allow. Any tag parsed while the STRICT-TAGS list is non-empty will be checked against the list, and an |
| 421 |
error will be thrown if the tag is not on the list. |
| 422 |
</para> |
| 423 |
<para> |
| 424 |
It is currently only possible to add to the list, hence <code>+=</code>. |
| 425 |
Removing and assigning can be added if anyone needs those. |
| 426 |
</para> |
| 427 |
<screen> |
| 428 |
STRICT-TAGS += N V ADJ etc ... ; |
| 429 |
</screen> |
| 430 |
<para> |
| 431 |
By default, STRICT-TAGS always allows wordforms, baseforms, regular expressions, case-insensitive, and VISL-style secondary tags |
| 432 |
(<code>"<…>"</code>, <code>"…"</code>, <code><…></code>), since those are too prolific to list |
| 433 |
individually. If you are extra paranoid, you can change that with <link linkend="grammar-options">OPTIONS</link>. |
| 434 |
</para> |
| 435 |
<para> |
| 436 |
To get a list of unique used tags, pass --show-tags to CG-3. To filter this list to the default set of interesting tags, |
| 437 |
<link linkend="cg-strictify">cg-strictify</link> can be used: |
| 438 |
<screen> |
| 439 |
cg-strictify grammar-goes-here |
| 440 |
</screen> |
| 441 |
<emphasis>For comparison, this yields 285 tags for VISL's 10000-rule Danish grammar.</emphasis> |
| 442 |
Edit the resulting list to remove any tags you can see are typos or should otherwise not be allowed, |
| 443 |
stuff it at the top of the grammar, and recompile the grammar. |
| 444 |
Any errors you get will be lines where forbidden tags are used, which can be whole sets if those sets aren't used in any rules. |
| 445 |
</para> |
| 446 |
<para> |
| 447 |
Once you have a suitable STRICT-TAGS list, you can further trim the grammar by taking advantage the fact that any |
| 448 |
tag listed in STRICT-TAGS may be used as an implicit set that contains only the tag itself. |
| 449 |
No more need for <code>LIST N = N ;</code> constructs. |
| 450 |
</para> |
| 451 |
</section> |
| 452 |
|
| 453 |
<section id="list-tags"> |
| 454 |
<title>LIST-TAGS</title> |
| 455 |
<para> |
| 456 |
Very similar to <link linkend="strict-tags">STRICT-TAGS</link>, but only performs the final part of making |
| 457 |
<code>LIST N = N ;</code> superfluous. Any tag listed in <code>LIST-TAGS</code> has an implicit set created for it. |
| 458 |
</para> |
| 459 |
</section> |
| 460 |
</chapter> |