Debian Codegraph @

manual/tags.xml

raw
1 <?xml version="1.0" encoding="utf-8"?>
2 <!DOCTYPE chapter SYSTEM "docbook-dtd-45/docbookx.dtd">
3
4 <chapter id="tags">
5 <title>Tags</title>
6
7 <para>
8 First some example tags as we know them from CG-2 and VISLCG:
9 <screen>
10 "&lt;wordform&gt;"
11 "baseform"
12 &lt;W-max&gt;
13 ADV
14 @error
15 (&lt;civ&gt; N)
16 </screen>
17 </para>
18 <para>
19 Now some example tags as they may look in VISL CG-3:
20 <screen>
21 "&lt;Wordform&gt;"i
22 "^[Bb]ase.*"r
23 /^@&lt;?ADV&gt;?$/r
24 &lt;W&gt;65&gt;
25 (&lt;F&gt;=15&gt; &lt;F&lt;=30&gt;)
26 !ADV
27 ^&lt;dem&gt;
28 (N &lt;civ&gt;)
29 </screen>
30 </para>
31
32 <para>
33 The tag '&gt;&gt;&gt;' is added to the 0th (invisible) cohort in the window, and the tag '&lt;&lt;&lt;' is added to the last cohort in the window. They can be used as markers to see if scans have reached those positions.
34 </para>
35
36 <section id="tag-order">
37 <title>Tag Order</title>
38 <para>
39 Starting with the latter, (N &lt;civ&gt;), as this merely signifies that tags
40 with multiple parts do not have to match in-order; (N &lt;civ&gt;) is the same
41 as (&lt;civ&gt; N). This is different from previous versions of CG, but I deemed
42 it unncecessary to spend extra time checking the tag order when hash lookups can
43 verify the existence so fast.
44 </para>
45 </section>
46
47 <section id="literal-string-modifiers">
48 <title>Literal String Modifiers</title>
49 <para>
50 The first two additions to the feature sheet all display what I refer to as
51 literal string modifiers, and there are two of such: 'i' for case-insensitive,
52 and 'r' for a regular expression match.
53 Using these modifiers will significantly slow down the matching as a hash lookup will no
54 longer be enough. You can combine 'ir' for case-insensitive regular expressions.
55 Regular expressions are evaluated via ICU, so
56 <ulink url="http://icu-project.org/userguide/regexp.html">their documentation</ulink> is a good source.
57 Regular expressions may also contain groupings that can later be used in variable string tags (see below).
58 </para>
59 <para>
60 Due to tags themselves needing the occasional escaping, regular expressions need double-escaping of
61 symbols that have special meaning to CG-3. E.g. literal non-grouping ()
62 need to be written as "a\\(b\\)c"r. Metacharacters also need double-escaping, so \w needs to be written as \\w.
63 </para>
64 <para>
65 This will not work for wordforms used as the first qualifier of a rule, e.g:
66 <screen>
67 "&lt;wordform&gt;"i SELECT (tag) ;
68 </screen>
69 but those can be rewritten in a form such as
70 <screen>
71 SELECT ("&lt;wordform&gt;"i) + (tag) ;
72 </screen>
73 which will work, but be slightly slower.
74 </para>
75 </section>
76
77 <section id="regex-icase">
78 <title>Regular Expressions</title>
79 <para>
80 Tags in the form //r and //i and //ri are general purpose regular expression and case insensitive matches
81 that may act on any tag type, and unlike Literal String Modifiers they can do partial matches. Thus a tag like
82 /^@&lt;?ADV&gt;?$/r will match any of @&lt;ADV, @&lt;ADV&gt;, @ADV&gt;, and plain @ADV.
83 A tag like /word/ri will match any tag containing a substring with any case-variation of the text 'word'.
84 Asides from that, the rules and gotchas are the same as for Literal String Modifiers.
85 </para>
86 </section>
87
88 <section id="line-match">
89 <title>Line Matching</title>
90 <para>
91 Tags in the form <code>//l</code> are regular expressions matched on the whole literal reading. Used to match exact tag sequences. Special helper <code>__</code> will expand to <code>(^|$| | .+? )</code> and can be used to ensure there are only whole tags before/after/between something.
92 </para>
93 </section>
94
95 <section id="variable-strings">
96 <title>Variable Strings</title>
97 <para>
98 Variable string tags contain markers that are replaced with matches from the previously run grouping regular expression tag.
99 Regular expression tags with no groupings will not have any effect on this behavior.
100 Time also has no effect, so one could theoretically perform a group match in a previous rule and use the results later,
101 though that would be highly unpredictable in practice.
102 </para>
103 <para>
104 Variable string tags are in the form of "string"v, "&lt;string&gt;"v, and &lt;string&gt;v, where variables matching
105 $1 through $9 will be replaced with the corresponding group from the regular expression match.
106 Multiple occurances of a single variable is allowed, so e.g. "$1$2$1"v would contain group 1 twice.
107 </para>
108 <para>
109 Alternative syntax is prefixing with <code>VSTR:</code>. This is used to build tags that are not textual, or tags
110 that need secondary processing such as regex or case-insensitive matching. E.g., <code>VSTR:@m$1</code> would
111 create a mapping tag or <code>VSTR:"$1.*"r</code> to create a regex tag. To include spaces in such tags, escape them
112 with a backslash, e.g. <code>VSTR:"$1\ $2"r</code> (otherwise it is treated as two tags).
113 </para>
114 <para>
115 One can also manipulate the case of the resulting tag via %U, %u, %L, and %l.
116 %U upper-cases the entire following string.
117 %u upper-cases the following single letter.
118 %L lower-cases the entire following string.
119 %l lower-cases the following single letter.
120 The case folding is performed right-to-left one-by-one.
121 </para>
122 <para>
123 It is also possible to include references to unified $$sets or &amp;&amp;sets in {} where they will be replaced
124 with the tags that the unification resulted in. If there are multiple tags, they will be delimited by an underscore _.
125 </para>
126 <para>
127 It should be noted that you can use varstring tags anywhere, not just when manipulating tags.
128 When used in a contextual test they are fleshed out with the information available at the time and then attempted matched.
129 </para>
130 <para>
131 <screen>
132 # Adds a lower-case &lt;wordform&gt; to all readings.
133 ADD (&lt;%L$1&gt;v) TARGET ("&lt;(.*)&gt;"r) ;
134
135 # Adds a reading with a normalized baseform for all suspicious wordforms ending in 'ies'
136 APPEND ("$1y"v N P NOM) TARGET N + ("&lt;(.*)ies&gt;"r) IF (1 VFIN) ;
137
138 # Merge results from multiple unified $$sets into a single tag
139 LIST ROLE = human anim inanim (bench table) ;
140 LIST OTHER = crispy waffles butter ;
141 MAP (&lt;{$$ROLE}/{$$OTHER}&gt;v) (target tags) (-1 $$OTHER) (-2C $$ROLE) ;
142 </screen>
143 </para>
144 </section>
145
146 <section id="numerical-matches">
147 <title>Numerical Matches</title>
148 <para>
149 Then there are the numerical matches, e.g. &lt;W&gt;65&gt;.
150 This will match tags such as &lt;W:204&gt; and &lt;W=156&gt; but not &lt;W:32&gt;.
151 The second tag, (&lt;F&gt;15&gt; &lt;F&lt;30&gt;), matches values 15&gt;F&gt;30.
152 These constructs are also slower than simple hash lookups.
153 </para>
154 <para>
155 The two special values MIN and MAX (both case-sensitive) will scan the cohort for their respective minimum
156 or maximum value, and use that for the comparison.
157 Internally the value is stored in a double, and the range is capped between -281474976710656.0 to +281474976710655.0,
158 and using values beyond that range will also act as those limits.
159 <screen>
160 # Select the maximum value of W. Readings with no W will also be removed.
161 SELECT (&lt;W=MAX&gt;) ;
162
163 # Remove the minimum F. Readings with no F will not be removed.
164 REMOVE (&lt;N=MIN&gt;) ;
165 </screen>
166 </para>
167 <para>
168 <table id="numeric-operators">
169 <title>Valid Operators</title>
170 <tgroup cols="2">
171 <thead>
172 <row>
173 <entry>Operator</entry>
174 <entry>Meaning</entry>
175 </row>
176 </thead>
177 <tbody>
178 <row>
179 <entry>=</entry>
180 <entry>Equal to</entry>
181 </row>
182 <row>
183 <entry>!=</entry>
184 <entry>Not equal to</entry>
185 </row>
186 <row>
187 <entry>&lt;</entry>
188 <entry>Less than</entry>
189 </row>
190 <row>
191 <entry>&gt;</entry>
192 <entry>Greater than</entry>
193 </row>
194 <row>
195 <entry>&lt;=</entry>
196 <entry>Less than or equal to</entry>
197 </row>
198 <row>
199 <entry>&gt;=</entry>
200 <entry>Greater than or equal to</entry>
201 </row>
202 <row>
203 <entry>&lt;&gt;</entry>
204 <entry>Not equal to</entry>
205 </row>
206 </tbody>
207 </tgroup>
208 </table>
209 Anywhere that an = is valid you can also use : for backwards compatibility.
210 </para>
211 <para>
212 <table id="numeric-truths">
213 <title>Comparison Truth Table</title>
214 <tgroup cols="3">
215 <thead>
216 <row>
217 <entry>A</entry>
218 <entry>B</entry>
219 <entry>Result</entry>
220 </row>
221 </thead>
222 <tbody>
223 <row>
224 <entry>= x</entry>
225 <entry>= y</entry>
226 <entry>True if x = y</entry>
227 </row>
228 <row>
229 <entry>= x</entry>
230 <entry>!= y</entry>
231 <entry>True if x != y</entry>
232 </row>
233 <row>
234 <entry>= x</entry>
235 <entry>&lt; y</entry>
236 <entry>True if x &lt; y</entry>
237 </row>
238 <row>
239 <entry>= x</entry>
240 <entry>&gt; y</entry>
241 <entry>True if x &gt; y</entry>
242 </row>
243 <row>
244 <entry>= x</entry>
245 <entry>&lt;= y</entry>
246 <entry>True if x &lt;= y</entry>
247 </row>
248 <row>
249 <entry>= x</entry>
250 <entry>&gt;= y</entry>
251 <entry>True if x &gt;= y</entry>
252 </row>
253
254 <row>
255 <entry>&lt; x</entry>
256 <entry>!= y</entry>
257 <entry>Always true</entry>
258 </row>
259 <row>
260 <entry>&lt; x</entry>
261 <entry>&lt; y</entry>
262 <entry>Always true</entry>
263 </row>
264 <row>
265 <entry>&lt; x</entry>
266 <entry>&gt; y</entry>
267 <entry>True if x &gt; y</entry>
268 </row>
269 <row>
270 <entry>&lt; x</entry>
271 <entry>&lt;= y</entry>
272 <entry>Always true</entry>
273 </row>
274 <row>
275 <entry>&lt; x</entry>
276 <entry>&gt;= y</entry>
277 <entry>True if x &gt; y</entry>
278 </row>
279
280 <row>
281 <entry>&gt; x</entry>
282 <entry>!= y</entry>
283 <entry>Always true</entry>
284 </row>
285 <row>
286 <entry>&gt; x</entry>
287 <entry>&gt; y</entry>
288 <entry>Always true</entry>
289 </row>
290 <row>
291 <entry>&gt; x</entry>
292 <entry>&lt;= y</entry>
293 <entry>True if x &lt; y</entry>
294 </row>
295 <row>
296 <entry>&gt; x</entry>
297 <entry>&gt;= y</entry>
298 <entry>Always true</entry>
299 </row>
300
301 <row>
302 <entry>&lt;= x</entry>
303 <entry>!= y</entry>
304 <entry>Always true</entry>
305 </row>
306 <row>
307 <entry>&lt;= x</entry>
308 <entry>&lt;= y</entry>
309 <entry>Always true</entry>
310 </row>
311 <row>
312 <entry>&lt;= x</entry>
313 <entry>&gt;= y</entry>
314 <entry>True if x &gt;= y</entry>
315 </row>
316
317 <row>
318 <entry>&gt;= x</entry>
319 <entry>!= y</entry>
320 <entry>Always true</entry>
321 </row>
322 <row>
323 <entry>&gt;= x</entry>
324 <entry>&gt;= y</entry>
325 <entry>Always true</entry>
326 </row>
327
328 <row>
329 <entry>!= x</entry>
330 <entry>!= y</entry>
331 <entry>True if x = y</entry>
332 </row>
333 </tbody>
334 </tgroup>
335 </table>
336 </para>
337 </section>
338
339 <section id="stream-metadata">
340 <title>Stream Metadata</title>
341 <para>
342 CG-3 will <link linkend="streamformats">store and forward</link> any data between cohorts,
343 attached to the preceding cohort.
344 <code>META:/.../r</code> and <code>META:/.../ri</code> lets you query and capture from this data with regular expressions.
345 Data before the first cohort is not accessible.
346 </para>
347 <screen>
348 ADD (@header) (*) IF (-1 (META:/&lt;h\d+&gt;$/ri)) ;
349 </screen>
350 <para>
351 I recommend keeping META tags in the contextual tests, since they cannot currently be cached and will be checked
352 every time.
353 </para>
354 </section>
355
356 <section id="stream-static-tags">
357 <title>Stream Static Tags</title>
358 <para>
359 In the CG and Apertium stream formats, it is allowed to have tags after the word form / surface form.
360 These tags behave as if they contextually exist in every reading of the cohort - they will not be seen by rule targets.
361 </para>
362 </section>
363
364 <section id="global-variables">
365 <title>Global Variables</title>
366 <para>
367 Global variables are manipulated with rule types <link linkend="setvariable">SETVARIABLE</link> and
368 <link linkend="remvariable">REMVARIABLE</link>, plus the stream commands <link linkend="cmd-setvar">SETVAR</link>
369 and <link linkend="cmd-remvar">REMVAR</link>.
370 Global variables persist until unset and are not bound to any window, cohort, or reading.
371 </para>
372 <para>
373 You can query a global variable with the form <code>VAR:name</code> or query whether a variable has a
374 specific value with <code>VAR:name=value</code>. Both the name and value test can be in the form of
375 <link linkend="regex-icase">// regular expressions</link>, a'la <code>VAR:/ame/r=value</code> or <code>VAR:name=/val.*/r</code>,
376 including capturing parts.
377 </para>
378 <para>
379 The runtime value of mapping prefix is stored in the special variable named <code>_MPREFIX</code>.
380 </para>
381 <screen>
382 REMOVE (@poetry) IF (0 (VAR:news)) ;
383 SELECT (&lt;historical&gt;) IF (0 (VAR:year=1764)) ;
384 </screen>
385 <para>
386 I recommend keeping VAR tags in the contextual tests, since they cannot currently be cached and will be checked
387 every time.
388 </para>
389 </section>
390
391 <section id="local-variables">
392 <title>Local Variables</title>
393 <para>
394 Almost identical to <link linkend="global-variables">global variables</link>, but uses <code>LVAR</code> instead of <code>VAR</code>, and variables are bound to windows.
395 </para>
396 <para>
397 Global variables are remembered on a per-window basis. When the current window has no more possible rules, the current variable state is recorded in the window. Later windows looking back with <link linkend="test-spanning">W</link> can then query what a given variable's value was at that time. It is also possible to query future windows' variable values if the stream contains <link linkend="cmd-setvar">SETVAR</link> and the window is in the lookahead buffer.
398 </para>
399 <para>
400 When LVAR queries the current window, it is the same as VAR.
401 </para>
402 </section>
403
404 <section id="fail-fast-tag">
405 <title>Fail-Fast Tag</title>
406 <para>
407 A Fail-Fast tag is the ^ prefix, such as ^&lt;dem&gt;.
408 This will be checked first of a set and if found will block the set from matching,
409 regardless of whether later independent tags could match.
410 It is mostly useful for sets such as LIST SetWithFail = (N &lt;bib&gt;) (V TR) ^&lt;dem&gt;.
411 This set will never match a reading with a &lt;dem&gt; tag, even if the reading matches (V TR).
412 </para>
413 </section>
414
415 <section id="strict-tags">
416 <title>STRICT-TAGS</title>
417 <para>
418 If you are worried about typos or need to otherwise enforce a strict tagset, <code>STRICT-TAGS</code> is your friend.
419 You can add tags to the list of allowed tags with <code>STRICT-TAGS += ... ;</code> where <code>...</code> is a list
420 of tags to allow. Any tag parsed while the STRICT-TAGS list is non-empty will be checked against the list, and an
421 error will be thrown if the tag is not on the list.
422 </para>
423 <para>
424 It is currently only possible to add to the list, hence <code>+=</code>.
425 Removing and assigning can be added if anyone needs those.
426 </para>
427 <screen>
428 STRICT-TAGS += N V ADJ etc ... ;
429 </screen>
430 <para>
431 By default, STRICT-TAGS always allows wordforms, baseforms, regular expressions, case-insensitive, and VISL-style secondary tags
432 (<code>"&lt;…&gt;"</code>, <code>"…"</code>, <code>&lt;…&gt;</code>), since those are too prolific to list
433 individually. If you are extra paranoid, you can change that with <link linkend="grammar-options">OPTIONS</link>.
434 </para>
435 <para>
436 To get a list of unique used tags, pass --show-tags to CG-3. To filter this list to the default set of interesting tags,
437 <link linkend="cg-strictify">cg-strictify</link> can be used:
438 <screen>
439 cg-strictify grammar-goes-here
440 </screen>
441 <emphasis>For comparison, this yields 285 tags for VISL's 10000-rule Danish grammar.</emphasis>
442 Edit the resulting list to remove any tags you can see are typos or should otherwise not be allowed,
443 stuff it at the top of the grammar, and recompile the grammar.
444 Any errors you get will be lines where forbidden tags are used, which can be whole sets if those sets aren't used in any rules.
445 </para>
446 <para>
447 Once you have a suitable STRICT-TAGS list, you can further trim the grammar by taking advantage the fact that any
448 tag listed in STRICT-TAGS may be used as an implicit set that contains only the tag itself.
449 No more need for <code>LIST N = N ;</code> constructs.
450 </para>
451 </section>
452
453 <section id="list-tags">
454 <title>LIST-TAGS</title>
455 <para>
456 Very similar to <link linkend="strict-tags">STRICT-TAGS</link>, but only performs the final part of making
457 <code>LIST N = N ;</code> superfluous. Any tag listed in <code>LIST-TAGS</code> has an implicit set created for it.
458 </para>
459 </section>
460 </chapter>