Debian Codegraph @

manual/streamformats.xml

raw
1 <?xml version="1.0" encoding="utf-8"?>
2 <!DOCTYPE chapter SYSTEM "docbook-dtd-45/docbookx.dtd">
3
4 <chapter id="streamformats">
5 <title>Input/Output Stream Format</title>
6
7 <para>
8 The <link linkend="cg-conv">cg-conv</link> tool converts between various stream formats.
9 </para>
10
11 <section id="stream-apertium">
12 <title>Apertium Format</title>
13 <indexterm>
14 <primary>Apertium Format</primary>
15 </indexterm>
16 <para>
17 The <link linkend="cg-proc">cg-proc</link> front-end processes the
18 <ulink url="http://wiki.apertium.org/wiki/Apertium_stream_format">Apertium stream format</ulink>,
19 or can convert for use via <link linkend="cg-conv">cg-conv</link>.
20 </para>
21 </section>
22
23 <section id="stream-fst">
24 <title>HFST/XFST Format</title>
25 <indexterm>
26 <primary>HFST/XFST Format</primary>
27 </indexterm>
28 <para>
29 HFST/XFST input can be converted for use via <link linkend="cg-conv">cg-conv</link>.
30 </para>
31 </section>
32
33 <section id="stream-vislcg">
34 <title>VISL CG Format</title>
35 <indexterm>
36 <primary>VISL CG Format</primary>
37 </indexterm>
38 <para>
39 The VISL CG stream format is a verticalized list of word forms with readings and optional plain text in between.
40 For example, the sentence "<emphasis>They went to the zoo to look at the bear.</emphasis>" would in VISL format look akin to:
41 <screen>
42 "&lt;They&gt;"
43 "they" &lt;*&gt; PRON PERS NOM PL3 SUBJ
44 "&lt;went&gt;"
45 "go" V PAST VFIN
46 "&lt;to&gt;"
47 "to" PREP
48 "&lt;the&gt;"
49 "the" DET CENTRAL ART SG/PL
50 "&lt;zoo&gt;"
51 "zoo" N NOM SG
52 "&lt;to&gt;"
53 "to" INFMARK&gt;
54 "&lt;look&gt;"
55 "look" V INF
56 "&lt;at&gt;"
57 "at" PREP
58 "&lt;the&gt;"
59 "the" DET CENTRAL ART SG/PL
60 "&lt;bear&gt;"
61 "bear" N NOM SG
62 "&lt;.&gt;"
63 </screen>
64
65 Or in CG terms:
66 <screen>
67 "&lt;word form&gt;" <link linkend="stream-static-tags">static_tags</link>
68 "base form" tags
69 </screen>
70
71 Also known as:
72 <screen>
73 "&lt;surface form&gt;" <link linkend="stream-static-tags">static_tags</link>
74 "lexeme" tags
75 </screen>
76 </para>
77
78 <para>
79 In more formal rules:
80 <itemizedlist mark='bullet'>
81 <listitem><para>
82 If the line begins with "&lt; followed by non-quotes and/or escaped quotes followed by &gt;"
83 (regex <code>/^"&lt;(.|\\")*&gt;"/</code>) then it opens a new cohort.
84 </para></listitem>
85 <listitem><para>
86 If the line begins with whitespace followed by " followed by non-quotes and/or escaped quotes followed by "
87 (regex <code>/^\s+"(.|\\")*"/</code>) then it is parsed as a reading, but only if a cohort is open at the time.
88 Thus, any such lines seen before the first cohort is treated as text.
89 </para></listitem>
90 <listitem><para>
91 Any line not matching the above is treated as text. Text is handled in two ways: If no cohort is open at the time,
92 then it is output immediately. If a cohort is open, then it is appended to that cohort's buffer and output after
93 the cohort. Note that text between readings will thus be moved to after the readings.
94 Re-arranging cohorts will also re-arrange the text attached to them.
95 Removed cohorts will still output their attached text.
96 </para></listitem>
97 </itemizedlist>
98
99 This means that you can embed all kinda of extra information in the stream as long as you don't hit those exact
100 patterns. For example, we use <code>&lt;s id="unique-1234"&gt; &lt;/s&gt;</code> tags around sentences to keep track of them
101 for corpus markup.
102 </para>
103 </section>
104
105 <section id="stream-niceline">
106 <title>Niceline CG Format</title>
107 <indexterm>
108 <primary>Niceline CG Format</primary>
109 </indexterm>
110 <para>
111 Niceline input can be converted for use via <link linkend="cg-conv">cg-conv</link>.
112 </para>
113 <para>
114 The Niceline format is primarily used in VISL and GrammarSoft chains to make the output more readable.
115 Using the same example as for VISL CG format, that would look like:
116 <screen>
117 They [they] &lt;*&gt; PRON PERS NOM PL3 SUBJ
118 went [go] V PAST VFIN
119 to [to] PREP
120 the [the] DET CENTRAL ART SG/PL
121 zoo [zoo] N NOM SG
122 to [to] INFMARK&gt;
123 look [look] V INF
124 at [at] PREP
125 the [the] DET CENTRAL ART SG/PL
126 bear [bear] N NOM SG
127 .
128 </screen>
129
130 Or in CG terms:
131 <screen>
132 word form TAB [base form] tags TAB [base form] tags
133 ...or quotes...
134 word form TAB "base form" tags TAB "base form" tags
135 ...or mixed...
136 word form TAB "base form" tags TAB [base form] tags
137 </screen>
138 </para>
139 <para>
140 In more formal rules:
141 <itemizedlist mark='bullet'>
142 <listitem><para>
143 If the line does not begin with &lt; and contains a tab (\t, 0x09), then it is a cohort.
144 Anything up to the first tab is the word form. Readings are tab delimited, where if the first tag is
145 contained in [] or "" then it is taken as the base form. Tags are otherwise whitespace delimited.
146 </para></listitem>
147 <listitem><para>
148 Any line not matching the above is treated as text, same rules as for VISL CG format.
149 Note that a tab character is required for it to be a cohort
150 - a word or punctuation without the tab will be treated as text.
151 </para></listitem>
152 </itemizedlist>
153 </para>
154 </section>
155
156 <section id="stream-plain">
157 <title>Plain Text Format</title>
158 <indexterm>
159 <primary>Plain Text Format</primary>
160 </indexterm>
161 <para>
162 Plain text can be tokenized for use via <link linkend="cg-conv">cg-conv</link>.
163 It is a naive tokenizer that you should not use, and is only included as a last resort.
164 Five minutes in any scripting language should give you a much better tokenizer.
165 </para>
166 <para>
167 The tokenization rules are simple:
168 <itemizedlist mark='bullet'>
169 <listitem><para>
170 Split tokens on any kind of whitespace.
171 </para></listitem>
172 <listitem><para>
173 Split punctuation from the start and end of tokens into tokens.
174 Each punctuation character becomes a separate token.
175 </para></listitem>
176 <listitem><para>
177 Detect whether the token is ALLUPPER, Firstupper, or MiXeDCaSe and add a tag denoting it.
178 </para></listitem>
179 <listitem><para>
180 The token then becomes a cohort with one reading with a lower-case variant of the token as base form.
181 </para></listitem>
182 </itemizedlist>
183 </para>
184 </section>
185
186 </chapter>