Debian Codegraph @

texmf-dist/doc/generic/pgf/pgfmanual-en-dv-formats.tex

raw
1 % Copyright 2019 by Till Tantau
2 %
3 % This file may be distributed and/or modified
4 %
5 % 1. under the LaTeX Project Public License and/or
6 % 2. under the GNU Free Documentation License.
7 %
8 % See the file doc/generic/pgf/licenses/LICENSE for more details.
9
10
11 \section{Providing Data for a Data Visualization}
12 \label{section-dv-formats}
13
14 \subsection{Overview}
15
16 The data visualization system needs a stream of data points as input. These
17 data points can be directly generated by repeatedly calling the |\pgfdatapoint|
18 command, but usually data is available in some special (text) format and one
19 would like to visualize this data. The present section explains how data in
20 some specific format can be fed to the data visualization system.
21
22 This section starts with an explanation of the main concepts. Then, the
23 standard formats are listed in the reference section. It is also possible to
24 define new formats, but this an advanced concept which requires an
25 understanding of some of the internals of the parsing mechanism, explained in
26 Section~\ref{section-dv-parsing}, and the usage of a rather low-level command,
27 explained in Section~\ref{section-dv-declaring-formats}.
28
29
30 \subsection{Concepts}
31
32 For the purposes of this section, let call a \emph{data format} some
33 standardized way of writing down a list of data points. A simple example of a
34 data format is the \textsc{csv} format (the acronym stands for \emph{comma
35 separated values}), where each line contains a data point, specified by values
36 separated by commas. A different format is the \emph{key--value format}, where
37 data points are specified by lists of key--value pairs. A far more complex
38 format is the \textsc{pdb}-format used by the protein database to describe
39 molecules.
40
41 The data visualization system does not use any specific format. Instead,
42 whenever data is read by the data visualization system, you must specify a
43 format parser (or it is chosen automatically for you). It is the job of the
44 parser to read (parse) the data lines and to turn them into data points, that
45 is, to setup appropriate subkeys of |/data point/|.
46
47 To give a concrete example, suppose a file contains the following
48 lines:
49 %
50 \begin{codeexample}[code only]
51 x, y, z
52 0, 0, 0
53 1, 1, 0
54 1, 1, 0.5
55 0, 1, 0.5
56 \end{codeexample}
57 %
58 This file is in the \textsc{csv}-format. This format can be read by the |table|
59 parser (which is called thus, rather than ``|csv|'', since it can also read
60 files in which the columns are separated by, say, a semicolon or a space). The
61 |table| format will then read the data and for each line of the data, except
62 for the headline of course, it will produce one data point. For instance, for
63 the last data point the key |/data point/x| will be set to |0|, the key
64 |/data point/y| will be set to |1|, and the key |/data point/z| will be set to
65 |0.5|.
66
67 All parsers are basically line-oriented. This means that, normally, each line
68 in the input data should contain one data point. This rule may not always
69 apply, for instance empty lines are typically ignored and sometimes a data
70 point may span several lines, but deviating from this ``one data point per
71 line'' rule makes parsers harder to program.
72
73
74 \subsection{Reference: Built-In Formats}
75
76 The following format is the default format, when no |format=...| is specified.
77
78 \begin{dataformat}{table}
79 This format is used to parse data that is formatted in the following
80 manner: Basically, each line consists of \emph{values} that are separated
81 by a \emph{separator} like a comma or a space. The values are stored in
82 different \emph{attributes}, that is, subkeys of |/data point| like
83 |/data point/x|. In order to decide which attribute is chosen for a give
84 value, the headline is important. This is the first non-empty line of a
85 table. It is formatted in the same way as normal data lines (value
86 separated by the separator), but the meaning of the values is different:
87 The first value in the headline is the name of the attribute where the
88 first values in the following lines should go each time. Similarly, the
89 second value in the headline is the name of the attribute for the second
90 values in the following lines, and so on.
91
92 A simple example is the following:
93 %
94 \begin{codeexample}[code only]
95 angle, radius
96 0, 1
97 45, 2
98 90, 3
99 135, 4
100 \end{codeexample}
101 %
102 The headline states that the values in the first column should be stored in
103 the |angle| attribute (|/data point/angle| to be precise) and that the
104 values in the second column should be stored in the |radius| attribute.
105 There are four data points in this data set.
106
107 The format will tolerate too few or too many values in a line. If there are
108 less values in a line than in the headline, the last attributes will simply
109 be empty. If there are more values in a line than in the headline, the
110 values are stored in attributes called |/data point/attribute |\meta{column
111 number}, where the first value of a line gets \meta{column number} equal to
112 |1| and so on.
113
114 The |table| format can be configured using the following options:
115 %
116 \begin{key}{/pgf/data/separator=\meta{character} (initially ,)}
117 Use this key to change which character is used to separate values in
118 the headline and in the data lines. To set the separator to a space,
119 either set this key to an empty value or say |separator=\space|. Note
120 that you must surround a comma by curly braces if you which to (re)set
121 the separator character to a space.
122 %
123 \begin{codeexample}[preamble={\usetikzlibrary{datavisualization}}]
124 \begin{tikzpicture}
125 \datavisualization [school book axes, visualize as line]
126 data [separator=\space] {
127 x y
128 0 0
129 1 1
130 2 1
131 3 0
132 }
133 data [separator=;] {
134 x; y; z
135 3; 1; 0
136 2; 2; 0
137 };
138 \end{tikzpicture}
139 \end{codeexample}
140 \end{key}
141 %
142 \begin{key}{/pgf/data/headline=\meta{headline}}
143 When this key is set to a non-empty value, the value of \meta{headline}
144 is used as the headline and the first line of the data is treated as a
145 normal line rather than as a headline.
146 %
147 \begin{codeexample}[preamble={\usetikzlibrary{datavisualization}}]
148 \begin{tikzpicture}
149 \datavisualization [school book axes, visualize as line]
150 data [headline={x, y}] {
151 0, 0
152 1, 1
153 2, 1
154 3, 0
155 };
156 \end{tikzpicture}
157 \end{codeexample}
158 \end{key}
159 \end{dataformat}
160
161 \begin{dataformat}{named}
162 Basically, each line of the data must consist of a comma-separated sequence
163 of attribute--values pairs like |x=5, lo=500|. This will cause the
164 attribute |/data point/x| to be set to |5| and |/data point/lo| to be set
165 to |500|.
166 %
167 \begin{codeexample}[preamble={\usetikzlibrary{datavisualization}}]
168 \begin{tikzpicture}
169 \datavisualization [school book axes, visualize as line]
170 data [format=named] {
171 x=0, y=0
172 x=1, y=1
173 x=2, y=1
174 x=3, y=0
175 };
176 \end{tikzpicture}
177 \end{codeexample}
178 %
179 However, instead of just specifying a single value for an attribute as in
180 |x=5|, you may also specify a whole set of values as in |x={1,2,3}|. In
181 this case, three data points will be created, one for each value in the
182 list. Indeed, the |\foreach| statement is used to iterate over the list of
183 values, so you can write things like |x={1,...,5}|.
184
185 It is also permissible to specify lists of values for more than one
186 attribute. In this case, a data point is created for each possible
187 combination of values in the different lists:
188 %
189 \begin{codeexample}[
190 width=7cm,
191 preamble={\usetikzlibrary{datavisualization}},
192 ]
193 \tikz \datavisualization
194 [scientific axes=clean,
195 visualize as scatter/.list={a,b,c},
196 style sheet=cross marks]
197 data [format=named] {
198 x=0, y={1,2,3}, set=a
199 x={2,3,4}, y={3,4,5,7}, set=b
200 x=6, y={5,7,...,15}, set=c
201 };
202 \end{codeexample}
203 %
204 \end{dataformat}
205
206 \begin{dataformat}{TeX code}
207 This format will simply execute each line of the data, each of which should
208 contain some normal TeX code. Note that at the end of each line control
209 returns to the format handler, so for instance the arguments of a command
210 may not be spread over several lines. However, not each line needs to
211 produce a data point.
212 %
213 \begin{codeexample}[preamble={\usetikzlibrary{datavisualization}}]
214 \begin{tikzpicture}
215 \datavisualization [school book axes, visualize as line]
216 data [format=TeX code] {
217 \pgfkeys{/data point/.cd,x=0, y=0} \pgfdatapoint
218 \pgfkeys{/data point/.cd,x=1, y=1} \pgfdatapoint
219 \pgfkeys{/data point/x=2} \pgfdatapoint
220 \pgfkeyssetvalue{/data point/x}{3}
221 \pgfkeyssetvalue{/data point/y}{0} \pgfdatapoint
222 };
223 \end{tikzpicture}
224 \end{codeexample}
225 %
226 \end{dataformat}
227
228
229 \subsection{Reference: Advanced Formats}
230
231 \begin{tikzlibrary}{datavisualization.formats.functions}
232 This library defines the formats described in the following, which allow
233 you to specify the data points indirectly, namely via a to-be-evaluated
234 function.
235
236 \begin{dataformat}{function}
237 This format allows you to specify a function that is then evaluated in
238 order to create the desired data points. In other words, the data lines
239 do not contain the data itself, but rather a functional description of
240 the data.
241
242 The format used to specify the function works as follows: Each nonempty
243 line of the data should contain at least one of either a \emph{variable
244 declaration} or a \emph{function declaration}. A variable declaration
245 signals that a certain attribute will range over a given interval. The
246 function declarations will then, later, be evaluated for values inside
247 this interval. The syntax for a variable declaration is one of the
248 following:
249 %
250 \begin{enumerate}
251 \item |var |\declare{\meta{variable}}| : interval[|\meta{low}|:|\meta{high}|]|
252 \opt{|samples |\meta{number}}|;|
253 \item |var |\declare{\meta{variable}}| : interval[|\meta{low}|:|\meta{high}%
254 |] step |\meta{step}|;|
255 \item |var |\declare{\meta{variable}}| : {|\meta{values}|};|
256 \end{enumerate}
257 %
258 In the first case, if the optional |samples| part is missing, the
259 number of |samples| is taken from the value stored in the following
260 key:
261 %
262 \begin{key}{/pgf/data/samples=\meta{number} (initially 25)}
263 Sets the number of samples to be used when no sample number is
264 specified.
265 \end{key}
266 %
267 The meaning of declaring a variable declaration to range over an
268 |interval| is that the attribute named \meta{variable}, that is, the
269 key |/data point/|\meta{variable}, will range over the interval
270 $[\meta{low},\meta{high}]$. If the number of |samples| is given
271 (directly or indirectly), the interval is evenly divided into
272 \meta{number} many points and the attribute is set to each of these
273 values. Similarly, when a \meta{step} is specified, this stepping is
274 used to increase \meta{low} iteratively up to the largest value that is
275 still less or equal to \meta{high}.
276
277 The meaning of declaring a variable using a list of \meta{values} is
278 that the variable will simply iterate over the values using |\foreach|.
279
280 You can specify more than one variable. In this case, each variable is
281 varied independently of the other variables. For instance, if you
282 declare an $x$-variable to range over the interval $[0,1]$ in $25$
283 steps and you also declare a $y$-variable to range over the same
284 interval, you get a total of $625$ value pairs.
285
286 The variable declarations specify which (input) variables will take
287 which values. It is the job of the \emph{function declarations} to
288 specify how some additional attributes are to be computed. The syntax
289 of a function declaration is as follows:
290 %
291 \begin{quote}
292 |func |\declare{\meta{attribute}}| = |\meta{expression}|;|
293 \end{quote}
294 %
295 The meaning of such a declaration is the following: For each setting of
296 the input variables (the variables specified using the |var|
297 declaration), evaluate the \meta{expression} using the standard
298 mathematical parser of \tikzname. The resulting value is then stored in
299 |/data point/|\meta{attribute}.
300
301 Inside \meta{expression} you can reference data point attributes using
302 the following command, which is only defined inside such an expression:
303 %
304 \begin{command}{\value\marg{variable}}
305 This expands to the current value of the key
306 |/data point/|\meta{variable}.
307 \end{command}
308
309 There can be multiple function declarations in a single data
310 specification. In this case, all of these functions will be evaluated
311 for each setting of input variables.
312 %
313 \begin{codeexample}[preamble={\usetikzlibrary{datavisualization.formats.functions}}]
314 \tikz
315 \datavisualization [school book axes, visualize as smooth line]
316 data [format=function] {
317 var x : interval [-1.5:1.5];
318
319 func y = \value x * \value x;
320 };
321 \end{codeexample}
322 %
323 \begin{codeexample}[
324 width=6cm,
325 preamble={\usetikzlibrary{datavisualization.formats.functions}},
326 ]
327 \tikz \datavisualization [
328 school book axes,
329 all axes={unit length=5mm, ticks={step=2}},
330 visualize as smooth line]
331 data [format=function] {
332 var t : interval [0:2*pi];
333
334 func x = \value t * cos(\value t r);
335 func y = \value t * sin(\value t r);
336 };
337 \end{codeexample}
338 %
339 \begin{codeexample}[
340 width=7cm,
341 preamble={\usetikzlibrary{datavisualization.formats.functions}},
342 ]
343 \tikz \datavisualization [
344 scientific axes=clean,
345 y axis={ticks={style={
346 /pgf/number format/fixed,
347 /pgf/number format/fixed zerofill,
348 /pgf/number format/precision=2}}},
349 x axis={ticks={tick suffix=${}^\circ$}},
350 visualize as smooth line/.list={1,2,3,4,5,6},
351 style sheet=vary hue]
352 data [format=function] {
353 var set : {1,...,6};
354 var x : interval [0:50];
355 func y = sin(\value x * (\value{set}+10))/(\value{set}+5);
356 };
357 \end{codeexample}
358 \end{dataformat}
359 \end{tikzlibrary}
360
361
362 \subsection{Advanced: The Data Parsing Process}
363 \label{section-dv-parsing}
364
365 Whenever data is fed to the data visualization system, it will be handled by
366 the |\pgfdata| command, declared in the |datavisualization| module. The command
367 is both used to parse data stored in external sources (that is, in external
368 files or which is produced on the fly by calling an external command) as well
369 as data given inline. A data format does not need to know whether data comes
370 from a file or is given inline, the |\pgfdata| command will take care of this.
371
372 Since \TeX\ will always read files in a line-wise fashion, data is always fed
373 to data format parsers in such a fashion. Thus, even it would make more sense
374 for a format to ignore line-breaks, the parser must still handle data given
375 line-by-line.
376
377 Let us now have a look at how |\pgfdata| works.
378
379 \begin{command}{\pgfdata\opt{\oarg{options}\marg{inline data}}}
380 This command is used to feed data to the visualization pipeline. This
381 command can only be used when a data visualization object has been properly
382 setup, see Section~\ref{section-dv-main-setup}.
383
384
385 \medskip
386 \textbf{Basic options.}
387 The |\pgfdata| command may be followed by \meta{options}, which are
388 executed with the path |/pgf/data/|. Depending on these options, the
389 \meta{options} may either be followed by \meta{inline data} or,
390 alternatively, no \meta{inline data} is present and the data is read from
391 an external source.
392
393 The first important option is \meta{source}, which governs which of these
394 two alternatives applies:
395 %
396 \begin{key}{/pgf/data/read from file=\meta{filename} (initially \normalfont empty)}
397 If you set the |read from file| attribute to a non-empty
398 \meta{filename}, the data will be read from this file. In this case, no
399 \meta{inline data} may be present, not even empty curly braces should
400 be provided. If |read from file| is empty, the data must directly
401 follow as \meta{inline data}.
402 %
403 \begin{codeexample}[code only]
404 % Data is read from two external files:
405 \pgfdata[format=table, read from file=file1.csv]
406 \pgfdata[format=table, read from file=file2.csv]
407 \end{codeexample}
408 %
409 \begin{codeexample}[code only]
410 % Data is given inline:
411 \pgfdata[format=table]
412 {
413 x, y
414 1, 2
415 2, 3
416 }
417 \end{codeexample}
418 \end{key}
419 %
420 \begin{key}{/pgf/data/inline}
421 This is a shorthand file |read from file={}|. You can add this to make
422 it clear(er) to the reader that data follows inline.
423 \end{key}
424 %
425 The second important key is |format|, which is used to specify the data
426 format:
427 %
428 \begin{key}{/pgf/data/format=\meta{format} (initially table)}
429 Use this key to locally set the format used for parsing the data. The
430 \meta{format} must be a format that has been previously declared using
431 the |\pgfdeclaredataformat| command. See the reference section for a
432 list of the predefined formats.
433 \end{key}
434 %
435 In case all your data is in a certain format, you may wish to generally set
436 the above key somewhere at the beginning of your file. Alternatively, you
437 can use the following style to setup the |format| key and possibly further
438 keys concerning the data format:
439 %
440 \begin{stylekey}{/pgf/every data}
441 This style is executed by |\pgfdata| before the \meta{options} are
442 parsed.
443
444 Note that the path of this key is just |/pgf/|, not |/pgf/data/|. Also
445 note that \tikzname\ internally sets the value of this key up in such a
446 way that the keys |/tikz/every data| and also
447 |/tikz/data visualization/every data| are executed. The bottom line of
448 this is that when using \tikzname, you should not set this key
449 directly, set |/tikz/every data| instead.
450 \end{stylekey}
451
452 \medskip
453 \textbf{Gathering of the data.}
454 Once the data format and the source have been decided upon, the data is
455 ``gathered''. During this phase the data is not actually parsed in detail,
456 but just gathered so that it can later be parsed during the visualization.
457 There are two different ways in which the data is gathered:
458 %
459 \begin{itemize}
460 \item In case you have specified an external source, the data
461 visualization object is told (by means of invoking the |add data|
462 method) that it should (later) read data from the file specified
463 by the |source| key using the format specified by the |format| key.
464 The file is not read at this point, but only later during the
465 actual visualization.
466 \item Otherwise, namely when data is given inline, depending on which
467 format is used, some catcodes get changed. This is necessary since
468 \TeX's special characters are often not-so-special in a certain
469 format.
470
471 Independently of the format, the end-of-line character (carriage
472 return) is made an active character.
473
474 Finally, the \meta{inline data} is then read as a normal argument
475 and the data visualization object is told that later on it should
476 parse this data using the given format parser. Note that in this
477 case the data visualization object must store the whole data
478 internally.
479 \end{itemize}
480 %
481 In both cases the ``data visualization object'' is the object stored
482 in the |/pgf/data visualization/obj| key.
483
484
485 \medskip
486 \textbf{Parsing of the data.}
487 During the actual data visualization, all code that has been added to the
488 data visualization object by means of the |add data| method is executed
489 several times. It is the job of this code to call the |\pgfdatapoint|
490 method for all data points present in the data.
491
492 When the |\pgfdata| method calls |add data|, the code that is passed to the
493 data visualization object is just a call to internal macros of |\pgfdata|,
494 which are able to parse the data stored in an external file or in the
495 inlined data. Independently of where the data is stored, these macros
496 always do the following:
497 %
498 \begin{enumerate}
499 \item The catcodes are setup according to what the data format
500 requires.
501 \item Format-specific startup code gets called, which can initialize
502 internal variables of the parsing process. (The catcode changes are
503 not part of the startup code since in order to read inline data
504 |\pgfdata| must be able to setup to temporarily setup the catcodes
505 needed later on by the parsers, but since no reading is to be done,
506 no startup code should be called at this point.)
507 \item For each line of the data a format-specific code handler, which
508 depends on the data format, is called. This handler gets the
509 current line as input and should call |\pgfdatapoint| once for each
510 data point that is encoded by this line (a line might define
511 multiple data points or none at all). Empty lines are handled by
512 special format-specific code.
513 \item At the end, format-specific end code is executed.
514 \end{enumerate}
515 %
516 For an example of how this works, see the description of the
517 |\pgfdeclaredataformat| command.
518
519
520 \medskip
521 \textbf{Data sets.}
522 There are three options that allow you to create \emph{data sets}. Such a
523 data set is essentially a macro that stores a pre-parsed set of data that
524 can be used multiple times in subsequent visualizations (or even in the
525 same visualization).
526 %
527 \begin{key}{/pgf/data/new set=\meta{name}}
528 Creates an empty data set called \meta{name}. If a data set of the same
529 name already exists, it is overwritten and made empty. Data sets are
530 global.
531 \end{key}
532 %
533 \begin{key}{/pgf/data/store in set=\meta{name}}
534 When this key is set to any non-empty \meta{name} and if this
535 \meta{name} has previously been used with the |new set| key, then the
536 following happens: For the current |\pgfdata| command, all parsed data
537 is not passed to the rendering pipeline. Instead, the parsed data is
538 appended to the data set \meta{name}. This includes all options parsed
539 to the |\pgfdata| command, which is why neither this key nor the
540 previous key should be passed as options to a |\pgfdata| command.
541 \end{key}
542 %
543 \begin{key}{/pgf/data/use set=\meta{name}}
544 This works similar to |read from file|. When this key is used with a
545 |\pgfdata| command, no inline data may follow. Instead, the data stored
546 in the data set \meta{name} is used.
547 \end{key}
548 \end{command}
549
550
551 \subsection{Advanced: Defining New Formats}
552 \label{section-dv-declaring-formats}
553
554 In order to define a new data format you can use the following command, which
555 is basic layer command defined in the module |datavisualization|:
556
557 \begin{command}{\pgfdeclaredataformat\marg{format name}\marg{catcode
558 code}\marg{startup code}\marg{line arguments}\\\marg{line
559 code}\marg{empty line code}\marg{end code}%
560 }
561 This command defines a new data format called \meta{format name}, which can
562 subsequently be used in the |\pgfdata| command. (The \tikzname's |data|
563 maps directly to |\pgfdata|, so the following applies to \tikzname\ as
564 well.)
565
566 As explained in the description of the |\pgfdata| command, when data is
567 being parsed that is formatted according to \meta{format name}, the
568 following happens:
569 %
570 \begin{enumerate}
571 \item The \meta{catcode code} is executed. This code should just
572 contain catcode changes. The \meta{catcode code} will also be
573 executed when inline data is read.
574 \item Next, the \meta{startup code} is executed.
575 \item Next, for each non-empty line of the data, the line is passed to
576 a macro whose argument list is given by \meta{line arguments} and
577 whose body is given by \meta{line code}. The idea is that you can
578 use \TeX's powerful pattern matching capabilities to parse the
579 non-empty lines. See also the below example.
580 \item Empty lines are not processed by the \meta{line code}, but rather
581 by the \meta{empty line code}. Typically, empty lines can simply be
582 ignored and in this case you can let this parameter be empty.
583 \item At the end of the data, the \meta{end code} is executed.
584 \end{enumerate}
585
586 As an example, let us now define a simple data format for reading files
587 formatted in the following manner: Each line should contain a coordinate
588 pair as in |(1.2,3.2)|, so two numbers separated by a comma and surrounded
589 by parentheses. To make things more interesting, suppose that the hash mark
590 symbol can be used to indicate comments. Here is an example of some data
591 given in this format:
592 %
593 \begin{codeexample}[code only]
594 # This is some data formatted according to the "coordinates" format
595 (0,0)
596 (0.5,0.25)
597 (1,1)
598 (1.5,2.25)
599 (2,4)
600 \end{codeexample}
601
602 A format parser for this format could be defined as follows:
603 %
604 \begin{codeexample}[code only]
605 \pgfdeclaredataformat{coordinates}
606 % First comes the catcode argument. We turn the hash mark into a comment character.
607 {\catcode`\#=14\relax}
608 % Second comes the startup code. Since we do not need to setup things, we can leave
609 % it empty. Note that we could also set it to something like \begingroup, provided we
610 % put an \endgroup in the end code
611 {}
612 % Now comes the arguments for non-empty lines. Well, these should be of the form
613 % (#1,#2), so we specify that:
614 {(#1,#2)}
615 % Now we must do something with a line of this form. We store the #1 argument in
616 % /data point/x and #2 in /data point/y. Then we call \pgfdatapoint to create a data point.
617 {
618 \pgfkeyssetvalue{/data point/x}{#1}
619 \pgfkeyssetvalue{/data point/y}{#2}
620 \pgfdatapoint
621 }
622 % We ignore empty lines:
623 {}
624 % And we also have no end-of-line code.
625 {}
626 \end{codeexample}
627 %
628 This format could now be used as follows:
629 %
630 \begin{codeexample}[code only]
631 \begin{tikzpicture}
632 \datavisualization[school book axes, visualize as smooth line]
633 data [format=coordinates] {
634 # This is some data formatted according
635 # to the "coordinates" format
636 (0,0)
637 (0.5,0.25)
638 (1,1)
639 (1.5,2.25)
640 (2,4)
641 };
642 \end{tikzpicture}
643 \end{codeexample}
644 %
645 \end{command}