Debian Codegraph @

README

raw
1 NAME
2
3 Catmandu::Breaker - Package that exports data in a Breaker format
4
5 SYNOPSIS
6
7 # From the command line
8
9 # Using the default breaker
10 $ catmandu convert JSON to Breaker < data.json
11
12 # Break a OAI-PMH harvest
13 $ catmandu convert OAI --url http://biblio.ugent.be/oai to Breaker
14
15 # Using a MARC breaker
16 $ catmandu convert MARC to Breaker --handler marc < data.mrc
17
18 # Using an XML breaker plus create a list of unique record fields
19 $ catmandu convert XML --path book to Breaker --handler xml --fields data.fields < t/book.xml > data.breaker
20
21 # Find the usage statistics of fields in the XML file above
22 $ catmandu breaker data.breaker
23
24 # Use the list of unique fields in the report
25 $ catmandu breaker --fields data.fields data.breaker
26
27 # verbose output
28 $ catmandu breaker -v data.breaker
29
30 # The breaker commands needs to know the unique fields in the dataset to build statistics.
31 # By default it will scan the whole file for fields. This can be a very
32 # time consuming process. With --maxscan one can limit the number of lines
33 # in the breaker file that can be scanned for unique fields
34 $ catmandu breaker -v --maxscan 1000000 data.breaker
35
36 # Alternatively the fields option can be used to specify the unique fields
37 $ catmandu breaker -v --fields 245a,022a data.breaker
38
39 $ cat data.breaker | cut -f 2 | sort -u > data.fields
40 $ catmandu breaker -v --fields data.fields data.breaker
41
42 # Export statistics as CSV. See L<Catmandu::Exporter::Stat> for supported formats.
43 $ catmandu breaker --as CSV data.breaker
44
45 DESCRIPTION
46
47 Inspired by the article "Metadata Analysis at the Command-Line" by Mark
48 Phillips in http://journal.code4lib.org/articles/7818 this exporter
49 breaks metadata records into the Breaker format which can be analyzed
50 further by command line tools.
51
52 BREAKER FORMAT
53
54 When breaking a input using 'catmandu convert {format} to Breaker' each
55 metadata fields gets transformed into a 'breaker' format:
56
57 <record-identifier><tab><metadata-field><tab><metadata-value><tab><metadatavalue>...
58
59 For the default JSON breaker the input format is broken down into
60 JSON-like Paths. E.g. when give this YAML input:
61
62 ---
63 name: John
64 colors:
65 - black
66 - yellow
67 - red
68 institution:
69 name: Acme
70 years:
71 - 1949
72 - 1950
73 - 1951
74 - 1952
75
76 the breaker command 'catmandu convert YAML to Breaker < file.yml' will
77 generate:
78
79 1 colors[] black
80 1 colors[] yellow
81 1 colors[] red
82 1 institution.name Acme
83 1 institution.years[] 1949
84 1 institution.years[] 1950
85 1 institution.years[] 1951
86 1 institution.years[] 1952
87 1 name John
88
89 The first column is a counter for each record (or the content of the
90 _id field when present). The second column provides a JSON path to the
91 data (with the array-paths translated to []). The third column is the
92 field value.
93
94 One can use this output in combination with Unix tools like grep, sort,
95 cut, etc to inspect the breaker output:
96
97 $ catmandu convert YAML to Breaker < file.yml | grep 'institution.years'
98
99 Some input formats, like MARC, the JSON-path format doesn't provide
100 much information which fields are present in the MARC because field
101 names are part of the data. It is then possible to use a special
102 handler to create a more verbose breaker output.
103
104 For instance, without a special handler:
105
106 $ catmandu convert MARC to Breaker < t/camel.usmarc
107 fol05731351 record[][] LDR
108 fol05731351 record[][] _
109 fol05731351 record[][] 00755cam 22002414a 4500
110 fol05731351 record[][] 001
111 fol05731351 record[][] _
112 fol05731351 record[][] fol05731351
113 fol05731351 record[][] 082
114 fol05731351 record[][] 0
115 fol05731351 record[][] 0
116 fol05731351 record[][] a
117
118 With the special marc handler:
119
120 $ catmandu convert MARC to Breaker --handler marc < t/camel.usmarc
121
122 fol05731351 LDR 00755cam 22002414a 4500
123 fol05731351 001 fol05731351
124 fol05731351 003 IMchF
125 fol05731351 005 20000613133448.0
126 fol05731351 008 000107s2000 nyua 001 0 eng
127 fol05731351 010a 00020737
128 fol05731351 020a 0471383147 (paper/cd-rom : alk. paper)
129 fol05731351 040a DLC
130 fol05731351 040c DLC
131 fol05731351 040d DLC
132
133 For the Catmandu::PICA tools a pica handler is available.
134
135 For the Catmandu::MAB2 tools a mab handler is available.
136
137 For the Catmandu::XML tools an xml handler is available:
138
139 $ catmandu convert XML --path book to Breaker --handler xml < t/book.xml
140
141 BREAKER STATISTICS
142
143 Statistical information can be calculated from a breaker output using
144 the 'catmandu breaker' command:
145
146 $ catmandu convert MARC to Breaker --handler marc < t/camel.usmarc > data.breaker
147 $ catmandu breaker data.breaker
148
149 | name | count | zeros | zeros% | min | max | mean | median | mode | variance | stdev | uniq%| entropy |
150 |------|-------|-------|--------|-----|-----|------|--------|--------|----------|-------|------|---------|
151 | 001 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 |
152 | 003 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 10 | 0.0/3.3 |
153 | 005 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 |
154 | 008 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 |
155 | 010a | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 |
156 | 020a | 9 | 1 | 10.0 | 0 | 1 | 0.9 | 1 | 1 | 0.09 | 0.3 | 90 | 3.3/3.3 |
157 | 040a | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 10 | 0.0/3.3 |
158 | 040c | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 10 | 0.0/3.3 |
159 | 040d | 5 | 5 | 50.0 | 0 | 1 | 0.5 | 0.5 | [0, 1] | 0.25 | 0.5 | 10 | 1.0/3.3 |
160
161 The output table provides statistical information on the usage of
162 fields in the original format. We see that the 001 field was counted 10
163 times in the data set, but the 040d value is only present 5 times. The
164 020a is empty in 10% (zeros%) of the records. The 001 has very unique
165 values (entropy is maximum), but all 040c fields contain the same
166 information (entropy is minimum).
167
168 See Catmandu::Exporter::Stat for more information about the statistical
169 fields and supported output formats.
170
171 MODULES
172
173 * Catmandu::Exporter::Breaker
174
175 * Catmandu::Cmd::breaker
176
177 SEE ALSO
178
179 Catmandu, Catmandu::MARC, Catmandu::XML, Catmandu::Stat
180
181 AUTHOR
182
183 Patrick Hochstenbach, <patrick.hochstenbach at ugent.be>
184
185 CONTRIBUTORS
186
187 Jakob Voss, nichtich at cpan.org
188
189 Johann Rolschewski, jorol at cpan.org
190