| 1 |
NAME |
| 2 |
|
| 3 |
Catmandu::Breaker - Package that exports data in a Breaker format |
| 4 |
|
| 5 |
SYNOPSIS |
| 6 |
|
| 7 |
# From the command line |
| 8 |
|
| 9 |
# Using the default breaker |
| 10 |
$ catmandu convert JSON to Breaker < data.json |
| 11 |
|
| 12 |
# Break a OAI-PMH harvest |
| 13 |
$ catmandu convert OAI --url http://biblio.ugent.be/oai to Breaker |
| 14 |
|
| 15 |
# Using a MARC breaker |
| 16 |
$ catmandu convert MARC to Breaker --handler marc < data.mrc |
| 17 |
|
| 18 |
# Using an XML breaker plus create a list of unique record fields |
| 19 |
$ catmandu convert XML --path book to Breaker --handler xml --fields data.fields < t/book.xml > data.breaker |
| 20 |
|
| 21 |
# Find the usage statistics of fields in the XML file above |
| 22 |
$ catmandu breaker data.breaker |
| 23 |
|
| 24 |
# Use the list of unique fields in the report |
| 25 |
$ catmandu breaker --fields data.fields data.breaker |
| 26 |
|
| 27 |
# verbose output |
| 28 |
$ catmandu breaker -v data.breaker |
| 29 |
|
| 30 |
# The breaker commands needs to know the unique fields in the dataset to build statistics. |
| 31 |
# By default it will scan the whole file for fields. This can be a very |
| 32 |
# time consuming process. With --maxscan one can limit the number of lines |
| 33 |
# in the breaker file that can be scanned for unique fields |
| 34 |
$ catmandu breaker -v --maxscan 1000000 data.breaker |
| 35 |
|
| 36 |
# Alternatively the fields option can be used to specify the unique fields |
| 37 |
$ catmandu breaker -v --fields 245a,022a data.breaker |
| 38 |
|
| 39 |
$ cat data.breaker | cut -f 2 | sort -u > data.fields |
| 40 |
$ catmandu breaker -v --fields data.fields data.breaker |
| 41 |
|
| 42 |
# Export statistics as CSV. See L<Catmandu::Exporter::Stat> for supported formats. |
| 43 |
$ catmandu breaker --as CSV data.breaker |
| 44 |
|
| 45 |
DESCRIPTION |
| 46 |
|
| 47 |
Inspired by the article "Metadata Analysis at the Command-Line" by Mark |
| 48 |
Phillips in http://journal.code4lib.org/articles/7818 this exporter |
| 49 |
breaks metadata records into the Breaker format which can be analyzed |
| 50 |
further by command line tools. |
| 51 |
|
| 52 |
BREAKER FORMAT |
| 53 |
|
| 54 |
When breaking a input using 'catmandu convert {format} to Breaker' each |
| 55 |
metadata fields gets transformed into a 'breaker' format: |
| 56 |
|
| 57 |
<record-identifier><tab><metadata-field><tab><metadata-value><tab><metadatavalue>... |
| 58 |
|
| 59 |
For the default JSON breaker the input format is broken down into |
| 60 |
JSON-like Paths. E.g. when give this YAML input: |
| 61 |
|
| 62 |
--- |
| 63 |
name: John |
| 64 |
colors: |
| 65 |
- black |
| 66 |
- yellow |
| 67 |
- red |
| 68 |
institution: |
| 69 |
name: Acme |
| 70 |
years: |
| 71 |
- 1949 |
| 72 |
- 1950 |
| 73 |
- 1951 |
| 74 |
- 1952 |
| 75 |
|
| 76 |
the breaker command 'catmandu convert YAML to Breaker < file.yml' will |
| 77 |
generate: |
| 78 |
|
| 79 |
1 colors[] black |
| 80 |
1 colors[] yellow |
| 81 |
1 colors[] red |
| 82 |
1 institution.name Acme |
| 83 |
1 institution.years[] 1949 |
| 84 |
1 institution.years[] 1950 |
| 85 |
1 institution.years[] 1951 |
| 86 |
1 institution.years[] 1952 |
| 87 |
1 name John |
| 88 |
|
| 89 |
The first column is a counter for each record (or the content of the |
| 90 |
_id field when present). The second column provides a JSON path to the |
| 91 |
data (with the array-paths translated to []). The third column is the |
| 92 |
field value. |
| 93 |
|
| 94 |
One can use this output in combination with Unix tools like grep, sort, |
| 95 |
cut, etc to inspect the breaker output: |
| 96 |
|
| 97 |
$ catmandu convert YAML to Breaker < file.yml | grep 'institution.years' |
| 98 |
|
| 99 |
Some input formats, like MARC, the JSON-path format doesn't provide |
| 100 |
much information which fields are present in the MARC because field |
| 101 |
names are part of the data. It is then possible to use a special |
| 102 |
handler to create a more verbose breaker output. |
| 103 |
|
| 104 |
For instance, without a special handler: |
| 105 |
|
| 106 |
$ catmandu convert MARC to Breaker < t/camel.usmarc |
| 107 |
fol05731351 record[][] LDR |
| 108 |
fol05731351 record[][] _ |
| 109 |
fol05731351 record[][] 00755cam 22002414a 4500 |
| 110 |
fol05731351 record[][] 001 |
| 111 |
fol05731351 record[][] _ |
| 112 |
fol05731351 record[][] fol05731351 |
| 113 |
fol05731351 record[][] 082 |
| 114 |
fol05731351 record[][] 0 |
| 115 |
fol05731351 record[][] 0 |
| 116 |
fol05731351 record[][] a |
| 117 |
|
| 118 |
With the special marc handler: |
| 119 |
|
| 120 |
$ catmandu convert MARC to Breaker --handler marc < t/camel.usmarc |
| 121 |
|
| 122 |
fol05731351 LDR 00755cam 22002414a 4500 |
| 123 |
fol05731351 001 fol05731351 |
| 124 |
fol05731351 003 IMchF |
| 125 |
fol05731351 005 20000613133448.0 |
| 126 |
fol05731351 008 000107s2000 nyua 001 0 eng |
| 127 |
fol05731351 010a 00020737 |
| 128 |
fol05731351 020a 0471383147 (paper/cd-rom : alk. paper) |
| 129 |
fol05731351 040a DLC |
| 130 |
fol05731351 040c DLC |
| 131 |
fol05731351 040d DLC |
| 132 |
|
| 133 |
For the Catmandu::PICA tools a pica handler is available. |
| 134 |
|
| 135 |
For the Catmandu::MAB2 tools a mab handler is available. |
| 136 |
|
| 137 |
For the Catmandu::XML tools an xml handler is available: |
| 138 |
|
| 139 |
$ catmandu convert XML --path book to Breaker --handler xml < t/book.xml |
| 140 |
|
| 141 |
BREAKER STATISTICS |
| 142 |
|
| 143 |
Statistical information can be calculated from a breaker output using |
| 144 |
the 'catmandu breaker' command: |
| 145 |
|
| 146 |
$ catmandu convert MARC to Breaker --handler marc < t/camel.usmarc > data.breaker |
| 147 |
$ catmandu breaker data.breaker |
| 148 |
|
| 149 |
| name | count | zeros | zeros% | min | max | mean | median | mode | variance | stdev | uniq%| entropy | |
| 150 |
|------|-------|-------|--------|-----|-----|------|--------|--------|----------|-------|------|---------| |
| 151 |
| 001 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 | |
| 152 |
| 003 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 10 | 0.0/3.3 | |
| 153 |
| 005 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 | |
| 154 |
| 008 | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 | |
| 155 |
| 010a | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 100 | 3.3/3.3 | |
| 156 |
| 020a | 9 | 1 | 10.0 | 0 | 1 | 0.9 | 1 | 1 | 0.09 | 0.3 | 90 | 3.3/3.3 | |
| 157 |
| 040a | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 10 | 0.0/3.3 | |
| 158 |
| 040c | 10 | 0 | 0.0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 10 | 0.0/3.3 | |
| 159 |
| 040d | 5 | 5 | 50.0 | 0 | 1 | 0.5 | 0.5 | [0, 1] | 0.25 | 0.5 | 10 | 1.0/3.3 | |
| 160 |
|
| 161 |
The output table provides statistical information on the usage of |
| 162 |
fields in the original format. We see that the 001 field was counted 10 |
| 163 |
times in the data set, but the 040d value is only present 5 times. The |
| 164 |
020a is empty in 10% (zeros%) of the records. The 001 has very unique |
| 165 |
values (entropy is maximum), but all 040c fields contain the same |
| 166 |
information (entropy is minimum). |
| 167 |
|
| 168 |
See Catmandu::Exporter::Stat for more information about the statistical |
| 169 |
fields and supported output formats. |
| 170 |
|
| 171 |
MODULES |
| 172 |
|
| 173 |
* Catmandu::Exporter::Breaker |
| 174 |
|
| 175 |
* Catmandu::Cmd::breaker |
| 176 |
|
| 177 |
SEE ALSO |
| 178 |
|
| 179 |
Catmandu, Catmandu::MARC, Catmandu::XML, Catmandu::Stat |
| 180 |
|
| 181 |
AUTHOR |
| 182 |
|
| 183 |
Patrick Hochstenbach, <patrick.hochstenbach at ugent.be> |
| 184 |
|
| 185 |
CONTRIBUTORS |
| 186 |
|
| 187 |
Jakob Voss, nichtich at cpan.org |
| 188 |
|
| 189 |
Johann Rolschewski, jorol at cpan.org |
| 190 |
|