Debian Codegraph @

lib/Catmandu/MARC/Tutorial.pod

raw
1 =encoding utf8
2
3 =head1 NAME
4
5 Catmandu::MARC::Tutorial - A documentation-only module for new users of Catmandu::MARC
6
7 =head1 SYNOPSIS
8
9 perldoc Catmandu::MARC::Tutorial
10
11 =head1 UTF-8
12
13 =head2 MARC8 and UTF-8
14
15 The current Catmandu MARC tools are targetted for processing UTF-8 encoded files.
16 When you have MARC8 encoded data tools like MarcEdit <https://marcedit.reeset.net/>
17 or C<yaz-marcdump> <https://software.indexdata.com/yaz/doc/yaz-marcdump.html> can
18 be used to create a UTF-8 encoded file:
19
20 $ yaz-marcdump -f MARC-8 -t UTF-8 -o marc -l 9=97 marc21.raw > marc21.utf8.raw
21
22 =head2 Unicode errors
23
24 If you process UTF-8 encoded files which contain faulty characters, you will get a fatal error message like:
25
26 utf8 "\xD8" does not map to Unicode at ...
27
28 Use the iconv (libc6-dev Linux package) tool, to preprocess the data and discard faulty characters:
29
30 $ iconv -c -f UTF-8 -t UTF-8 marc21.utf8.raw | catmandu convert MARC --type RAW to JSON
31
32 =head2 Convert a decomposed UTF-8 file to a combined UTF-8 file and vice versa
33
34 For example, the character ä can be represented as
35
36 "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), or as
37 "ä", that is the two codepoints U+0061 U+0308 (three bytes 61 cc 88 in UTF-8).
38
39 The uconv tool (from the libicu-dev Linux package) can be used to convert these types of
40 files:
41
42 $ uconv -x any-nfc < decomposed.txt > combined.txt
43 $ uconv -x any-nfd < combined.txt > decomposed.txt
44
45 =head1 READING
46
47 =head2 Convert MARC21 records into JSON
48
49 The command below converts file data.mrc into JSON:
50
51 $ catmandu convert MARC to JSON < data.mrc
52
53 =head2 Convert MARC21 records into MARC-XML
54
55 $ catmandu convert MARC to MARC --type XML < data.mrc
56
57 =head2 Convert UNIMARC records into JSON, XML, ...
58
59 To read UNIMARC records use the RAW parser to get the correct character
60 encoding.
61
62 $ catmandu convert MARC --type RAW to JSON < data.mrc
63 $ catmandu convert MARC --type RAW to MARC --type XML < data.mrc
64
65 =head2 Create a CSV file containing all the titles
66
67 To extract data from a MARC record on needs a Fix routine. This
68 is a small language to manipulate data. In the example below
69 we extract all 245 fields from MARC:
70
71 $ catmandu convert MARC to CSV --fix 'marc_map(245,title); retain(title)' < data.mrc
72
73 The Fix C<marc_map> puts the MARC 245 field in the C<title> field.
74 The Fix C<retain> makes sure only the title field ends up in the
75 CSV file.
76
77 =head2 Create a CSV file containing only the 245$a and 245$c subfields
78
79 The C<marc_map> Fix can get one or more subfields to extract from MARC:
80
81 $ catmandu convert MARC to CSV --fix 'marc_map(245ac,title); retain(title)' < data.mrc
82
83 =head2 Create a CSV file which contains a repeated field
84
85 In the example below the 650a field can be repeated in some MARC records.
86 We will join all the repetitions in a comma delimited list for each record.
87
88 $ catmandu convert MARC to CSV --fix 'marc_map(650a,subject,join:","); retain(subject)' < data.mrc
89
90 =head2 Create a list of all ISBN numbers in the data
91
92 In the previous example we saw how all subjects can be printed using a few Fix commands.
93 When a subject is repeated in a record, it will be written on one line joined by a comma:
94
95 subject1
96 subject2,subject3
97 subject4
98
99 In this example, record 1 contained 'subject1', record 2 'subject2' and 'subject3' and
100 record 3 'subject4'. What should we use when we want a list of all values in a single long list?
101
102 In the example below we'll print all ISBN numbers in a batch of MARC records in one long list
103 using the Text exporter:
104
105 $ catmandu convert MARC to Text --field_sep "\n" --fix 'marc_map(020a,isbn.$append); retain(isbn)' < data.mrc
106
107 The first new thing is C<$append> in the marc_map. This will create in C<isbn> a
108 list of all ISBN numbers found in the C<020a> field.
109 The C<Text> exporter with the C<field_sep>
110 option will use all list values in the C<isbn> field and writ them using new line as separator.
111
112 =head2 Create a list of all unique ISBN numbers in the data
113
114 Given the result of the previous command, it is now easy to create a unique list of ISBN numbers
115 with the UNIX C<uniq> command:
116
117 $ catmandu convert MARC to Text --field_sep "\n" --fix 'marc_map(020a,isbn.$append); retain(isbn)' < data.mrc | sort | uniq
118
119 =head2 Create a list of the number of subjects per record
120
121 We will create a list of subjects (650a) and count the number of items
122 in this list for each record. The CSV file will contain the C<_id> (record
123 identifier) and C<subject> the number of 650a fields.
124
125 Writing all Fixes on the command line can become tedious. In Catmandu it is possible
126 to create a Fix script that contains all the Fix commands.
127
128 Open a text editor and create the C<myfix.fix> file with content:
129
130 marc_map(650a,subject.$append)
131 count(subject)
132 retain(_id, subject)
133
134 And execute the command:
135
136 $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc
137
138 =head2 Create a list of all ISBN numbers for records with type 920a == book
139
140 In the example we need an extra condition for match the content of the
141 920a field against the string C<book>.
142
143 Open a text editor and create the C<myfix.fix> file with content:
144
145 marc_map(020a,isbn.$append)
146 marc_map(920a,type)
147
148 select all_match(type,"book") # select only the books
149 select exists(isbn) # select only the records with ISBN numbers
150
151 retain(isbn) # only keep this field
152
153 Text after the C<#> sign are inline code comments.
154
155 And run the command:
156
157 $ catmandu convert MARC to Text --field_sep "\n" --fix myfix.fix < data.mrc
158
159 =head2 Show which MARC records don't contain a 900a field matching some list of values
160
161 First we need to create a list of keys that need to be matched against our MARC records.
162 In the example below we create a CSV file with a C<key> , C<value>
163 header and all the keys that are OK:
164
165 $ cat mylist.txt
166 key,value
167 book,OK
168 article,OK
169 journal,OK
170
171 Next we create a Fix script that maps the MARC 900a field to a field called
172 C<type>. This C<type> field we lookup in the C<mylist.txt> file. If a match
173 is found, then the C<type> field will contain the value in the list (OK). When
174 no match is found then the C<type> will contain the original value. We reject
175 all records that have OK as C<type> and keep only the ones that weren't matched
176 in the file.
177
178 Open a text editor and create the C<myfix.fix> file with content:
179
180 marc_map(900a,type)
181
182 lookup(type,'/tmp/mylist.txt')
183
184 reject all_match(type,OK)
185
186 retain(_id,type)
187
188 And now run the command:
189
190 $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc
191
192 =head1 Create a CSV file of all ISSN numbers found at any MARC field
193
194 To process this information we need to create a Fix script like the
195 one below (line numbers are added here to explain the working of this script
196 but should not be included in the script):
197
198 01: marc_map('***',text.$append)
199 02:
200 03: filter(text,'(\b\d{4}-?\d{3}[\dxX]\b)')
201 04: replace_all(text.*,'.*(\b\d{4}-?\d{3}[\dxX]\b).*',$1)
202 05:
203 06: do list(path:text)
204 07: unless is_valid_issn(.)
205 08: reject()
206 09: end
207 10: end
208 11:
209 12: vacuum()
210 13:
211 14: select exists(text)
212 15:
213 16: join_field(text,' ; ')
214 17:
215 18: retain(_id,text)
216
217 On line 01 all the text in the MARC record is mapped into a C<text> array.
218 On line 03 we filter out this array all the lines that contain an ISSN string
219 using a regular expression.
220 On line 04 the C<replace_all> is used to delete everything in the C<text>
221 array that isn't an ISSN number.
222 On line 06-10 we go over every ISSN string and check if it has a valid checksum
223 and erase it when not.
224 On line 12 we use the C<vacuum> function to remove any remaining empty fields
225 On line 14 we select only the records that contain a valid ISSN number
226 On line 16 the ISSN get joined by a semicolon ';' into a long string
227 On line 18 we keep only the record id and the ISSNs in for the report.
228
229 Run this Fix script (without the line number) using this command
230
231 $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc
232
233 =head2 Create a MARC validator
234
235 For this example we need a Fix script that contains validation rules we need to
236 check. For instance, we require to have a 245 field and at least a 008 control
237 field with a date filled in. This can be coded as in:
238
239 # Check if a 245 field is present
240 unless marc_has('245')
241 log("no 245 field",level:ERROR)
242 end
243
244 # Check if there is more than one 245 field
245 if marc_has_many('245')
246 log("more than one 245 field?",level:ERROR)
247 end
248
249 # Check if in 008 position 7 to 10 contains a 4 digit number ('\d' means digit)
250 unless marc_match('008/07-10','\d{4}')
251 log("no 4-digit year in 008 position 7 -> 10",level:ERROR)
252 end
253
254 Put this Fix script in a file C<myfix.fix> and execute the Catmandu command
255 with the "-D" option for logging and the Null exporter to discard the normal
256 output
257
258 $ catmandu -D convert MARC to Null --fix myfix.fix < data.mrc
259
260 =head1 TRANSFORMING
261
262 =head2 Add a new MARC field
263
264 In the example bellow we add new 856 field to the record with a $u subfield containing
265 the Google homepage:
266
267 marc_add(856,u,"http://www.google.com")
268
269 A control field can be added by using the '_' subfield
270
271 marc_add(009,_,0123456789)
272
273 Maybe you want to copy the data from one subfield to another. Use the marc_map to
274 store the data first in a temporary field and add it later to the new field:
275
276 # copy a subfield
277 marc_map(001,tmp)
278
279 # maybe process the data a bit
280 append(tmp,"-mytest")
281
282 # add the contents of the tmp field to the new 009 field
283 marc_add(009,_,$.tmp)
284
285 =head2 Set a MARC subfield
286
287 Set the $h subfield to a new value (or create it when it doesn't exist yet):
288
289 marc_set(100h, test123)
290
291 Only set the 100 field if the first indicator is 3
292
293 marc_set(100[3]h, test123)
294
295 =head2 Remove a MARC (sub)field
296
297 Remove all fields 500 , 501 , 5** :
298
299 marc_remove(5**)
300
301 Remove all 245h fields:
302
303 marc_remove(245h)
304
305 =head2 Append text to a MARC field
306
307 Append a period to the 500 field is there isn't already there:
308
309 do marc_each()
310 unless marc_match(500, "\.$") # Only if the current field 500 doesn't end with a period
311 marc_append(500,".") # Add to the current 500 field a period
312 end
313 end
314
315 Use the L<Catmandu::Fix::Bind::marc_each> Bind to loop over all MARC fields. In the
316 context of the C<do -- end> only one MARC field at a time is visible for the C<marc_*> fixes.
317
318 =head2 The marc_each binder
319
320 All C<marc_*> fixes will operate on all MARC fields matching a MARC path. For example,
321
322 marc_remove(856)
323
324 will remove all 856 MARC fields. In some cases you may want to change only some of the fields
325 in a record. You could write:
326
327 if marc_match(856u,"google")
328 marc_remove(856)
329 end
330
331 in the hope it would remove the 856 fields that contain the text "google" in the $u subfield.
332 Alas, this is not what will happen. The C<if> condition will match when the record contains one or
333 more 856u fields containing "google". The C<marc_remove> Fix will delete B<all> 856 fields. To
334 correctly remove only the 856 fields in the context of the C<if> statement the C<marc_each> binder
335 is required:
336
337 do marc_each()
338 if marc_match(856u,"google")
339 marc_remove(856)
340 end
341 end
342
343 The C<marc_each> will loop over all MARC fields one at a time. The if statement will only match when
344 the current MARC field is 856 and the $u field contains "google". The C<marc_remove(856)> will only
345 delete the current 856 field.
346
347 In C<marc_each> binder, it seems for all Fixes as if there is only one field at a time visible in the record.
348 This Fix will not work:
349
350 do marc_each()
351 if marc_match(856u,"google")
352 marc_remove(900) # <-- there is only a 856 field in the current context
353 end
354 end
355
356 =head2 marc_copy, marc_cut and marc_paste
357
358 The L<Catmandu::Fix::marc_copy>, L<Catmandu::Fix::marc_cut>, L<Catmandu::Fix::marc_paste> Fixes
359 are needed when complicated edits are needed in MARC record.
360
361 The C<marc_copy> fill copy parts of a MARC record matching a MARC_PATH to a temporary variable.
362 This tempoarary variable will contain an ARRAY of HASHes containing the content of the MARC field.
363
364 For instance,
365
366 marc_copy(650, tmp)
367
368 The C<tmp> will contain something like:
369
370 tmp:[
371 {
372 "subfields" : [
373 {
374 "a" : "Perl (Computer program language)"
375 }
376 ],
377 "ind1" : " ",
378 "ind2" : "0",
379 "tag" : "650"
380 },
381 {
382 "ind1" : " ",
383 "subfields" : [
384 {
385 "a" : "Web servers."
386 }
387 ],
388 "tag" : "650",
389 "ind2" : "0"
390 }
391 ]
392
393 This structure can be edited with all the Catmandu fixes. For instance you can set the first
394 indicator to '1':
395
396 set_field(tmp.*.ind1 , 1)
397
398 The JSON path C<tmp.*.ind1> will match all the first indicators. The JSON path
399 C<tmp.*.tag> will match all the MARC tags. The JSON path C<tmp.*.subfields.*.a> will
400 match all the $a subfields. For instance, to change all 'Perl' into 'Python' in the $a subfield
401 use this Fix:
402
403 replace_all(tmp.*.subfields.*.a,"Perl","Python")
404
405 When the fields need to be places back into the record the C<marc_paste> command can be used:
406
407 marc_paste(subjects)
408
409 This will add all 650 fields in the C<tmp> temporary variable at the B<end> of the record. You can
410 change the MARC fields in place using the C<march_each> binder:
411
412 do marc_each()
413 # Select only the 650 fields
414 if marc_has(650)
415 # Create a working copy
416 marc_copy(650,tmp)
417
418 # Change some fields
419 set_field(tmp.*.ind1 , 1)
420
421 # Paste the result back
422 marc_paste(tmp)
423 end
424 end
425
426 The C<marc_cut> Fix works like C<marc_copy> but will delete the matching MARC field from the record.
427
428 =head2 Rename MARC subfields
429
430 In the example below we rename each $1 subfield in the MARC record to $0 using
431 the L<Catmandu::Fix::marc_cut>, L<Catmandu::Fix::marc_paste> and L<Catmandu::Fix::rename>
432 fixes:
433
434 # For each marc field...
435 do marc_each()
436 # Cut the field into tmp..
437 marc_cut(***,tmp)
438
439 # Rename every 1 subfield to 0
440 rename(tmp.*.subfields.*,1,0)
441
442 # And paste it back
443 marc_paste(tmp)
444 end
445
446 The C<marc_each> bind will loop over all the MARC fields. With C<marc_cut> we
447 store any field (C<***> matches every field) into a C<tmp> field. The C<marc_cut>
448 creates an array structure in C<tmp> which is easy to process using the Fix
449 language. Using the C<rename> function we search for all the subfields, and replace
450 the field matching the regular expression C<1> with C<0>. At the end, we paste
451 back the C<tmp> field into the record.
452
453 =head2 Setting and remove MARC indicators
454
455 In the example below we set every indicator1 of the 500 field to the value "0".
456 We will use the L<Catmandu::Fix::Bind::marc_each> bind with a loop variable:
457
458 # For each marc field...
459 do marc_each(var:this)
460 # If the marc field is a 500 field
461 if marc_has(500)
462 # Set the indicator1 to value "0"
463 set_field(this.ind1,0)
464 # Store the result back into the MARC record
465 marc_remove(500)
466 marc_paste(this)
467 end
468 end
469
470 Using the same method indicators can also be deleted by setting their value to
471 a space " ".
472
473 =head2 Adding a new MARC subfield
474
475 In the example below we append a new MARC subfield $z to the 500 field with value test.
476 We will use the L<Catmandu::Fix::Bind::marc_each> bind with a loop variable:
477
478 # For each marc field...
479 do marc_each(var:this)
480 # If the marc field is a 500 field
481 if marc_has(500)
482 # add a new subfield z
483 add_field(this.subfields.$append.z,Test)
484 # Store the result back into the MARC record
485 marc_remove(500)
486 marc_paste(this)
487 end
488 end
489
490 =head2 Remove all non-numeric fields from the MARC record
491
492 # For each marc field...
493 do marc_each(var:this)
494 # If we have a non-numeric fields
495 unless all_match(this.tag,"\d{3}")
496 # Remove this tag
497 marc_remove(***)
498 end
499 end
500
501 =head1 WRITING
502
503 =head2 Convert a MARC record into a MARC record (do nothing)
504
505 $ catmandu convert MARC to MARC < data.mrc > output.mrc
506
507 =head2 Add a 920a field with value 'checked' to all records
508
509 $ catmandu convert MARC to MARC --fix 'marc_add("900",a,"checked")' < data.mrc > output.mrc
510
511 =head2 Delete the 024 fields from all MARC records
512
513 $ catmandu convert MARC to MARC --fix 'marc_remove("024")' < data.mrc > output.mrc
514
515 =head2 Set the 650p field to 'test' for all records
516
517 $ catmandu convert MARC to MARC --fix 'marc_add("650p","test")' < data.mrc > output.mrc
518
519 =head2 Select only the records with 900a == book
520
521 $ catmandu convert MARC to MARC --fix 'marc_map(900a,type); select all_match(type,book)' < data.mrc > output.mrc
522
523 The C<all_match> also allows a regular expressions:
524
525 $ catmandu convert MARC to MARC --fix 'marc_map(900a,type); select all_match(type,"[Bb]ook")' < data.mrc > output.mrc
526
527 =head2 Select only the records with 900a values in a given CSV file
528
529 Create a CSV file with name,value pairs (need two columns):
530
531 $ cat values.csv
532 name,values
533 book,1
534 journal,1
535 movie,1
536
537 $ catmandu convert MARC to MARC --fix myfixes.txt < data.mrc > output.mrc
538
539 with myfixes.txt like:
540
541 do marc_each()
542 marc_map(900a,test)
543 lookup(test,values.csv,default:0)
544 select all_match(test,1)
545 remove_field(test)
546 end
547
548 We use a "do marc_each() ... end" loop because 900a fields can be repeated. If a
549 MARC tag isn't repeatable this loop not isn't needed. With marc_map we copy
550 first the value of a marc subfield to a 'test' field. This test we lookup against
551 the CSV file. Then, we select only the records that are found in the CSV file
552 (and return the correct value).
553
554 =head1 DEDUPLICATION
555
556 =head2 Check for duplicate ISBN numbers in a MARC file
557
558 In this example we extract from a MARC file all the ISBN numbers from
559 the 020 and do a little bit of data cleaning using the L<Catmandu::Identifier>
560 project. To install this package, we run this command:
561
562 $ cpanm Catmandu::Identifier
563
564 To extract all the ISBN numbers we use this Fix script 'dedup.fix':
565
566 marc_map(020a, identifier.$append)
567 replace_all(identifier.*,"\s+.*","")
568 do list(path:identifier)
569 isbn13(.)
570 end
571 do hashmap(exporter:YAML)
572 copy_field(identifier,key)
573 copy_field(_id,value)
574 end
575
576 The first C<marc_map> fix maps every 020 field to an identifier array.
577 The C<replace_all> cleans the data a bit and deletes some unwanted text.
578 The C<do list> will transform all the ISBN numbers to ISBN13.
579 The C<do hashmap> will create an internal mapping table of identifier,_id key
580 value pairs. For very identifier, one or more _id can be stored. At the end
581 of all MARC processing this mapping table is dumped from memory as a YAML document.
582
583 Run this fix as:
584
585 $ catmandu convert MARC to Null --fix dedup.fix < marc.mrc > output.yml
586
587 The output YAML file will contain the ISBN to document ID mapping. We
588 only need the ISBN numbers with more than one hit. We need a little bit
589 of cleanup on this YAML file to reach our final result. Use the following
590 'cleanup.fix' script:
591
592 select exists(value.1)
593 join_field(value,",")
594
595 This first C<select> fix selects only the records with more than one hit.
596 The C<join_field> will turn the array of results into a string. Execute
597 this Fix like:
598
599 $ catmandu convert YAML to TSV --fix cleanup.fix < output.yml > result.csv
600
601 This will provide a tab delimited file of double isbn numbers in the MARC
602 input file.