=encoding utf8 =head1 NAME Catmandu::MARC::Tutorial - A documentation-only module for new users of Catmandu::MARC =head1 SYNOPSIS perldoc Catmandu::MARC::Tutorial =head1 UTF-8 =head2 MARC8 and UTF-8 The current Catmandu MARC tools are targetted for processing UTF-8 encoded files. When you have MARC8 encoded data tools like MarcEdit or C can be used to create a UTF-8 encoded file: $ yaz-marcdump -f MARC-8 -t UTF-8 -o marc -l 9=97 marc21.raw > marc21.utf8.raw =head2 Unicode errors If you process UTF-8 encoded files which contain faulty characters, you will get a fatal error message like: utf8 "\xD8" does not map to Unicode at ... Use the iconv (libc6-dev Linux package) tool, to preprocess the data and discard faulty characters: $ iconv -c -f UTF-8 -t UTF-8 marc21.utf8.raw | catmandu convert MARC --type RAW to JSON =head2 Convert a decomposed UTF-8 file to a combined UTF-8 file and vice versa For example, the character ä can be represented as "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), or as "ä", that is the two codepoints U+0061 U+0308 (three bytes 61 cc 88 in UTF-8). The uconv tool (from the libicu-dev Linux package) can be used to convert these types of files: $ uconv -x any-nfc < decomposed.txt > combined.txt $ uconv -x any-nfd < combined.txt > decomposed.txt =head1 READING =head2 Convert MARC21 records into JSON The command below converts file data.mrc into JSON: $ catmandu convert MARC to JSON < data.mrc =head2 Convert MARC21 records into MARC-XML $ catmandu convert MARC to MARC --type XML < data.mrc =head2 Convert UNIMARC records into JSON, XML, ... To read UNIMARC records use the RAW parser to get the correct character encoding. $ catmandu convert MARC --type RAW to JSON < data.mrc $ catmandu convert MARC --type RAW to MARC --type XML < data.mrc =head2 Create a CSV file containing all the titles To extract data from a MARC record on needs a Fix routine. This is a small language to manipulate data. In the example below we extract all 245 fields from MARC: $ catmandu convert MARC to CSV --fix 'marc_map(245,title); retain(title)' < data.mrc The Fix C puts the MARC 245 field in the C field. The Fix C<retain> makes sure only the title field ends up in the CSV file. =head2 Create a CSV file containing only the 245$a and 245$c subfields The C<marc_map> Fix can get one or more subfields to extract from MARC: $ catmandu convert MARC to CSV --fix 'marc_map(245ac,title); retain(title)' < data.mrc =head2 Create a CSV file which contains a repeated field In the example below the 650a field can be repeated in some MARC records. We will join all the repetitions in a comma delimited list for each record. $ catmandu convert MARC to CSV --fix 'marc_map(650a,subject,join:","); retain(subject)' < data.mrc =head2 Create a list of all ISBN numbers in the data In the previous example we saw how all subjects can be printed using a few Fix commands. When a subject is repeated in a record, it will be written on one line joined by a comma: subject1 subject2,subject3 subject4 In this example, record 1 contained 'subject1', record 2 'subject2' and 'subject3' and record 3 'subject4'. What should we use when we want a list of all values in a single long list? In the example below we'll print all ISBN numbers in a batch of MARC records in one long list using the Text exporter: $ catmandu convert MARC to Text --field_sep "\n" --fix 'marc_map(020a,isbn.$append); retain(isbn)' < data.mrc The first new thing is C<$append> in the marc_map. This will create in C<isbn> a list of all ISBN numbers found in the C<020a> field. The C<Text> exporter with the C<field_sep> option will use all list values in the C<isbn> field and writ them using new line as separator. =head2 Create a list of all unique ISBN numbers in the data Given the result of the previous command, it is now easy to create a unique list of ISBN numbers with the UNIX C<uniq> command: $ catmandu convert MARC to Text --field_sep "\n" --fix 'marc_map(020a,isbn.$append); retain(isbn)' < data.mrc | sort | uniq =head2 Create a list of the number of subjects per record We will create a list of subjects (650a) and count the number of items in this list for each record. The CSV file will contain the C<_id> (record identifier) and C<subject> the number of 650a fields. Writing all Fixes on the command line can become tedious. In Catmandu it is possible to create a Fix script that contains all the Fix commands. Open a text editor and create the C<myfix.fix> file with content: marc_map(650a,subject.$append) count(subject) retain(_id, subject) And execute the command: $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc =head2 Create a list of all ISBN numbers for records with type 920a == book In the example we need an extra condition for match the content of the 920a field against the string C<book>. Open a text editor and create the C<myfix.fix> file with content: marc_map(020a,isbn.$append) marc_map(920a,type) select all_match(type,"book") # select only the books select exists(isbn) # select only the records with ISBN numbers retain(isbn) # only keep this field Text after the C<#> sign are inline code comments. And run the command: $ catmandu convert MARC to Text --field_sep "\n" --fix myfix.fix < data.mrc =head2 Show which MARC records don't contain a 900a field matching some list of values First we need to create a list of keys that need to be matched against our MARC records. In the example below we create a CSV file with a C<key> , C<value> header and all the keys that are OK: $ cat mylist.txt key,value book,OK article,OK journal,OK Next we create a Fix script that maps the MARC 900a field to a field called C<type>. This C<type> field we lookup in the C<mylist.txt> file. If a match is found, then the C<type> field will contain the value in the list (OK). When no match is found then the C<type> will contain the original value. We reject all records that have OK as C<type> and keep only the ones that weren't matched in the file. Open a text editor and create the C<myfix.fix> file with content: marc_map(900a,type) lookup(type,'/tmp/mylist.txt') reject all_match(type,OK) retain(_id,type) And now run the command: $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc =head1 Create a CSV file of all ISSN numbers found at any MARC field To process this information we need to create a Fix script like the one below (line numbers are added here to explain the working of this script but should not be included in the script): 01: marc_map('***',text.$append) 02: 03: filter(text,'(\b\d{4}-?\d{3}[\dxX]\b)') 04: replace_all(text.*,'.*(\b\d{4}-?\d{3}[\dxX]\b).*',$1) 05: 06: do list(path:text) 07: unless is_valid_issn(.) 08: reject() 09: end 10: end 11: 12: vacuum() 13: 14: select exists(text) 15: 16: join_field(text,' ; ') 17: 18: retain(_id,text) On line 01 all the text in the MARC record is mapped into a C<text> array. On line 03 we filter out this array all the lines that contain an ISSN string using a regular expression. On line 04 the C<replace_all> is used to delete everything in the C<text> array that isn't an ISSN number. On line 06-10 we go over every ISSN string and check if it has a valid checksum and erase it when not. On line 12 we use the C<vacuum> function to remove any remaining empty fields On line 14 we select only the records that contain a valid ISSN number On line 16 the ISSN get joined by a semicolon ';' into a long string On line 18 we keep only the record id and the ISSNs in for the report. Run this Fix script (without the line number) using this command $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc =head2 Create a MARC validator For this example we need a Fix script that contains validation rules we need to check. For instance, we require to have a 245 field and at least a 008 control field with a date filled in. This can be coded as in: # Check if a 245 field is present unless marc_has('245') log("no 245 field",level:ERROR) end # Check if there is more than one 245 field if marc_has_many('245') log("more than one 245 field?",level:ERROR) end # Check if in 008 position 7 to 10 contains a 4 digit number ('\d' means digit) unless marc_match('008/07-10','\d{4}') log("no 4-digit year in 008 position 7 -> 10",level:ERROR) end Put this Fix script in a file C<myfix.fix> and execute the Catmandu command with the "-D" option for logging and the Null exporter to discard the normal output $ catmandu -D convert MARC to Null --fix myfix.fix < data.mrc =head1 TRANSFORMING =head2 Add a new MARC field In the example bellow we add new 856 field to the record with a $u subfield containing the Google homepage: marc_add(856,u,"http://www.google.com") A control field can be added by using the '_' subfield marc_add(009,_,0123456789) Maybe you want to copy the data from one subfield to another. Use the marc_map to store the data first in a temporary field and add it later to the new field: # copy a subfield marc_map(001,tmp) # maybe process the data a bit append(tmp,"-mytest") # add the contents of the tmp field to the new 009 field marc_add(009,_,$.tmp) =head2 Set a MARC subfield Set the $h subfield to a new value (or create it when it doesn't exist yet): marc_set(100h, test123) Only set the 100 field if the first indicator is 3 marc_set(100[3]h, test123) =head2 Remove a MARC (sub)field Remove all fields 500 , 501 , 5** : marc_remove(5**) Remove all 245h fields: marc_remove(245h) =head2 Append text to a MARC field Append a period to the 500 field is there isn't already there: do marc_each() unless marc_match(500, "\.$") # Only if the current field 500 doesn't end with a period marc_append(500,".") # Add to the current 500 field a period end end Use the L<Catmandu::Fix::Bind::marc_each> Bind to loop over all MARC fields. In the context of the C<do -- end> only one MARC field at a time is visible for the C<marc_*> fixes. =head2 The marc_each binder All C<marc_*> fixes will operate on all MARC fields matching a MARC path. For example, marc_remove(856) will remove all 856 MARC fields. In some cases you may want to change only some of the fields in a record. You could write: if marc_match(856u,"google") marc_remove(856) end in the hope it would remove the 856 fields that contain the text "google" in the $u subfield. Alas, this is not what will happen. The C<if> condition will match when the record contains one or more 856u fields containing "google". The C<marc_remove> Fix will delete B<all> 856 fields. To correctly remove only the 856 fields in the context of the C<if> statement the C<marc_each> binder is required: do marc_each() if marc_match(856u,"google") marc_remove(856) end end The C<marc_each> will loop over all MARC fields one at a time. The if statement will only match when the current MARC field is 856 and the $u field contains "google". The C<marc_remove(856)> will only delete the current 856 field. In C<marc_each> binder, it seems for all Fixes as if there is only one field at a time visible in the record. This Fix will not work: do marc_each() if marc_match(856u,"google") marc_remove(900) # <-- there is only a 856 field in the current context end end =head2 marc_copy, marc_cut and marc_paste The L<Catmandu::Fix::marc_copy>, L<Catmandu::Fix::marc_cut>, L<Catmandu::Fix::marc_paste> Fixes are needed when complicated edits are needed in MARC record. The C<marc_copy> fill copy parts of a MARC record matching a MARC_PATH to a temporary variable. This tempoarary variable will contain an ARRAY of HASHes containing the content of the MARC field. For instance, marc_copy(650, tmp) The C<tmp> will contain something like: tmp:[ { "subfields" : [ { "a" : "Perl (Computer program language)" } ], "ind1" : " ", "ind2" : "0", "tag" : "650" }, { "ind1" : " ", "subfields" : [ { "a" : "Web servers." } ], "tag" : "650", "ind2" : "0" } ] This structure can be edited with all the Catmandu fixes. For instance you can set the first indicator to '1': set_field(tmp.*.ind1 , 1) The JSON path C<tmp.*.ind1> will match all the first indicators. The JSON path C<tmp.*.tag> will match all the MARC tags. The JSON path C<tmp.*.subfields.*.a> will match all the $a subfields. For instance, to change all 'Perl' into 'Python' in the $a subfield use this Fix: replace_all(tmp.*.subfields.*.a,"Perl","Python") When the fields need to be places back into the record the C<marc_paste> command can be used: marc_paste(subjects) This will add all 650 fields in the C<tmp> temporary variable at the B<end> of the record. You can change the MARC fields in place using the C<march_each> binder: do marc_each() # Select only the 650 fields if marc_has(650) # Create a working copy marc_copy(650,tmp) # Change some fields set_field(tmp.*.ind1 , 1) # Paste the result back marc_paste(tmp) end end The C<marc_cut> Fix works like C<marc_copy> but will delete the matching MARC field from the record. =head2 Rename MARC subfields In the example below we rename each $1 subfield in the MARC record to $0 using the L<Catmandu::Fix::marc_cut>, L<Catmandu::Fix::marc_paste> and L<Catmandu::Fix::rename> fixes: # For each marc field... do marc_each() # Cut the field into tmp.. marc_cut(***,tmp) # Rename every 1 subfield to 0 rename(tmp.*.subfields.*,1,0) # And paste it back marc_paste(tmp) end The C<marc_each> bind will loop over all the MARC fields. With C<marc_cut> we store any field (C<***> matches every field) into a C<tmp> field. The C<marc_cut> creates an array structure in C<tmp> which is easy to process using the Fix language. Using the C<rename> function we search for all the subfields, and replace the field matching the regular expression C<1> with C<0>. At the end, we paste back the C<tmp> field into the record. =head2 Setting and remove MARC indicators In the example below we set every indicator1 of the 500 field to the value "0". We will use the L<Catmandu::Fix::Bind::marc_each> bind with a loop variable: # For each marc field... do marc_each(var:this) # If the marc field is a 500 field if marc_has(500) # Set the indicator1 to value "0" set_field(this.ind1,0) # Store the result back into the MARC record marc_remove(500) marc_paste(this) end end Using the same method indicators can also be deleted by setting their value to a space " ". =head2 Adding a new MARC subfield In the example below we append a new MARC subfield $z to the 500 field with value test. We will use the L<Catmandu::Fix::Bind::marc_each> bind with a loop variable: # For each marc field... do marc_each(var:this) # If the marc field is a 500 field if marc_has(500) # add a new subfield z add_field(this.subfields.$append.z,Test) # Store the result back into the MARC record marc_remove(500) marc_paste(this) end end =head2 Remove all non-numeric fields from the MARC record # For each marc field... do marc_each(var:this) # If we have a non-numeric fields unless all_match(this.tag,"\d{3}") # Remove this tag marc_remove(***) end end =head1 WRITING =head2 Convert a MARC record into a MARC record (do nothing) $ catmandu convert MARC to MARC < data.mrc > output.mrc =head2 Add a 920a field with value 'checked' to all records $ catmandu convert MARC to MARC --fix 'marc_add("900",a,"checked")' < data.mrc > output.mrc =head2 Delete the 024 fields from all MARC records $ catmandu convert MARC to MARC --fix 'marc_remove("024")' < data.mrc > output.mrc =head2 Set the 650p field to 'test' for all records $ catmandu convert MARC to MARC --fix 'marc_add("650p","test")' < data.mrc > output.mrc =head2 Select only the records with 900a == book $ catmandu convert MARC to MARC --fix 'marc_map(900a,type); select all_match(type,book)' < data.mrc > output.mrc The C<all_match> also allows a regular expressions: $ catmandu convert MARC to MARC --fix 'marc_map(900a,type); select all_match(type,"[Bb]ook")' < data.mrc > output.mrc =head2 Select only the records with 900a values in a given CSV file Create a CSV file with name,value pairs (need two columns): $ cat values.csv name,values book,1 journal,1 movie,1 $ catmandu convert MARC to MARC --fix myfixes.txt < data.mrc > output.mrc with myfixes.txt like: do marc_each() marc_map(900a,test) lookup(test,values.csv,default:0) select all_match(test,1) remove_field(test) end We use a "do marc_each() ... end" loop because 900a fields can be repeated. If a MARC tag isn't repeatable this loop not isn't needed. With marc_map we copy first the value of a marc subfield to a 'test' field. This test we lookup against the CSV file. Then, we select only the records that are found in the CSV file (and return the correct value). =head1 DEDUPLICATION =head2 Check for duplicate ISBN numbers in a MARC file In this example we extract from a MARC file all the ISBN numbers from the 020 and do a little bit of data cleaning using the L<Catmandu::Identifier> project. To install this package, we run this command: $ cpanm Catmandu::Identifier To extract all the ISBN numbers we use this Fix script 'dedup.fix': marc_map(020a, identifier.$append) replace_all(identifier.*,"\s+.*","") do list(path:identifier) isbn13(.) end do hashmap(exporter:YAML) copy_field(identifier,key) copy_field(_id,value) end The first C<marc_map> fix maps every 020 field to an identifier array. The C<replace_all> cleans the data a bit and deletes some unwanted text. The C<do list> will transform all the ISBN numbers to ISBN13. The C<do hashmap> will create an internal mapping table of identifier,_id key value pairs. For very identifier, one or more _id can be stored. At the end of all MARC processing this mapping table is dumped from memory as a YAML document. Run this fix as: $ catmandu convert MARC to Null --fix dedup.fix < marc.mrc > output.yml The output YAML file will contain the ISBN to document ID mapping. We only need the ISBN numbers with more than one hit. We need a little bit of cleanup on this YAML file to reach our final result. Use the following 'cleanup.fix' script: select exists(value.1) join_field(value,",") This first C<select> fix selects only the records with more than one hit. The C<join_field> will turn the array of results into a string. Execute this Fix like: $ catmandu convert YAML to TSV --fix cleanup.fix < output.yml > result.csv This will provide a tab delimited file of double isbn numbers in the MARC input file.