| 1 |
======================= |
| 2 |
CMIP5 product detection |
| 3 |
======================= |
| 4 |
|
| 5 |
drslib includes an algorithm for detecting the required CMIP5 product |
| 6 |
from filenames generated by CMOR2_ developed by Martin Juckes. The |
| 7 |
module :mod:`drslib.p_cmip5` contains an API to the algorithm and |
| 8 |
``drs_tool`` will use the algorithm to allocate a product for incoming |
| 9 |
data when used with the ``--detect-product`` option. |
| 10 |
|
| 11 |
A detailed discussion of the algorithm used is available as the document `Identifying the product`_ [PDF]. |
| 12 |
|
| 13 |
.. _`Identifying the product`: doc/requested_subset_decision_tree_v0_5.pdf |
| 14 |
|
| 15 |
|
| 16 |
``drs_tool`` configuration |
| 17 |
========================== |
| 18 |
|
| 19 |
Product detection requires extra information to be configured about |
| 20 |
the CMIP5 experiment and the data being processed. Before first using |
| 21 |
the product detection feature you should initialise the CMIP5 |
| 22 |
experiment description data using ``drs_tool init``: |
| 23 |
|
| 24 |
.. code-block:: bash |
| 25 |
|
| 26 |
$ drs_tool init --shelve-dir=DIR |
| 27 |
|
| 28 |
where ``DIR`` is a directory suitable for containing the data files. On an ESG datanode a suitable command might be: |
| 29 |
|
| 30 |
.. code-block:: bash |
| 31 |
|
| 32 |
$ drs_tool init --shelve-dir=/usr/local/share/p_cmip5/data |
| 33 |
|
| 34 |
In addition to the shelve directory ``p_cmip5`` requires additional |
| 35 |
information about the model output being processed to be included in |
| 36 |
an external configuration file (see `Input configuration file`_). |
| 37 |
|
| 38 |
Both these parameters can be configured using metaconfig as follows: |
| 39 |
|
| 40 |
.. code-block:: ini |
| 41 |
|
| 42 |
[metaconfig] |
| 43 |
configs = drslib |
| 44 |
|
| 45 |
[drslib:p_cmip5] |
| 46 |
shelve-dir = /usr/local/share/p_cmip5/data |
| 47 |
config = /usr/local/share/p_cmip5/model.ini |
| 48 |
|
| 49 |
|
| 50 |
.. include:: p_cmip5/configuration.rst |
| 51 |
|
| 52 |
|
| 53 |
Invoking drs_tool with product detection |
| 54 |
======================================== |
| 55 |
|
| 56 |
``drs_tool`` will allocate a product to files in the incoming |
| 57 |
directory when invoked with the ``--detect-product`` option. |
| 58 |
|
| 59 |
``drs_tool list`` will show the product deduced in each dataset_id and |
| 60 |
list datasets for which the product could not be determined as |
| 61 |
incomplete. ``drs_tool todo`` and ``drs_tool upgrade`` will only |
| 62 |
operate on datasets for which the product could be determined. |
| 63 |
|
| 64 |
In the following example data from the UK Met Office Hadley Centre is |
| 65 |
processed into the DRS hierarchy. |
| 66 |
|
| 67 |
.. code-block:: bash |
| 68 |
|
| 69 |
$ drs_tool list -I ./mohc_holding/ -R ./cmip5 --detect-product |
| 70 |
... |
| 71 |
[INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1 |
| 72 |
[INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2001120106-2002120100> |
| 73 |
[INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1 |
| 74 |
[INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2002120106-2003120100> |
| 75 |
[INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1 |
| 76 |
[INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2003120106-2004120100> |
| 77 |
[INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1 |
| 78 |
[INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2004120106-2005120100> |
| 79 |
[INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1 |
| 80 |
============================================================================== |
| 81 |
DRS Tree at . |
| 82 |
------------------------------------------------------------------------------ |
| 83 |
cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrLev.r1i1p1 0:0 1120:1371721676416 |
| 84 |
cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1 0:0 224:116004074368 |
| 85 |
------------------------------------------------------------------------------ |
| 86 |
2 datasets awaiting upgrade |
| 87 |
============================================================================== |
| 88 |
|
| 89 |
# Select the 6hrPlev dataset for publishing and check commands to be issued |
| 90 |
$ drs_tool todo -I ./mohc_holding/ -R . --detect-product cmip5.%.%.%.%.%.%.6hrPlev |
| 91 |
============================================================================== |
| 92 |
DRS Tree at . |
| 93 |
------------------------------------------------------------------------------ |
| 94 |
Publisher Tree cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1 todo for version 20101006 |
| 95 |
|
| 96 |
mv ./mohc_holding/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/psl_20101006/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc |
| 97 |
ln -s ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/psl_20101006/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/v20101006/psl/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc |
| 98 |
... |
| 99 |
mv ./mohc_holding/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/va_20101006/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc |
| 100 |
ln -s ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/va_20101006/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/v20101006/va/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc |
| 101 |
============================================================================== |
| 102 |
|
| 103 |
# Do the upgrade |
| 104 |
$ drs_tool upgrade -I ./mohc_holding/ -R . --detect-product cmip5.%.%.%.%.%.%.6hrPlev |
| 105 |
... |
| 106 |
|
| 107 |
# List the results. |
| 108 |
# Not including --detect-product lists datasets that are incompletely specified |
| 109 |
$ drs_tool list -I ./mohc_holding -R mohc_dryrun |
| 110 |
============================================================================== |
| 111 |
DRS Tree at mohc_dryrun |
| 112 |
------------------------------------------------------------------------------ |
| 113 |
cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.v20101005 224:116004074368 |
| 114 |
------------------------------------------------------------------------------ |
| 115 |
Incompletely specified incoming datasets |
| 116 |
------------------------------------------------------------------------------ |
| 117 |
cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrLev.r1i1p1 |
| 118 |
============================================================================== |
| 119 |
|
| 120 |
|
| 121 |
.. _CMOR2: http://www2-pcmdi.llnl.gov/cmor/documentation/ |