Debian Codegraph @

doc/product.rst

raw
1 =======================
2 CMIP5 product detection
3 =======================
4
5 drslib includes an algorithm for detecting the required CMIP5 product
6 from filenames generated by CMOR2_ developed by Martin Juckes. The
7 module :mod:`drslib.p_cmip5` contains an API to the algorithm and
8 ``drs_tool`` will use the algorithm to allocate a product for incoming
9 data when used with the ``--detect-product`` option.
10
11 A detailed discussion of the algorithm used is available as the document `Identifying the product`_ [PDF].
12
13 .. _`Identifying the product`: doc/requested_subset_decision_tree_v0_5.pdf
14
15
16 ``drs_tool`` configuration
17 ==========================
18
19 Product detection requires extra information to be configured about
20 the CMIP5 experiment and the data being processed. Before first using
21 the product detection feature you should initialise the CMIP5
22 experiment description data using ``drs_tool init``:
23
24 .. code-block:: bash
25
26 $ drs_tool init --shelve-dir=DIR
27
28 where ``DIR`` is a directory suitable for containing the data files. On an ESG datanode a suitable command might be:
29
30 .. code-block:: bash
31
32 $ drs_tool init --shelve-dir=/usr/local/share/p_cmip5/data
33
34 In addition to the shelve directory ``p_cmip5`` requires additional
35 information about the model output being processed to be included in
36 an external configuration file (see `Input configuration file`_).
37
38 Both these parameters can be configured using metaconfig as follows:
39
40 .. code-block:: ini
41
42 [metaconfig]
43 configs = drslib
44
45 [drslib:p_cmip5]
46 shelve-dir = /usr/local/share/p_cmip5/data
47 config = /usr/local/share/p_cmip5/model.ini
48
49
50 .. include:: p_cmip5/configuration.rst
51
52
53 Invoking drs_tool with product detection
54 ========================================
55
56 ``drs_tool`` will allocate a product to files in the incoming
57 directory when invoked with the ``--detect-product`` option.
58
59 ``drs_tool list`` will show the product deduced in each dataset_id and
60 list datasets for which the product could not be determined as
61 incomplete. ``drs_tool todo`` and ``drs_tool upgrade`` will only
62 operate on datasets for which the product could be determined.
63
64 In the following example data from the UK Met Office Hadley Centre is
65 processed into the DRS hierarchy.
66
67 .. code-block:: bash
68
69 $ drs_tool list -I ./mohc_holding/ -R ./cmip5 --detect-product
70 ...
71 [INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1
72 [INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2001120106-2002120100>
73 [INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1
74 [INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2002120106-2003120100>
75 [INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1
76 [INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2003120106-2004120100>
77 [INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1
78 [INFO] drslib.p_cmip5: Deducing product for <DRS cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.%.va.2004120106-2005120100>
79 [INFO] drslib.p_cmip5: Product deduced as output1, selected years [112/56] assigned to output1
80 ==============================================================================
81 DRS Tree at .
82 ------------------------------------------------------------------------------
83 cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrLev.r1i1p1 0:0 1120:1371721676416
84 cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1 0:0 224:116004074368
85 ------------------------------------------------------------------------------
86 2 datasets awaiting upgrade
87 ==============================================================================
88
89 # Select the 6hrPlev dataset for publishing and check commands to be issued
90 $ drs_tool todo -I ./mohc_holding/ -R . --detect-product cmip5.%.%.%.%.%.%.6hrPlev
91 ==============================================================================
92 DRS Tree at .
93 ------------------------------------------------------------------------------
94 Publisher Tree cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1 todo for version 20101006
95
96 mv ./mohc_holding/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/psl_20101006/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc
97 ln -s ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/psl_20101006/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/v20101006/psl/psl_6hrPlev_HadGEM2-ES_historical_r1i1p1_194912010600-195012010000.nc
98 ...
99 mv ./mohc_holding/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/va_20101006/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc
100 ln -s ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/files/va_20101006/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc ./cmip5/output1/MOHC/HadGEM2-ES/historical/6hr/atmos/6hrPlev/r1i1p1/v20101006/va/va_6hrPlev_HadGEM2-ES_historical_r1i1p1_200412010600-200512010000.nc
101 ==============================================================================
102
103 # Do the upgrade
104 $ drs_tool upgrade -I ./mohc_holding/ -R . --detect-product cmip5.%.%.%.%.%.%.6hrPlev
105 ...
106
107 # List the results.
108 # Not including --detect-product lists datasets that are incompletely specified
109 $ drs_tool list -I ./mohc_holding -R mohc_dryrun
110 ==============================================================================
111 DRS Tree at mohc_dryrun
112 ------------------------------------------------------------------------------
113 cmip5.output1.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrPlev.r1i1p1.v20101005 224:116004074368
114 ------------------------------------------------------------------------------
115 Incompletely specified incoming datasets
116 ------------------------------------------------------------------------------
117 cmip5.%.MOHC.HadGEM2-ES.historical.6hr.atmos.6hrLev.r1i1p1
118 ==============================================================================
119
120
121 .. _CMOR2: http://www2-pcmdi.llnl.gov/cmor/documentation/