Debian Codegraph @

doc/command.rst

raw
1 ``drs_tool`` command-line interface
2 ===================================
3
4 The ``drs_tool`` command can be used to invoke functions from the
5 :mod:`drslib.drs_tree` API. The tool is designed to help ESGF
6 datanode managers to
7
8 1. Prepare incoming data for publication, placing files in the DRS
9 directory structure.
10 2. Manage multiple versions of publication-level datasets to minimise
11 disk usage.
12 3. Deduce the CMIP5 product into which data should be published
13 according to the rules definied in the CMIP5 experiment definition.
14
15
16 Concepts
17 --------
18
19 DRS root
20 The root of the published DRS directory tree. This is synonimous
21 with the directory representing the DRS activity component.
22
23 Incoming directory
24 The directory scanned for data files to be added to
25 the DRS directory tree. This directory is scanned recursively to
26 find NetCDF files in the DRS filename encoding. It defaults to
27 ``<drs-root>/output`` for compatibility with the output structure of
28 CMOR2. However, the directory names under the incoming directory
29 are not taken into account when deducing the DRS components of
30 files.
31
32 DRS pattern
33 Each invocation of ``drs_tool`` is given a set of DRS component
34 values that define the portion of the DRS space on which it acts.
35
36
37 Usage
38 -----
39
40 Usage: drs_tool [command] [options] [drs-pattern]
41
42 command:
43
44 ======= ====================================================================
45 list list publication-level datasets
46 todo show file operations pending for the next version
47 upgrade make changes to the selected datasets to upgrade to the next version
48 mapfile make a mapfile of the selected dataset
49 history list all versions of the selected dataset
50 init initialise CMIP5 product detection data
51 ======= ====================================================================
52
53 drs-pattern:
54 A dataset identifier in '.'-separated notation. Use the '%' to indicate unknown DRS components, e.g. ``cmip5.%.MPI-M`` will match any product.
55
56
57 Options:
58 -h, --help show this help message and exit
59 -R ROOT, --root=ROOT Root directory of the DRS tree
60 -I INCOMING, --incoming=INCOMING
61 Incoming directory for DRS files. Defaults to
62 <root>/output
63 -a ACTIVITY, --activity=ACTIVITY
64 Set DRS attribute activity for dataset discovery
65 -p PRODUCT, --product=PRODUCT
66 Set DRS attribute product for dataset discovery
67 -i INSTITUTE, --institute=INSTITUTE
68 Set DRS attribute institute for dataset discovery
69 -m MODEL, --model=MODEL
70 Set DRS attribute model for dataset discovery
71 -e EXPERIMENT, --experiment=EXPERIMENT
72 Set DRS attribute experiment for dataset discovery
73 -f FREQUENCY, --frequency=FREQUENCY
74 Set DRS attribute frequency for dataset discovery
75 -r REALM, --realm=REALM
76 Set DRS attribute realm for dataset discovery
77 -v VERSION, --version=VERSION
78 Force version upgrades to this version
79 -P FILE, --profile=FILE
80 Profile the script exectuion into FILE
81 --detect-product Automatically detect the DRS product of incoming data
82
83 -j JSON_DRS, --json-drs=JSON_DRS
84 Use the JSON output from the `ceda-cc` quality control tool
85 to define the incoming set of files and their associated DRS terms.
86
87 An Example
88 ----------
89
90 This example walks through using ``drs_tool`` to prepare datasets for
91 publication. It uses some dummy data files based on test runs of the
92 Met Office Hadley Centre HadGEM2-ES model. You can repeat the example
93 from within the drslib source distribution directory.
94
95 First generate a dummy set of incoming data files. In this example we
96 assume all incoming files are in a single flat directory, however
97 ``drs_tool`` will recursively travers the incoming directory to find
98 all DRS-compliant NetCDF files.
99
100 .. code-block:: bash
101
102 $ mkdir mohc_eg
103 $ python test/gen_drs.py test/mohc_delivery.ls mohc_eg/incoming
104 $ ls mohc_eg/incoming/ | head
105 baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
106 baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
107 baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
108 baresoilFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc
109 c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
110 c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
111 c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
112 c3PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc
113 c4PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc
114 c4PftFrac_Lmon_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
115 $ ls mohc_eg/incoming/ | wc -l
116 494
117
118 We now have about 500 dummy NetCDF files in ``mohc_eg/incoming``. You
119 can ask ``drs_tool`` to list which publication-level datasets these
120 files would be put in using the ``drs_tool list`` subcommand. For
121 this to work ``drs_tool`` requires 2 DRS components not decidable from
122 the filenames: activity and product [*]_. ``drs_tool list`` will list
123 all publication-level datasets with the criteria given, including
124 those that would be created by processing the incoming directory.
125
126 .. [*] later versions of drslib will be able to decide the product
127 component from other components and by inspecting the NetCDF.
128
129 .. code-block:: bash
130
131 $ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
132 ==============================================================================
133 DRS Tree at mohc_eg/
134 ------------------------------------------------------------------------------
135 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
136 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1 *
137 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
138 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
139 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
140 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1 *
141 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 *
142 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
143 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1 *
144 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1 *
145 ==============================================================================
146
147 The asterisk against each dataset_id indicates there are files in the
148 incoming directory to add to the dataset. In this case all datasets
149 are empty.
150
151 We can restrict ``drs_tool list`` output by using a dataset_id
152 wildcard. For instance to select only datasets in the ``atmos`` realm:
153
154 .. code-block:: bash
155
156 $ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1.%.%.%.%.atmos
157 ==============================================================================
158 DRS Tree at mohc_eg/
159 ------------------------------------------------------------------------------
160 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
161 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
162 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
163 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
164 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
165 ==============================================================================
166
167 The same effect can be achieved with individual component options:
168
169 .. code-block:: bash
170
171 $ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=atmos
172 ==============================================================================
173 DRS Tree at mohc_eg/
174 ------------------------------------------------------------------------------
175 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
176 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
177 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
178 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
179 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
180 ==============================================================================
181
182 Now we will focus on a single dataset in the ``aerosol`` realm and
183 show how to move files into the DRS directory structure ready for
184 publication. We can check what filesystem commands will be done using
185 the ``drs_tool todo`` subcommand.
186
187 .. code-block:: bash
188
189 $ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol
190 ==============================================================================
191 DRS Tree at mohc_eg/
192 ------------------------------------------------------------------------------
193 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 *
194 ==============================================================================
195 $ drs_tool todo -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol | head
196 ==============================================================================
197 DRS Tree at mohc_eg/
198 ------------------------------------------------------------------------------
199 Publisher Tree cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 todo for version 20100927
200 ------------------------------------------------------------------------------
201 mv mohc_eg/incoming/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/emidust_20100927/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
202 ln -s /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/emidust_20100927/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/emidust/emidust_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc
203 mv mohc_eg/incoming/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/reffclwtop_20100927/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
204 ln -s /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/reffclwtop_20100927/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/reffclwtop/reffclwtop_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
205 mv mohc_eg/incoming/dryso2_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc /home/spascoe/git/esgf-drslib/mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/files/dryso2_20100927/dryso2_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc
206
207 You can see here that drslib will move files into datestamped
208 directories under ``<dataset-dir>/files`` then symbolically link them
209 into the DRS directory structure. To do the actual moving use
210 ``drs_tool upgrade``. Then use ``drs_tool list`` to view the result.
211
212 .. code-block:: bash
213
214 $ drs_tool upgrade -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol
215 ==============================================================================
216 DRS Tree at mohc_eg/
217 ------------------------------------------------------------------------------
218 Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1 to version 20100927 ... done
219 ==============================================================================
220 $ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
221 ==============================================================================
222 DRS Tree at mohc_eg/
223 ------------------------------------------------------------------------------
224 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
225 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1 *
226 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 *
227 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 *
228 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
229 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1 *
230 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1.v20100927 -
231 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
232 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1 *
233 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1 *
234 ==============================================================================
235
236 Using ``drs_tool``'s criteria options you can upgrade multiple datasets in one command:
237
238 .. code-block:: bash
239
240 $ drs_tool upgrade -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=atmos --frequency=6hr
241 ==============================================================================
242 DRS Tree at mohc_eg/
243 ------------------------------------------------------------------------------
244 Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1 to version 20100927 ... done
245 Upgrading cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1 to version 20100927 ... done
246 ==============================================================================
247 $ drs_tool list -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1
248 ==============================================================================
249 DRS Tree at mohc_eg/
250 ------------------------------------------------------------------------------
251 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.atmos.3hr.r1i1p1 *
252 cmip5.output1.MOHC.HadGEM2-ES.rcp45.3hr.land.3hr.r1i1p1 *
253 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrLev.r1i1p1.v20100927 -
254 cmip5.output1.MOHC.HadGEM2-ES.rcp45.6hr.atmos.6hrPlev.r1i1p1.v20100927 -
255 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.atmos.day.r1i1p1 *
256 cmip5.output1.MOHC.HadGEM2-ES.rcp45.day.land.day.r1i1p1 *
257 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1.v20100927 -
258 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.atmos.Amon.r1i1p1 *
259 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.land.Lmon.r1i1p1 *
260 cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.landIce.LImon.r1i1p1 *
261 ==============================================================================
262
263 Finally you need to send publish the datasets with ``esgpublish``. To make this easier ``drs_tool`` can create a mapfile of a dataset:
264
265 .. code-block:: bash
266
267 $ drs_tool mapfile -R mohc_eg/ -I mohc_eg/incoming/ cmip5.output1 --realm=aerosol >rcp45.mon.aerosol.map
268 $ head rcp45.mon.aerosol.map
269 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
270 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
271 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
272 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadsoa/loadsoa_aero_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
273 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_201512-204011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
274 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_206512-209011.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
275 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
276 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/loadbc/loadbc_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
277 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/wetbc/wetbc_aero_HadGEM2-ES_rcp45_r1i1p1_209012-209911.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
278 mohc_eg/output1/MOHC/HadGEM2-ES/rcp45/mon/aerosol/aero/r1i1p1/v20100927/wetbc/wetbc_aero_HadGEM2-ES_rcp45_r1i1p1_204012-206511.nc | cmip5.output1.MOHC.HadGEM2-ES.rcp45.mon.aerosol.aero.r1i1p1
279
280
281
282 Some further examples of usage can be found in the doctest file
283 ``test/test_command.txt``.