| 1 |
The p_cmip5 module will decide whether data should be assigned to DRS |
| 2 |
product=output1 or output2. Background and the algorithm steps are |
| 3 |
given in `requested_subset_decision_tree_v0_5.pdf`__. |
| 4 |
|
| 5 |
__ doc/requested_subset_decision_tree_v0_5.pdf |
| 6 |
|
| 7 |
Set-up |
| 8 |
...... |
| 9 |
|
| 10 |
There is a set-up step performed by the code in p_cmip5/init.py, using |
| 11 |
the "init(shelve_dir)" function. This modules takes information from |
| 12 |
the spreadsheets CMIP5_archive_size_template.xls and |
| 13 |
standard_output_17Sep2010_mod.xls and stores it in python shelves used |
| 14 |
by the main code. The shelves are placed in "shelve_dir". |
| 15 |
|
| 16 |
The cmip5_product class |
| 17 |
....................... |
| 18 |
|
| 19 |
The :mod:`drslib.p_cmip5.product` module provides a ``cmip5_product`` class with a |
| 20 |
"find_product" method. At instantiation, the class picks up |
| 21 |
configuration tables The configuration file (sample in |
| 22 |
ini/sample_1.ini, ini/smaple_2.ini) needs to contain information about |
| 23 |
each model. This information is used for a small number of cases and |
| 24 |
the format is not yet stable: |
| 25 |
|
| 26 |
.. code-block:: python |
| 27 |
|
| 28 |
class cmip5_product: |
| 29 |
|
| 30 |
def __init__(self,mip_table_shelve='sh/standard_output_mip', |
| 31 |
template='sh/template', |
| 32 |
stdo='sh/standard_output', |
| 33 |
config='ini/sample_1.ini', |
| 34 |
override_product_change_warning=False, |
| 35 |
policy_opt1='all_rel',not_ok_excpt=False): |
| 36 |
|
| 37 |
Optional arguments |
| 38 |
'''''''''''''''''' |
| 39 |
:mip_table_shelve: shelve containing information about MIP tables; |
| 40 |
:template: shelve containing information mapping experiment names to labels used in the standard_output spreadsheet; |
| 41 |
:stdo: shelve containing information from the standard_output spreadsheet; |
| 42 |
:config: configuration file; |
| 43 |
:override_product_change_warning: in some cases it is possible that adding new data to previously published data can change the product |
| 44 |
designation of the previously published data. The default behaviour is to givw an error return at this point. This can be over-ridden, |
| 45 |
the code will then provide the product of the file to be added and lists of changes that need to be made to previously published data. |
| 46 |
It is not expected, however, that the ESG publisher will support such updates: the user should instead compile a new set of files to |
| 47 |
submit using all the previously published files and the new files. |
| 48 |
:policy_opt1: this controls two options for treatment of data blocks in which time slices are requested. The default is that, if the time |
| 49 |
slices are specified using relative dates (e.g. relative to start of experiment) and the number of years submitted is les than the number |
| 50 |
of years requested, all years submitted will be assigned to output1 without examining the dates. There is an option (depricated) to extend |
| 51 |
this catch-all approach to time slices specified with absolute dates. |
| 52 |
:not_ok_excpt: if True, raise an exception if product can not be designated as output1 or output2. |
| 53 |
|
| 54 |
The find_product method |
| 55 |
....................... |
| 56 |
|
| 57 |
The principal interface to the cmip5_product class is the find_product method: |
| 58 |
|
| 59 |
.. code-block:: python |
| 60 |
|
| 61 |
def find_product(self,var,table,expt,model,path,startyear=None,endyear=None,verbose=False, |
| 62 |
path_output1=None, path_output2=None,selective_ads_scan=True): |
| 63 |
|
| 64 |
Required arguments |
| 65 |
'''''''''''''''''' |
| 66 |
:var: DRS variable name |
| 67 |
:table: MIP table |
| 68 |
:expt: DRS experiment name |
| 69 |
:model: Model name |
| 70 |
:path: Path to directory containing all the files of one atomic dataset. |
| 71 |
|
| 72 |
Optional arguments |
| 73 |
'''''''''''''''''' |
| 74 |
:startyear: first year of the file to be assessed |
| 75 |
:endyear: last year of file to be assessed (not currently used) |
| 76 |
:verbose: if True, provide additional comments to logger |
| 77 |
:path_output1: path to last published output1 data for this atomic dataset, if new data is to be considered as an addition; |
| 78 |
:path_output2: path to last published output1 data for this atomic dataset, if new data is to be considered as an addition; |
| 79 |
:selective_ads_scan: when scanning the atomic dataset directory, look only at files matching the variable, table, experiment and model. |
| 80 |
The False option is provided to facilitate testing using dummy data files, and should not otherwise be used. |
| 81 |
:return: if "not_ok_excp=True", the method will return True if it can assign output1 or output2, otherwise an exception will be raised. |
| 82 |
if "not_ok_excpt=False", the method will return False when it cannot assign output1 or output2, with a message in pc.reason (where pc |
| 83 |
is an instance of the cmip5_product class). |
| 84 |
|
| 85 |
Attributes containing results |
| 86 |
''''''''''''''''''''''''''''' |
| 87 |
After successful completion of the find_product method, the followin attibutes of the instance contain information: |
| 88 |
|
| 89 |
:product: the product to be assigned to the file; |
| 90 |
:reason: a short summary of the reasons for the assigment; |
| 91 |
:rc: a return code, 'OKnnn' if successful, 'ERRnnn' if not. |
| 92 |
|
| 93 |
|
| 94 |
Usage |
| 95 |
..... |
| 96 |
|
| 97 |
The following code fragment illustrates usage of the module:: |
| 98 |
|
| 99 |
## import module |
| 100 |
import p_cmip5_v5 as p |
| 101 |
|
| 102 |
## create and instance of the cmip5_product class, specifying a configuration file |
| 103 |
## |
| 104 |
pc2 = p.cmip5_product( config='ini/sample_2.ini') |
| 105 |
|
| 106 |
## test a file: using the variable, mip table, experiment id, model and specifying the path of the atomic dataset directory containing all |
| 107 |
## the submitted files. |
| 108 |
## In some cases the decision as to which product the file belongs in will depend on the contents of this directory. |
| 109 |
## verbose=True results in additional messages being printed to standard out. |
| 110 |
|
| 111 |
if pc2.find_product( var, mip, expt,model,path,startyear=startyear, verbose=verbose): |
| 112 |
print 'product is: ', pc2.product |
| 113 |
|
| 114 |
## A True return means the method has identified the product |
| 115 |
else: |
| 116 |
## A False return means the product could not be identified |
| 117 |
print 'Dont know what to do with this data:: ',pc2.reason |
| 118 |
|
| 119 |
|
| 120 |
Testing the p_cmip5 module: test_p_cmip5.py |
| 121 |
........................................... |
| 122 |
|
| 123 |
The test_p_cmip5 module can be used to test the p_cmip5 module. E.g. run the following from the directory containing the "test" subdirectory: |
| 124 |
|
| 125 |
.. code-block:: bash |
| 126 |
|
| 127 |
$ nosetests --tests=test/test_p_cmip5.py |