Debian Codegraph @

doc/intro.rst

raw
1 ============
2 Introduction
3 ============
4
5 Use Cases
6 =========
7
8 The design of the library has been driven by the following requirements:
9
10 1. Deduce the DRS directory structure a DRS-compliant filename.
11 2. Validate filenames and directory paths against the DRS.
12 3. Convert the CMIP3 directory structure into a DRS-compliant form.
13 4. Manage multiple versions of DRS publication-level datasets on the filesystem.
14 5. Detect the CMIP5 product DRS component during data publication.
15
16
17 Installation
18 ============
19
20 drslib is written in Python_. It requires one of the installation
21 tools setuptools_ or pip_. Other requirements are downloaded
22 automatically during installation. Download and install with:
23
24 .. code-block:: bash
25
26 # Using pip
27 $ pip install drslib
28
29 # OR using setuptools
30 $ easy_install drslib
31
32 If you are upgrading drslib you will need to add the "-U" option to
33 either pip or easy_install.
34
35 .. pull-quote::
36
37 **Note:** To install onto an ESG datanode you can use use the
38 ``easy_install`` in ``/usr/local/uv-cdat/bin``. This will install
39 ``drslib`` into ``/usr/local/uv-cdat/bin/python`` and install
40 ``drs_tool`` as ``/usr/local/uv-cdat/bin/drs_tool``.
41
42
43
44
45 Installing for development
46 --------------------------
47
48 The source of drslib is available on github from the ESGF `github repo`_.
49
50 Once you have the code you must activate the distribution
51 in `develop` mode. To do this execute the following using a python
52 interpreter with setuptools_ installed:
53
54 .. code-block:: bash
55
56 $ python setup.py develop
57
58
59 __ esgf-drslib.gitweb_
60
61
62 Configuration
63 =============
64
65 ``drslib`` will use CMIP5 MIP tables to deduce realm and other
66 metadata. MIP tables are not required for other projects as selected
67 with the ``-s`` scheme switch. You can point to your MIP table location
68 using the environment variable ``MIP_TABLE_PATH``. All tables should
69 be named ``CMIP5_*``.
70
71 Alternatively you can use metaconfig_ to configure the location of
72 your MIP tables. Create the following in your ``$HOME/.metaconfig.conf`` file:
73
74 .. code-block:: ini
75
76 [metaconfig]
77 configs = drslib
78
79 [drslib:tables]
80 path = /path/to/mip/tables
81 model_table = /path/to/model_table
82
83 .. _metaconfig: http://pypi.python.org/pypi/metaconfig
84
85 The option ``model_table`` is optional and if specified should point
86 to the "CMIP5 Modelling Group" spreadsheet in CSV format. This table
87 is used to map model names to institute names. The drslib package is
88 distributed with a recent version of this file.
89
90 Default DRS attributes used by the ``drs_tool`` command can be set in
91 the ``drs`` section:
92
93 .. code-block:: ini
94
95 [drslib:drs]
96 root = /path/to/drs/activity
97 institute = MOHC
98 model = HadCM3
99
100 It is usually convenient to set at least ``root`` and ``activity`` in
101 the configuration file.
102
103 Overriding DRS vocabularies
104 ----------------------
105
106 Drslib ships with reasonable defaults for the DRS vocabularies,
107 however sometimes you will want to override or extend the defaults
108 supplied. At this time drslib supports extending the vocabularies of
109 institutes, models and experiments.
110
111 To define all CMIP5 experiments including individual decadal
112 experiments you can define the ``drslib:vocabularies`` section as follows:
113
114 .. code-block:: ini
115
116 [drslib:vocabularies]
117 experiments =
118 1pctto2x 2xco2 pdcntrl sresa1b 1pctto4x amip picntrl sresa2 20c3m commit
119 slabcntl sresb1
120 decadal1960 decadal1961 decadal1962 decadal1963 decadal1964 decadal1965
121 decadal1966 decadal1967 decadal1968 decadal1969 decadal1970 decadal1971
122 decadal1972 decadal1973 decadal1974 decadal1975 decadal1976 decadal1977
123 decadal1978 decadal1979 decadal1980 decadal1981 decadal1982 decadal1983
124 decadal1984 decadal1985 decadal1986 decadal1987 decadal1988 decadal1989
125 decadal1990 decadal1991 decadal1992 decadal1993 decadal1994 decadal1995
126 decadal1996 decadal1997 decadal1998 decadal1999 decadal2000 decadal2001
127 decadal2002 decadal2003 decadal2004 decadal2005 decadal2006 decadal2007
128 decadal2008 decadal2009
129
130 Institutes and models can be defined with the ``institutes`` option.
131 This is interpreted as a newline separated list of lines, each line
132 being the institute name a colon then space separated list of models.
133
134 .. code-block:: ini
135
136 institutes =
137 NOAA-GFDL:GFDL-ESM2G
138 MOHC:HadGEM2-ES HadCM3 HadGEM2-CC
139
140
141
142 Logging
143 =======
144
145 ``drslib`` uses Python's standard logging infrastructure to give
146 details of it's operation. Messages are sent to loggers under the
147 ``drslib`` logger. You can configure logging via metaconfig by
148 pointing to a separate logging configuration file:
149
150 .. code-block:: ini
151
152 [metaconfig]
153 configs = drslib
154 logging = /path/to/logging.conf
155
156 The format of ``logging.conf`` should conform to the Python logging
157 `configuration file format`__. An example logging configuration is
158 given below which will log product detection decisions separately from
159 general drslib warnings:
160
161 .. code-block:: ini
162
163 #
164 # Basic logging configuration for drs_tool
165 #
166 # This configuration prints product detection decisions to STDERR and logs
167 # warnings to ./drs_tool.log
168 #
169
170 [loggers]
171 keys=root,drslib,p_cmip5
172
173 [handlers]
174 keys=drslib_h,p_cmip5_h
175
176 [formatters]
177 keys=f1,f2
178
179 #---------------------------------------------------------------------------
180 # Loggers
181
182 # No catch-all logging
183 [logger_root]
184 handlers=
185 level=NOTSET
186
187 [logger_drslib]
188 qualname=drslib
189 handlers=drslib_h
190
191 [logger_p_cmip5]
192 qualname=drslib.p_cmip5
193 handlers=p_cmip5_h
194 propagate=0
195
196 #---------------------------------------------------------------------------
197 # Handlers & Formatters
198
199 [handler_drslib_h]
200 class=FileHandler
201 args=('./drs_tool.log', )
202 formatter=f1
203 level=INFO
204
205 [handler_p_cmip5_h]
206 class=StreamHandler
207 args=(sys.stderr, )
208 formatter=f2
209 level=INFO
210
211 [formatter_f1]
212 format=%(asctime)s [%(levelname)s] %(name)s: %(message)s
213 datefmt=
214
215 [formatter_f2]
216 format=[%(levelname)s] %(name)s: %(message)s
217
218
219 __ http://docs.python.org/library/logging.html#configuration-file-format
220
221
222
223 Testing
224 =======
225
226 drslib ships with a test suite compliant with nose_. The suite can be
227 run in various ways. The test suite uses the extension NoseXUnit_ to
228 produce XML reports of the test results. NoseXUnit will be
229 automatically installed if you run the tests via ``setup.py``::
230
231 $ python setup.py test
232
233 Or if the depencencies are satisfied you can run all tests with::
234
235 $ nosetests
236
237
238 Reporting Bugs
239 ==============
240
241 Please report bugs to the `github repo`_.
242
243 .. _CMIP5: http://cmip-pcmdi.llnl.gov/cmip5/
244 .. _DRS: http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf
245 .. _nose: http://somethingaboutorange.com/mrl/projects/nose
246 .. _setuptools: http://pypi.python.org/pypi/setuptools
247 .. _pip: http://pypi.python.org/pypi/pip
248 .. _NoseXUnit: http://pypi.python.org/pypi/NoseXUnit
249 .. _esgf-drslib.git: git://esgf.org/esgf-drslib.git
250 .. _esgf-drslib.gitweb: http://esgf.org/gitweb/?p=esgf-drslib.git;a=summary
251 .. _CEDA: http://www.ceda.ac.uk
252 .. _`Stephen Pascoe`: mailto:[email protected]
253 .. _Python: http://www.python.org
254 .. _`esgf.org bugzilla`: http://esgf.org/bugzilla/enter_bug.cgi?product=drslib
255 .. _`github repo`: http://github.com/ESGF/esgf-drslib