Debian Codegraph @

docs/tutorial.rst

raw
1 .. _drms-tutorial:
2
3 ********
4 Tutorial
5 ********
6
7 This tutorial gives an introduction on how to use the ``drms`` Python library.
8 More detailed information on the different classes and functions can be found in the :ref:`API Reference <reference>`.
9
10 Basic usage
11 ===========
12
13 We start with looking at data series that are available from `JSOC <http://jsoc.stanford.edu/>`__ and perform some basic DRMS queries to obtain keyword data (metadata) and segment file (data) locations.
14 This is essentially what you can do on the `JSOC Lookdata <http://jsoc.stanford.edu/ajax/lookdata.html>`__ website.
15
16 To be able to access the JSOC DRMS from Python, we first need to import the ``drms`` module and create an instance of the `~drms.client.Client` class:
17
18 .. code-block:: python
19
20 >>> import drms
21 >>> client = drms.Client() # doctest: +REMOTE_DATA
22
23 All available data series can be now retrieved by calling :meth:`drms.client.Client.series`.
24 HMI series names start with ``"hmi."``, AIA series names with ``"aia."`` and the names of MDI series with ``"mdi."``.
25
26 The first (optional) parameter of this method takes a regular expression that allows you to filter the result.
27 If for example, you want to obtain a list of HMI series, with a name that start with the string ``"m_"``, you can write:
28
29 .. code-block:: python
30
31 >>> client.series(r'hmi\.m_') # doctest: +REMOTE_DATA
32 ['hmi.M_45s', 'hmi.M_45s_dcon', 'hmi.M_720s', 'hmi.M_720s_dcon', 'hmi.M_720s_dconS', 'hmi.m_720s_mod', 'hmi.m_720s_nrt']
33
34 Keep in mind to escape the dot character (``'.'``), like it is shown in the example above, if you want to include it in your filter string.
35 Also note that series names are handled in a case-insensitive way.
36
37 DRMS records can be selected by creating a query string that contains a series name, followed by one or more fields, which are surrounded by square brackets.
38 Each of those fields corresponds to a specific primekey, that is specified in the series definition.
39 A complete set of primekeys represents a unique identifier for a record in that particular series.
40 For more detailed information on building record set queries, including additional non-primekey fields, see the `JSOC Help <http://jsoc.stanford.edu/ajax/RecordSetHelp.html>`__ page about this.
41
42 With the ``drms`` module you can use :meth:`drms.client.Client.pkeys` to obtain a list of all primekeys of a series:
43
44 .. code-block:: python
45
46 >>> client.pkeys('hmi.m_720s') # doctest: +REMOTE_DATA
47 ['T_REC', 'CAMERA']
48 >>> client.pkeys('hmi.v_sht_modes') # doctest: +REMOTE_DATA
49 ['T_START', 'LMIN', 'LMAX', 'NDT']
50
51 A list of all (regular) keywords can be obtained using :meth:`drms.client.Client.keys`.
52 You can also use :meth:`drms.client.Client.info` to get more detailed information about a series:
53
54 .. code-block:: python
55
56 >>> series_info = client.info('hmi.v_sht_2drls') # doctest: +REMOTE_DATA
57 >>> series_info.segments # doctest: +REMOTE_DATA
58 type units protocol dims note
59 name
60 split string none generic calculated splittings
61 rot string none generic rotation profile
62 err string none generic errors
63 mesh string none generic radial grid points
64 parms string none generic input parameters
65 log string none generic standard output
66
67 All table-like structures, returned by routines in the ``drms`` module, are `Pandas DataFrames <https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html>`__.
68 If you are new to `Pandas <https://pandas.pydata.org/>`__, you should have a look at the introduction to `Pandas Data Structures <https://pandas.pydata.org/pandas-docs/stable/dsintro.html>`__.
69
70 Record set queries, used to obtain keyword data and get the location of data segments, can be performed using :meth:`drms.client.Client.query`.
71 To get, for example, the record time and the mean value for some of the HMI Dopplergrams that were recorded on April 1, 2016, together with the spacecraft's radial velocity in respect to the Sun, you can write:
72
73 .. code-block:: python
74
75 >>> query = client.query('hmi.v_45s[2016.04.01_TAI/1d@6h]',
76 ... key='T_REC, DATAMEAN, OBS_VR') # doctest: +REMOTE_DATA
77 >>> query # doctest: +REMOTE_DATA
78 T_REC DATAMEAN OBS_VR
79 0 2016.04.01_00:00:00_TAI 3313.104980 3309.268006
80 1 2016.04.01_06:00:00_TAI 878.075195 887.864139
81 2 2016.04.01_12:00:00_TAI -2289.062500 -2284.690263
82 3 2016.04.01_18:00:00_TAI 128.609283 137.836168
83
84 JSOC time strings can be converted to a naive `~datetime.datetime` representation using :meth:`drms.utils.to_datetime`:
85
86 .. code-block:: python
87
88 >>> timestamps = drms.to_datetime(query.T_REC) # doctest: +REMOTE_DATA
89 >>> timestamps # doctest: +REMOTE_DATA
90 0 2016-04-01 00:00:00
91 1 2016-04-01 06:00:00
92 2 2016-04-01 12:00:00
93 3 2016-04-01 18:00:00
94 Name: T_REC, dtype: datetime64[us]
95
96 For most of the HMI and MDI data sets, the `TAI <https://en.wikipedia.org/wiki/International_Atomic_Time>`__ time standard is used which, in contrast to `UTC <https://en.wikipedia.org/wiki/Coordinated_Universal_Time>`__, does not make use of any leap seconds.
97 The TAI standard is currently not supported by the Python standard libraries.
98 If you need to convert timestamps between TAI and UTC, you can use `Astropy <https://www.astropy.org/>`__:
99
100 .. code-block:: python
101
102 >>> from astropy.time import Time
103 >>> start_time = Time(timestamps[0], format='datetime', scale='tai') # doctest: +REMOTE_DATA
104 >>> start_time # doctest: +REMOTE_DATA
105 <Time object: scale='tai' format='datetime' value=2016-04-01 00:00:00>
106 >>> start_time.utc # doctest: +REMOTE_DATA
107 <Time object: scale='utc' format='datetime' value=2016-03-31 23:59:24>
108
109 The ``"hmi.v_45s"`` series has a data segment with the name ``"Dopplergram"``, which contains Dopplergrams for each record in the series, that are stored as `FITS <https://fits.gsfc.nasa.gov/>`__ files.
110 The location of the FITS files for the record set query in the example above, can be obtained by using the ``seg`` parameter of :meth:`drms.client.Client.query`:
111
112 .. code-block:: python
113
114 >>> query = client.query('hmi.v_45s[2016.04.01_TAI/1d@6h]', seg='Dopplergram') # doctest: +REMOTE_DATA
115 >>> query # doctest: +REMOTE_DATA
116 Dopplergram
117 0 /SUM58/D803708321/S00008/Dopplergram.fits
118 1 /SUM41/D803708361/S00008/Dopplergram.fits
119 2 /SUM71/D803720859/S00008/Dopplergram.fits
120 3 /SUM70/D803730119/S00008/Dopplergram.fits
121
122 Note that the ``key`` and ``seg`` parameters can also be used together in one :meth:`drms.client.Client.query` call:
123
124 .. code-block:: python
125
126 >>> keys, segments = client.query('hmi.v_45s[2016.04.01_TAI/1d@6h]',
127 ... key='T_REC, DATAMEAN, OBS_VR', seg='Dopplergram') # doctest: +REMOTE_DATA
128
129 The file paths listed above are the storage location on the JSOC server.
130 You can access these files, even if you do not have direct NFS access to the filesystem, by prepending the JSOC URL to segment file path:
131
132 .. code-block:: python
133
134 >>> url = 'http://jsoc.stanford.edu' + segments.Dopplergram[0] # doctest: +REMOTE_DATA
135 >>> url # doctest: +REMOTE_DATA
136 'http://jsoc.stanford.edu/SUM58/D803708321/S00008/Dopplergram.fits'
137
138 >>> from astropy.io import fits
139 >>> data = fits.getdata(url) # doctest: +REMOTE_DATA
140 >>> print(data.shape, data.dtype) # doctest: +REMOTE_DATA
141 (4096, 4096) float32
142
143 Note that FITS files which are accessed in this way, do not contain any keyword data in their headers.
144 This is perfectly fine in many cases, because you can just use :meth:`drms.client.Client.query` to obtain the data of all required keywords.
145 If you need FITS files with headers that contain all the keyword data, you need to submit an export request to JSOC, which is described in the next section.
146
147 Export requests can also be useful, if you want to download more than only one or two files (even without keyword headers), because you can then use :meth:`drms.client.ExportRequest.download`, which takes care of creating URLs, downloading the data and (if necessary) generating suitable local filenames.
148
149 Data export requests
150 --------------------
151
152 Data export requests can be interactively built and submitted on the `JSOC Export Data <http://jsoc.stanford.edu/ajax/exportdata.html>`__ webpage, where you can also find more information about the different export options that are available.
153 Note that a registered email address is required to for submitting export requests.
154 You can register your email address on the `JSOC email registration <http://jsoc.stanford.edu/ajax/register_email.html>`__ webpage.
155
156 It is advisable to have a closer look at the export webpage before submitting export requests using the ``drms`` library.
157 It is also possible to submit an export request on the webpage and then use the Python routines to query the request status and download files.
158
159 .. warning::
160 Please replace the email below with your own registered email.
161
162 .. code-block:: python
163
164 >>> import os
165 >>> email_address = os.environ["JSOC_EMAIL"] # doctest: +REMOTE_DATA
166
167 First, we start again with importing the ``drms`` library and creating a `~drms.client.Client` instance:
168
169 .. code-block:: python
170
171 >>> import drms
172 >>> client = drms.Client(email=email_address) # doctest: +REMOTE_DATA
173
174 In this case we also provide an email address (which needs to be already registered at JSOC).
175
176 We now create a download directory for our downloads, in case it does not exist yet:
177
178 .. code-block:: python
179
180 >>> import os
181 >>> out_dir = 'downloads'
182 >>> if not os.path.exists(out_dir):
183 ... os.mkdir(out_dir)
184
185 Data export requests can be submitted using :meth:`drms.client.Client.export`.
186 The most important parameters of this method, besides the export query string, are the parameters ``method`` and ``protocol``.
187 There are many different export methods and protocols available.
188 In the following examples we confine ourselves to the methods ``url_quick`` and ``url`` and the protocols ``as-is`` and ``fits``.
189
190 url_quick / as-is
191 ^^^^^^^^^^^^^^^^^
192
193 The most direct and quickest way of downloading files is the combination ``url_quick`` / ``as-is``.
194 This (in most cases) does not create an actual export request, where you would have to wait for it being finished, but rather compiles a list of files from your data export query, which can then be directly downloaded.
195 This also means that this kind of export usually has no ``ExportID`` assigned to it.
196 The only time it is treated as a "real" export request (including an ``ExportID`` and some wait time) is, when the requested data segments are not entirely online, and parts of the requested files need to be restored from tape drives.
197
198 As an example, we now create an ``url_quick`` / ``as-is`` export request for the same record set that was used in the previous section.
199 For export requests, the segment name is specified using an additional field in the query string, surrounded by curly braces.
200 Note that :meth:`drms.client.Client.export` performs an ``url_quick`` / ``as-is`` export request by default, so you do not need to explicitly use ``method='url_quick'`` and ``protocol='as-is'`` in this case:
201
202 .. code-block:: python
203
204 >>> export_request = client.export('hmi.v_45s[2016.04.01_TAI/1d@6h]{Dopplergram}') # doctest: +REMOTE_DATA
205 >>> export_request # doctest: +SKIP
206 <ExportRequest: id=None, status=0>
207
208 >>> export_request.data.filename # doctest: +REMOTE_DATA
209 0 /SUM58/D803708321/S00008/Dopplergram.fits
210 1 /SUM41/D803708361/S00008/Dopplergram.fits
211 2 /SUM71/D803720859/S00008/Dopplergram.fits
212 3 /SUM70/D803730119/S00008/Dopplergram.fits
213 Name: filename, dtype: str
214
215 Download URLs can now be generated using the :attr:`drms.client.ExportRequest.urls` attribute:
216
217 .. code-block:: python
218
219 >>> export_request.urls.url[0] # doctest: +REMOTE_DATA
220 'http://jsoc.stanford.edu/SUM58/D803708321/S00008/Dopplergram.fits'
221
222 Files can be downloaded using :meth:`drms.client.ExportRequest.download`.
223 You can (optionally) select which file(s) you want to download, by using the ``index`` parameter of this method.
224 The following, for example, only downloads the first file of the request:
225
226 .. code-block:: python
227
228 >>> res = export_request.download(out_dir, index=0) # doctest: +REMOTE_DATA
229
230 Being a direct ``as-is`` export, there are no keyword data written to any FITS headers.
231 If you need keyword data added to the headers, you have to use the ``fits`` export protocol instead, which is described below.
232
233 url / fits
234 ^^^^^^^^^^
235
236 Using the ``fits`` export protocol, allows you to request FITS files that include all keyword data in their headers.
237 Note that this protocol *does not convert* other file formats into the FITS format.
238 The only purpose of ``protocol='fits'`` is to add keyword data to headers of segment files, that are already stored using the FITS format.
239
240 In contrast to ``url_quick`` / ``as-is`` exports, described in the previous subsection, ``url`` / ``fits`` exports always create a "real" data export request on the server, which needs to be processed before you can download the requested files.
241 For each request you will get an unique ``ExportID``, which can be accessed using the :attr:`drms.client.ExportRequest.id` attribute.
242 In addition you will get an email notification (including the ``ExportID``), which is sent to your registered email address when the requested files are ready for download.
243
244 In the following example, we use the ``hmi.sharp_720s`` series, which contains `Spaceweather HMI Active Region Patches <http://jsoc.stanford.edu/doc/data/hmi/sharp/sharp.htm>`__ (SHARPs), and download some data files from this series.
245
246 First we have a look at the content of the series, by using :meth:`drms.client.Client.info` to get a `~drms.client.SeriesInfo` instance for this particular series:
247
248 .. code-block:: python
249
250 >>> series_info = client.info('hmi.sharp_720s') # doctest: +REMOTE_DATA
251 >>> series_info.note # doctest: +SKIP
252 'Spaceweather HMI Active Region Patch (SHARP): CCD coordinates'
253 >>> series_info.primekeys # doctest: +SKIP
254 ['HARPNUM', 'T_REC']
255
256 This series contains a total of 31 different data segments:
257
258 .. code-block:: python
259
260 >>> len(series_info.segments) # doctest: +SKIP
261 31
262 >>> series_info.segments.index.values # doctest: +SKIP
263 array(['magnetogram', 'bitmap', 'Dopplergram', 'continuum', 'inclination',
264 'azimuth', 'field', 'vlos_mag', 'dop_width', 'eta_0', 'damping',
265 'src_continuum', 'src_grad', 'alpha_mag', 'chisq', 'conv_flag',
266 'info_map', 'confid_map', 'inclination_err', 'azimuth_err',
267 'field_err', 'vlos_err', 'alpha_err', 'field_inclination_err',
268 'field_az_err', 'inclin_azimuth_err', 'field_alpha_err',
269 'inclination_alpha_err', 'azimuth_alpha_err', 'disambig',
270 'conf_disambig'], dtype=object)
271
272 Here, we are only interested in magnetograms and continuum intensity maps:
273
274 .. code-block:: python
275
276 >>> series_info.segments.loc[['continuum', 'magnetogram']] # doctest: +SKIP
277 type units protocol dims note
278 name
279 continuum int DN/s fits VARxVAR continuum intensity
280 magnetogram int Gauss fits VARxVAR magnetogram
281
282 which are stored as FITS files with varying dimensions.
283
284 If we now want to submit an export request for a magnetogram and an intensity map of HARP number 10490, recorded at eight am on December 7th, 2023, we can use the following export query string:
285
286 .. code-block:: python
287
288 >>> query_string = 'hmi.sharp_720s[10490][2023.12.07_08:00:00_TAI]{continuum}' # doctest: +REMOTE_DATA
289
290 In order to obtain FITS files that include keyword data in their headers, we then need to use ``protocol='fits'`` when submitting the request using :meth:`drms.client.Client.export`:
291
292 .. code-block:: python
293
294 >>> export_request = client.export(query_string, method='url', protocol='fits') # doctest: +REMOTE_DATA
295 >>> export_request # doctest: +SKIP
296 <ExportRequest: id=JSOC_..., status=2>
297
298 We now need to wait for the server to prepare the requested files:
299
300 .. code-block:: python
301
302 >>> export_request.wait() # doctest: +SKIP
303 True
304
305 >>> export_request.status # doctest: +SKIP
306 0
307
308 Note that calling :meth:`drms.client.ExportRequest.wait` is optional.
309 It gives you some control over the waiting process, but it can be usually omitted, in which case :meth:`~drms.client.ExportRequest.wait` is called implicitly, when you for example try to download the requested files.
310
311 After the export request is finished, a unique request URL is created for you, which points to the location where all your requested files are stored.
312 You can use the :attr:`drms.client.ExportRequest.request_url` attribute to obtain this URL:
313
314 .. code-block:: python
315
316 >>> export_request.request_url # doctest: +SKIP
317 'http://jsoc.stanford.edu/.../S00000'
318
319 Note that this location is only temporary and that all files will be deleted after a couple of days.
320
321 Downloading the data works exactly like in the previous example, by using :meth:`drms.client.ExportRequest.download`:
322
323 .. code-block:: python
324
325 >>> export_request.download(out_dir) # doctest: +SKIP
326 record url download
327 0 warning=No FITS files were exported. The reque... http://jsoc.stanford.edu/... /...
328
329 If you want to access an existing export request that you have submitted earlier, or if you submitted an export request using the `JSOC Export Data <http://jsoc.stanford.edu/ajax/exportdata.html>`__ webpage.
330 You can use :meth:`drms.client.Client.export_from_id` with the corresponding ``ExportID`` to create an `drms.client.ExportRequest` instance for this particular request.