| 1 |
<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> |
| 2 |
<html> |
| 3 |
<head> |
| 4 |
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> |
| 5 |
<meta name="GENERATOR" content="Mozilla/4.6 [en] (WinNT; I) [Netscape]"> |
| 6 |
<title> Disk Resident Arrays Operations </title> |
| 7 |
<!-- DO NOT EDIT: this is a cleared public file --> |
| 8 |
</head> |
| 9 |
<body text="#000000" bgcolor="#F5F3D3" link="#0000EF" vlink="#55188A" alink="#FF0000" background="magenta"> |
| 10 |
|
| 11 |
<center> |
| 12 |
<h3> |
| 13 |
Disk Resident Arrays</h3></center> |
| 14 |
Disk Resident Arrays (DRA) extend the Global Arrays (GA) programming model |
| 15 |
to disk. The library encapsulates the details of data layout, addressing |
| 16 |
and I/O transfer in disk arrays objects. Disk resident arrays resemble |
| 17 |
global arrays except that they reside on the disk instead of the main memory. |
| 18 |
The main features of this model are: |
| 19 |
<ul> |
| 20 |
<li> |
| 21 |
<font size=-1>Data can be transfered (copied) between disk and global memory.</font></li> |
| 22 |
|
| 23 |
<li> |
| 24 |
<font size=-1>I/O operations have nonblocking interface to allow overlapping |
| 25 |
of I/O with computations.</font></li> |
| 26 |
|
| 27 |
<li> |
| 28 |
<font size=-1>All I/O operations are collective.</font></li> |
| 29 |
|
| 30 |
<li> |
| 31 |
<font size=-1>Either whole or sections of global arrays can be transferred |
| 32 |
between GA memory and the disk.</font></li> |
| 33 |
|
| 34 |
<li> |
| 35 |
<font size=-1>Reshaping and transpose operation are allowed during the |
| 36 |
transfer.</font></li> |
| 37 |
|
| 38 |
<li> |
| 39 |
<font size=-1>Disk resident arrays might be temporary or persistent (saved |
| 40 |
after program termination and available in next runs).</font></li> |
| 41 |
|
| 42 |
<li> |
| 43 |
<font size=-1>Persistent disk arrays can be accessed by any program executing |
| 44 |
on arbitrary number of processors.</font></li> |
| 45 |
|
| 46 |
<li> |
| 47 |
<font size=-1>Distribution and internal data layout on the disk is optimized |
| 48 |
for large data transfer.</font></li> |
| 49 |
|
| 50 |
<li> |
| 51 |
<font size=-1>Hints provided by the user are utilized to guide optimization |
| 52 |
of the library performance for the specific I/O patterns.</font></li> |
| 53 |
</ul> |
| 54 |
DRA can take advantage of a shared file system of a collection of independent |
| 55 |
filesystems accessible from individual computing nodes. |
| 56 |
<br> |
| 57 |
<hr ALIGN=LEFT SIZE=1 NOSHADE WIDTH="100%"> |
| 58 |
<p><a NAME="dra_init"></a> |
| 59 |
<h2> |
| 60 |
dra_init</h2> |
| 61 |
|
| 62 |
<pre>status = dra_init(max_arrays, max_array_size, total_disk_space, max_memory) |
| 63 |
integer max_arrays [input] |
| 64 |
double precision max_array_size [input] |
| 65 |
double precision total_disk_space [input] |
| 66 |
double precision max_memory [input]</pre> |
| 67 |
Initializes Disk Resident Array I/O subsystem. |
| 68 |
<p><i>max_array_size </i>, <i>total_disk_space</i> and <i>max_memory</i> |
| 69 |
are given in bytes. |
| 70 |
<p><i>max_memory</i> specifies how much local memory per processor the |
| 71 |
application is willing to provide to the DRA I/O subsystem for buffering. |
| 72 |
<p>The value of -1 for any of input arguments means: "don't care ", "don't |
| 73 |
know" , or "use defaults ". |
| 74 |
<p> |
| 75 |
<hr> |
| 76 |
<p><a NAME="dra_terminate"></a> |
| 77 |
<h2> |
| 78 |
dra_terminate</h2> |
| 79 |
|
| 80 |
<pre> status=dra_terminate()</pre> |
| 81 |
Close all open disk resident arrays and shut down DRA I/O subsystem. |
| 82 |
<p> |
| 83 |
<hr> |
| 84 |
<p><a NAME="dra_create"></a> |
| 85 |
<h2> |
| 86 |
dra_create</h2> |
| 87 |
|
| 88 |
<pre> status = dra_create(type, dim1, dim2, name, filename, mode,rdim1,rdim2,d_a) |
| 89 |
integer type [input] ! MA type identifier |
| 90 |
integer dim1 [input] |
| 91 |
integer dim2 [input] |
| 92 |
character*(*) name [input] |
| 93 |
character*(*) filename [input] |
| 94 |
integer mode [input] |
| 95 |
integer rdim1 [input] |
| 96 |
integer rdim2 [input] |
| 97 |
integer d_a [output] ! DRA handle</pre> |
| 98 |
Creates a new disk resident array with specified dimensions and data |
| 99 |
<i>type</i> |
| 100 |
(represented by MA type handle). |
| 101 |
<p>The string <i>filename</i> specifies name of an abstract meta-file that |
| 102 |
will store the data on the disk. The meta-file might be implemented as |
| 103 |
multiple files that will contain parts of the disk resident array. The |
| 104 |
component files will have names derived from the string <i>filename</i> |
| 105 |
according to some established scheme(s). Only one DRA object can be stored |
| 106 |
in DRA meta-file identified by <i>filename</i> . |
| 107 |
<p>DRA objects persist on the disk after calling <a href="#dra_close">dra_close</a>. |
| 108 |
<a href="#dra_delete">dra_delete</a> |
| 109 |
should be used instead of <a href="#dra_close">dra_close</a> to delete |
| 110 |
disk array and associated meta-file on the disk. |
| 111 |
<p>String <i>name</i> can be used for more informative (longer)names. |
| 112 |
<p>The data in disk resident array is implicitly initialized to 0 . |
| 113 |
<p>Access permissions (read, write, read& write) are set in <i>mode</i> |
| 114 |
. These are set using defined in the header files DRA.fh (Fortran) and |
| 115 |
DRA.h (C) preprocessor constants: DRA_R, DRA_W, DRA_RW. |
| 116 |
<p>The pair [<i>rdim1</i>, <i>rdim2</i>] specifies dimensions of a typical |
| 117 |
request. The value of -1 for either of them means "unspecified". The layout |
| 118 |
of the data on the disk(s) is determined based on the values of these arguments. |
| 119 |
Performance of the DRA operations depends on the dimensions (section shape) |
| 120 |
of the requests. If data layout is optimized for column-like sections, |
| 121 |
performance of DRA operations for row-like sections might be seriously |
| 122 |
degraded. This is analogous to the effect of wrong loop ordering iyielding |
| 123 |
frequent cache misses in the following example . |
| 124 |
<pre> double precision a(1000, 1000) |
| 125 |
do i = 1, 1000 |
| 126 |
do j = 1, 1000 |
| 127 |
a(i,j) = dfloat(i+j) |
| 128 |
enddo |
| 129 |
enddo |
| 130 |
|
| 131 |
</pre> |
| 132 |
instead of |
| 133 |
<pre> do j = 1, 1000 |
| 134 |
do i = 1, 1000 |
| 135 |
a(i,j) = dfloat(i+j) |
| 136 |
enddo |
| 137 |
enddo</pre> |
| 138 |
|
| 139 |
<hr> |
| 140 |
<p><a NAME="dra_open"></a> |
| 141 |
<h2> |
| 142 |
dra_open</h2> |
| 143 |
|
| 144 |
<pre> status = dra_open(filename, mode, d_a) |
| 145 |
character*(*) filename [input] |
| 146 |
integer mode [input] |
| 147 |
integer d_a [output] ! DRA handle</pre> |
| 148 |
Open and assign DRA handle to disk resident array stored in DRA meta-file |
| 149 |
<i>filename.</i> |
| 150 |
Disk resident arrays that are created with <a href="#dra_create">dra_create</a> |
| 151 |
and saved by calling <a href="#dra_close">dra_close</a> can be later opened |
| 152 |
and accessed by the same or different application. |
| 153 |
<p>Attributes of the disk resident array can be found by calling <a href="#dra_inquire">dra_inquire</a>. |
| 154 |
<p> |
| 155 |
<hr> |
| 156 |
<p><a NAME="dra_write"></a> |
| 157 |
<h2> |
| 158 |
dra_write</h2> |
| 159 |
|
| 160 |
<pre> status = dra_write(g_a, d_a, request) |
| 161 |
integer g_a [input] ! GA handle |
| 162 |
integer d_a [input] ! DRA handle |
| 163 |
integer request [output] ! request id</pre> |
| 164 |
Write asynchronously specified global array to specified disk resident |
| 165 |
array. |
| 166 |
<p>The dimensions and type of arrays represented by handles <i>g_a</i> |
| 167 |
and <i>d_a</i> must match. If dimensions don't match, <a href="#dra_write_section">dra_write_section</a> |
| 168 |
should be used instead. |
| 169 |
<p>The operation is by definition asynchronous but it might be implemented |
| 170 |
as synchronous i.e., it would return only when the I/O is completed. |
| 171 |
<p><i>request</i> can be used to <a href="#dra_probe">dra_probe</a> or |
| 172 |
<a href="#dra_wait">dra_wait</a> |
| 173 |
for completion of the associated operation. |
| 174 |
<p> |
| 175 |
<hr> |
| 176 |
<p><a NAME="dra_write_section"></a> |
| 177 |
<h2> |
| 178 |
dra_write_section</h2> |
| 179 |
|
| 180 |
<pre> status = dra_write_section(transp, g_a, gilo, gihi, gjlo, gjhi, |
| 181 |
d_a, dilo, dihi, djlo, djhi, request) |
| 182 |
logical transp [input] ! transpose operator |
| 183 |
integer g_a [input] ! GA handle |
| 184 |
integer d_a [input] ! DRA handle |
| 185 |
integer gilo [input] |
| 186 |
integer gihi [input] |
| 187 |
integer gjlo [input] |
| 188 |
integer gjhi [input] |
| 189 |
integer dilo [input] |
| 190 |
integer dihi [input] |
| 191 |
integer djlo [input] |
| 192 |
integer djhi [input] |
| 193 |
integer request [output] ! request id</pre> |
| 194 |
Write asynchronously specified global array section to specified disk resident |
| 195 |
array section: |
| 196 |
<pre> OP(g_a[ gilo:gihi, gjlo:gjhi]) --> d_a[ dilo:dihi, djlo:djhi]</pre> |
| 197 |
where <i>OP</i> is the transpose operator (.<i>true./.false</i>.). Return |
| 198 |
error if the two section's types or sizes mismatch. See <a href="#dra_write">dra_write</a> |
| 199 |
specs for discussion of <i>request</i> . |
| 200 |
<p> |
| 201 |
<hr> |
| 202 |
<p><a NAME="dra_read"></a> |
| 203 |
<h2> |
| 204 |
dra_read</h2> |
| 205 |
|
| 206 |
<pre> status = dra_read(g_a, d_a, request) |
| 207 |
integer g_a [input] ! GA handle |
| 208 |
integer d_a [input] ! DRA handle |
| 209 |
integer request [output] ! request id</pre> |
| 210 |
Read asynchronously the specified global array from the specified disk |
| 211 |
resident array. |
| 212 |
<p>Dimensions and type of arrays referred to by handles <i>g_a</i> and |
| 213 |
<i>d_a</i> |
| 214 |
must match. If dimensions don't match, <a href="#dra_read_section">dra_read_section</a> |
| 215 |
could be used instead. |
| 216 |
<p>See <a href="#dra_write">dra_write</a> specs for discussion of <i>request |
| 217 |
</i>. |
| 218 |
<p> |
| 219 |
<hr> |
| 220 |
<p><a NAME="dra_read_section"></a> |
| 221 |
<h2> |
| 222 |
dra_read_section</h2> |
| 223 |
|
| 224 |
<pre> status = dra_read_section(transp, g_a, gilo, gihi, gjlo, gjhi, |
| 225 |
d_a, dilo, dihi, djlo, djhi, request) |
| 226 |
logical transp [input] ! transpose operator |
| 227 |
integer g_a [input] ! GA handle |
| 228 |
integer d_a [input] ! DRA handle |
| 229 |
integer gilo [input] |
| 230 |
integer gihi [input] |
| 231 |
integer gjlo [input] |
| 232 |
integer gjhi [input] |
| 233 |
integer dilo [input] |
| 234 |
integer dihi [input] |
| 235 |
integer djlo [input] |
| 236 |
integer djhi [input] |
| 237 |
integer request [output] ! request id</pre> |
| 238 |
Read asynchronously specified global array section from specified disk |
| 239 |
resident array section: |
| 240 |
<pre> OP(d_a[ dilo:dihi, djlo:djhi]) --> g_a[ gilo:gihi, gjlo:gjhi]</pre> |
| 241 |
where <i>OP </i>is the transpose operator (.<i>true./.false</i>.). |
| 242 |
<p>See <a href="#dra_write">dra_write</a> specs for discussion of <i>request</i> |
| 243 |
. |
| 244 |
<p> |
| 245 |
<hr> |
| 246 |
<p><a NAME="dra_probe"></a> |
| 247 |
<h2> |
| 248 |
dra_probe</h2> |
| 249 |
|
| 250 |
<pre> status = dra_probe(request, compl_status) |
| 251 |
integer request [input] ! request id |
| 252 |
integer compl_status [output] ! completion status</pre> |
| 253 |
Tests for completion of <a href="#dra_write">dra_write</a>/<a href="#dra_read">read</a> |
| 254 |
or <a href="#dra_write_section">dra_write</a>/<a href="#dra_read_section">read_section</a> |
| 255 |
operation which set the value passed in request argument. |
| 256 |
<p><i>compl_status .eq. </i>0 means the operation has been completed. |
| 257 |
<p><i>compl_status .ne. </i>0 means "not done yet ". |
| 258 |
<p> |
| 259 |
<hr> |
| 260 |
<p><a NAME="dra_wait"></a> |
| 261 |
<h2> |
| 262 |
dra_wait</h2> |
| 263 |
|
| 264 |
<pre> status = dra_wait(request) |
| 265 |
integer request [input] ! request id</pre> |
| 266 |
Blocks until completion of <a href="#dra_write">dra_write</a>/<a href="#dra_read">read</a> |
| 267 |
or <a href="#dra_write_section">dra_write</a>/<a href="#dra_read_section">read_section</a> |
| 268 |
operation which set the value passed in <i>request</i> argument. |
| 269 |
<p> |
| 270 |
<hr> |
| 271 |
<p><a NAME="dra_inquire"></a> |
| 272 |
<h2> |
| 273 |
dra_inquire</h2> |
| 274 |
|
| 275 |
<pre> status = dra_inquire(d_a, type, dim1, dim2, name, filename) |
| 276 |
integer d_a [input] ! DRA handle |
| 277 |
integer type [output] |
| 278 |
integer dim1 [output] |
| 279 |
integer dim2 [output] |
| 280 |
character*(*) name [output] |
| 281 |
character*(*) filename [output]</pre> |
| 282 |
Return <i>dimensions</i>, <i>type</i> , <i>name</i> of disk resident array, |
| 283 |
and <i>filename</i> of DRA meta-file associated with <i>d_a</i> handle. |
| 284 |
<p> |
| 285 |
<hr> |
| 286 |
<p><a NAME="dra_delete"></a> |
| 287 |
<h2> |
| 288 |
dra_delete</h2> |
| 289 |
|
| 290 |
<pre> status = dra_delete(d_a) |
| 291 |
integer d_a [input] ! DRA handle</pre> |
| 292 |
Delete a disk resident array associated with <i>d_a</i> handle. Invalidate |
| 293 |
handle. The corresponding DRA meta-file is destroyed. |
| 294 |
<p> |
| 295 |
<hr> |
| 296 |
<p><a NAME="dra_close"></a> |
| 297 |
<h2> |
| 298 |
dra_close</h2> |
| 299 |
|
| 300 |
<pre> status = dra_close(d_a) |
| 301 |
integer d_a [input] ! DRA handle</pre> |
| 302 |
Close DRA meta-file associated with <i>d_a</i> handle and deallocate data |
| 303 |
structures corresponding to this disk array. Invalidate <i>d_a</i> handle. |
| 304 |
The array on the disk is persistent. |
| 305 |
<p> |
| 306 |
<hr> |
| 307 |
<p><a NAME="dra_flick"></a> |
| 308 |
<h2> |
| 309 |
dra_flick</h2> |
| 310 |
|
| 311 |
<pre> subroutine dra_flick()</pre> |
| 312 |
Returns control to DRA for a VERY short time to improve progress of pending |
| 313 |
asynchronous operations. |
| 314 |
</body> |
| 315 |
</html> |