| 1 |
Introduction |
| 2 |
============ |
| 3 |
|
| 4 |
ARMCI has support for aggregating messages via its aggregate non-blocking |
| 5 |
handles. Any ARMCI non-blocking handle can be explicitly made as an aggregate |
| 6 |
handle and all subsequent messages that use the same handle are aggregated and |
| 7 |
sent as a single message. The details of the implementation can be found in |
| 8 |
the cluster 2003 paper from the ARMCI publications website. |
| 9 |
|
| 10 |
Description |
| 11 |
============ |
| 12 |
|
| 13 |
The implicit aggregation of data transfers is implemented using the |
| 14 |
generalized I/O vector operations available in ARMCI. This interface enables |
| 15 |
the representation of a data transfer as a combination of multiple sets of |
| 16 |
equally sized contiguous data segments. When the first call involving |
| 17 |
aggregate nonblocking handle is executed, the library starts building a vector |
| 18 |
descriptor stored in one of the preallocated internal buffers. The actual data |
| 19 |
transfer takes place when the user calls wait operation or the buffer storing |
| 20 |
the vector descriptor fills up. |
| 21 |
|
| 22 |
Function Definitions |
| 23 |
==================== |
| 24 |
|
| 25 |
ARMCI_SET_AGGREGATE_HANDLE (armci_hdl_t* handle) |
| 26 |
------------------------------------------------ |
| 27 |
|
| 28 |
handle - Pointer to a desciptor associated with a particular non-blocking |
| 29 |
transfer. |
| 30 |
|
| 31 |
PURPOSE: Mark a handle as aggregate. This will allow ARMCI to combine |
| 32 |
nonblocking operations that use that particular handle and process them as a |
| 33 |
single operation. In the initial implementation only contiguous puts or gets |
| 34 |
could use aggregate handle. Specifying the same handle for a mix of put anmd |
| 35 |
get calls is not allowed i.e., only multiple put or only multiple get calls |
| 36 |
can use the same handle. |
| 37 |
|
| 38 |
ARMCI_UNSET_AGGREGATE_HANDLE (armci_hdl_t* handle) |
| 39 |
-------------------------------------------------- |
| 40 |
|
| 41 |
handle - Pointer to a desciptor associated with a particular non-blocking |
| 42 |
transfer. |
| 43 |
|
| 44 |
PURPOSE: Clears a handle that has been marked as aggregate. |
| 45 |
|
| 46 |
Sample Programs |
| 47 |
=============== |
| 48 |
|
| 49 |
1. Sparse matrix-vector multiplication |
| 50 |
|
| 51 |
Sparse matrix-vector multiplication is one of the common computational |
| 52 |
kernels, for example in solving linear systems using conjugate gradient |
| 53 |
method. It is described as Ax = b, where A is an nxn nonsingular sparse |
| 54 |
matrix, b is an n-dimensional vector, and x is an n-dimensional vector of |
| 55 |
unknowns. In this benchmark, one of the sparse matrices (Figure 8a) from |
| 56 |
the Harwell-Boeing collection is used [9] to test the matrix-vector |
| 57 |
multiplication. The sparse matrix size is 41092 and has 1683902 (~.1%) |
| 58 |
nonzero elements. |
| 59 |
|
| 60 |
The experiments were conducted on the Linux cluster (dual node, 1GHz |
| 61 |
Itanium-2, Myrinet-2000 interconnect) at PNNL. Sparse matrix-vector |
| 62 |
multiplication was done with aggregation enabled and disabled. The sparse |
| 63 |
matrix and the vector are distributed among processors. Instead of |
| 64 |
gathering the entire vector, each process caches the vector elements |
| 65 |
corresponding to the non-zero element columns of its locally owned part of |
| 66 |
the matrix. When aggregation is enabled, all the get calls corresponding to |
| 67 |
a single processor are aggregated into a single request, thus reducing the |
| 68 |
overall latency and improving the data transfer rate. |
| 69 |
|
| 70 |
This sample program is in the sparse_matvecmul directory. There is an input |
| 71 |
required for this program. The method to obtain the input is given in the |
| 72 |
sparse_matvecmul directory in a README file |