| 1 |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> |
| 2 |
|
| 3 |
<!--Converted with LaTeX2HTML 2019.2 (Released June 5, 2019) --> |
| 4 |
<HTML lang="EN"> |
| 5 |
<HEAD> |
| 6 |
<TITLE>3.3 Parallelization levels</TITLE> |
| 7 |
<META NAME="description" CONTENT="3.3 Parallelization levels"> |
| 8 |
<META NAME="keywords" CONTENT="user_guide"> |
| 9 |
<META NAME="resource-type" CONTENT="document"> |
| 10 |
<META NAME="distribution" CONTENT="global"> |
| 11 |
|
| 12 |
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8"> |
| 13 |
<META NAME="viewport" CONTENT="width=device-width, initial-scale=1.0"> |
| 14 |
<META NAME="Generator" CONTENT="LaTeX2HTML v2019.2"> |
| 15 |
|
| 16 |
<LINK REL="STYLESHEET" HREF="user_guide.css"> |
| 17 |
|
| 18 |
<LINK REL="next" HREF="node21.html"> |
| 19 |
<LINK REL="previous" HREF="node19.html"> |
| 20 |
<LINK REL="next" HREF="node21.html"> |
| 21 |
</HEAD> |
| 22 |
|
| 23 |
<BODY > |
| 24 |
<!--Navigation Panel--> |
| 25 |
<A |
| 26 |
HREF="node21.html"> |
| 27 |
<IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A> |
| 28 |
<A |
| 29 |
HREF="node17.html"> |
| 30 |
<IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A> |
| 31 |
<A |
| 32 |
HREF="node19.html"> |
| 33 |
<IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A> |
| 34 |
<A ID="tex2html225" |
| 35 |
HREF="node1.html"> |
| 36 |
<IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A> |
| 37 |
<BR> |
| 38 |
<B> Next:</B> <A |
| 39 |
HREF="node21.html">3.4 Tricks and problems</A> |
| 40 |
<B> Up:</B> <A |
| 41 |
HREF="node17.html">3 Parallelism</A> |
| 42 |
<B> Previous:</B> <A |
| 43 |
HREF="node19.html">3.2 Running on parallel</A> |
| 44 |
<B> <A ID="tex2html226" |
| 45 |
HREF="node1.html">Contents</A></B> |
| 46 |
<BR> |
| 47 |
<BR> |
| 48 |
<!--End of Navigation Panel--> |
| 49 |
<!--Table of Child-Links--> |
| 50 |
<A ID="CHILD_LINKS"><STRONG>Subsections</STRONG></A> |
| 51 |
|
| 52 |
<UL> |
| 53 |
<LI><UL> |
| 54 |
<LI><A ID="tex2html227" |
| 55 |
HREF="node20.html#SECTION00043010000000000000">3.3.0.1 About communications</A> |
| 56 |
<LI><A ID="tex2html228" |
| 57 |
HREF="node20.html#SECTION00043020000000000000">3.3.0.2 Choosing parameters</A> |
| 58 |
<LI><A ID="tex2html229" |
| 59 |
HREF="node20.html#SECTION00043030000000000000">3.3.0.3 Massively parallel calculations</A> |
| 60 |
</UL> |
| 61 |
<BR> |
| 62 |
<LI><A ID="tex2html230" |
| 63 |
HREF="node20.html#SECTION00043100000000000000">3.3.1 Understanding parallel I/O</A> |
| 64 |
</UL> |
| 65 |
<!--End of Table of Child-Links--> |
| 66 |
<HR> |
| 67 |
|
| 68 |
<H2><A ID="SECTION00043000000000000000"> |
| 69 |
3.3 Parallelization levels</A> |
| 70 |
</H2> |
| 71 |
|
| 72 |
<P> |
| 73 |
In Q<SMALL>UANTUM </SMALL>ESPRESSO several MPI parallelization levels are |
| 74 |
implemented, in which both calculations |
| 75 |
and data structures are distributed across processors. |
| 76 |
Processors are organized in a hierarchy of groups, |
| 77 |
which are identified by different MPI communicators level. |
| 78 |
The groups hierarchy is as follow: |
| 79 |
|
| 80 |
<UL> |
| 81 |
<LI><B>world</B>: is the group of all processors (MPI_COMM_WORLD). |
| 82 |
</LI> |
| 83 |
<LI><B>images</B>: Processors can then be divided into different "images", each corresponding to a |
| 84 |
different self-consistent or linear-response |
| 85 |
calculation, loosely coupled to others. |
| 86 |
</LI> |
| 87 |
<LI><B>pools</B>: each image can be subpartitioned into |
| 88 |
"pools", each taking care of a group of k-points. |
| 89 |
</LI> |
| 90 |
<LI><B>bands</B>: each pool is subpartitioned into |
| 91 |
"band groups", each taking care of a group |
| 92 |
of Kohn-Sham orbitals (also called bands, or |
| 93 |
wavefunctions). Especially useful for calculations |
| 94 |
with hybrid functionals. |
| 95 |
</LI> |
| 96 |
<LI><B>PW</B>: orbitals in the PW basis set, |
| 97 |
as well as charges and density in either |
| 98 |
reciprocal or real space, are distributed |
| 99 |
across processors. |
| 100 |
This is usually referred to as "PW parallelization". |
| 101 |
All linear-algebra operations on array of PW / |
| 102 |
real-space grids are automatically and effectively parallelized. |
| 103 |
3D FFT is used to transform electronic wave functions from |
| 104 |
reciprocal to real space and vice versa. The 3D FFT is |
| 105 |
parallelized by distributing planes of the 3D grid in real |
| 106 |
space to processors (in reciprocal space, it is columns of |
| 107 |
G-vectors that are distributed to processors). |
| 108 |
</LI> |
| 109 |
<LI><B>tasks</B>: |
| 110 |
In order to allow good parallelization of the 3D FFT when |
| 111 |
the number of processors exceeds the number of FFT planes, |
| 112 |
FFTs on Kohn-Sham states are redistributed to |
| 113 |
``task'' groups so that each group |
| 114 |
can process several wavefunctions at the same time. |
| 115 |
Alternatively, when this is not possible, a further |
| 116 |
subdivision of FFT planes is performed. |
| 117 |
</LI> |
| 118 |
<LI><B>linear-algebra group</B>: |
| 119 |
A further level of parallelization, independent on |
| 120 |
PW or k-point parallelization, is the parallelization of |
| 121 |
subspace diagonalization / iterative orthonormalization. |
| 122 |
Both operations required the diagonalization of |
| 123 |
arrays whose dimension is the number of Kohn-Sham states |
| 124 |
(or a small multiple of it). All such arrays are distributed block-like |
| 125 |
across the ``linear-algebra group'', a subgroup of the pool of processors, |
| 126 |
organized in a square 2D grid. As a consequence the number of processors |
| 127 |
in the linear-algebra group is given by <I>n</I><SUP>2</SUP>, where <I>n</I> is an integer; |
| 128 |
<I>n</I><SUP>2</SUP> must be smaller than the number of processors in the PW group. |
| 129 |
The diagonalization is then performed |
| 130 |
in parallel using standard linear algebra operations. |
| 131 |
(This diagonalization is used by, but should not be confused with, |
| 132 |
the iterative Davidson algorithm). The preferred option is to use |
| 133 |
ELPA and ScaLAPACK; alternative built-in algorithms are anyway available. |
| 134 |
</LI> |
| 135 |
</UL> |
| 136 |
Note however that not all parallelization levels |
| 137 |
are implemented in all codes. |
| 138 |
|
| 139 |
<P> |
| 140 |
|
| 141 |
<H4><A ID="SECTION00043010000000000000"> |
| 142 |
3.3.0.1 About communications</A> |
| 143 |
</H4> |
| 144 |
Images and pools are loosely coupled: inter-processors communication |
| 145 |
between different images and pools is modest. Processors within each |
| 146 |
pool are instead tightly coupled and communications are significant. |
| 147 |
This means that fast communication hardware is needed if |
| 148 |
your pool extends over more than a few processors on different nodes. |
| 149 |
|
| 150 |
<P> |
| 151 |
|
| 152 |
<H4><A ID="SECTION00043020000000000000"> |
| 153 |
3.3.0.2 Choosing parameters</A> |
| 154 |
</H4>: |
| 155 |
To control the number of processors in each group, |
| 156 |
command line switches: |
| 157 |
<TT>-nimage</TT>, <TT>-npools</TT>, <TT>-nband</TT>, |
| 158 |
<TT>-ntg</TT>, <TT>-ndiag</TT> or <TT>-northo</TT> |
| 159 |
(shorthands, respectively: <TT>-ni</TT>, <TT>-nk</TT>, <TT>-nb</TT>, |
| 160 |
<TT>-nt</TT>, <TT>-nd</TT>) |
| 161 |
are used. |
| 162 |
As an example consider the following command line: |
| 163 |
<PRE> |
| 164 |
mpirun -np 4096 ./neb.x -ni 8 -nk 2 -nt 4 -nd 144 -i my.input |
| 165 |
</PRE> |
| 166 |
This executes a NEB calculation on 4096 processors, 8 images (points in the configuration |
| 167 |
space in this case) at the same time, each of |
| 168 |
which is distributed across 512 processors. |
| 169 |
k-points are distributed across 2 pools of 256 processors each, |
| 170 |
3D FFT is performed using 4 task groups (64 processors each, so |
| 171 |
the 3D real-space grid is cut into 64 slices), and the diagonalization |
| 172 |
of the subspace Hamiltonian is distributed to a square grid of 144 |
| 173 |
processors (12x12). |
| 174 |
|
| 175 |
<P> |
| 176 |
Default values are: <TT>-ni 1 -nk 1 -nt 1</TT> ; |
| 177 |
<TT>nd</TT> is set to 1 if ScaLAPACK is not compiled, |
| 178 |
it is set to the square integer smaller than or equal to the number of |
| 179 |
processors of each pool. |
| 180 |
|
| 181 |
<P> |
| 182 |
|
| 183 |
<H4><A ID="SECTION00043030000000000000"> |
| 184 |
3.3.0.3 Massively parallel calculations</A> |
| 185 |
</H4> |
| 186 |
For very large jobs (i.e. O(1000) atoms or more) or for very long jobs, |
| 187 |
to be run on massively parallel machines (e.g. IBM BlueGene) it is |
| 188 |
crucial to use in an effective way all available parallelization levels: |
| 189 |
on linear algebra (requires compilation with ELPA and/or ScaLAPACK), |
| 190 |
on "task groups" (requires run-time option "-nt N"), and mixed |
| 191 |
MPI-OpenMP (requires OpenMP compilation: <TT>configure</TT>–enable-openmp). |
| 192 |
Without a judicious choice of parameters, large jobs will find a |
| 193 |
stumbling block in either memory or CPU requirements. Note that I/O |
| 194 |
may also become a limiting factor. |
| 195 |
|
| 196 |
<P> |
| 197 |
|
| 198 |
<H3><A ID="SECTION00043100000000000000"> |
| 199 |
3.3.1 Understanding parallel I/O</A> |
| 200 |
</H3> |
| 201 |
In parallel execution, each processor has its own slice of data |
| 202 |
(Kohn-Sham orbitals, charge density, etc), that have to be written |
| 203 |
to temporary files during the calculation, |
| 204 |
or to data files at the end of the calculation. |
| 205 |
This can be done in two different ways: |
| 206 |
|
| 207 |
<UL> |
| 208 |
<LI>``collected'': all slices are |
| 209 |
collected by the code to a single processor |
| 210 |
that writes them to disk, in a single file, |
| 211 |
using a format that doesn't depend upon |
| 212 |
the number of processors or their distribution. |
| 213 |
This is the default since v.6.2 for final data. |
| 214 |
</LI> |
| 215 |
<LI>``portable'': as above, but data can be |
| 216 |
copied to and read from a different machines |
| 217 |
(this is not guaranteed with Fortran binary files). |
| 218 |
Requires compilation with <code>-D__HDF5</code> |
| 219 |
preprocessing option and HDF |
| 220 |
</LI> |
| 221 |
</UL> |
| 222 |
There is a third format, no longer used for final |
| 223 |
data but used for scratch and restart files: |
| 224 |
|
| 225 |
<UL> |
| 226 |
<LI>``distributed'': each processor |
| 227 |
writes its own slice to disk in its internal |
| 228 |
format to a different file. |
| 229 |
The ``distributed'' format is fast and simple, |
| 230 |
but the data so produced is readable only by |
| 231 |
a job running on the same number of processors, |
| 232 |
with the same type of parallelization, as the |
| 233 |
job who wrote the data, and if all |
| 234 |
files are on a file system that is visible to all |
| 235 |
processors (i.e., you cannot use local scratch |
| 236 |
directories: there is presently no way to ensure |
| 237 |
that the distribution of processes across |
| 238 |
processors will follow the same pattern |
| 239 |
for different jobs). |
| 240 |
</LI> |
| 241 |
</UL> |
| 242 |
|
| 243 |
<P> |
| 244 |
The directory for data is specified in input variables |
| 245 |
<TT>outdir</TT> and <TT>prefix</TT> (the former can be specified |
| 246 |
as well in environment variable ESPRESSO_TMPDIR): |
| 247 |
<TT>outdir/prefix.save</TT>. A copy of pseudopotential files |
| 248 |
is also written there. If some processor cannot access the |
| 249 |
data directory, the pseudopotential files are read instead |
| 250 |
from the pseudopotential directory specified in input data. |
| 251 |
Unpredictable results may follow if those files |
| 252 |
are not the same as those in the data directory! |
| 253 |
|
| 254 |
<P> |
| 255 |
<EM>IMPORTANT:</EM> |
| 256 |
Avoid I/O to network-mounted disks (via NFS) as much as you can! |
| 257 |
Ideally the scratch directory <TT>outdir</TT> should be a modern |
| 258 |
Parallel File System. If you do not have any, you can use local |
| 259 |
scratch disks (i.e. each node is physically connected to a disk |
| 260 |
and writes to it) but you may run into trouble anyway if you |
| 261 |
need to access your files that are scattered in an unpredictable |
| 262 |
way across disks residing on different nodes. |
| 263 |
|
| 264 |
<P> |
| 265 |
You can use input variable <TT>disk_io</TT> to vary the |
| 266 |
amount of I/O done by <TT>pw.x</TT>. Since v.5.1, the dafault value is |
| 267 |
<TT>disk_io='low'</TT>, so the code will store wavefunctions |
| 268 |
into RAM and not on disk during the calculation. Specify |
| 269 |
<TT>disk_io='medium'</TT> only if you have too many k-points |
| 270 |
and you run into trouble with memory; choose <TT>disk_io='none'</TT> |
| 271 |
if you do not need to keep final data files. |
| 272 |
|
| 273 |
<P> |
| 274 |
<HR> |
| 275 |
<!--Navigation Panel--> |
| 276 |
<A |
| 277 |
HREF="node21.html"> |
| 278 |
<IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A> |
| 279 |
<A |
| 280 |
HREF="node17.html"> |
| 281 |
<IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A> |
| 282 |
<A |
| 283 |
HREF="node19.html"> |
| 284 |
<IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A> |
| 285 |
<A ID="tex2html225" |
| 286 |
HREF="node1.html"> |
| 287 |
<IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A> |
| 288 |
<BR> |
| 289 |
<B> Next:</B> <A |
| 290 |
HREF="node21.html">3.4 Tricks and problems</A> |
| 291 |
<B> Up:</B> <A |
| 292 |
HREF="node17.html">3 Parallelism</A> |
| 293 |
<B> Previous:</B> <A |
| 294 |
HREF="node19.html">3.2 Running on parallel</A> |
| 295 |
<B> <A ID="tex2html226" |
| 296 |
HREF="node1.html">Contents</A></B> |
| 297 |
<!--End of Navigation Panel--> |
| 298 |
|
| 299 |
</BODY> |
| 300 |
</HTML> |