Debian Codegraph @

GUI/PWgui/doc/pwdocs/user_guide/node20.html

raw
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
2
3 <!--Converted with LaTeX2HTML 2019.2 (Released June 5, 2019) -->
4 <HTML lang="EN">
5 <HEAD>
6 <TITLE>3.3 Parallelization levels</TITLE>
7 <META NAME="description" CONTENT="3.3 Parallelization levels">
8 <META NAME="keywords" CONTENT="user_guide">
9 <META NAME="resource-type" CONTENT="document">
10 <META NAME="distribution" CONTENT="global">
11
12 <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
13 <META NAME="viewport" CONTENT="width=device-width, initial-scale=1.0">
14 <META NAME="Generator" CONTENT="LaTeX2HTML v2019.2">
15
16 <LINK REL="STYLESHEET" HREF="user_guide.css">
17
18 <LINK REL="next" HREF="node21.html">
19 <LINK REL="previous" HREF="node19.html">
20 <LINK REL="next" HREF="node21.html">
21 </HEAD>
22
23 <BODY >
24 <!--Navigation Panel-->
25 <A
26 HREF="node21.html">
27 <IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A>
28 <A
29 HREF="node17.html">
30 <IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A>
31 <A
32 HREF="node19.html">
33 <IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A>
34 <A ID="tex2html225"
35 HREF="node1.html">
36 <IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A>
37 <BR>
38 <B> Next:</B> <A
39 HREF="node21.html">3.4 Tricks and problems</A>
40 <B> Up:</B> <A
41 HREF="node17.html">3 Parallelism</A>
42 <B> Previous:</B> <A
43 HREF="node19.html">3.2 Running on parallel</A>
44 <B> <A ID="tex2html226"
45 HREF="node1.html">Contents</A></B>
46 <BR>
47 <BR>
48 <!--End of Navigation Panel-->
49 <!--Table of Child-Links-->
50 <A ID="CHILD_LINKS"><STRONG>Subsections</STRONG></A>
51
52 <UL>
53 <LI><UL>
54 <LI><A ID="tex2html227"
55 HREF="node20.html#SECTION00043010000000000000">3.3.0.1 About communications</A>
56 <LI><A ID="tex2html228"
57 HREF="node20.html#SECTION00043020000000000000">3.3.0.2 Choosing parameters</A>
58 <LI><A ID="tex2html229"
59 HREF="node20.html#SECTION00043030000000000000">3.3.0.3 Massively parallel calculations</A>
60 </UL>
61 <BR>
62 <LI><A ID="tex2html230"
63 HREF="node20.html#SECTION00043100000000000000">3.3.1 Understanding parallel I/O</A>
64 </UL>
65 <!--End of Table of Child-Links-->
66 <HR>
67
68 <H2><A ID="SECTION00043000000000000000">
69 3.3 Parallelization levels</A>
70 </H2>
71
72 <P>
73 In Q<SMALL>UANTUM </SMALL>ESPRESSO several MPI parallelization levels are
74 implemented, in which both calculations
75 and data structures are distributed across processors.
76 Processors are organized in a hierarchy of groups,
77 which are identified by different MPI communicators level.
78 The groups hierarchy is as follow:
79
80 <UL>
81 <LI><B>world</B>: is the group of all processors (MPI_COMM_WORLD).
82 </LI>
83 <LI><B>images</B>: Processors can then be divided into different "images", each corresponding to a
84 different self-consistent or linear-response
85 calculation, loosely coupled to others.
86 </LI>
87 <LI><B>pools</B>: each image can be subpartitioned into
88 "pools", each taking care of a group of k-points.
89 </LI>
90 <LI><B>bands</B>: each pool is subpartitioned into
91 "band groups", each taking care of a group
92 of Kohn-Sham orbitals (also called bands, or
93 wavefunctions). Especially useful for calculations
94 with hybrid functionals.
95 </LI>
96 <LI><B>PW</B>: orbitals in the PW basis set,
97 as well as charges and density in either
98 reciprocal or real space, are distributed
99 across processors.
100 This is usually referred to as "PW parallelization".
101 All linear-algebra operations on array of PW /
102 real-space grids are automatically and effectively parallelized.
103 3D FFT is used to transform electronic wave functions from
104 reciprocal to real space and vice versa. The 3D FFT is
105 parallelized by distributing planes of the 3D grid in real
106 space to processors (in reciprocal space, it is columns of
107 G-vectors that are distributed to processors).
108 </LI>
109 <LI><B>tasks</B>:
110 In order to allow good parallelization of the 3D FFT when
111 the number of processors exceeds the number of FFT planes,
112 FFTs on Kohn-Sham states are redistributed to
113 ``task'' groups so that each group
114 can process several wavefunctions at the same time.
115 Alternatively, when this is not possible, a further
116 subdivision of FFT planes is performed.
117 </LI>
118 <LI><B>linear-algebra group</B>:
119 A further level of parallelization, independent on
120 PW or k-point parallelization, is the parallelization of
121 subspace diagonalization / iterative orthonormalization.
122 Both operations required the diagonalization of
123 arrays whose dimension is the number of Kohn-Sham states
124 (or a small multiple of it). All such arrays are distributed block-like
125 across the ``linear-algebra group'', a subgroup of the pool of processors,
126 organized in a square 2D grid. As a consequence the number of processors
127 in the linear-algebra group is given by <I>n</I><SUP>2</SUP>, where <I>n</I> is an integer;
128 <I>n</I><SUP>2</SUP> must be smaller than the number of processors in the PW group.
129 The diagonalization is then performed
130 in parallel using standard linear algebra operations.
131 (This diagonalization is used by, but should not be confused with,
132 the iterative Davidson algorithm). The preferred option is to use
133 ELPA and ScaLAPACK; alternative built-in algorithms are anyway available.
134 </LI>
135 </UL>
136 Note however that not all parallelization levels
137 are implemented in all codes.
138
139 <P>
140
141 <H4><A ID="SECTION00043010000000000000">
142 3.3.0.1 About communications</A>
143 </H4>
144 Images and pools are loosely coupled: inter-processors communication
145 between different images and pools is modest. Processors within each
146 pool are instead tightly coupled and communications are significant.
147 This means that fast communication hardware is needed if
148 your pool extends over more than a few processors on different nodes.
149
150 <P>
151
152 <H4><A ID="SECTION00043020000000000000">
153 3.3.0.2 Choosing parameters</A>
154 </H4>:
155 To control the number of processors in each group,
156 command line switches:
157 <TT>-nimage</TT>, <TT>-npools</TT>, <TT>-nband</TT>,
158 <TT>-ntg</TT>, <TT>-ndiag</TT> or <TT>-northo</TT>
159 (shorthands, respectively: <TT>-ni</TT>, <TT>-nk</TT>, <TT>-nb</TT>,
160 <TT>-nt</TT>, <TT>-nd</TT>)
161 are used.
162 As an example consider the following command line:
163 <PRE>
164 mpirun -np 4096 ./neb.x -ni 8 -nk 2 -nt 4 -nd 144 -i my.input
165 </PRE>
166 This executes a NEB calculation on 4096 processors, 8 images (points in the configuration
167 space in this case) at the same time, each of
168 which is distributed across 512 processors.
169 k-points are distributed across 2 pools of 256 processors each,
170 3D FFT is performed using 4 task groups (64 processors each, so
171 the 3D real-space grid is cut into 64 slices), and the diagonalization
172 of the subspace Hamiltonian is distributed to a square grid of 144
173 processors (12x12).
174
175 <P>
176 Default values are: <TT>-ni 1 -nk 1 -nt 1</TT> ;
177 <TT>nd</TT> is set to 1 if ScaLAPACK is not compiled,
178 it is set to the square integer smaller than or equal to the number of
179 processors of each pool.
180
181 <P>
182
183 <H4><A ID="SECTION00043030000000000000">
184 3.3.0.3 Massively parallel calculations</A>
185 </H4>
186 For very large jobs (i.e. O(1000) atoms or more) or for very long jobs,
187 to be run on massively parallel machines (e.g. IBM BlueGene) it is
188 crucial to use in an effective way all available parallelization levels:
189 on linear algebra (requires compilation with ELPA and/or ScaLAPACK),
190 on "task groups" (requires run-time option "-nt N"), and mixed
191 MPI-OpenMP (requires OpenMP compilation: <TT>configure</TT>&ndash;enable-openmp).
192 Without a judicious choice of parameters, large jobs will find a
193 stumbling block in either memory or CPU requirements. Note that I/O
194 may also become a limiting factor.
195
196 <P>
197
198 <H3><A ID="SECTION00043100000000000000">
199 3.3.1 Understanding parallel I/O</A>
200 </H3>
201 In parallel execution, each processor has its own slice of data
202 (Kohn-Sham orbitals, charge density, etc), that have to be written
203 to temporary files during the calculation,
204 or to data files at the end of the calculation.
205 This can be done in two different ways:
206
207 <UL>
208 <LI>``collected'': all slices are
209 collected by the code to a single processor
210 that writes them to disk, in a single file,
211 using a format that doesn't depend upon
212 the number of processors or their distribution.
213 This is the default since v.6.2 for final data.
214 </LI>
215 <LI>``portable'': as above, but data can be
216 copied to and read from a different machines
217 (this is not guaranteed with Fortran binary files).
218 Requires compilation with <code>-D__HDF5</code>
219 preprocessing option and HDF
220 </LI>
221 </UL>
222 There is a third format, no longer used for final
223 data but used for scratch and restart files:
224
225 <UL>
226 <LI>``distributed'': each processor
227 writes its own slice to disk in its internal
228 format to a different file.
229 The ``distributed'' format is fast and simple,
230 but the data so produced is readable only by
231 a job running on the same number of processors,
232 with the same type of parallelization, as the
233 job who wrote the data, and if all
234 files are on a file system that is visible to all
235 processors (i.e., you cannot use local scratch
236 directories: there is presently no way to ensure
237 that the distribution of processes across
238 processors will follow the same pattern
239 for different jobs).
240 </LI>
241 </UL>
242
243 <P>
244 The directory for data is specified in input variables
245 <TT>outdir</TT> and <TT>prefix</TT> (the former can be specified
246 as well in environment variable ESPRESSO_TMPDIR):
247 <TT>outdir/prefix.save</TT>. A copy of pseudopotential files
248 is also written there. If some processor cannot access the
249 data directory, the pseudopotential files are read instead
250 from the pseudopotential directory specified in input data.
251 Unpredictable results may follow if those files
252 are not the same as those in the data directory!
253
254 <P>
255 <EM>IMPORTANT:</EM>
256 Avoid I/O to network-mounted disks (via NFS) as much as you can!
257 Ideally the scratch directory <TT>outdir</TT> should be a modern
258 Parallel File System. If you do not have any, you can use local
259 scratch disks (i.e. each node is physically connected to a disk
260 and writes to it) but you may run into trouble anyway if you
261 need to access your files that are scattered in an unpredictable
262 way across disks residing on different nodes.
263
264 <P>
265 You can use input variable <TT>disk_io</TT> to vary the
266 amount of I/O done by <TT>pw.x</TT>. Since v.5.1, the dafault value is
267 <TT>disk_io='low'</TT>, so the code will store wavefunctions
268 into RAM and not on disk during the calculation. Specify
269 <TT>disk_io='medium'</TT> only if you have too many k-points
270 and you run into trouble with memory; choose <TT>disk_io='none'</TT>
271 if you do not need to keep final data files.
272
273 <P>
274 <HR>
275 <!--Navigation Panel-->
276 <A
277 HREF="node21.html">
278 <IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A>
279 <A
280 HREF="node17.html">
281 <IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A>
282 <A
283 HREF="node19.html">
284 <IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A>
285 <A ID="tex2html225"
286 HREF="node1.html">
287 <IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A>
288 <BR>
289 <B> Next:</B> <A
290 HREF="node21.html">3.4 Tricks and problems</A>
291 <B> Up:</B> <A
292 HREF="node17.html">3 Parallelism</A>
293 <B> Previous:</B> <A
294 HREF="node19.html">3.2 Running on parallel</A>
295 <B> <A ID="tex2html226"
296 HREF="node1.html">Contents</A></B>
297 <!--End of Navigation Panel-->
298
299 </BODY>
300 </HTML>