| 1 |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> |
| 2 |
|
| 3 |
<!--Converted with LaTeX2HTML 2019.2 (Released June 5, 2019) --> |
| 4 |
<HTML lang="EN"> |
| 5 |
<HEAD> |
| 6 |
<TITLE>3.4 Tricks and problems</TITLE> |
| 7 |
<META NAME="description" CONTENT="3.4 Tricks and problems"> |
| 8 |
<META NAME="keywords" CONTENT="user_guide"> |
| 9 |
<META NAME="resource-type" CONTENT="document"> |
| 10 |
<META NAME="distribution" CONTENT="global"> |
| 11 |
|
| 12 |
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8"> |
| 13 |
<META NAME="viewport" CONTENT="width=device-width, initial-scale=1.0"> |
| 14 |
<META NAME="Generator" CONTENT="LaTeX2HTML v2019.2"> |
| 15 |
|
| 16 |
<LINK REL="STYLESHEET" HREF="user_guide.css"> |
| 17 |
|
| 18 |
<LINK REL="previous" HREF="node20.html"> |
| 19 |
<LINK REL="next" HREF="node22.html"> |
| 20 |
</HEAD> |
| 21 |
|
| 22 |
<BODY > |
| 23 |
<!--Navigation Panel--> |
| 24 |
<A |
| 25 |
HREF="node22.html"> |
| 26 |
<IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A> |
| 27 |
<A |
| 28 |
HREF="node17.html"> |
| 29 |
<IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A> |
| 30 |
<A |
| 31 |
HREF="node20.html"> |
| 32 |
<IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A> |
| 33 |
<A ID="tex2html231" |
| 34 |
HREF="node1.html"> |
| 35 |
<IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A> |
| 36 |
<BR> |
| 37 |
<B> Next:</B> <A |
| 38 |
HREF="node22.html">About this document ...</A> |
| 39 |
<B> Up:</B> <A |
| 40 |
HREF="node17.html">3 Parallelism</A> |
| 41 |
<B> Previous:</B> <A |
| 42 |
HREF="node20.html">3.3 Parallelization levels</A> |
| 43 |
<B> <A ID="tex2html232" |
| 44 |
HREF="node1.html">Contents</A></B> |
| 45 |
<BR> |
| 46 |
<BR> |
| 47 |
<!--End of Navigation Panel--> |
| 48 |
<!--Table of Child-Links--> |
| 49 |
<A ID="CHILD_LINKS"><STRONG>Subsections</STRONG></A> |
| 50 |
|
| 51 |
<UL> |
| 52 |
<LI><UL> |
| 53 |
<LI><A ID="tex2html233" |
| 54 |
HREF="node21.html#SECTION00044010000000000000">3.4.0.1 Trouble with input files</A> |
| 55 |
<LI><A ID="tex2html234" |
| 56 |
HREF="node21.html#SECTION00044020000000000000">3.4.0.2 Trouble with MKL and MPI parallelization</A> |
| 57 |
<LI><A ID="tex2html235" |
| 58 |
HREF="node21.html#SECTION00044030000000000000">3.4.0.3 Trouble with compilers and MPI libraries</A> |
| 59 |
</UL></UL> |
| 60 |
<!--End of Table of Child-Links--> |
| 61 |
<HR> |
| 62 |
|
| 63 |
<H2><A ID="SECTION00044000000000000000"></A> |
| 64 |
<A ID="SubSec:badpara"></A> |
| 65 |
<BR> |
| 66 |
3.4 Tricks and problems |
| 67 |
</H2> |
| 68 |
|
| 69 |
<P> |
| 70 |
Many problems in parallel execution derive from the mixup of different |
| 71 |
MPI libraries and runtime environments. There are two major MPI |
| 72 |
implementations, OpenMPI and MPICH, coming in various versions, |
| 73 |
not necessarily compatible; plus vendor-specific implementations |
| 74 |
(e.g. Intel MPI). A parallel machine may have multiple parallel |
| 75 |
compilers (typically, <TT>mpif90</TT> scripts calling different |
| 76 |
serial compilers), multiple MPI libraries, multiple launchers |
| 77 |
for parallel codes (different versions of <TT>mpirun</TT> and/or |
| 78 |
<TT>mpiexec</TT>). You have to figure out the proper combination |
| 79 |
of all of the above, which may require using command <TT>module</TT> |
| 80 |
or manually setting environment variables and execution paths. |
| 81 |
What exactly has to be done depends upon the configuration of your |
| 82 |
machine. You should inquire with your system administrator or user |
| 83 |
support (if available; if not, YOU are the system administrator |
| 84 |
and user support and YOU have to solve your problems). |
| 85 |
|
| 86 |
<P> |
| 87 |
Always verify if your executable is actually compiled for |
| 88 |
parallel execution or not: it is declared in the first lines |
| 89 |
of output. Running several instances of a serial code with |
| 90 |
<TT>mpirun</TT> or <TT>mpiexec</TT> produces strange crashes. |
| 91 |
|
| 92 |
<P> |
| 93 |
|
| 94 |
<H4><A ID="SECTION00044010000000000000"> |
| 95 |
3.4.0.1 Trouble with input files</A> |
| 96 |
</H4> |
| 97 |
Some implementations of the MPI library have problems with input |
| 98 |
redirection in parallel. This typically shows up under the form of |
| 99 |
mysterious errors when reading data. If this happens, use the option |
| 100 |
<TT>-i</TT> (or <TT>-in</TT>, <TT>-inp</TT>, <TT>-input</TT>), |
| 101 |
followed by the input file name. |
| 102 |
Example: |
| 103 |
<PRE> |
| 104 |
pw.x -i inputfile -nk 4 > outputfile |
| 105 |
</PRE> |
| 106 |
Of course the |
| 107 |
input file must be accessible by the processor that must read it |
| 108 |
(only one processor reads the input file and subsequently broadcasts |
| 109 |
its contents to all other processors). |
| 110 |
|
| 111 |
<P> |
| 112 |
Apparently the LSF implementation of MPI libraries manages to ignore or to |
| 113 |
confuse even the <TT>-i/in/inp/input</TT> mechanism that is present in all |
| 114 |
Q<SMALL>UANTUM </SMALL>ESPRESSO codes. In this case, use the <TT>-i</TT> option of <TT>mpirun.lsf</TT> |
| 115 |
to provide an input file. |
| 116 |
|
| 117 |
<P> |
| 118 |
|
| 119 |
<H4><A ID="SECTION00044020000000000000"> |
| 120 |
3.4.0.2 Trouble with MKL and MPI parallelization</A> |
| 121 |
</H4> |
| 122 |
If you notice very bad parallel performances with MPI and MKL libraries, |
| 123 |
it is very likely that the OpenMP parallelization performed by the latter |
| 124 |
is colliding with MPI. Recent versions of MKL enable autoparallelization |
| 125 |
by default on multicore machines. You must set the environment variable |
| 126 |
OMP_NUM_THREADS to 1 to disable it. |
| 127 |
Note that if for some reason the correct setting of variable |
| 128 |
OMP_NUM_THREADS |
| 129 |
does not propagate to all processors, you may equally run into trouble. |
| 130 |
Lorenzo Paulatto (Nov. 2008) suggests to use the <TT>-x</TT> option to <TT>mpirun</TT> to |
| 131 |
propagate OMP_NUM_THREADS to all processors. |
| 132 |
Axel Kohlmeyer suggests the following (April 2008): |
| 133 |
"(I've) found that Intel is now turning on multithreading without any |
| 134 |
warning and that is for example why their FFT seems faster than |
| 135 |
FFTW. For serial and OpenMP based runs this makes no difference (in |
| 136 |
fact the multi-threaded FFT helps), but if you run MPI locally, you |
| 137 |
actually lose performance. Also if you use the 'numactl' tool on linux |
| 138 |
to bind a job to a specific cpu core, MKL will still try to use all |
| 139 |
available cores (and slow down badly). The cleanest way of avoiding |
| 140 |
this mess is to either link with |
| 141 |
<BLOCKQUOTE> |
| 142 |
<TT>-lmkl_intel_lp64 -lmkl_sequential -lmkl_core</TT> (on 64-bit: |
| 143 |
x86_64, ia64) |
| 144 |
<BR><TT>-lmkl_intel -lmkl_sequential -lmkl_core</TT> (on 32-bit, i.e. ia32 ) |
| 145 |
|
| 146 |
</BLOCKQUOTE> |
| 147 |
or edit the <TT>libmkl_'platform'.a</TT> file. I'm using now a file |
| 148 |
<TT>libmkl10.a</TT> with: |
| 149 |
<PRE> |
| 150 |
GROUP (libmkl_intel_lp64.a libmkl_sequential.a libmkl_core.a) |
| 151 |
</PRE> |
| 152 |
It works like a charm". UPDATE: Since v.4.2, <TT>configure</TT> links by |
| 153 |
default MKL without multithreaded support. |
| 154 |
|
| 155 |
<P> |
| 156 |
|
| 157 |
<H4><A ID="SECTION00044030000000000000"> |
| 158 |
3.4.0.3 Trouble with compilers and MPI libraries</A> |
| 159 |
</H4> |
| 160 |
Many users of Q<SMALL>UANTUM </SMALL>ESPRESSO, in particular those working on PC clusters, |
| 161 |
have to rely on themselves (or on less-than-adequate system managers) for |
| 162 |
the correct configuration of software for parallel execution. Mysterious and |
| 163 |
irreproducible crashes in parallel execution are sometimes due to bugs |
| 164 |
in Q<SMALL>UANTUM </SMALL>ESPRESSO, but more often than not are a consequence of buggy |
| 165 |
compilers or of buggy or miscompiled MPI libraries. |
| 166 |
|
| 167 |
<P> |
| 168 |
<HR> |
| 169 |
<!--Navigation Panel--> |
| 170 |
<A |
| 171 |
HREF="node22.html"> |
| 172 |
<IMG WIDTH="37" HEIGHT="24" ALT="next" SRC="next.png"></A> |
| 173 |
<A |
| 174 |
HREF="node17.html"> |
| 175 |
<IMG WIDTH="26" HEIGHT="24" ALT="up" SRC="up.png"></A> |
| 176 |
<A |
| 177 |
HREF="node20.html"> |
| 178 |
<IMG WIDTH="63" HEIGHT="24" ALT="previous" SRC="prev.png"></A> |
| 179 |
<A ID="tex2html231" |
| 180 |
HREF="node1.html"> |
| 181 |
<IMG WIDTH="65" HEIGHT="24" ALT="contents" SRC="contents.png"></A> |
| 182 |
<BR> |
| 183 |
<B> Next:</B> <A |
| 184 |
HREF="node22.html">About this document ...</A> |
| 185 |
<B> Up:</B> <A |
| 186 |
HREF="node17.html">3 Parallelism</A> |
| 187 |
<B> Previous:</B> <A |
| 188 |
HREF="node20.html">3.3 Parallelization levels</A> |
| 189 |
<B> <A ID="tex2html232" |
| 190 |
HREF="node1.html">Contents</A></B> |
| 191 |
<!--End of Navigation Panel--> |
| 192 |
|
| 193 |
</BODY> |
| 194 |
</HTML> |