| 1 |
|
| 2 |
To RedHat Engineers: |
| 3 |
|
| 4 |
Paul Howarth, |
| 5 |
Denis Fateyev, |
| 6 |
|
| 7 |
Near the very end of the 1.8 cycle were several revisions. Each time, an |
| 8 |
email was sent to you stating that MCE was completed. But each morning, |
| 9 |
I woke up with a sense that something else was missed. |
| 10 |
|
| 11 |
(a) Running MCE and MCE::Shared inside an eval block. |
| 12 |
(b) Check for EINTR condition during sysread and syswrite. |
| 13 |
(c) Check the OS while running with ~ 200 workers. |
| 14 |
(d) Test MCE with the Wx toolkit - wxWidgets. |
| 15 |
(e) Test MCE and MCE::Shared by running taint mode via perl -T. |
| 16 |
(f) Validate DESTROY for shared Condvar and Queue objects. |
| 17 |
(g) Improve MCE::Shared::Handle for non-shared use, was not 100%. |
| 18 |
|
| 19 |
Going from 99.7% to 100.0% required so much effort. I am deeply sorry |
| 20 |
for the many emails sent. Although far away in miles, you were there |
| 21 |
in keeping me from going insane. For this I thank you. |
| 22 |
|
| 23 |
Best, |
| 24 |
Mario |
| 25 |
|
| 26 |
-- |
| 27 |
|
| 28 |
############################################################################### |
| 29 |
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # |
| 30 |
############################################################################### |
| 31 |
|
| 32 |
What follows are many people reporting problems, fixes or solutions. |
| 33 |
More specifically in alphabetical order by last name. |
| 34 |
|
| 35 |
Also, not to forget all the people mentioned in the change log. |
| 36 |
|
| 37 |
Baumgart, Philippe |
| 38 |
|
| 39 |
For trying the MCE interval option and calling MCE->yield. I felt bad for |
| 40 |
Philippe spending many hours that it caused me to simulate the interval |
| 41 |
option over and over again. It turns out that there can be unnecessary |
| 42 |
delays from a worker taking extra time. MCE 1.815 resolves the issue by |
| 43 |
allowing the next available worker to run versus workers running orderly. |
| 44 |
Thank you, Philippe for your patience. |
| 45 |
|
| 46 |
Berger, Joel |
| 47 |
|
| 48 |
The inspiration for the sixth model (MCE::Step) came from reading a blog |
| 49 |
by Joel, "Writing Non-Blocking Applications with Mojolicious: Part 3": |
| 50 |
|
| 51 |
https://blogs.perl.org/users/joel_berger/2014/01/ |
| 52 |
writing-non-blocking-applications-with-mojolicious-part-3.html |
| 53 |
|
| 54 |
Bouras, George |
| 55 |
|
| 56 |
For reporting sockets failing under the Windows environment due to a |
| 57 |
period of inactivity. An example is when workers execute an external |
| 58 |
command taking beyond 4 minutes to complete. On returning, the sockets |
| 59 |
have gone stale with a CPU core going 100% at that point. |
| 60 |
|
| 61 |
The MCE 1.601 release is attributed to the many use cases I received |
| 62 |
from George. |
| 63 |
|
| 64 |
Cantrell, David (DrHyde on perlmonks.org) |
| 65 |
|
| 66 |
For reporting on MCE hanging with cpan -t MCE under his environment. Also, |
| 67 |
thank you for providing me a guest account via ssh. I was able to reproduce |
| 68 |
the issue. Thus, added writeable check and croak if tmp dir is not writeable. |
| 69 |
One cannot assume that /dev/shm is writeable by everyone. :) |
| 70 |
|
| 71 |
Collet, Eric |
| 72 |
|
| 73 |
For the inspiration to the tre-agrep binary. |
| 74 |
|
| 75 |
I first learned of tre-agrep from Eric. He emailed me running times for a |
| 76 |
couple scenarios utilizing a single core. Thus increasing the need to have |
| 77 |
a MCE-driven wrapper to run tre-agrep in parallel. |
| 78 |
|
| 79 |
The bin/mce_grep script was created to show folks how one may go about |
| 80 |
chunking data between Perl and an external C binary efficiently without |
| 81 |
invoking the shell. The script supports grep, egrep, fgrep, agrep and |
| 82 |
tre-agrep. |
| 83 |
|
| 84 |
Eldai, Hisham |
| 85 |
|
| 86 |
While working on the biofasta examples, Hisham introduced me to hg19.fa. |
| 87 |
A couple records inside hg19 (driven by $/ = "\n>") are 250 MiB in size. |
| 88 |
Memory optimizations in v1.600 came about by running against hg19.fa |
| 89 |
hundreds of times. |
| 90 |
|
| 91 |
############################################################################### |
| 92 |
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # |
| 93 |
############################################################################### |
| 94 |
|
| 95 |
Farrell, David |
| 96 |
|
| 97 |
David wrote a nice article about MCE (love the picture). What stood out was |
| 98 |
his example. I never thought about using mce_grep { ... } <$fileHandle> |
| 99 |
without the _f (mce_grep_f). At the time, updated MCE::Grep and MCE::Map |
| 100 |
to be fully aware of wantarray. |
| 101 |
|
| 102 |
The blog "Make your code run faster with Perl's secret turbo module" is |
| 103 |
located at: |
| 104 |
|
| 105 |
https://www.perl.com/article/61/2014/1/21/ |
| 106 |
Make-your-code-run-faster-with-Perl-s-secret-turbo-module/ |
| 107 |
|
| 108 |
Gasper, Felipe |
| 109 |
|
| 110 |
For test scripts found in IO::SigGuard. I was able to get MCE::Mutex |
| 111 |
to fail with similar testing. The result from testing is that MCE too, |
| 112 |
must check for EINTR during sysread and syswrite operations. The same |
| 113 |
is true for MCE::Shared. |
| 114 |
|
| 115 |
Golden, David |
| 116 |
|
| 117 |
For the Hash::Ordered module. I learned a lot from this *GEM* on CPAN. |
| 118 |
Particularly, the use of overload. I didn't know that was possible. |
| 119 |
|
| 120 |
For the time during collaborative efforts in making Hash::Ordered faster. |
| 121 |
I learned some new tricks along the way. |
| 122 |
|
| 123 |
Thank you for alarming me of the memory leaks with the on-demand hash |
| 124 |
dereferencing inside MCE::Shared::Ordhash. MCE::Shared::Object was also |
| 125 |
impacted. I'd never forget that dreadful day. |
| 126 |
|
| 127 |
MCE::Shared::{ Array, Hash, and Ordhash } are inspired by Hash::Ordered. |
| 128 |
In particular, the clone and iterator methods. Also, keys and values on |
| 129 |
taking arguments. I confess for never getting used to the "as_list" |
| 130 |
method name, initially. :) |
| 131 |
|
| 132 |
Gorwits, Oliver |
| 133 |
|
| 134 |
For reporting on a couple issues with MCE along with a solution for each. |
| 135 |
|
| 136 |
Pointed me in the right direction for improving the logic for the die |
| 137 |
handler. Basically, eval { die 'this should not cause MCE to die' }; |
| 138 |
This has reached 100% (all cases, threads and processes) in MCE 1.600. |
| 139 |
|
| 140 |
Halpenny, Shawn |
| 141 |
|
| 142 |
For reporting an issue (bug RT#94706) with signal handling in MCE::Signal. |
| 143 |
Also, thank you for tip on getpgrp. |
| 144 |
|
| 145 |
Hedden, Jerry |
| 146 |
|
| 147 |
From reading threads::shared. In particular the make_shared function. |
| 148 |
That was helpful in getting MCE::Shared working. |
| 149 |
|
| 150 |
Hluchan, Yary |
| 151 |
|
| 152 |
For providing a benchmark script which I used to compare IPC performance |
| 153 |
between 1.6 and 1.7. For many kind words near the end of the development |
| 154 |
cycle for MCE 1.7 and MCE::Shared. |
| 155 |
|
| 156 |
############################################################################### |
| 157 |
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # |
| 158 |
############################################################################### |
| 159 |
|
| 160 |
Jacobsen, Dana |
| 161 |
|
| 162 |
For reporting an issue with get_ncpu on AIX. In addition, suggesting |
| 163 |
various solutions. Also, thank you for your time with Sandboxing with |
| 164 |
Perl + MCE + Inline::C at https://github.com/marioroy/mce-sandbox/. |
| 165 |
|
| 166 |
Karasik, Dmitry |
| 167 |
|
| 168 |
For bug (RT#102802) and elegant solution for the die handler inside |
| 169 |
MCE::Core::Worker and MCE::Signal. |
| 170 |
|
| 171 |
Kharchenko, Oleksandr |
| 172 |
|
| 173 |
Someone once asked Oleksandr why not use MCE. Oleksandr created 2 modules |
| 174 |
on CPAN, Parallel::DataPipe and Parallel::parallel_map. I used his example |
| 175 |
heavily in comparing IPC between pipe-driven and socket-driven. Not pleased |
| 176 |
with MCE's performance with chunk_size => 1, although 2 was as fast if not |
| 177 |
faster, I tried various things to include IO::Select. Finally, realized |
| 178 |
increasing the number of data channels was all that was needed (v1.500). |
| 179 |
|
| 180 |
Kulow, Stephan |
| 181 |
|
| 182 |
For making the OpenSUSE package for MCE. |
| 183 |
https://build.opensuse.org/package/show/devel:languages:perl:CPAN-M/perl-MCE |
| 184 |
|
| 185 |
Mabon, Philip |
| 186 |
|
| 187 |
For reporting on a couple issues with MCE in particular bug RT#92627. |
| 188 |
|
| 189 |
McKeown, Benjamin |
| 190 |
|
| 191 |
For reporting several edge cases with MCE in particular bugs RT#105557, |
| 192 |
RT#105559, and RT#111780. |
| 193 |
|
| 194 |
Mertens, David |
| 195 |
|
| 196 |
I am thankful for the tip on making PDL + MCE a possibility. Workers were |
| 197 |
crashing during exiting. Adding "sub PDL::CLONE_SKIP { 1 }" inside MCE.pm |
| 198 |
solved this issue entirely. Also, for tips on effectively using PDL in |
| 199 |
his example at https://gist.github.com/run4flat/4942132. |
| 200 |
|
| 201 |
Barrier synchronization in MCE came from reading parallel_sync in David's |
| 202 |
Perl module; PDL::Parallel::threads::SIMD. |
| 203 |
|
| 204 |
Mughal, Zakariyya |
| 205 |
|
| 206 |
First, Demian Riccardi introduced me to Devel::IPerl by Zakariyya. The die |
| 207 |
handlers in MCE broke the user experience. It seems that looking at $^S |
| 208 |
(for Perl state) is not enough. MCE v1.600 fixes this once and for all. |
| 209 |
The answer is in perl-5.20.1/cpan/CGI/lib/CGI/Carp.pm (Carp::longmess). |
| 210 |
|
| 211 |
############################################################################### |
| 212 |
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # |
| 213 |
############################################################################### |
| 214 |
|
| 215 |
Ogulin, Joseph |
| 216 |
|
| 217 |
For providing the initial perl-MCE.spec file. |
| 218 |
For the cool remarks at work about MCE being really big :) |
| 219 |
|
| 220 |
Riccardi, Demian |
| 221 |
|
| 222 |
I received a message from Demian asking for mce_map. His message requested |
| 223 |
( my @a = mce_map { ... } 1..100 ). Five models were introduced in v1.500; |
| 224 |
MCE::Flow, MCE::Grep, MCE::Loop, MCE::Map, and MCE::Stream. |
| 225 |
|
| 226 |
Find capability were limited in MCE::Shared 1.001. For example, not able to |
| 227 |
mix :AND(s) and :OR(s) together. One day, I clicked on recent distributions |
| 228 |
on metacpan.org and came across HackaMol::Roles::SelectionRole. The trick |
| 229 |
is to eval the query string together with grep or map. Thank you Demian. |
| 230 |
|
| 231 |
Rowe, Jillian |
| 232 |
|
| 233 |
For reporting on IO::File handles not working with MCE. Fixed in 1.515. |
| 234 |
For believing in possibilities beyond text-book thinking. |
| 235 |
|
| 236 |
Roy, Sylvia (spouse) |
| 237 |
|
| 238 |
For supporting me from v1.0, especially during later 1.6 revisions and 1.7. |
| 239 |
But not to forget the journey during 1.8. So little was known on the hurdles |
| 240 |
to cross. It's all over now with MCE 1.827 and MCE::Shared 1.823. |
| 241 |
|
| 242 |
Šabata, Petr |
| 243 |
|
| 244 |
For initial package submission at Red Hat. |
| 245 |
https://bugzilla.redhat.com/show_bug.cgi?id=1162531 |
| 246 |
|
| 247 |
Sasser, Tom |
| 248 |
|
| 249 |
For reporting on bin/mce_grep failing with Perl 5.8.x. |
| 250 |
|
| 251 |
Schlichting, Florian |
| 252 |
|
| 253 |
For making a Debian package for MCE: libmce-perl |
| 254 |
|
| 255 |
Shen, Wei |
| 256 |
|
| 257 |
For the BioUtil CPAN module. BioUtil::Seq::FastaReader inspired me to |
| 258 |
create parallel examples (biofasta folder) in MCE 1.600. We tried to |
| 259 |
make FastaReader in BioUtil::Seq even faster. |
| 260 |
|
| 261 |
############################################################################### |
| 262 |
# * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # |
| 263 |
############################################################################### |
| 264 |
|
| 265 |
Sjøgren, Adam |
| 266 |
|
| 267 |
For reporting on running /usr/bin/time mce_script.pl and pressing Ctrl-C |
| 268 |
failing. The default behavior in MCE::Signal is to call setpgrp. MCE 1.403 |
| 269 |
adds a new option to MCE::Signal to not call setpgrp. This also gives the |
| 270 |
developer finer control as to where setpgrp should be called, if ever |
| 271 |
needed, such as in the child process and not the manager process. |
| 272 |
|
| 273 |
use MCE::Signal qw(-no_setpgrp); |
| 274 |
use MCE; |
| 275 |
|
| 276 |
Smith, Marcus |
| 277 |
|
| 278 |
For reporting a problem with MCE including a sample script demonstrating |
| 279 |
MCE not 100% UTF-8 safe. All this time, totally overlooked the fact that |
| 280 |
the length function deals in logical characters, not physical bytes. :) |
| 281 |
|
| 282 |
Timmermans, Leon |
| 283 |
|
| 284 |
The inspiration for not having to backslash variables for MCE::Shared came |
| 285 |
from reading Const::Fast: e.g. mce_share my @array => ( list ); |
| 286 |
|
| 287 |
Thalhammer, Jeffrey Ryan |
| 288 |
|
| 289 |
For submitting a feature request for lazy arrays support. |
| 290 |
|
| 291 |
Although a lazy array cannot be used directly for input_data, support for |
| 292 |
iterator objects was added to MCE 1.505; e.g. input_data => $iterator_ref; |
| 293 |
This enabled support for lazy arrays using closures. |
| 294 |
|
| 295 |
I am thankful for the small code snippet highlighting a use-case of MCE. |
| 296 |
I would have failed initially without it. Your example was placed under |
| 297 |
the "SYNTAX for INPUT_DATA" section in MCE::Core for others to see on |
| 298 |
how to process output from Path::Iterator::Rule in parallel. |
| 299 |
|
| 300 |
Warm regards, |
| 301 |
Mario |
| 302 |
|