Debian Codegraph @

Credits

raw
1
2 To RedHat Engineers:
3
4 Paul Howarth,
5 Denis Fateyev,
6
7 Near the very end of the 1.8 cycle were several revisions. Each time, an
8 email was sent to you stating that MCE was completed. But each morning,
9 I woke up with a sense that something else was missed.
10
11 (a) Running MCE and MCE::Shared inside an eval block.
12 (b) Check for EINTR condition during sysread and syswrite.
13 (c) Check the OS while running with ~ 200 workers.
14 (d) Test MCE with the Wx toolkit - wxWidgets.
15 (e) Test MCE and MCE::Shared by running taint mode via perl -T.
16 (f) Validate DESTROY for shared Condvar and Queue objects.
17 (g) Improve MCE::Shared::Handle for non-shared use, was not 100%.
18
19 Going from 99.7% to 100.0% required so much effort. I am deeply sorry
20 for the many emails sent. Although far away in miles, you were there
21 in keeping me from going insane. For this I thank you.
22
23 Best,
24 Mario
25
26 --
27
28 ###############################################################################
29 # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
30 ###############################################################################
31
32 What follows are many people reporting problems, fixes or solutions.
33 More specifically in alphabetical order by last name.
34
35 Also, not to forget all the people mentioned in the change log.
36
37 Baumgart, Philippe
38
39 For trying the MCE interval option and calling MCE->yield. I felt bad for
40 Philippe spending many hours that it caused me to simulate the interval
41 option over and over again. It turns out that there can be unnecessary
42 delays from a worker taking extra time. MCE 1.815 resolves the issue by
43 allowing the next available worker to run versus workers running orderly.
44 Thank you, Philippe for your patience.
45
46 Berger, Joel
47
48 The inspiration for the sixth model (MCE::Step) came from reading a blog
49 by Joel, "Writing Non-Blocking Applications with Mojolicious: Part 3":
50
51 https://blogs.perl.org/users/joel_berger/2014/01/
52 writing-non-blocking-applications-with-mojolicious-part-3.html
53
54 Bouras, George
55
56 For reporting sockets failing under the Windows environment due to a
57 period of inactivity. An example is when workers execute an external
58 command taking beyond 4 minutes to complete. On returning, the sockets
59 have gone stale with a CPU core going 100% at that point.
60
61 The MCE 1.601 release is attributed to the many use cases I received
62 from George.
63
64 Cantrell, David (DrHyde on perlmonks.org)
65
66 For reporting on MCE hanging with cpan -t MCE under his environment. Also,
67 thank you for providing me a guest account via ssh. I was able to reproduce
68 the issue. Thus, added writeable check and croak if tmp dir is not writeable.
69 One cannot assume that /dev/shm is writeable by everyone. :)
70
71 Collet, Eric
72
73 For the inspiration to the tre-agrep binary.
74
75 I first learned of tre-agrep from Eric. He emailed me running times for a
76 couple scenarios utilizing a single core. Thus increasing the need to have
77 a MCE-driven wrapper to run tre-agrep in parallel.
78
79 The bin/mce_grep script was created to show folks how one may go about
80 chunking data between Perl and an external C binary efficiently without
81 invoking the shell. The script supports grep, egrep, fgrep, agrep and
82 tre-agrep.
83
84 Eldai, Hisham
85
86 While working on the biofasta examples, Hisham introduced me to hg19.fa.
87 A couple records inside hg19 (driven by $/ = "\n>") are 250 MiB in size.
88 Memory optimizations in v1.600 came about by running against hg19.fa
89 hundreds of times.
90
91 ###############################################################################
92 # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
93 ###############################################################################
94
95 Farrell, David
96
97 David wrote a nice article about MCE (love the picture). What stood out was
98 his example. I never thought about using mce_grep { ... } <$fileHandle>
99 without the _f (mce_grep_f). At the time, updated MCE::Grep and MCE::Map
100 to be fully aware of wantarray.
101
102 The blog "Make your code run faster with Perl's secret turbo module" is
103 located at:
104
105 https://www.perl.com/article/61/2014/1/21/
106 Make-your-code-run-faster-with-Perl-s-secret-turbo-module/
107
108 Gasper, Felipe
109
110 For test scripts found in IO::SigGuard. I was able to get MCE::Mutex
111 to fail with similar testing. The result from testing is that MCE too,
112 must check for EINTR during sysread and syswrite operations. The same
113 is true for MCE::Shared.
114
115 Golden, David
116
117 For the Hash::Ordered module. I learned a lot from this *GEM* on CPAN.
118 Particularly, the use of overload. I didn't know that was possible.
119
120 For the time during collaborative efforts in making Hash::Ordered faster.
121 I learned some new tricks along the way.
122
123 Thank you for alarming me of the memory leaks with the on-demand hash
124 dereferencing inside MCE::Shared::Ordhash. MCE::Shared::Object was also
125 impacted. I'd never forget that dreadful day.
126
127 MCE::Shared::{ Array, Hash, and Ordhash } are inspired by Hash::Ordered.
128 In particular, the clone and iterator methods. Also, keys and values on
129 taking arguments. I confess for never getting used to the "as_list"
130 method name, initially. :)
131
132 Gorwits, Oliver
133
134 For reporting on a couple issues with MCE along with a solution for each.
135
136 Pointed me in the right direction for improving the logic for the die
137 handler. Basically, eval { die 'this should not cause MCE to die' };
138 This has reached 100% (all cases, threads and processes) in MCE 1.600.
139
140 Halpenny, Shawn
141
142 For reporting an issue (bug RT#94706) with signal handling in MCE::Signal.
143 Also, thank you for tip on getpgrp.
144
145 Hedden, Jerry
146
147 From reading threads::shared. In particular the make_shared function.
148 That was helpful in getting MCE::Shared working.
149
150 Hluchan, Yary
151
152 For providing a benchmark script which I used to compare IPC performance
153 between 1.6 and 1.7. For many kind words near the end of the development
154 cycle for MCE 1.7 and MCE::Shared.
155
156 ###############################################################################
157 # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
158 ###############################################################################
159
160 Jacobsen, Dana
161
162 For reporting an issue with get_ncpu on AIX. In addition, suggesting
163 various solutions. Also, thank you for your time with Sandboxing with
164 Perl + MCE + Inline::C at https://github.com/marioroy/mce-sandbox/.
165
166 Karasik, Dmitry
167
168 For bug (RT#102802) and elegant solution for the die handler inside
169 MCE::Core::Worker and MCE::Signal.
170
171 Kharchenko, Oleksandr
172
173 Someone once asked Oleksandr why not use MCE. Oleksandr created 2 modules
174 on CPAN, Parallel::DataPipe and Parallel::parallel_map. I used his example
175 heavily in comparing IPC between pipe-driven and socket-driven. Not pleased
176 with MCE's performance with chunk_size => 1, although 2 was as fast if not
177 faster, I tried various things to include IO::Select. Finally, realized
178 increasing the number of data channels was all that was needed (v1.500).
179
180 Kulow, Stephan
181
182 For making the OpenSUSE package for MCE.
183 https://build.opensuse.org/package/show/devel:languages:perl:CPAN-M/perl-MCE
184
185 Mabon, Philip
186
187 For reporting on a couple issues with MCE in particular bug RT#92627.
188
189 McKeown, Benjamin
190
191 For reporting several edge cases with MCE in particular bugs RT#105557,
192 RT#105559, and RT#111780.
193
194 Mertens, David
195
196 I am thankful for the tip on making PDL + MCE a possibility. Workers were
197 crashing during exiting. Adding "sub PDL::CLONE_SKIP { 1 }" inside MCE.pm
198 solved this issue entirely. Also, for tips on effectively using PDL in
199 his example at https://gist.github.com/run4flat/4942132.
200
201 Barrier synchronization in MCE came from reading parallel_sync in David's
202 Perl module; PDL::Parallel::threads::SIMD.
203
204 Mughal, Zakariyya
205
206 First, Demian Riccardi introduced me to Devel::IPerl by Zakariyya. The die
207 handlers in MCE broke the user experience. It seems that looking at $^S
208 (for Perl state) is not enough. MCE v1.600 fixes this once and for all.
209 The answer is in perl-5.20.1/cpan/CGI/lib/CGI/Carp.pm (Carp::longmess).
210
211 ###############################################################################
212 # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
213 ###############################################################################
214
215 Ogulin, Joseph
216
217 For providing the initial perl-MCE.spec file.
218 For the cool remarks at work about MCE being really big :)
219
220 Riccardi, Demian
221
222 I received a message from Demian asking for mce_map. His message requested
223 ( my @a = mce_map { ... } 1..100 ). Five models were introduced in v1.500;
224 MCE::Flow, MCE::Grep, MCE::Loop, MCE::Map, and MCE::Stream.
225
226 Find capability were limited in MCE::Shared 1.001. For example, not able to
227 mix :AND(s) and :OR(s) together. One day, I clicked on recent distributions
228 on metacpan.org and came across HackaMol::Roles::SelectionRole. The trick
229 is to eval the query string together with grep or map. Thank you Demian.
230
231 Rowe, Jillian
232
233 For reporting on IO::File handles not working with MCE. Fixed in 1.515.
234 For believing in possibilities beyond text-book thinking.
235
236 Roy, Sylvia (spouse)
237
238 For supporting me from v1.0, especially during later 1.6 revisions and 1.7.
239 But not to forget the journey during 1.8. So little was known on the hurdles
240 to cross. It's all over now with MCE 1.827 and MCE::Shared 1.823.
241
242 Šabata, Petr
243
244 For initial package submission at Red Hat.
245 https://bugzilla.redhat.com/show_bug.cgi?id=1162531
246
247 Sasser, Tom
248
249 For reporting on bin/mce_grep failing with Perl 5.8.x.
250
251 Schlichting, Florian
252
253 For making a Debian package for MCE: libmce-perl
254
255 Shen, Wei
256
257 For the BioUtil CPAN module. BioUtil::Seq::FastaReader inspired me to
258 create parallel examples (biofasta folder) in MCE 1.600. We tried to
259 make FastaReader in BioUtil::Seq even faster.
260
261 ###############################################################################
262 # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * # * #
263 ###############################################################################
264
265 Sjøgren, Adam
266
267 For reporting on running /usr/bin/time mce_script.pl and pressing Ctrl-C
268 failing. The default behavior in MCE::Signal is to call setpgrp. MCE 1.403
269 adds a new option to MCE::Signal to not call setpgrp. This also gives the
270 developer finer control as to where setpgrp should be called, if ever
271 needed, such as in the child process and not the manager process.
272
273 use MCE::Signal qw(-no_setpgrp);
274 use MCE;
275
276 Smith, Marcus
277
278 For reporting a problem with MCE including a sample script demonstrating
279 MCE not 100% UTF-8 safe. All this time, totally overlooked the fact that
280 the length function deals in logical characters, not physical bytes. :)
281
282 Timmermans, Leon
283
284 The inspiration for not having to backslash variables for MCE::Shared came
285 from reading Const::Fast: e.g. mce_share my @array => ( list );
286
287 Thalhammer, Jeffrey Ryan
288
289 For submitting a feature request for lazy arrays support.
290
291 Although a lazy array cannot be used directly for input_data, support for
292 iterator objects was added to MCE 1.505; e.g. input_data => $iterator_ref;
293 This enabled support for lazy arrays using closures.
294
295 I am thankful for the small code snippet highlighting a use-case of MCE.
296 I would have failed initially without it. Your example was placed under
297 the "SYNTAX for INPUT_DATA" section in MCE::Core for others to see on
298 how to process output from Path::Iterator::Rule in parallel.
299
300 Warm regards,
301 Mario
302