Debian Codegraph @

Changes

raw
1
2 Revision history for Perl module MCE.
3
4 1.902 Tue Sep 09 15:00:00 EST 2025
5
6 * Add support for Iterator:: classes.
7
8 1.901 Thu Jan 02 17:30:00 EST 2025
9
10 * Add MCE::Core package for future development.
11
12 1.900 Mon Sep 09 20:30:00 EST 2024
13
14 * Improve MCE::Child exiting when signaled.
15
16 1.899 Fri Sep 06 00:15:00 EST 2024
17
18 * Fix for MCE::Child and MCE::Channel signal anomaly #24.
19 Thank you, @exodist.
20
21 1.898 Wed Aug 21 15:30:00 EST 2024
22
23 * Fix for MCE::Child #22, Can't call method "len" on an undefined value
24 during global destruction. Thanks, @exodist.
25
26 1.897 Wed Jun 19 22:00:00 EST 2024
27
28 * In scalar context, the init function in MCE Child and models Flow, Grep,
29 Loop, Map, Step, and Stream returns a guard to call finish automatically
30 upon leaving the { scope i.e. omitting finish }.
31
32 * Add out_iter_callback to MCE::Candy.
33
34 1.896 Tue Jun 11 16:00:00 EST 2024
35
36 * Weaken internal core MCE reference to reap workers automatically
37 upon leaving the { scope i.e. omitting shutdown }. Note: No change
38 to MCE models Flow, Grep, Loop, Map, Step, and Stream. Call finish
39 explicitly to reap workers.
40
41 This resolves the case when omitting calling either one of $mce->run(),
42 $mce->run(1), or $mce->shutdown() inside a scope, causing workers to
43 linger around until completion of the script.
44
45 1.895 Sun Jun 09 01:00:00 EST 2024
46
47 * Revert back to calling CORE::rand() to set the internal seed.
48 MCE and MCE::Child cannot assume the srand or setter function used
49 by the application for predictability.
50 1. https://perlmonks.org/?node_id=11159834
51 2. https://perlmonks.org/?node_id=11159827
52
53 * Add class methods MCE->seed and MCE::Child->seed to retrieve the seed.
54
55 1.894 Sun Jun 09 15:30:00 EST 2024
56
57 * Improve support for PDL.
58
59 1.893 Sat Jun 08 08:30:00 EST 2024
60
61 * Preserve functionality for older Perl, non-threads.
62
63 1.892 Sat Jun 08 08:00:00 EST 2024
64
65 * Remove check if spinning threads i.e. use_threads.
66 Predictable output matches non-threads for CORE,
67 Math::Prime::Util, and Math::Random::MT::Auto.
68 https://perlmonks.org/?node_id=11159834
69
70 1.891 Fri Jun 06 04:00:00 EST 2024
71
72 * Apply workaround for PDL::srand in MCE and MCE::Child. Thank you, PerlMonks.
73 https://www.perlmonks.org/?node_id=11159773
74 * Add PDL::srand (v2.062 ~ v2.089) and PDL::srandom (v2.089_01+).
75 * Call CORE::srand inside child processes, only.
76
77 1.890 Fri May 24 19:00:00 EST 2024
78
79 * Improve reaping completed workers in MCE::Child.
80 * Fix the _sprintf function, failing multiple arguments.
81
82 1.889 Wed Sep 13 18:00:00 EST 2023
83
84 * Add Android support. Thank you, Dimitrios Kechagias.
85 * Revert defer signal-handling in MCE::Channel (send2 method).
86 * Improve mutex synchronize (aka enter) with guard capability.
87 Thank you, José Joaquín Atria.
88 * Fix mutex re-entrant lock on the Windows platform.
89 * Add mutex guard_lock method.
90
91 1.888 Wed Jun 21 17:00:00 EST 2023
92
93 * Fix typos caught by lintian. Thank you, Étienne Mollier.
94
95 1.887 Fri Jun 09 08:00:00 EST 2023
96
97 * Fix typo in Queue dequeue_timed documentation.
98 Thank you, Łukasz Strzelecki.
99
100 1.886 Tue Jun 06 12:00:00 EST 2023
101
102 * Added dequeue_timed method to MCE::Queue.
103 * Fixed taint mode in MCE->printf and _sprintf.
104 * Improved reliability on the Windows platform.
105
106 1.885 Tue May 30 20:00:00 EST 2023
107
108 * Improved reliability on the Windows platform.
109
110 1.884 Thu Jan 05 10:00:00 EST 2023
111
112 * Disabled non-blocking dequeue_nb and recv_nb tests on the Windows platform.
113 Reason: Author cannot reproduce failing tests reported by CPAN Tester aero.
114 Copied nb tests to xt folder: nonblocking_channel.t and nonblocking_queue.t
115
116 1.883 Tue Jan 03 20:00:00 EST 2023
117
118 * Fix typo in MCE::Channel::SimpleFast documentation.
119 * Improve 05_mce_child.t test.
120
121 1.882 Fri Dec 02 21:00:00 EST 2022
122
123 * Added ABRT to the list of signals to trap in MCE::Signal.
124 * Added a guard to MCE::Core::Worker for checking if exited prematurely.
125 * Added init_relay and use_threads import options to MCE and MCE Models.
126 * Separated input mutexes from the rest of IPC for lesser latency.
127
128 * Auto-detect if init_relay is defined and set chunk_size to 1 in
129 MCE::Grep, MCE::Map, and MCE::Stream.
130 * Update the import function in MCE models, detecting if the caller
131 is another MCE module, to not export model functions.
132
133 * Update the error status if MCE::Child died due to receiving a signal.
134 * Improved reaping in MCE::Child, before creating a new child.
135 * Improved the timeout handler in MCE::Child and MCE::Mutex::Channel.
136 * Fixed private functions _quit and _trap not setting the return value.
137
138 1.881 Thu Oct 13 23:45:00 EST 2022
139
140 * Improved the private _parse_chunk_size function. For better
141 utilization of CPU cores in MCE::Grep, MCE::Map, and MCE::Stream,
142 processing small input sizes.
143
144 Previously, chunk_size => 'auto' equals 2 minimally.
145 Starting with MCE v1.881, 'auto' equals 1 minimally.
146
147 1.880 Mon Oct 10 04:00:00 EST 2022
148
149 * Improved reliability on the Windows platform.
150 * Improved MCE::Mutex::Channel::timedwait on the Windows platform.
151 * Improved MCE::Mutex::Channel performance on UNIX platforms.
152 * Resolved edge case in MCE::Child reaching deadlock.
153
154 1.879 Tue May 24 05:00:00 EST 2022
155
156 * Replace http with https in documentation and meta files.
157 * Call PDL::set_autopthread_targ(1); disables PDL auto-threading.
158
159 1.878 Sun Feb 20 06:45:00 EST 2022
160
161 * Fix for the fast channel implementations.
162 Thank you, twata for the test report.
163
164 1.877 Sun Feb 20 02:30:00 EST 2022
165
166 * Improved suppressing the PDL CLONE warning. Piddles should not be
167 naively copied into new threads.
168
169 * Added fast channel implementations optimized for non-Unicode strings.
170 The main difference is that these lack freeze-thaw serialization.
171 MCE::Channel::MutexFast, MCE::Channel::SimpleFast, and
172 MCE::Channel::ThreadsFast.
173
174 1.876 Thu Dec 02 18:00:00 EST 2021
175
176 * Allow percentage above 100% for max_workers in MCE.
177
178 * MCE::Child update.
179 Improved _ordhash.
180 Renamed JOINED to REAPED in code for better clarity.
181 Specify a percentage for max_workers.
182 Added t/05_mce_child_max_workers.t
183
184 1.875 Tue Nov 16 04:00:00 EST 2021
185
186 * Specify a percentage for max_workers.
187 Thank you, kcott@PerlMonks (Ken) for the idea.
188 https://www.perlmonks.org/?node_id=11134439
189
190 * Added t/03_max_workers.t
191
192 1.874 Tue Aug 18 16:00:00 EST 2020
193
194 * Improved MCE->yield when used together with MCE::Relay.
195
196 1.873 Sat Aug 01 16:00:00 EST 2020
197
198 * Removed unused variable in MCE::Mutex::Channel.
199
200 * Fixed typo in prior change log.
201 Replaced "completed" with "feature complete".
202
203 1.872 Sun Jun 14 22:30:00 EST 2020
204
205 * Added open to required dependencies.
206 * Set default encodings on standard filehandles in tests using UTF-8.
207
208 * Bumped minimal Perl version to 5.8.1.
209 * Bumped MCE version to 1.872 to align with MCE::Shared.
210
211 * The MCE project is feature complete.
212
213 1.868 Sun May 10 22:00:00 EST 2020
214
215 * Completed threads-like detach capability in MCE::Child.
216
217 * Resolved MCE::Channel failing when calling dequeue multiple times
218 on an ended channel.
219
220 * MCE->say, MCE->print, and MCE->printf now return 1.
221
222 1.867 Sun May 03 18:00:00 EST 2020
223
224 * Bug fix for UTF-8 issues during inter-process communication.
225 This update required undoing optimizations specific to scalar args.
226 Essentially, IPC involves serialization for everything going forward.
227 Install Sereal::Encoder and Sereal::Decoder for better performance
228 in Perl 5.8.8+.
229
230 * MCE options flush_stdout, flush_stderr, and flush_file now default to
231 enabled for the MCE->print, MCE->printf, and MCE->say output routines.
232
233 * Improved MCE::Child with threads-like detach capability. See POD.
234
235 * Improved IPC in MCE::Queue with permanent fast-like dequeue including
236 dequeue_nb. Going forward, the fast and barrier options are silently
237 ignored if specified (i.e. no-op).
238
239 * Improved IPC performance on Linux.
240
241 1.866 Sat Feb 08 21:00:00 EST 2020
242
243 * Bug fix for restart_worker, race condition introduced in 1.863.
244 Thank you, Oliver Gorwits for reporting the issue.
245
246 1.865 Wed Dec 25 18:00:00 EST 2019
247
248 * Bug fix for two-way IPC stalling on Windows in MCE::Channel::Threads.
249 See https://www.perlmonks.org/?node_id=11110612
250
251 * Remove the check for MSWin32 in MCE::Channel::Mutex. MCE::Channel since
252 the 1st release silently defaults to MCE::Channel::Threads on Windows.
253
254 * Small tweak to MCE::Signal.
255
256 1.864 Wed Dec 04 13:00:00 EST 2019
257
258 * Bug fix in MCE::Signal - Shared manager not exiting, introduced in 1.863.
259
260 * Use monotonic clock if available in MCE->yield and MCE::Child->yield.
261 See https://www.perlmonks.org/?node_id=11109673
262
263 1.863 Sun Nov 26 20:00:00 EST 2019
264
265 * On Cygwin, silently use Mutex in MCE::Channel when Threads is specified
266 for better performance.
267
268 * New defer capability in MCE::Signal. This applies to MCE::Shared 1.863.
269 See POD section labled "DEFER SIGNAL" in MCE::Signal.
270
271 * Reverted $child->exit back to sending the SIGQUIT signal in MCE::Child
272 now that MCE::Shared::Server 1.863 defers signal during IPC.
273
274 * Improved reliability for spawning MCE and MCE::Child inside threads
275 including nested parallelization, made possible using a global lock
276 $MCE::_GMUTEX.
277
278 * Updated signal handling in mce-examples/framebuffer on GitHub.
279
280 1.862 Wed Sep 18 22:00:00 EST 2019
281
282 * Hotfix for 1.861.
283
284 1.861 Wed Sep 18 08:30:00 EST 2019
285
286 * Hotfix for 1.849-1.860. The edge cases regarding signal handling have
287 been finally resolved for MCE::Child. Thank you, Richard Kelsch for the
288 use case involving ctrl-c. See mce-examples/framebuffer on GitHub.
289
290 1.860 Sun Sep 15 20:00:00 EST 2019
291
292 * Signal-handling update release.
293
294 * Localized input and output record separators in MCE::Channel.
295 * IPC safety in MCE::Child during SIGINT and SIGTERM.
296
297 * Method $child->exit in MCE::Child now sends the SIGINT signal
298 for extra reliability with MCE::Shared (previously SIGQUIT).
299
300 1.850 Mon Sep 09 12:30:00 EST 2019
301
302 * Bumped version to coincide with the stable MCE::Shared 1.850 release.
303
304 1.849 Sun Sep 08 23:30:00 EST 2019
305
306 * Fixed edge case in MCE::Child when reaping inside a signal handler.
307 * Added list_pids class method to MCE::Child.
308
309 1.848 Tue Sep 03 23:30:00 EST 2019
310
311 * Improved IO::All::{ File, Pipe, STDIO } output via MCE->print($io, ...),
312 printf, and say. This resolves a bug introduced in 1.845 when using
313 App::Cmd::Tester to capture output.
314
315 1.847 Mon Sep 02 23:30:00 EST 2019
316
317 * Obsolete RedHat MCE-1.840-Sereal-deps.patch file. This patch file
318 is no longer needed and finally resolved with this release.
319
320 * PDL random numbers now unique between threads. Thank you, PerlMonks (vr).
321 See https://www.perlmonks.org/?node_id=1214439.
322
323 * Replaced "PF_UNIX" with "AF_UNIX" in MCE::Util.
324
325 1.846 Mon Aug 26 21:30:00 EST 2019
326
327 * Fixed code tags in documentation. Thank you, Mohammad S Anwar.
328
329 1.845 Sun Aug 25 22:00:00 EST 2019
330
331 * Croak if is_joinable, is_running, or join is called by a non-managed
332 process in MCE::Child. Added LIMITATION section to the documentation.
333
334 * Improved is_joinable, is_running, list_joinable, and list_running
335 in MCE::Child. Thank you, Philippe Baumgart.
336
337 * Added example (Consumer requests item) to MCE::Channel documentation.
338
339 * Support the task_end option regardless if user_tasks is specified.
340
341 * Support IO::All::{ File, Pipe, STDIO } for input data including
342 output via MCE->print($io, ...), printf, and say.
343
344 * Support gather => MCE::Candy::out_iter_fh($io) using MCE::Candy.
345
346 1.844 Wed Aug 14 21:30:00 EST 2019
347
348 * Resolved MCE stalling when specifying max_retries with init_relay.
349 Ditto for loop_timeout with init_relay on UNIX platforms.
350 Thank you, Chris Denley.
351
352 * Enhanced loop_timeout to handle workers dieing uncontrollably from
353 any user_tasks (i.e. task_id >= 0). Previously, only task_id == 0.
354
355 * Improved IPC on the Windows platform for edge case when a worker is
356 awaiting input while the manager process is restarting a worker.
357
358 * MCE, MCE::Child workers exit immediately upon receiving a SIGSEGV signal.
359 This safeguards IPC from stalling inside the manager process.
360
361 * Enhanced the _wait_one private function in MCE::Child.
362
363 * Removed Prima from the list for auto-enabling the posix_exit option.
364 Prima (since 1.52) is parallel safe during global cleanup.
365
366 * Reached 100% Pod coverage.
367
368 1.843 Tue Jul 23 22:30:00 EST 2019
369
370 * Updated results in MCE::Child (Parallel::ForkManager-like demonstration).
371
372 * Completed missing interrupt signal-safety for the non-blocking methods
373 in MCE::Channel::Mutex and MCE::Channel::Threads.
374
375 1.842 Sun Jul 21 19:00:00 EST 2019
376
377 * Fixed race condition abnormalities in MCE::Child.
378 * Added Parallel::ForkManager-like demonstration to MCE::Child.
379
380 1.841 Sun Jul 07 23:30:00 EST 2019
381
382 * Disabled t/04_channel_threads testing on Unix platforms for Perl less than
383 5.10.1. Basically, the MCE::Channel::Threads implementation is not supported
384 on older Perls unless the OS vendor applied upstream patches (i.e. works on
385 RedHat/CentOS 5.x running Perl 5.8.x).
386
387 * Added LIMITATIONS section to MCE::Channel::Threads.
388
389 1.840 Sun Jul 07 05:00:00 EST 2019
390
391 * Update MCE::Channel POD documentation.
392
393 1.839 Sun Jul 07 04:30:00 EST 2019
394
395 [NEW FEATURES]
396
397 * Added MCE::Mutex::Channel2 providing two locks using a single channel.
398 The secondary lock is accessible by calling methods with the '2' suffix.
399 E.g. primary mutex ->lock, ->unlock; secondary mutex ->lock2, ->unlock2
400
401 * Added MCE::Channel providing queue-like and two-way communication
402 supporting threads and processes.
403
404 * Added MCE::Child and compatibility with Perl 5.8. MCE::Child is based on
405 MCE::Hobo, but using MCE::Channel for data retrieval without involving
406 a shared-manager process.
407
408 * Added MCE::Channel examples { channel1.pl and channel2.pl } using
409 threads and MCE::Child respectively.
410 https://github.com/marioroy/mce-examples/tree/master/chameneos
411
412 [ENHANCEMENTS]
413
414 * IPC update; removed unnecessary overhead including private methods
415 _sysseek and _syswrite from MCE::Util (no longer needed).
416
417 * Improved MCE->do, now callable by workers and the manager process.
418
419 * Updated MCE::{ Flow, Grep, Loop, Map, Step, and Stream } documentation
420 on passing an array reference versus a list for deeply input data.
421
422 * Updated and re-organized the top-level MCE documentation, particularly
423 improved clarity for the 'MCE Models' section.
424
425 * Removed MANIFEST.SKIP.
426
427 1.838 Wed Jan 23 08:30:00 EST 2019
428
429 * IPC update, raising reliability across multiple platforms.
430 * Improved hack for the Windows platform for nested MCE sessions.
431 * Added _sysread, _sysseek, _syswrite, and _nonblocking to MCE::Util.
432 * Added barrier option to MCE::Queue allowing one to disable.
433
434 1.837 Sat Aug 25 13:00:00 EST 2018
435
436 * Seeds the Math::Random::MT::Auto generator automatically when present
437 for non-threads, similarly to Math::Random and Math::Prime::Util, to
438 avoid child processes sharing the same seed value as the parent and
439 each other. The new seed is computed using the current seed.
440
441 1.836 Mon Jun 25 08:30:00 EST 2018
442
443 * Moved validation code from MCE::Util to MCE::Core::Validation.
444 * Applied small optimizations.
445
446 1.835 Tue Mar 13 15:00:00 EST 2018
447
448 * Added gather and relay demonstrations to MCE::Relay.
449 * Load IO::Handle for extra stability, preventing workers loading uniquely.
450
451 * Load Net::HTTP and Net::HTTPS before spawning if present LWP::UserAgent.
452 See https://www.perlmonks.org/?node_id=1199760
453 and https://www.perlmonks.org/?node_id=1199891.
454
455 1.834 Tue Jan 23 08:00:00 EST 2018
456
457 * Improved Queue await and dequeue performance on the Windows platform.
458
459 * Added chameneos-redux parallel demonstrations on GitHub:
460 https://github.com/marioroy/mce-examples/tree/master/chameneos
461
462 1.833 Thu Dec 28 16:00:00 EST 2017
463
464 * Fixed bug with sequence (#10), broken in 1.832. Thank you, @djerius.
465
466 1.832 Tue Nov 21 15:30:00 EST 2017
467
468 * Added LWP::UserAgent to list for enabling posix_exit.
469 * Improved number-sequence generation for big integers.
470 * Improved wantarray support in MCE::Mutex synchronize.
471 * Removed limit check on chunk_size option.
472
473 1.831 Sun Oct 08 20:30:00 EST 2017
474
475 * Added STFL (Terminal UI) to list for enabling posix_exit.
476 See https://www.perlmonks.org/?node_id=1200923.
477
478 * Math::Prime::Util random numbers now unique between MCE workers.
479 See https://www.perlmonks.org/?node_id=1200960.
480
481 1.830 Tue Sep 12 17:00:00 EST 2017
482
483 [BUG FIXES]
484
485 * Fixed MCE and MCE::Relay stalling when setting the input record separator.
486 See https://www.perlmonks.org/?node_id=1196701. Thank you, JediWombat.
487
488 * Fixed bug with dequeue_nb in MCE::Queue (#8). Thank you, @bokutin.
489 * Fixed signal handler (#9). Thank you, @chrisdenley.
490
491 [ENHANCEMENTS]
492
493 * Added Coro and Win32::GUI to list for enabling posix_exit.
494 * Added support for Haiku to get_ncpu in MCE::Util.
495 * Allow gathering to a shared array in MCE::Candy.
496
497 * Improved CPU count on the AIX platform in MCE::Util.
498 * Improved signal handling, including nested parallel-sessions.
499 * Improved running MCE::Hobo inside MCE workers.
500 * Improved running MCE with PDL.
501
502 * Refactored logic for MCE->do, bi-directional callback feature.
503 * Preserve lexical type for numbers during IPC: MCE->do and MCE::Queue.
504
505 * No longer loads threads on the Windows platform in MCE::Signal.
506 This enables MCE::Hobo 1.827 to spin faster, including lesser
507 memory consumption. Threads isn't required to run MCE::Hobo.
508
509 * Removed extra white-space from POD documentation.
510 * Validated MCE on SmartOS.
511
512 1.829 Wed May 03 03:00:00 EST 2017
513
514 * Reduced memory consumption.
515
516 1.828 Fri Apr 28 16:00:00 EST 2017
517
518 [BUG FIXES]
519
520 * Do not enable barrier mode for Queue on the Windows platform.
521 * Fixed MCE::Mutex::Flock, tmp_file missing script name in path.
522
523 [ENHANCEMENTS]
524
525 * Added Curses and Prima to list for enabling the posix_exit option.
526 * Allow a hash as input_data: Core API, MCE::{ Flow, Loop, Step }.
527 * Improved API documentation on MCE models with more synopsis.
528 * Enhanced IPC and signal handling. Reduced memory consumption.
529 * Make tmp_dir on demand in MCE::Signal. Ditto for sess_dir in MCE.
530 * Load Fcntl, File::Path, Symbol on demand.
531
532 1.827 Wed Apr 05 01:30:00 EST 2017
533
534 * Do not enable barrier mode for Queue if constructed inside a thread
535 or by MCE models (e.g. Step, Stream). Ditto for fast => 1 option.
536
537 * Updated MCE to not croak when running Perl in taint mode via perl -T.
538 Failing -T was MCE::Core::Input::{ Generator, Sequence }, MCE::Signal,
539 and MCE::Util.
540
541 * Added Denis Fateyev, Felipe Gasper, and Paul Howarth to Credits.
542
543 1.826 Sun Apr 02 23:00:00 EST 2017
544
545 * Is now safe running MCE with the Wx GUI toolkit (wxWidgets).
546
547 1.825 Sun Apr 02 07:00:00 EST 2017
548
549 * Updated MCE::Queue. The following provides a comparison for the
550 enhancements made regarding IPC during 1.822 through 1.825, in order
551 to run on machines having "many" cores. I ran with 12, 96, and 192
552 workers on an 8 core box.
553
554 A. MCE queue, dequeue 100k items.
555
556 my $Q = MCE::Queue->new();
557
558 $Q->enqueue( 1 .. 100000 );
559 $Q->end(); MCE 1.608 $Q->enqueue((undef) x 12, 96, or 192);
560
561 MCE->new(
562 max_workers => 12, 96, or 192,
563 user_func => sub {
564 while ( defined ( my $item = $Q->dequeue ) ) {
565 ;
566 }
567 }
568 )->run();
569
570 MCE 1.608: 12 ~ 1.799 secs 96 ~ 8.702 secs 192 ~ 18.083 secs
571 MCE 1.821: ~ 1.450 secs ~ 5.231 secs ~ 8.102 secs
572 MCE 1.825: ~ 0.976 secs ~ 1.067 secs ~ 1.509 secs
573
574 B. Input file containing 250k lines ( 300 MiB ).
575
576 MCE->new(
577 input_data => "/dev/shm/file_250k.txt",
578 max_workers => 12, 96, or 192,
579 chunk_size => 1,
580 use_slurpio => 1,
581 user_func => sub { }
582 )->run();
583
584 MCE 1.608: 12 ~ 3.605 secs 96 ~ 8.074 secs 192 ~ 8.465 secs
585 MCE 1.821: ~ 3.613 secs ~ 8.058 secs ~ 8.607 secs
586 MCE 1.825: ~ 3.567 secs ~ 2.601 secs ~ 3.199 secs
587
588 C. Sequence of numbers from 1 to 200k.
589
590 MCE->new(
591 sequence => [ 1, 200000 ],
592 max_workers => 12, 96, or 192,
593 chunk_size => 1,
594 user_func => sub { }
595 )->run();
596
597 MCE 1.608: 12 ~ 1.236 secs 96 ~ 2.922 secs 192 ~ 3.113 secs
598 MCE 1.821: ~ 1.227 secs ~ 2.915 secs ~ 3.106 secs
599 MCE 1.825: ~ 1.250 secs ~ 1.203 secs ~ 1.581 secs
600
601 * Results were captured on a fast 8 core system running CentOS Linux 7.
602 The thing to take from this is that running many workers "no longer"
603 results in up to 5.6x penalty regarding IPC.
604
605 1.824 Sat Apr 01 01:00:00 EST 2017
606
607 * Completed validation for running MCE on a box having 100+ cores.
608
609 * Tuned the number of data-channels for IPC. Set upper limit in
610 MCE::Core::Input::{ Handle and Sequence } to not impact the
611 OS kernel. The result is better performance, yet graceful.
612
613 1.823 Fri Mar 31 19:30:00 EST 2017
614
615 * Calibrated the number of data-channels for IPC.
616
617 1.822 Fri Mar 31 11:00:00 EST 2017
618
619 * Check for EINTR during sysread and syswrite.
620 * Improved reliability when running nested MCE sessions.
621 * Updated MCE::Mutex with Channel and Fcntl implementations.
622
623 * Completed validation for using MCE with 200+ cores.
624
625 1.821 Sun Mar 19 04:00:00 EST 2017
626
627 * Improved reliability when running MCE with threads.
628
629 * Added parallel Net::Pcap and Ping demonstrations on GitHub:
630 https://github.com/marioroy/mce-examples/tree/master/network
631
632 * Optimized 'dequeue' method in MCE::Queue.
633 * Optimized 'synchronize' method in MCE::Mutex.
634
635 1.820 Thu Mar 09 02:00:00 EST 2017
636
637 * Improved reliability when running MCE inside an eval block.
638
639 1.819 Fri Mar 03 23:00:00 EST 2017
640
641 * Fixed issue with localizing AUTOFLUSH variable before autoflush handles.
642 Thank you, Charles Hendry for raising the issue.
643
644 1.818 Wed Mar 01 22:00:00 EST 2017
645
646 * Updated bin/mce_grep for determining chunk level. Ditto for chunk size.
647 Fixed an issue for not seeing STDERR output with '--chunk-level=file'.
648 Added support for zgrep, zegrep, and zfgrep. Thank you, Jeff Rouse.
649
650 * Changed Sereal to Sereal::Decoder and Sereal::Encoder in recommends
651 section inside Makefile and META files.
652
653 * Refactored MCE::Queue. Merged local and manager code base into one.
654 Removed t/04_norm_que_local.t and t/04_prio_que_local.t.
655
656 * Added 'end' method to MCE::Queue.
657 Updated documentation on dequeue and pending.
658
659 1.817 Sat Feb 25 02:00:00 EST 2017
660
661 * Improved bin/mce_grep. When -r is specified and zero paths are given,
662 start recursively in the current directory versus await data from STDIN.
663 Set chunk-level accordingly to list mode.
664
665 1.816 Fri Feb 24 19:00:00 EST 2017
666
667 * Revised the description on max_retries in MCE::Core.pod.
668
669 1.815 Fri Feb 24 01:00:00 EST 2017
670
671 * Fixed divide-by-zero error in MCE->yield.
672
673 * Refactored code for the interval option by moving the code to the manager
674 process. This allows the manager process to accomodate the next available
675 worker and ready to run. Previoulsy, a worker taking a long time resulted
676 in empty time slots. Thank you, Philippe Baumgart for your patience.
677
678 * Revised the description on posix_exit in MCE::Core.pod.
679
680 1.814 Mon Feb 20 05:30:00 EST 2017
681
682 * Enhanced the progress option for use with MCE->process.
683 Updated demonstrations in MCE::Core.pod.
684
685 1.813 Thu Feb 16 02:30:00 EST 2017
686
687 * Last minute request by Philippe Baumgart (reminder and long overdue).
688 Added progress option, a code block for receiving info on progress made.
689 See MCE::Core.pod for demonstrations accommodating all input data types.
690
691 1.812 Tue Feb 14 17:00:00 EST 2017
692
693 * Bumped minimum requirement for Sereal to 3.015 when available.
694 Added check ensuring matching version for Encoder and Decoder.
695
696 1.811 Mon Feb 13 23:30:00 EST 2017
697
698 * Fixed bug in MCE::Queue (dequeue_nb) when queue has zero items.
699 * Applied small optimization in MCE::Core::Input::Sequence and Generator.
700
701 * Added cross-platform template to MCE::Examples for making an executable.
702 * Removed signal handling for XCPU and XFSZ from MCE::Signal.
703
704 * Imply posix_exit => 1 if Gearman::XS or Gearman::Util is present during
705 MCE construction.
706
707 * Added MCE + Gearman demonstrations (xs and non-xs) on GitHub:
708 https://github.com/marioroy/mce-examples/tree/master/gearman_xs
709 https://github.com/marioroy/mce-examples/tree/master/gearman
710
711 * Changed kilobytes and megabytes to kibiBytes (KiB) and mebiBytes (MiB)
712 respectively inside the documentation.
713
714 1.810 Fri Dec 09 23:30:00 EST 2016
715
716 * Updated check for IO handle allowed. This allows $gather_fh = *STDOUT{IO},
717 construction in Perl <= 5.10.1. Thank you, Qiang Wang.
718
719 1.809 Wed Nov 23 16:00:00 EST 2016
720
721 * Bug fixes for running MCE inside threads.
722 * Random numbers are unique between workers.
723
724 1.808 Sat Nov 05 02:00:00 EST 2016
725
726 * Workers persist unless shutdown explicity while running alongside
727 the Mojolicious framework.
728
729 1.807 Tue Nov 01 16:00:00 EST 2016
730
731 * Enhanced relay capabilities. Added Mandelbrot example to MCE::Example.
732 Added extra demonstrations to MCE::Relay. Also, added test script.
733
734 * Tweaked manager-loop delay for special cases -- applies to MSWin32 only.
735
736 1.806 Tue Oct 11 21:30:00 EST 2016
737
738 * Fixed two typos. Thank you, Florian Schlichting.
739 * Support input_data with nested arrays in MCE models.
740
741 1.805 Thu Sep 01 16:00:00 EST 2016
742
743 * Fixed bug in MCE::Queue (#4). Thank you, Mary Ehlers.
744
745 * Improved support for running MCE with Tk. Added Tk demonstrations to
746 MCE::Examples. Thank you, Götz Meyer.
747
748 1.804 Thu Jul 28 23:00:00 EST 2016
749
750 * Removed the sleep statement in MCE->restart_worker.
751
752 * Added FCGI::ProcManager demonstrations to MCE::Examples.
753 * Automatically set posix_exit to 1 whenever (F)CGI.pm is present.
754 * Thank you Kai Wasserbäch (TheRealCuran) for the cool test case.
755 https://github.com/marioroy/mce-perl/issues/1
756
757 1.803 Sun Jul 10 23:30:00 EST 2016
758
759 * Re-enabled Sereal 3.008+ for Perl < v5.12.0, if available.
760 * Optimized dequeue methods in MCE::Queue.
761
762 1.802 Mon Jul 04 03:30:00 EST 2016
763
764 * Default to Storable for serialization in Perl less than v5.12.0.
765 Sereal 3.008+, if available, is loaded automatically in Perl v5.12+.
766
767 1.801 Sun Jul 03 00:30:00 EST 2016
768
769 [BUG FIXES]
770
771 * Fixed race condition in Queue->await.
772
773 [ENHANCEMENTS]
774
775 * MCE 1.801 is stable on all supported platforms.
776
777 * Completed work supporting cyclical include of MCE Core / models.
778 * Updated MCE to support Perl included with Git Bash.
779
780 * Renamed temp dir from 'mce' to 'Perl-MCE' under user's %TEMP%
781 location on Windows. E.g. Native Perl, Cygwin, Git Bash.
782
783 1.800 Sat Jun 18 16:30:00 EST 2016
784
785 [BUG FIXES]
786
787 * Fixed dequeue (count) in MCE::Queue for standalone mode.
788
789 [ENHANCEMENTS]
790
791 * On Windows, improved stablity and feature parity with UNIX.
792 * Use Sereal 3.008+ automatically if available on the box.
793
794 [NEW FEATURES]
795
796 * Added support for cyclical include of MCE Core, MCE models, and
797 MCE Queue by scoping the configuration to the local package.
798 This resolves (RT#107384), bug reported by Kai Wasserbäch.
799
800 1.708 Sat May 28 14:00:00 EST 2016
801
802 [BUG FIXES]
803
804 * Improved import routine in MCE models and MCE::Subs. This resolves an
805 issue where functions are not exported; e.g. mce_flow, mce_flow_s.
806
807 [ENHANCEMENTS]
808
809 * Added support for IO::TieCombine handles. This enables MCE->print and
810 MCE->sendto to work reliably with App::Cmd and App::Cmd::Tester.
811 See Testing and Capturing Output in MCE::Examples.
812
813 1.707 Wed May 25 16:00:00 EST 2016
814
815 [BUG FIXES]
816
817 * Fixed logic when workers exit. Improved reliability on Windows.
818
819 [ENHANCEMENTS]
820
821 * Applied MCE-1.700-provides.patch from RedHat. Thank you, Paul Howarth.
822
823 * Added META.json to the distribution.
824
825 1.706 Fri Apr 22 21:30:00 EST 2016
826
827 [ENHANCEMENTS]
828
829 * Time::HiRes sleep resolution is 15 milliseconds on Windows and Cygwin.
830 Adjusted timeout values accordingly. Thank you, Daniel Dragan.
831
832 * Reinstated the hack for faster IO when use_slurpio => 1 is specified.
833 Tuned chunk_size => 'auto'.
834
835 1.705 Thu Apr 14 10:00:00 EST 2016
836
837 * Bumped version for Test::More to 0.88. Thank you, Paul Howarth.
838
839 1.704 Thu Apr 14 05:00:00 EST 2016
840
841 [BUG FIXES]
842
843 * Fixed restart on the Windows platform, bug introduced in 1.700.
844 * Reached *stable* on all major platforms for MCE 1.7x.
845
846 [ENHANCEMENTS]
847
848 * Enabled auto-destroy for MCE objects.
849 * Enabled freeze callbacks for Sereal.
850 * Switched bug tracking to GitHub.
851 * Tweaked test scripts.
852
853 1.703 Sat Mar 19 23:00:00 EST 2016
854
855 * Completed IPC optimizations for 1.7.
856
857 1.702 Tue Mar 15 17:00:00 EST 2016
858
859 * Bumped version.
860
861 1.701 Tue Mar 15 12:00:00 EST 2016
862
863 [ENHANCEMENTS]
864
865 * Some folks have expressed a wish for running MCE 1.7 with Perl v5.8.
866 To restore support for Perl v5.8, removed MCE::Shared and MCE::Hobo
867 from the MCE 1.701 distribution.
868
869 * MCE::Shared will be released after MCE 1.700 is deleted from CPAN.
870 Thank you for your patience during this transition.
871
872 1.700 Tue Mar 08 15:30:00 EST 2016
873
874 [BUG FIXES]
875
876 * Fixed race condition on Windows for non-threaded workers.
877
878 * Updated MCE models to not fail when running inside an eval statement.
879 This addresses (RT#105557) and (RT#105559). Thank you, Benjamin McKeown.
880
881 * Added new MCE option loop_timeout to prevent the MCE Manager process
882 from hanging perpetually. The manager process wrongly assumes a worker
883 is still running when the worker died in an uncontrollable manner.
884 This resolves (RT#111780). Thank you, Benjamin McKeown.
885
886 [ENHANCEMENTS]
887
888 * Perl 5.10.1 or later is required to run MCE 1.7. Perl < 5.10.1 lacks
889 'overloading.pm'.
890
891 * Added code in MCE::Grep's documentation for parsing huge files.
892 * Added support for running MCE with Perl under MobaXterm on Windows.
893
894 * MCE/examples and MCE/images are no longer included with the distribution.
895 These are maintained separately at https://github.com/marioroy/mce-examples
896 and https://github.com/marioroy/mce-assets respectively.
897
898 * MCE performs channel locking via a pipe or socket depending on platform.
899 Previously, locking was through file locking using flock. This resolves
900 the slow locking performance on Cygwin.
901
902 * Optimized signal handling including improved support on Windows.
903
904 * Reduced overhead during spawning and job submissions on Windows and Cygwin.
905 This enables IPC to complete up to 20x faster, thus benefiting Monte-Carlo
906 simulations; e.g. calling ->run(0) or ->process(...) repeatedly.
907
908 * The MCE::Flow and MCE::Step models can take an anonymous array for
909 specifying use_threads uniquely for sub-tasks.
910
911 [NEW FEATURES]
912
913 * Added MCE::Hobo for running code asynchronously. This provides async/join
914 functionality for processes similarly to async/join in threads. It includes
915 ->is_joinable, ->is_running, ->join, ->kill, ->list, ->waitall, ->waitone,
916 and other methods not metioned here.
917
918 * Added MCE::Shared for sharing objects/data between threads/processes.
919 * Added MCE::Shared::{ Array, Handle, Hash, Ordhash, and Scalar }.
920 * Added MCE::Shared::{ Condvar, Minidb, Queue, and Sequence }.
921 * Added MCE::Shared::{ Server }.
922
923 * Added methods ->await, ->enq, and ->enqp to MCE::Step.
924 * Added method ->await to MCE::Queue.
925
926 * Added option max_retries => N for retrying a failed chunk from a worker
927 dying while processing input data or sequence of numbers.
928
929 * Added option posix_exit => 1 to avoid END and destructor processing.
930 This is necessary for running with Tk and child processes or with
931 use_threads => 0.
932
933 * Seeds the Math::Random generator automatically when present for non-threads
934 to avoid child processes sharing the same seed value as the parent and
935 each other. The new seed is computed using the current seed. Thus, okay
936 to set the seed at the application level for predictable results.
937
938 1.608 Fri Apr 10 03:00:00 EST 2015
939
940 * Correction for $prog_name (-e) to (perl) in MCE::Signal.
941
942 1.607 Fri Apr 10 01:00:00 EST 2015
943
944 [BUG FIXES]
945
946 * Updated t/01_load_signal_arg.t to address false-positive from bug fix
947 in 1.606. This was missed in the 1.606 release. Thank you Nigel Horne.
948
949 [ENHANCEMENTS]
950
951 * Replaced (-e) with (perl) for the $prog_name value in MCE::Signal;
952 e.g. from running perl -e 'command'.
953
954 1.606 Wed Apr 08 18:00:00 EST 2015
955
956 [BUG FIXES]
957
958 * Added -d and -w tests to ensure $ENV{TEMP} exists and writeable in
959 MCE::Signal. Otherwise /tmp is used as usual.
960
961 [ENHANCEMENTS]
962
963 * Determine running state in MCE->exit. Call stop_and_exit if not
964 already running to not hang the script.
965
966 1.605 Mon Apr 06 00:30:00 EST 2015
967
968 [BUG FIXES]
969
970 * Improved fix for the die handler in MCE::Signal and MCE::Core::Worker.
971 * Improved support for threads in MCE::Signal's stop_and_exit function.
972
973 1.604 Sat Mar 21 21:00:00 EST 2015
974
975 [BUG FIXES]
976
977 * All bugs found during testing of the upcoming 1.7 release have been
978 backported to the 1.6 branch.
979
980 [NEW FEATURES]
981
982 * Added out_iter_array and out_iter_fh to MCE::Candy. These preserve output
983 order and cover the two general use cases.
984
985 1.603 Tue Mar 17 21:00:00 EST 2015
986
987 [ENHANCEMENTS]
988
989 * A safer solution by Dmitry Karasik for the die handler in MCE::Signal
990 and MCE::Core::Worker.
991
992 * Moved ->forchunk, ->foreach, and ->forseq sugar methods to MCE::Candy.
993 Stubs exist in MCE. Thus, no breakage to existing apps.
994
995 * Removed the link to MCE::Shared on the main page. I decided to backport
996 all the fixes into 1.6. The MCE::Shared link was missed and requires
997 the upcoming 1.7 release.
998
999 1.602 Mon Mar 16 20:00:00 EST 2015
1000
1001 [BUG FIXES]
1002
1003 * Updated die handler in MCE::Core::Worker and MCE::Signal (RT#102802)
1004 Bug reported by Dmitry Karasik.
1005
1006 [ENHANCEMENTS]
1007
1008 * MCE child processes call CORE::exit during exiting; previously CORE::kill.
1009 * Improved IPC stability on Windows and Cygwin.
1010
1011 * Multiple calls to mutex->lock by the same process or thread is now safe.
1012 The mutex will remain locked until mutex->unlock is called.
1013
1014 1.601 Sun Mar 15 21:00:00 EST 2015
1015
1016 This release is attributed to the many use cases sent by George Bouras.
1017
1018 [BUG FIXES]
1019
1020 * Updated Makefile.PL allowing installation of bin/mce_grep (RT#102040).
1021 Running (MCE_INSTALL_TOOLS=1 perl Makefile.PL) installs bin/mce_grep.
1022
1023 * Ensure MCE instances spawned by workers have shutdown before leaving.
1024 * An exiting forked non-MCE process will not cause the worker to exit.
1025
1026 * Fixed sockets not closing immediately in Cygwin and Windows.
1027 * Fixed a pod error in MCE::Mutex.
1028
1029 * Fixed a rare condition where socket handles for MCE::Mutex and
1030 MCE::Queue were closing pre-maturely when using threads.
1031
1032 * Fixed an undefined variable inside croak statements in MCE models'
1033 import subroutine.
1034
1035 * Fixed automatic shutdown due to loading threads and specifying
1036 use_threads => 0 with workers persisting before exiting the script.
1037
1038 [ENHANCEMENTS]
1039
1040 * Moved relay methods, introduced in 1.600, from MCE to MCE::Relay.
1041 The MCE init_relay option loads and enables MCE::Relay automatically.
1042
1043 * Captured metrics from Linux (previously OS X) for forchunk.pl,
1044 foreach.pl, and forseq.pl. Updated Examples.pod.
1045
1046 * Default to 'auto' for max_workers in bin/mce_grep, examples/egrep.pl,
1047 forchunk.pl, foreach.pl, forseq.pl, iterator.pl, and wc.pl.
1048
1049 * Moved _create_socket_pair from MCE to MCE::Util as _make_socket_pair.
1050 Added _destroy_sockets. Updated MCE, Mutex, and Queue to use MCE::Util.
1051
1052 * Added a CARP_NOT line to MCE models.
1053 * Added support for ->next and ->last from any sub-tasks in MCE::Step.
1054 * Reverted a small change applied in 1.600 to ->do and ->gather.
1055 * The 'synchronize' method in Mutex is wantarray aware.
1056 * Updated POD header lines for method names; from =item to =head2.
1057 * Workers now set STDERR and STDOUT to flush automatically.
1058
1059 [NEW FEATURES]
1060
1061 * Added ->pid method to MCE.
1062
1063 1.600 Sat Jan 31 20:00:00 EST 2015
1064
1065 [BUG FIXES]
1066
1067 * Die handlers (in MCE::Signal and MCE::Core::Worker) are finally 100%.
1068 Furthermore, eval { die ... } statements behave correctly while running
1069 MCE itself inside an eval block. Thus, MCE on iPerl on top of iPython
1070 is possible via Devel::IPerl.
1071
1072 * Fixed queues stalling from running (MCE::Queue fast => 1) on Linux.
1073
1074 * MCE models now set $MCE::FREEZE, $MCE::THAW, and $MCE::TMP_DIR when
1075 overriding freeze, thaw, and tmp_dir respectively at load time.
1076
1077 * Pressing CTRL-D now ends STDIN the first time.
1078
1079 [ENHANCEMENTS]
1080
1081 * Added seven names to CREDITS; David Farrell, Demian Riccardi,
1082 Hisham Eldai, Joel Berger, Oleksandr Kharchenko, Wei Shen, and
1083 Zakariyya Mughal.
1084
1085 * Refactored ->print, ->printf, and ->say. Optimized ->print some more.
1086 * Shorten $_queue to $_Q in Queue. ID is always sent first during IPC.
1087 * The init method in MCE models can now take an array of options.
1088
1089 * Optimized memory consumption in Handle.pm, Iterator.pm, and Request.pm.
1090 * Optimized memory consumption for ->sendto, ->do, and ->gather.
1091 * Optimized memory consumption for overall IPC in general.
1092 * Refreshed the MCE->new method.
1093
1094 [NEW FEATURES]
1095
1096 * Added a new module; MCE::Mutex providing simple semaphore.
1097
1098 * Added ->relay_recv, ->relay, and ->relay_final methods to the Core API.
1099 Refreshed cat.pl and findnull.pl examples to relay the number of lines
1100 read. Workers output exclusively and orderly to STDOUT in cat.pl.
1101
1102 * Added several examples; biofasta (folder), mutex.pl, and relay.pl.
1103 The FASTA examples process by records "\n>", not by lines.
1104
1105 * MCE applies additional logic when RS begins with a newline character;
1106 e.g. RS => "\n>". It trims away characters after the newline and
1107 prepends them to the next record. This happens automatically when not
1108 slurping. Otherwise, the logic is applied to the first and last records
1109 only. This is illustrated in the Core API documentation.
1110
1111 * Updated the Core API documentation (RS, added relay methods).
1112
1113 1.522 Thu Dec 25 16:00:00 EST 2014
1114
1115 [BUG FIXES]
1116
1117 * Applied fix to MCE->shutdown so that MCE models do not err when
1118 receiving signal to terminate.
1119
1120 * Optimization for MCE->print, MCE->printf, MCE->say and MCE->sendto.
1121 This was calling fileno unnecessarily. MCE->print('STDERR', ...)
1122 is not supported. This works; MCE->print(\*STDERR, ...)
1123
1124 [ENHANCEMENTS]
1125
1126 * Inserts the actual lib-path at the head of @INC in example files.
1127 * Massive documentation updates throughout the entire distribution.
1128 * Renamed barrier_sync.pl to sync.pl; scaling_pings.pl to ping.pl.
1129 * Updated requires in META.yml, Makefile.PL and perl-MCE.spec.
1130 * Updated comment in MCE::Queue test scripts.
1131 * Updated the README file.
1132
1133 [NEW FEATURES]
1134
1135 * Added sampledb examples demonstrating DBI and SQLite with MCE.
1136 * Added step_demo.pl to examples.
1137
1138 1.521 Thu Dec 11 16:00:00 EST 2014
1139
1140 [BUG FIXES]
1141
1142 * Fixed broken MCE::Queue ->insert and ->peek methods. FIFO and LIFO
1143 are fully supported with this release.
1144
1145 * Support running in taint mode.
1146
1147 [ENHANCEMENTS]
1148
1149 * Added support for negative index in MCE::Queue ->insert and ->peek.
1150 Updated the documentation.
1151
1152 * CBOR::XS serialization is mentioned in documentation along with
1153 JSON::XS and Sereal.
1154
1155 * Completed code refactoring for the 1.5 branch.
1156 * Optimized argument parsing in import routines.
1157 * Removed the MCE spawn_delay option from test scripts.
1158
1159 [NEW FEATURES]
1160
1161 * Added 15 new test scripts for testing user_args, MCE::Queue
1162 and MCE models.
1163
1164 * An upper-limit of 8 is set when specifying max_workers => 'auto'.
1165 Several folks have requested this. More info at MCE::Util::get_ncpu
1166 for increasing or decreasing the limit.
1167
1168 1.520 Wed Nov 05 03:00:00 EST 2014
1169
1170 [ENHANCEMENTS]
1171
1172 * MCE::Step and MCE::Stream can take the 'fast' option when including
1173 the module. The 'fast' option was introduced in 1.518.
1174
1175 * Removed the type declaration for self (feedback from Sri).
1176 * Removed -s from files_flow.pl (was left behind).
1177 * Added support for GNU Hurd OS in get_ncpu.
1178
1179 1.519 Mon Oct 27 19:00:00 EST 2014
1180
1181 [ENHANCEMENTS]
1182
1183 * Unset the need for channel locking if only worker riding the channel.
1184 There are 8 data channels in MCE. Basically, a worker will obtain a
1185 lock only when sharing the data channel with another worker.
1186
1187 * Updated files_flow.pl, files_mce.pl and files_thr.pl to allow for
1188 many workers for the first task. Updated the synopsis in MCE::Queue.
1189 Synced example listing in Examples.pod with the examples folder.
1190
1191 * Remove period from summary line.
1192
1193 1.518 Mon Oct 27 10:00:00 EST 2014
1194
1195 [BUG FIXES]
1196
1197 * Corrected MCE::Queue's synopsis due to missing List::MoreUtils line.
1198 Changed the synopsis to use 1 worker for the 'dir' task in the event
1199 one were to copy/patse the code and use threads. The glob() function
1200 is not thread-safe in Perl 5.16.x; fixed in 5.18.2, okay in 5.8 - 14.
1201
1202 * Use portable syntax for setpgrp in MCE::Signal. This closes issue 1
1203 at https://code.google.com/p/many-core-engine-perl.
1204
1205 [NEW FEATURES]
1206
1207 * New 'fast' option for MCE::Queue. The 'fast' option speeds up ->dequeue
1208 ops and not enabled by default. It is beneficial for queues not needing
1209 ->clear or ->dequeue_nb and not altering the optional count value
1210 while running; e.g. ->dequeue( [ $count ] ).
1211
1212 * Added three examples: files_flow.pl, files_mce.pl, and files_thr.pl.
1213 * Benchmarked on several OSes and appended results to MCE::Queue synopsis.
1214
1215 1.517 Thu Oct 23 10:00:00 EST 2014
1216
1217 [BUG FIXES]
1218
1219 * Correction applied to MCE::Util::get_ncpu for Tru64 UNIX. This method
1220 will emit a warning (not croak) whenever the OS is unknown.
1221
1222 * Changed ${^CHILD_ERROR_NATIVE} to $? in examples/pipe2.pl. This was
1223 missed in the previous release.
1224
1225 [ENHANCEMENTS]
1226
1227 * Added support for DragonFly BSD, SCO OpenServer 5, SCO OpenServer 6 and
1228 SCO UnixWare 7 to MCE::Util::get_ncpu.
1229
1230 * Also, validated MCE on FreeBSD, NetBSD, OpenBSD, PC-BSD and JabirOS.
1231 No further changes to MCE::Util::get_ncpu.
1232
1233 1.516 Fri Oct 03 02:00:00 EST 2014
1234
1235 [BUG FIXES]
1236
1237 * Updated IPC for better stability across multiple environments. The fix
1238 addresses an issue on Windows where sockets fail to respond after a
1239 period of inactivity; i.e. 4 minutes. Added George Bouras to CREDITS
1240 for reporting the issue.
1241
1242 * Tip for folks developing on Windows:
1243 Open an explorer window and go to C:\Users\<user>\AppData\Local\Temp.
1244 Right-click on the 'mce' folder and create a shortcut on the desktop.
1245 Although MCE removes its temp files automatically, it is possible for
1246 files to remain from a failing app. Go inside the 'mce' folder and
1247 press ctrl-a to select all files. Then press shift-delete;
1248 (macbook[pro] folks: fn-shift-delete).
1249
1250 [ENHANCEMENTS]
1251
1252 * Corrections to documentation.
1253
1254 1.515 Thu Jul 24 23:30:00 EST 2014
1255
1256 [BUG FIXES]
1257
1258 * bin/mce_grep: ${^CHILD_ERROR_NATIVE} is not defined in Perl 5.8.x.
1259 Changed to $?.
1260
1261 [ENHANCEMENTS]
1262
1263 * Tweaked bin/mce_grep. Compute chunk_level => 'auto' to use 'file' when
1264 reading STDIN. Set chunk_size to 8M when not specified (4M previously).
1265
1266 * Added the following names to CREDITS.
1267
1268 Stephan Kulow ; for making the OpenSUSE MCE package
1269 Henry Lu ; for listening while I chatted away about MCE
1270 Jillian Rowe ; for reporting IO::File failing as input_data
1271 Sylvia Roy ; for driving while I worked in the passenger seat
1272 Tom Sasser ; for reporting bin/mce_grep failing with Perl 5.8.x
1273 Florian Schlichting ; for making the Debian MCE package
1274
1275 [NEW FEATURES]
1276
1277 * Added support for IO::File handles as valid input_data including
1278 IO:Uncompress:Gunzip.
1279
1280 Modified SYNTAX section for INPUT_DATA in MCE::Core.pod.
1281
1282 input_data => $fh, ## new IO::File "file", "r"
1283 input_data => $fh, ## new IO::Uncompress::Gunzip "file.gz"
1284
1285 1.514 Thu Jun 05 09:00:00 EST 2014
1286
1287 [BUG FIXES]
1288
1289 * Fixed typo in MCE::Step POD (RT#95250) (Florian Schlichting).
1290 * Updated MCE::Util's get_ncpu function for AIX (Dana Jacobsen).
1291 * Do not send a KILL signal after receiving a SIGPIPE.
1292 * Fixed issue with Makefile.PL on defining the minimum Perl version.
1293
1294 * Use Scalar::Util (looks_like_number) for validation logic. The regex
1295 previously were insufficient for large numbers containing scientific
1296 notation. This impacted the sequence option in MCE.
1297
1298 [ENHANCEMENTS]
1299
1300 * The mce-sandbox demo has been released on GitHub demonstrating
1301 Perl + MCE + Inline::C. The theme is Prime Numbers. This journey
1302 which began 2 years ago has been completed. There is also the
1303 mce-sort exercise demonstrating Perl + MCE + External C.
1304
1305 https://github.com/marioroy/mce-sandbox (is thread-safe)
1306 https://github.com/marioroy/mce-sort (not thread-safe)
1307
1308 * A hard decision had to made for MCE, particularly Perl under Windows
1309 excluding Cygwin. MCE will now load the 'threads' module automatically
1310 for Windows only. Folks may specify use_threads => 0 if threads is
1311 not desired.
1312
1313 The reason for this is from seeing Math::Prime::Util crashing once
1314 workers exit. The same is true without MCE and forking a child process.
1315 Threads does not exhibit this behavior. It is a hard problem to solve.
1316 Why not default to threads for Windows since forking is emulated.
1317
1318 1.513 Sat Apr 19 20:30:00 EST 2014
1319
1320 [BUG FIXES]
1321
1322 * Added fix for bug RT#94869 -- crash when restarting workers with 9+
1323 workers. Updated the perldoc for restart_worker in MCE::Core.pod.
1324
1325 [ENHANCEMENTS]
1326
1327 * Replaced $self with $mce in MCE::/Core.pod to be consistent with
1328 examples described in MCE::Examples.pod.
1329
1330 1.512 Fri Apr 18 21:00:00 EST 2014
1331
1332 [BUG FIXES]
1333
1334 * Signal-handling update for MCE::Signal. Passing the -setpgrp option is not
1335 necessary, even with Daemon::Control. Piping data into and out is better
1336 supported with this release (\*STDIN). This resolves bug RT#94706.
1337
1338 cat infile | mce_script | head
1339 mce_script < infile | head
1340
1341 Added Shawn Halpenny to the CREDITS file.
1342
1343 * The utf8.pl example now runs under the BSD 9.0 environment. This was
1344 failing due to $^H{charnames} is not defined error. Removed the constant
1345 from the list of unicode characters inside the script.
1346
1347 [ENHANCEMENTS]
1348
1349 * Added examples/pipe1.pl and pipe2.pl. These process STDIN or FILE in
1350 parallel. Processing is via Perl for pipe1.pl, whereas an external
1351 command for pipe2.pl.
1352
1353 1.511 Fri Apr 04 22:30:00 EST 2014
1354
1355 [BUG FIXES]
1356
1357 * Added "use bytes;" in several files to have length() return physical
1358 bytes, not logical characters. MCE is now UTF-8 safe when passing scalar
1359 data between workers and the manager process.
1360
1361 Added Marcus Smith to the CREDITS file for reporting this bug.
1362
1363 [ENHANCEMENTS]
1364
1365 * Added examples/utf8.pl
1366
1367 1.510 Thu Mar 27 10:00:00 EST 2014
1368
1369 [ENHANCEMENTS]
1370
1371 * The user_begin and user_end functions now receive 3 arguments.
1372 my ($mce, $task_id, $task_name) = @_;
1373
1374 * Pass the chunk_size value when calling the iterator function. Added a
1375 DBI example under "SYNTAX for INPUT_DATA" in MCE::Core.pod.
1376
1377 * Store the last scalar reference to not have workers re-spawn
1378 unnecessarily when input_data => \$same_scalar_ref.
1379
1380 [NEW FEATURES]
1381
1382 * New parallel_io option to further enhance slurp IO when specifying
1383 use_slurpio => 1 and chunk_size is greater than 8192. Try with
1384 chunk_size => '300k' or chunk_size => '2m'.
1385
1386 The parallel_io option is beneficial when reading from fast storage.
1387 However, possibly not recommended if running MCE on many compute nodes
1388 and having workers read various input_data from shared storage.
1389
1390 Enable parallel_io only if it makes sence and without impacting the
1391 environment such as nfsd. Because use_slurpio => 1, parallel_io => 1,
1392 and chunk_size => '2560k' is truly parallel.
1393
1394 1.509 Sat Feb 03 05:30:00 EST 2014
1395
1396 [BUG FIXES]
1397
1398 * Fixed an issue with all the models (Flow, Grep, Loop, Map, Step, and
1399 Stream) ending immediately on subsequent runs when input_data is
1400 specified through the init method.
1401
1402 * Things have finally settled down with all the models.
1403
1404 [ENHANCEMENTS]
1405
1406 * Refactored the fix applied in 1.508 for addressing bug #92627.
1407
1408 * MCE, for the most part, has been completed. This release touches up on
1409 many examples under the 'examples' directory.
1410
1411 * Brought the MCE::Examples Perl documentation up to date.
1412 Inlined MCE::Loop snippets under the chunking sections.
1413 Added a new section GLOBALLY SCOPED VARIABLES AND MCE MODELS.
1414 Added a new section MONTE CARLO SIMULATION.
1415
1416 1.508 Sat Feb 01 04:00:00 EST 2014
1417
1418 [BUG FIXES]
1419
1420 * Applied fix for bug #92627 submitted by Philip Mabon. MCE scripts may
1421 exit with a non zero error code due to leaving workers up thereby
1422 having MCE performing the shut down in its END block.
1423
1424 For folks running on an older release, the workaround is to shut down
1425 workers prior to exiting.
1426
1427 1.507 Fri Jan 30 23:00:00 EST 2014
1428
1429 [ENHANCEMENTS]
1430
1431 * Added clarity around preserving output order for several models;
1432 MCE::Flow, MCE::Loop, and MCE::Step.
1433
1434 * Inlined comments in the code for overriding MCE options when using
1435 an anonymous hash for the first argument; MCE::Flow, MCE::Step, and
1436 MCE::Stream.
1437
1438 * Removed unnecessary code due to one not able to pass MCE options other
1439 than through the init method for MCE::Loop, MCE::Grep, and MCE::Map.
1440
1441 * Added to doc a use-case example searching a large file with mce_grep_f.
1442 The emphasis is comparing the memory consumption against the native
1443 grep function as well as time to complete.
1444
1445 1.506 Fri Jan 30 03:00:00 EST 2014
1446
1447 [BUG FIXES]
1448
1449 * Fixed issue with all 5 models croaking for an internal '_file' option.
1450 This has been broken since the 1.502 release. Methods impacted were:
1451 mce_flow_f, mce_grep_f, mce_loop_f, mce_map_f, and mce_stream_f
1452
1453 * Specifying a different value for gather on subsequent runs is now taking
1454 effect for when workers persist after running. The models impacted are
1455 MCE::Flow and MCE::Loop.
1456
1457 * Updated _parse_chunk_size in MCE::Util to compute chunk_size correctly
1458 for new edge cases. One may call either mce_grep or mce_grep_f for a
1459 GLOB or scalar reference as input data. The same also applies for the
1460 other models; mce_flow, mce_loop, mce_map and mce_stream.
1461
1462 [ENHANCEMENTS]
1463
1464 * Updates to MCE::Grep and MCE::Map. The logic is now aware of wantarray
1465 for faster processing when storing to a scalar value. In addition, the
1466 use_slurpio option is enabled for efficient IO when processing large
1467 files.
1468
1469 * The task_end option can now be specified for MCE::Stream, although being
1470 used internally.
1471
1472 * The interval option can take a decimal number. Previously, this
1473 wanted a hash reference with up to 3 key/value pairs. Most often, all
1474 one needed was delay and not max_nodes or node_id.
1475
1476 interval => { delay => 0.05 } ## Choose either format for specifying
1477 interval => 0.05 ## delay in 1.506 and above. MCE will
1478 ## translate 0.05 to { delay => 0.05 }
1479 ## automatically.
1480
1481 * Slight optimization to get_ncpu (public method) in MCE::Util.
1482
1483 * Tweak to _parse_chunk_size (private method) in MCE::Util.
1484
1485 [NEW FEATURES]
1486
1487 * Added a new model MCE::Step for transparent use of MCE::Queue when passing
1488 data among sub-tasks. MCE::Step is basically a spin off from MCE::Flow
1489 with a touch of MCE::Stream.
1490
1491 This new model is crazy :)
1492
1493 1.505 Thu Jan 21 01:30:00 EST 2014
1494
1495 [BUG FIXES]
1496
1497 * Delete $self->{input_data} inside the worker immediately during spawning
1498 (if input_data is an ARRAY, GLOB, or Iterator reference).
1499
1500 * Reverted the logic for the RS (record separator) option from the 1.4x
1501 code base. This is now working as expected.
1502
1503 * Specifying chunk_size => 'auto' via the 'init' method (all 5 models)
1504 causes MCE validation to croak due to 'auto' being carried over to the
1505 core API. Note that only the 5 models are allowed 'auto' for chunk_size
1506 as of this time.
1507
1508 * Update to the MCE->do method. Previously, an undef sent back from the
1509 callback function ended up as a blank value "". Both "" and undef are
1510 now properly captured and sent back to the worker process.
1511
1512 * Added an if statement inside the 'abort' method.
1513 * Removed an old unlink _store.db statement (old code left behind).
1514 * Removed an unused _next variable inside Request.pm.
1515
1516 [ENHANCEMENTS]
1517
1518 * Enhanced egrep.pl to handle additional options including recursion.
1519 This script now supports many egrep options [ceHhiLlmnqRrsv].
1520
1521 * IO performance for examples/cat.pl was improved. The real focus here is
1522 demonstrating output order.
1523
1524 * The chunk_size option can take a suffix; K (Kilobytes) or M (Megabytes).
1525
1526 * The following examples except --max-workers=NUM --chunk-size=NUM options.
1527 cat.pl, egrep.pl, findnull.pl, scaling_pings.pl, and wc.pl
1528
1529 [NEW FEATURES]
1530
1531 * The input_data option can now receive an iterator reference. Added a
1532 new example iterator.pl for demonstraton. In addition, there are
1533 several examples listed under a new section "SYNTAX for INPUT_DATA"
1534 under MCE::Core.pod.
1535
1536 * Added a new demo script; bin/mce_grep. This is a wrapper script for the
1537 grep binary. This script supports agrep, grep, egrep, fgrep, & tre-agrep.
1538 Simply create a link to mce_grep or make a copy. Both Windows and Cygwin
1539 are supported as well. Recursion works for all binaries including agrep
1540 (-R, -r options).
1541
1542 ln mce_grep mce_agrep
1543 ln mce_grep mce_egrep
1544 ln mce_grep mce_fgrep
1545 ln mce_grep mce_tre-agrep
1546
1547 Try with the --lang=C option for faster execution time (-i runs faster).
1548 Try mce_agrep or mce_tre-agrep against very large files. The speedup is
1549 linear and makes good utilization of all available cores on the box.
1550
1551 One may specify the chunking level via the --chunk-level option. For large
1552 files, specify 'file' (chunks file). For many small files, use 'list'.
1553
1554 I have tested against the following GnuWin32 packages found at this URL.
1555 http://gnuwin32.sourceforge.net/packages.html
1556
1557 tre-0.7.5-bin.zip ## Contains agrep.exe (also runs under Cygwin)
1558 grep-2.5.4-bin.zip ## Contains egrep.exe, fgrep.exe, grep.exe
1559 libiconv-1.9.2-1-bin.zip ## These are required for the GnuWin32 binaries.
1560 libintl-0.14.4-bin.zip
1561 pcre-7.0-bin.zip
1562 regex-2.7-bin.zip
1563
1564 Btw, bin/mce_grep is optional and therefore not installed by default
1565 when running make install for the MCE module. Simply copy mce_grep as
1566 mce_agrep.pl for Windows. The '.pl' suffix is optional for other
1567 environments.
1568
1569 1.504 Tue Oct 29 16:00:00 EST 2013
1570
1571 [BUG FIXES]
1572
1573 * MCE::Grep, MCE::Map, and MCE::Stream were failing for mce_grep_s,
1574 mce_map_s, and mce_stream_s when specifying chunk_size => 1.
1575
1576 [ENHANCEMENTS]
1577
1578 * Calibrated 'auto' slightly in MCE::Util::_parse_chunk_size.
1579
1580 1.503 Mon Oct 28 17:00:00 EST 2013
1581
1582 [BUG FIXES]
1583
1584 * The previous release introduced a bug by removing the line closing
1585 STDERR and STDOUT prior to workers exiting. The proper fix intended
1586 is to flush (not close) the handles. Closing them was not the thing
1587 to do in the first place due to possibly needed by the END block or
1588 WARN/DIE handlers initiated from inside the END block.
1589
1590 [ENHANCEMENTS]
1591
1592 * Updated README under examples/tbray and examples/matmult.
1593 * Changed 0.499 to 0.5 inside the yield method.
1594 * Small cosmetic changes otherwise.
1595
1596 1.502 Mon Oct 21 16:00:00 EST 2013
1597
1598 [BUG FIXES]
1599
1600 * The END block for all models will return immediately when called by a
1601 worker thread or process. Removed the line closing STDERR and STDOUT
1602 prior to workers exiting. It turns out that workers were calling
1603 shutdown which is not allowed. This impacted MCE::Flow, MCE::Grep,
1604 MCE::Loop, MCE::Map, and MCE::Stream.
1605
1606 * An update was applied to DESTROY in MCE::Queue to address an edge
1607 case during additional testing across several environments.
1608
1609 * Added documentation describing the core methods in MCE. I had moved the
1610 missing sections to another file some time back and totally forgotten
1611 about them when finalizing on MCE::Core.pod for the 1.5 release.
1612
1613 [ENHANCEMENTS]
1614
1615 * All models will croak when specifying an invalid MCE option.
1616
1617 [NEW FEATURES]
1618
1619 * One can specify a hash reference for the gather option. Updated
1620 documentation describing the gather option and the gather method.
1621
1622 1.501 Wed Oct 16 01:20:00 EST 2013
1623
1624 [BUG FIXES]
1625
1626 * Addressed an issue with the worker signal "die" handler (#89538). Eval'd
1627 code should not raise an exception causing the app to die. This is working
1628 as expected for workers spawned as children.
1629
1630 For threads, a die called explicitly inside an eval block will cause the
1631 worker to exit (same as previously). However, an eval 'use MissingModule'
1632 will no longer cause the thread to die.
1633
1634 [ENHANCEMENTS]
1635
1636 * Changelog section uses a new format beginning with this release.
1637
1638 [NEW FEATURES]
1639
1640 * Added examples/flow_model.pl for demonstrating MCE::Flow, MCE::Queue, and
1641 MCE->gather.
1642
1643 1.500 Thu Oct 10 01:00:00 EST 2013
1644
1645 * MCE 1.5 is backwards compatible with 1.4 and below.
1646
1647 * The documentation, previously MCE.pod, moved to MCE::Core.pod with the
1648 examples section placed in MCE::Examples.pod. The MCE.pod file serves
1649 as an index page for the various documentation.
1650
1651 * IPC has been enhanced with 8 data channels. Many operations run 3x when
1652 compared with MCE 1.4.
1653
1654 * Five models: MCE::Flow, MCE::Grep, MCE::Loop, MCE::Map, and MCE::Stream.
1655
1656 * Hybrid queues via MCE::Queue allowing for normal and priority queues.
1657
1658 * MCE::Subs for exporting functions prefixed with mce_; e.g. mce_wid.
1659
1660 * All public methods can be called directly using the package name
1661 and method e.g. MCE->wid, MCE->run.
1662
1663 MCE->new(
1664 max_workers => 'auto',
1665 user_func => sub {
1666 my $wid = MCE->wid;
1667 MCE->sendto("STDOUT", "Hello from $wid\n");
1668 }
1669 );
1670
1671 MCE->run;
1672
1673 * Localize the input scalar $_ prior to calling user_func. Folks can
1674 use $_ for input_data and sequence of numbers. Added section to docs
1675 explaining DEFAULT INPUT SCALAR.
1676
1677 * New options (bounds_only, gather, interval, task_name).
1678
1679 The task_end option can now be specified at the top level.
1680 See docs for use case.
1681
1682 Input_data can be specified inside the first task instead of having
1683 to specify this at the top level. Input_data is ignored when specified
1684 for tasks other than the first task.
1685
1686 user_tasks => [{
1687 input_data => \@list,
1688 ...
1689 },{
1690 ...
1691 }
1692
1693 * New public methods
1694 chunk_id, gather, freeze, thaw, yield, task_name, print, printf, say
1695
1696 * New example (interval.pl).
1697
1698 * Optimized the egrep.pl and wc.pl examples. These run much faster.
1699 Try these out against large log files. Both examples fly.
1700
1701 * Barrier synchronization update. Two sockets are utilized instead of
1702 2 lock files. This, now works wonderfully under the Cygwin environment.
1703 For threading, the removal of 2 lock files increases the number of
1704 threads allowed from about 1/3rd previously to under 1/2 of ulimit,
1705 e.g. int(ulimit -n / 2 - 20).
1706
1707 * Removed the logic for determining MAX_OPEN_FILES and MAX_USER_PROCS.
1708 MCE no longer has a constraint on max_workers allowed.
1709
1710 * Code re-factor work. Added a private method _validate_runstate called
1711 by various methods. Organized the code slightly such as placement of
1712 methods.
1713
1714 1.415 Mon Jun 17 15:00:00 EST 2013
1715
1716 * Code-refactor in preparation for the upcoming 1.5 release.
1717 This completes the 1.4x branch for a very solid and stable release.
1718
1719 * IPC optimization. Localize $\ and $/ only as needed. Small tweak to
1720 logic when specifying RS (record separator) for input data.
1721
1722 Foreach is very communication intensive. Forseq is less so. Chunking
1723 was made faster by increasing chunk size from 500 to 2500. I've reached
1724 my goal for forseq by reaching 60,000+. A while back, foreach could
1725 barely reach 18,000.
1726
1727 Before and after results with IPC optimization.
1728
1729 (Before) (After)
1730 Parallel::Loops 600 600
1731 $mce->foreach 20,000 21,500 (+ 1,500)
1732 $mce->forseq 55,000 64,000 (+ 9,000)
1733 $mce->forchunk 395,000 450,000 (+ 55,000)
1734
1735 * Updated the process method. Allows sequence to be specified as an
1736 option. Also allow the hash to be specified as the 1st or 2nd
1737 argument.
1738
1739 $mce->process( \@input_array, { options } );
1740 $mce->process( { options }, [ 1..1000 ] );
1741 $mce->process( { sequence => [ 20, 40, 2 ] } );
1742
1743 * The forseq method will now honor chunk_size greater than 1. Updated
1744 docs with use case.
1745
1746 * Added CONST short for EXPORT_CONST, e.g. use MCE CONST => 1;
1747
1748 1.414 Sun Jun 16 13:00:00 EST 2013
1749
1750 * Upstream bug fixes. Mainly chunk_size was not honored when specifying
1751 user_tasks and sequence/chunk_size for the first task. Bug is with
1752 not setting abort_msg correctly.
1753
1754 1.413 Mon Jun 10 02:00:00 EST 2013
1755
1756 * Fixed typo: Changed local @_ to local $@ inside _parse_max_workers
1757 before evaling.
1758
1759 1.412 Sun Jun 09 14:00:00 EST 2013
1760
1761 * Updated the main README file and CHANGES under 1.411 below.
1762
1763 * Added support for multiplication and division as well as
1764 case insentivity to the private _parse_max_workers method.
1765
1766 max_workers => 'auto', ## = MCE::Util::get_ncpu()
1767 max_workers => 'Auto-1', ## = MCE::Util::get_ncpu() - 1
1768 max_workers => 'AUTO + 3', ## = MCE::Util::get_ncpu() + 3
1769 max_workers => 'AUTO * 1.5',
1770 max_workers => 'auto / 1.333',
1771
1772 1.411 Sun Jun 09 02:00:00 EST 2013
1773
1774 * Reverted the change made in MCE-1.410. That degraded slurp IO on
1775 large files. This restores slurp IO performance in MCE.
1776
1777 * Corrected an if statement towards the end of the shutdown method.
1778 The delay was occurring always when it was meant only if MCE was
1779 launched from inside a non-main thread.
1780
1781 * Addressed a race condition with barrier synchronization under the
1782 Cygwin environment. Was not able to reproduce this with other OS's.
1783 This takes barrier synchronization work to 100%. Delay statements
1784 for Cygwin were removed around barrier synchronization.
1785
1786 * Optimized writes to sockets. Arguments passed to print statements
1787 are concatenated as one big string. Removed local $/ = $LF when
1788 reading from the queue socket. It's not required there since using
1789 read and known size. The foreach.pl example (very communication
1790 intensive) can now do 20000 in 1 second on my Macbook Pro. Updated
1791 benchmark results inside foreach.pl, forchunk.pl, and forseq.pl.
1792
1793 * MAX_WORKERS, CHUNK_SIZE, TMP_DIR, FREEZE and THAW can be specified
1794 when loading the module. FREEZE and THAW allows one to choose an
1795 alternative serialization module if preferred for your project.
1796
1797 use Sereal qw(encode_sereal decode_sereal);
1798 use MCE FREEZE => \&encode_sereal, THAW => \&decode_sereal;
1799
1800 * Passing EXPORT_CONST => 1 will export 3 constants: SELF, CHUNK, CID
1801 The "my ($self, $chunk_ref, $chunk_id) = @_" line is not necessary.
1802 $_[SELF], $_[CHUNK], $_[CID] refers to $_[0], $_[1], $_[2].
1803
1804 * The MCE::Util module was created. It contains the get_ncpu function.
1805 This is largely borrowed from Test::Smoke::Util.pm. MCE supports
1806 'auto' when specifying max_workers. Read doc for other use case.
1807
1808 max_workers => 'auto'; ## Same as MCE::Util::get_ncpu()
1809
1810 * Re-factored code in preparation for the upcoming 1.5 release. The
1811 1.5 release will use just 3 socket pairs versus 4. I did not want
1812 to include that change in this release. Therefore, I decided to
1813 take the upstream bug fixes and apply them to the 1.4 base.
1814
1815 Re-factored documentation by making use of the full 78 character
1816 width per line.
1817
1818 * Updated the main README file under the top level dir.
1819
1820 1.410 Tue May 28 23:30:00 EST 2013
1821
1822 * Use threads under MSWin32 for 02* and 03* test scripts.
1823 * Removed sysopen/sysseek/sysread to simplify logic due to negligible
1824 performance gains over open/seek/read.
1825 * Minor updates to documentation.
1826
1827 1.409 Sun May 12 22:45:00 EST 2013
1828
1829 * Croak if user_func is not defined and input_data/sequence is specified.
1830 * Fix barrier synchronization when running multiple tasks via user_tasks.
1831 * Updated Perl documentation for easier reading through cpan/metapan.
1832 * Renamed private method _worker_sequence to _worker_sequence_queue.
1833
1834 1.408 Tue Mar 19 22:00:00 EST 2013
1835
1836 * Minor tweaks here and there to further increase reliability.
1837 * Updated the barrier synchronization logic to not stall. Updated the
1838 perl docs on mixing "sync" with "do" or "sendto" methods.
1839 * Added new "status" method for the manager process.
1840 * Added new arguments for MCE::Signal: -no_kill9, -no_sigmsg
1841
1842 1.407 Thu Mar 14 21:00:00 EST 2013
1843
1844 * This marks a tremendous effort in achieving parity across the board
1845 from Cygwin to Windows and obviously UNIX. MCE now works beautifully
1846 under the Cygwin environment including Perl for Windows.
1847 * Up'ed the maximum workers allowed for both Cygwin and Windows to
1848 56 (forking) and 80 (threading).
1849 * Barrier synchronization requires an extra semaphore file, therefore
1850 the maximum workers allowed under the UNIX environment for threading
1851 decreased. It shouldn't be a problem as there are many threads already.
1852 * Addressed an issue with barrier synchronization under Cygwin.
1853 * Addressed an issue with the die handler for the main worker method
1854 when threading.
1855 * Thank you for hanging in there. It took quite some time to get there.
1856 This is the most stable release thus far for MCE across the board.
1857
1858 1.406 Tue Mar 12 19:00:00 EST 2013
1859
1860 * Added support for barrier synchronization (via new sync method).
1861 Added barrier_sync.pl example.
1862 * Addressed rounding errors with the sequence generator.
1863 The sequence option now follows a bank-teller queuing model when
1864 generating numbers. This applies to task 0 only. Other tasks can
1865 still specify sequence where numbers will be distributed equally
1866 among workers like before.
1867 * Optimized the _worker_request_chunk private method.
1868 * A take 2 on the matrix multiplication examples. This is better
1869 organized with updated README file to include the script running
1870 time in the results.
1871
1872 1.405 Mon Mar 04 19:00:00 EST 2013
1873
1874 * Added strassen_pdl_t.pl in the event folks cannot make use of /dev/shm
1875 used by the strassen_pdl_s.pl example.
1876 * Optimized the 'send' method -- workers process immediately after receiving
1877 data. Updated run times in README for the strassen examples.
1878 * MCE no longer calls setpgrp by default as of MCE 1.405. There is only
1879 one reason to call setpgrp, but many reasons not to. The sole reason
1880 was for MCE to run correctly with Daemon::Control. If needed, one can
1881 pass the option to MCE::Signal qw(-setpgrp).
1882 * Return void in the shutdown method (previously was returning $self).
1883 * Tidy code inside sequence generator.
1884
1885 1.404 Sun Feb 24 13:00:00 EST 2013
1886
1887 * Added sess_dir method
1888 * Completed work with matmult/* examples
1889 Added matmult_pdl_q.pl, removed strassen_pdl_h.pl
1890 Added strassen_pdl_o/p/q/r/s.pl
1891 Added benchmark results from a 32-way box at the end of the readme
1892 * Removed lines setting max limit for files/procs
1893
1894 1.403 Sun Feb 17 15:00:00 EST 2013
1895
1896 * Wrap sub PDL::CLONE_SKIP into a no warnings 'redefine' block
1897 MCE now works with PDL::Parallel::threads without any warnings
1898 * Added missing examples/matmult/matmult_pdl_n.pl to MANIFEST
1899 * Refactored strassen examples, memory consumption reduced by > than 50%
1900 * Added matmult_pdl_o.pl -- uses PDL::Parallel::threads to share matrices
1901 * Added matmult_pdl_p.pl -- matrix b is read from shared memory, not mmap
1902 * Added strassen_pdl_n.pl -- additional improvements to memory reduction
1903 * Added strassen_pdl_h.pl -- shown running with 4 workers (half and half)
1904 * Re-ran matrix multiplication examples and updated results in README file
1905 * Added -no_setpgrp option to MCE::Signal.pm
1906 Ctrl-C does not respond when running /usr/bin/time mce_script.pl
1907 * Added undef $buffer in a couple of places within MCE.pm
1908 * Added David Mertens and Adam Sjøgren to CREDITS
1909 * The 'send' method now checks if sending > total workers after spawning
1910 not before
1911
1912 1.402 Thu Feb 14 07:30:00 EST 2013
1913
1914 * Updated matrix multiplication examples including README
1915 * Added examples/matmult/matmult_pdl_n.pl
1916
1917 1.401 Tue Feb 12 19:00:00 EST 2013
1918
1919 * Added sub PDL::CLONE_SKIP { 1 } to MCE.pm. Running PDL + MCE threads no
1920 longer crashes during exiting.
1921 * Updated matrix multiplication examples. All examples now work under the
1922 Windows environment no matter if threading or forking. Unix is stable as
1923 well if wanting to use PDL + MCE and use_threads => 1 or 0.
1924 * Added benchmark results for 2048x2048, 4096x4096, and 8192x8192 to the
1925 README file under examples/matmult/
1926 * Updated documentation
1927
1928 1.400 Mon Feb 11 07:00:00 EST 2013
1929
1930 * Slight optimization in the _do_callback method
1931 * Added 2 new options: user_args and RS (record separator)
1932 * Added new send method for sending data to workers after spawning and
1933 prior to running
1934 * The sequence option can now take an ARRAY reference
1935 * Updated documentation on new features
1936 * Added matrix multiplication examples
1937
1938 1.306 Sat Jan 05 18:00:00 EST 2013
1939
1940 * Added if statement around setpgrp(0,0). That function is not supported
1941 under Windows.
1942 * Updated logic for removing any remaining MCE session directories inside
1943 MCE::Signal.
1944
1945 1.305 Sat Jan 05 16:00:00 EST 2013
1946
1947 * Added check for $^S to the DIE handler inside the _worker_main method
1948 * Added setpgrp(0,0) to MCE::Signal's BEGIN block
1949 * MCE::Signal points to a _mce_sess_dir hash in the event of a signal,
1950 will remove the sess_dir(s) as well. This is needed when tmp_dir is
1951 specified during instantiation and pointing to another location than
1952 MCE::Signal::tmp_dir.
1953
1954 1.304 Wed Jan 02 22:00:00 EST 2013
1955
1956 * Added Oliver Gorwits to CREDITS for identifying 2 issues
1957 * Direct die to CORE::die inside handler if executing an eval
1958 * Undef $mce_spawned_ref if signal was caught (stop_and_exit)
1959 * Changed INIT to sub import in MCE.pm
1960
1961 1.303 Tue Jan 01 20:00:00 EST 2013
1962
1963 * Bump version -- MCE.pm VERSION now matches with META.yml
1964 * Sorted forchunk, foreach, forseq methods inside MCE.pm
1965 * Modified if statement in run method
1966 * Task ID is never undef, therefore removed check inside
1967 restart_worker and worker_do methods
1968 * Added 2 package variables to MCE::Signal
1969 $display_die_with_localtime and $display_warn_with_localtime
1970 * Completed updates to documentation
1971 * Update to forseq.pl and seq_demo.pl examples
1972
1973 1.302 Tue Jan 01 07:30:00 EST 2013
1974
1975 * Fixed validation logic for sequence
1976 * Updated the sequence generator -- now supports chunking
1977 * Updated seq_demo.pl example to demo user_tasks, sequence, and
1978 chunk_size applied together
1979 * Documentation updates
1980
1981 1.301 Mon Dec 31 15:00:00 EST 2012
1982
1983 * Emphasis on documentation -- better flow plus additional clarity
1984 * Minor updates to sequence option validation
1985 * Minor updates to included examples
1986
1987 1.300 Mon Dec 31 06:00:00 EST 2012
1988
1989 * New methods...: chunk_size, restart_worker, task_id, task_wid, tmp_dir
1990 * New options...: on_post_exit, on_post_run, sequence
1991 * New examples..: forseq.pl, seq_demo.pl
1992 * Overhaul to exit method
1993 Workers can exit or die without impacting the manager process
1994 * Enabled executable bit for test files
1995 * Removed localtime output in die and warn handlers
1996 * All 3 delay options are consistent whether user_tasks is specified
1997 * Removed logic around total_ended count -- replaced with new exit logic
1998 * Code refactoring plus documentation updates
1999 * Added LICENSE file
2000
2001 1.201 Fri Dec 21 00:00:00 EST 2012
2002
2003 * Added MCE.pod -- moved documentation from MCE.pm to pod file
2004 * Added missing use strict/warnings to test scripts
2005 * Default to 1 for chunk_size and max_workers if not specified
2006 * Test::More is not a requirement to run MCE, only for building
2007 * Changed the format for the change log file
2008
2009 1.200 Thu Dec 20 00:00:00 EST 2012
2010
2011 * Added new user_tasks option
2012 * Added space between method name and left-paren for header lines in POD
2013 * Remove not-needed BSD::Resource and forks inside BEGIN/INIT blocks
2014
2015 1.106 Wed Dec 19 05:00:00 EST 2012
2016
2017 * Added t/pod-coverage.t
2018 * Big overhaul of the MCE documentation -- all methods are documented
2019 * Croak if method suited for a MCE worker is called by the main MCE process
2020 * Croak if method suited for the main MCE process is called by a MCE worker
2021 * Updated Makefile.PL to declare the minimum Perl version
2022
2023 1.105 Sun Dec 16 23:00:00 EST 2012
2024
2025 * Completed code re-factoring
2026 * Added t/pod.t
2027
2028 1.104 Sun Nov 25 17:00:00 EST 2012
2029
2030 * Added 1 new example to MCE's Perl documentation
2031 * Use module::method name versus constant symbol when calling _croak
2032 * Croak if session directory is not writeable inside MCE::spawn
2033 * Renamed _mce_id to _mce_sid (meant to be spawn id actually)
2034 * Re-calibrated maximum workers allowed
2035
2036 1.103 Fri Nov 23 13:00:00 EST 2012
2037
2038 * Added writeable check on /dev/shm
2039 * Croak if tmp dir is not writeable inside MCE::Signal::import
2040
2041 1.102 Thu Nov 22 13:00:00 EST 2012
2042
2043 * Woohoot !!! MCE now passes with Perl 5.17.x
2044 * Added Copying file -- same as in Perl
2045
2046 1.101 Wed Nov 21 16:00:00 EST 2012
2047
2048 * Shifted white space to the left for code blocks inside documentation
2049
2050 1.100 Wed Nov 21 10:00:00 EST 2012
2051
2052 * Completed optimization and shakeout for MCE's existing API
2053 * File handles are cached when calling sendto and appending to a file
2054 * The sendto method now supports multiple arguments -- see perldoc
2055 * Added new option: flush_file
2056
2057 1.008 Sat Nov 17 23:00:00 EST 2012
2058
2059 * Updates to __DIE__ and __WARN__ handling in MCE. These address the
2060 unreferenced scalars seen in packaging logs at activestate.com for
2061 Perl under Windows: https://code.activestate.com/ppm/MCE/
2062 * Update t/01_load_signal_arg.t -- added check for $ENV{TEMP}
2063 This fixes issue seen under Cygwin
2064
2065 1.007 Thu Nov 15 21:30:00 EST 2012
2066
2067 * At last, the "Voila" release :)
2068 * Small change to __DIE__ and __WARN__ signal handling for spawn method
2069
2070 1.006 Thu Nov 15 03:30:00 EST 2012
2071
2072 * Added description section to MCE::Signal's Perl doc
2073 * Do not set trap on __DIE__ and __WARN__ inside MCE::Signal
2074 * Localized __DIE__ and __WARN__ handlers inside MCE instead
2075 * Clarify the use of threads in documentation
2076
2077 1.005 Tue Nov 13 06:11:00 EST 2012
2078
2079 * Removed underscore from package globals in MCE::Signal
2080 * Optimized _worker_read_handle method in MCE
2081 * Updated files under examples/tbray/
2082
2083 1.004 Mon Nov 12 01:50:00 EST 2012
2084
2085 * Updated examples/mce_usage.readme
2086 * Updated examples/wide_finder.pl
2087 * Added examples/tbray/README
2088 * Added examples/tbray/tbray_baseline1.pl
2089 * Added examples/tbray/tbray_baseline2.pl
2090 * Added examples/tbray/wf_mce1.pl
2091 * Added examples/tbray/wf_mce2.pl
2092 * Added examples/tbray/wf_mce3.pl (../wide_finder.pl moved here)
2093 * Added examples/tbray/wf_mmap.pl
2094
2095 1.003 Sat Nov 10 12:55:00 EST 2012
2096
2097 * Updated README
2098 * Updated images/06_Shared_Sockets.gif
2099 * Updated images/10_Scaling_Pings.gif
2100 * Added images/11_SNMP_Collection.gif
2101 * Minor updates to MCE::Signal
2102
2103 1.002 Thu Nov 08 01:13:10 EST 2012
2104
2105 * Renamed continue method to next
2106
2107 1.001 Wed Nov 07 23:58:20 EST 2012
2108
2109 * Added perl-MCE.spec to trunk
2110 https://code.google.com/p/many-core-engine-perl/source/browse/trunk/
2111 * Added CREDITS
2112 * Added 3 new methods to MCE.pm: continue, last, and exit
2113 * Both foreach & forchunk now call run(1, {...}) to auto-shutdown workers
2114
2115 1.000 Mon Nov 05 10:00:00 EST 2012
2116
2117 * First release
2118