| 1 |
NAME |
| 2 |
|
| 3 |
Parallel::Runner - An object to manage running things in parallel |
| 4 |
processes. |
| 5 |
|
| 6 |
DESCRIPTION |
| 7 |
|
| 8 |
There are several other modules to do this, you probably want one of |
| 9 |
them. This module exists as a super specialised parallel task manager. |
| 10 |
You create the object with a process limit and callbacks for what to do |
| 11 |
while waiting for a free process slot, as well as a callback for what a |
| 12 |
process should do just before exiting. |
| 13 |
|
| 14 |
You must explicitly call $runner->finish() when you are done. If the |
| 15 |
runner is destroyed before it's children are finished a warning will be |
| 16 |
generated and your child processes will be killed, by force if |
| 17 |
necessary. |
| 18 |
|
| 19 |
If you specify a maximum of 1 then no forking will occur, and run() |
| 20 |
will block until the coderef returns. You can force a fork by providing |
| 21 |
a boolean true value as the second argument to run(), this will force |
| 22 |
the runner to fork before running the coderef, however run() will still |
| 23 |
block until it the child exits. |
| 24 |
|
| 25 |
SYNOPSYS |
| 26 |
|
| 27 |
#!/usr/bin/perl |
| 28 |
use strict; |
| 29 |
use warnings; |
| 30 |
use Parallel::Runner; |
| 31 |
|
| 32 |
my $runner = Parallel::Runner->new(4); |
| 33 |
$runner->run( sub { ... } ); |
| 34 |
$runner->run( sub { ... } ); |
| 35 |
$runner->run( sub { ... } ); |
| 36 |
$runner->run( sub { ... } ); |
| 37 |
|
| 38 |
# This will block until one of the previous 4 finishes |
| 39 |
$runner->run( sub { ... } ); |
| 40 |
|
| 41 |
# Do not forget this. |
| 42 |
$runner->finish; |
| 43 |
|
| 44 |
CONSTRUCTOR |
| 45 |
|
| 46 |
$runner = $class->new( $max, $accessor => $value, ... ); |
| 47 |
|
| 48 |
Create a new instance of Parallel::Runner. $accessor can be anything |
| 49 |
listed under the ACCESSORS section. $max should be the maximum number |
| 50 |
of processes allowed, defaults to 1. |
| 51 |
|
| 52 |
ACCESSORS |
| 53 |
|
| 54 |
These are simple accessors, providing an argument sets the accessor to |
| 55 |
that argument, no argument it simply returns the current value. |
| 56 |
|
| 57 |
$val = $runner->data_callback( \&callback ) |
| 58 |
|
| 59 |
If this is specified than IPC will be automatically enabled, and the |
| 60 |
final return from each process will be passed into this handler in |
| 61 |
the main process. Due to the way IPC works only strings/numerical |
| 62 |
data is passed, if you need to pass a ref you will need to serialize |
| 63 |
it yourself before returning it, followed by deserializing it in your |
| 64 |
callback. |
| 65 |
|
| 66 |
Example: |
| 67 |
|
| 68 |
# Place to put the accumulated data |
| 69 |
my @accum_data; |
| 70 |
|
| 71 |
# Create the runner with a callback that pushes the data onto our array. |
| 72 |
$runner = $CLASS->new( 2, |
| 73 |
data_callback => sub { |
| 74 |
my ($data) = @_; |
| 75 |
push @accum_data => $data; |
| 76 |
}, |
| 77 |
); |
| 78 |
|
| 79 |
# 4 processes that return data |
| 80 |
$runner->run( sub { return "foo" }); |
| 81 |
$runner->run( sub { return "bar" }); |
| 82 |
$runner->run( sub { return "baz" }); |
| 83 |
$runner->run( sub { return "bat" }); |
| 84 |
$runner->finish; |
| 85 |
|
| 86 |
# Verify the data (order is not predictable) |
| 87 |
is_deeply( |
| 88 |
[ sort @accum_data ], |
| 89 |
[ sort qw/foo bar baz bat/ ], |
| 90 |
"Got all data returned by subprocesses" |
| 91 |
); |
| 92 |
|
| 93 |
$val = $runner->exit_callback( \&callback ) |
| 94 |
|
| 95 |
Codref to call just before a child exits (called within child) |
| 96 |
|
| 97 |
$val = $runner->iteration_delay( $float ); |
| 98 |
|
| 99 |
How long to wait per iterate if nothing has changed. |
| 100 |
|
| 101 |
$val = $runner->iteration_callback( $newval ) |
| 102 |
|
| 103 |
Coderef to call multiple times in a loop while run() is blocking |
| 104 |
waiting for a process slot. |
| 105 |
|
| 106 |
$val = $runner->reap_callback( $newval ) |
| 107 |
|
| 108 |
Codref to call whenever a pid is reaped using waitpid. The callback |
| 109 |
sub will be passed 3 values The first is the exit status of the child |
| 110 |
process. The second is the pid of the child process. The third used |
| 111 |
to be the return of waitpid, but this is deprecated as Child is now |
| 112 |
used and throws an exception when waitpid is not what it should be. |
| 113 |
The third is simply the pid of the child process again. The final |
| 114 |
argument is the child process object itself. |
| 115 |
|
| 116 |
$runner->reap_callback( sub { |
| 117 |
my ( $status, $pid, $pid_again, $proc ) = @_; |
| 118 |
|
| 119 |
# Status as returned from system, so 0 is good, 1+ is bad. |
| 120 |
die "Child $pid did not exit 0" |
| 121 |
if $status; |
| 122 |
}); |
| 123 |
|
| 124 |
@children = $runner->children( @append ) |
| 125 |
|
| 126 |
Returns a list of Child::Link::Proc objects. |
| 127 |
|
| 128 |
$val = $runner->pid() |
| 129 |
|
| 130 |
pid of the parent process |
| 131 |
|
| 132 |
$val = $runner->max( $newval ) |
| 133 |
|
| 134 |
Maximum number of children |
| 135 |
|
| 136 |
OBJECT METHODS |
| 137 |
|
| 138 |
run( $code ) |
| 139 |
|
| 140 |
run( $code, $force_fork ) |
| 141 |
|
| 142 |
Run the specified code in a child process. Blocks if no free slots |
| 143 |
are available. Force fork can be used to force a fork when max is 1, |
| 144 |
however it will still block until the child exits. |
| 145 |
|
| 146 |
finish() |
| 147 |
|
| 148 |
finish( $timeout ) |
| 149 |
|
| 150 |
finish( $timeout, $timeoutcallback ) |
| 151 |
|
| 152 |
Wait for all children to finish, then clean up after them. If a |
| 153 |
timeout is specified it will return after the timeout regardless of |
| 154 |
wether or not children have all exited. If there is a timeout call |
| 155 |
back then that code will be run upon timeout just before the method |
| 156 |
returns. |
| 157 |
|
| 158 |
NOTE: DO NOT LET YOUR RUNNER BE DESTROYED BEFORE FINISH COMPLETES |
| 159 |
WITHOUT A TIMEOUT. |
| 160 |
|
| 161 |
the runner will kill all children, possibly with force if your runner |
| 162 |
is destroyed with children still running, or not waited on. |
| 163 |
|
| 164 |
killall( $sig ) |
| 165 |
|
| 166 |
Send all children the specified kill signal. |
| 167 |
|
| 168 |
DESTROY() |
| 169 |
|
| 170 |
Automagically called when the object is destroyed. If called while |
| 171 |
children are running it will forcefully clean up after you as |
| 172 |
follows: |
| 173 |
|
| 174 |
1) Sends an ugly warning. |
| 175 |
|
| 176 |
2) Will first give all your children 1 second to complete. |
| 177 |
|
| 178 |
Windows) Strawberry fails with processes, so on windows DESTROY will |
| 179 |
wait as long as needed, possibly forever. |
| 180 |
|
| 181 |
3) Sends kill signal 15 to all children then waits up to 4 seconds. |
| 182 |
|
| 183 |
4) Sends kill signal 9 to any remaining children then waits up to 10 |
| 184 |
seconds |
| 185 |
|
| 186 |
5) Gives up and returns |
| 187 |
|
| 188 |
FENNEC PROJECT |
| 189 |
|
| 190 |
This module is part of the Fennec project. See Fennec for more details. |
| 191 |
Fennec is a project to develop an extendable and powerful testing |
| 192 |
framework. Together the tools that make up the Fennec framework provide |
| 193 |
a potent testing environment. |
| 194 |
|
| 195 |
The tools provided by Fennec are also useful on their own. Sometimes a |
| 196 |
tool created for Fennec is useful outside the greater framework. Such |
| 197 |
tools are turned into their own projects. This is one such project. |
| 198 |
|
| 199 |
Fennec - The core framework |
| 200 |
|
| 201 |
The primary Fennec project that ties them all together. |
| 202 |
|
| 203 |
AUTHORS |
| 204 |
|
| 205 |
Chad Granum [email protected] |
| 206 |
|
| 207 |
COPYRIGHT |
| 208 |
|
| 209 |
Copyright (C) 2010 Chad Granum |
| 210 |
|
| 211 |
Parallel-Runner is free software; Standard perl licence. |
| 212 |
|
| 213 |
Parallel-Runner is distributed in the hope that it will be useful, but |
| 214 |
WITHOUT ANY WARRANTY; without even the implied warranty of |
| 215 |
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the license |
| 216 |
for more details. |
| 217 |
|