| 1 |
NAME |
| 2 |
Message::Passing::ZeroMQ - input and output messages to ZeroMQ. |
| 3 |
|
| 4 |
SYNOPSIS |
| 5 |
# Terminal 1: |
| 6 |
$ message-passing --input STDIN --output ZeroMQ --output_options '{"connect":"tcp://127.0.0.1:5552"}' |
| 7 |
{"data":{"some":"data"},"@metadata":"value"} |
| 8 |
|
| 9 |
# Terminal 2: |
| 10 |
$ message-passing --output STDOUT --input ZeroMQ --input_options '{"socket_bind":"tcp://*:5552"}' |
| 11 |
{"data":{"some":"data"},"@metadata":"value"} |
| 12 |
|
| 13 |
DESCRIPTION |
| 14 |
A ZeroMQ transport for Message::Passing. |
| 15 |
|
| 16 |
Designed for use as a log transport and aggregation mechanism for perl |
| 17 |
applications, allowing you to aggregate structured and non-structured |
| 18 |
log messages across the network in a non-blocking manor. |
| 19 |
|
| 20 |
Clients (I.e. users of the Message::Passing::Output::ZeroMQ class) |
| 21 |
connect to a server (I.e. a user of the Message::Passing::Input::ZeroMQ |
| 22 |
class) via ZeroMQ's pub/sub sockets. These are setup to be lossy and |
| 23 |
non-blocking, meaning that if the log-receiver process is down or slow, |
| 24 |
then the application will queue a small (and configurable) amount of |
| 25 |
logs on it's side, and after that log messages will be dropped. |
| 26 |
|
| 27 |
Whilst throwing away log messages isn't a good thing to do, or something |
| 28 |
that you want to happen regularly, in many (especially web application) |
| 29 |
contexts, network logging being a single point of failure is not |
| 30 |
acceptable from a reliability and graceful degradation standpoint. |
| 31 |
|
| 32 |
The application grinding to a halt as a non-essential centralised |
| 33 |
resource is unavailable (e.g. the log aggregation server) is |
| 34 |
significantly less acceptable than the loss of non-essential logging |
| 35 |
data. |
| 36 |
|
| 37 |
HOW TO USE |
| 38 |
In your application emitting messages, you can either use |
| 39 |
Message::Passing::Output::ZeroMQ directly, or you can use it via |
| 40 |
Log::Dispatch::Message::Passing. |
| 41 |
|
| 42 |
use Log::Dispatch; |
| 43 |
use Log::Dispatch::Message::Passing; |
| 44 |
use Message::Passing::Output::ZeroMQ; |
| 45 |
use Message::Passing::Filter::Encode::JSON; |
| 46 |
|
| 47 |
my $log = Log::Dispatch->new; |
| 48 |
|
| 49 |
$log->add(Log::Dispatch::Message::Passing->new( |
| 50 |
name => 'myapp_aggregate_log', |
| 51 |
min_level => 'debug', |
| 52 |
output => Message::Passing::Filter::Encode::JSON->new( |
| 53 |
output_to => Message::Passing::Output::ZeroMQ->new( |
| 54 |
connect => 'tcp://192.168.0.1:5558', |
| 55 |
) |
| 56 |
), |
| 57 |
)); |
| 58 |
|
| 59 |
$log->warn($_) for qw/ foo bar baz /; |
| 60 |
|
| 61 |
On your log aggregation server, just run the message-passing utility: |
| 62 |
|
| 63 |
message-passing --input ZeroMQ --input_options '{"socket_bind":"tcp://*:5222"}' \ |
| 64 |
--output File --output_options '{"filename":"/tmp/my_test.log"}' |
| 65 |
|
| 66 |
SOCKET TYPES |
| 67 |
ZeroMQ supports multiple socket types, the only ones used in |
| 68 |
Message::Passing::ZeroMQ are: |
| 69 |
|
| 70 |
PUB/SUB |
| 71 |
Used for general message distribution - you can have either multiple |
| 72 |
producers (PUB) which connect to one consumer (SUB), or multiple |
| 73 |
consumers (SUB) which connect to one producer (PUB). |
| 74 |
|
| 75 |
All consumers will get a copy of every message. |
| 76 |
|
| 77 |
In Message::Passing terms, Message::Passing::Input::ZeroMQ is for SUB |
| 78 |
sockets, and Message::Passing::Output::ZeroMQ is for PUB sockets. |
| 79 |
|
| 80 |
PUSH/PULL |
| 81 |
Used for message distribution. A sever (PUSH) distributes messages |
| 82 |
between a number of connecting clients (PULL) |
| 83 |
|
| 84 |
In Message::Passing terms, Message::Passing::Input::ZeroMQ is for PULL |
| 85 |
sockets, and Message::Passing::Output::ZeroMQ is for PUSH sockets. |
| 86 |
|
| 87 |
CONNECTION DIRECTION |
| 88 |
Note that in ZeroMQ, the connection direction and the direction of |
| 89 |
message flow can be entirely opposite. I.e. a client can connect to a |
| 90 |
server and send messages to it, or receive messages from it (depending |
| 91 |
on the direction of the socket types). |
| 92 |
|
| 93 |
CONNECTION ATTRIBUTES |
| 94 |
Both Message::Passing::Input::ZeroMQ and |
| 95 |
Message::Passing::Output::ZeroMQ support either binding a server or |
| 96 |
connecting to a remote host, due to the fact that ZeroMQ connections can |
| 97 |
be in any direction, as noted above. |
| 98 |
|
| 99 |
Therefore, each input or output should have one (but not both!) of the |
| 100 |
following attributes: |
| 101 |
|
| 102 |
connect |
| 103 |
Connects to a remote server, e.g. "tcp://192.168.0.1:5222" |
| 104 |
|
| 105 |
socket_bind |
| 106 |
Binds a server and waits for connections from clients, e.g. |
| 107 |
"tcp://*:5222" |
| 108 |
|
| 109 |
socket_type |
| 110 |
This defaults to "SUB" for Message::Passing::Input::ZeroMQ and "PUB" for |
| 111 |
Message::Passing::Output::ZeroMQ, however you can override it to |
| 112 |
"PUSH"/"PULL" as appropriate for your use case if desired. |
| 113 |
|
| 114 |
MORE COMPLEX EXAMPLES |
| 115 |
With this in mind, we can easily create a system which aggregates |
| 116 |
messages from multiple publishers, and passes them out (in a round-robin |
| 117 |
fashion) to a pool of workers. |
| 118 |
|
| 119 |
# The message distributor: |
| 120 |
message-passing --input ZeroMQ --input_options '{"socket_bind":"tcp://*:5222"}' \ |
| 121 |
--output ZeroMQ --output_options '{"socket_bind":"tcp://*:5223","socket_type":"PUSH"}' |
| 122 |
|
| 123 |
# Workers |
| 124 |
{ |
| 125 |
package MyApp::MessageWorker; |
| 126 |
use Moo; |
| 127 |
|
| 128 |
with 'Message::Passing::Role::Filter'; |
| 129 |
|
| 130 |
sub filter { |
| 131 |
my ($self, $message) = @_; |
| 132 |
# .... process the message in any way you want here |
| 133 |
return undef; # Do not output the message.. |
| 134 |
} |
| 135 |
} |
| 136 |
|
| 137 |
message-passing --input ZeroMQ --input_options '{"connect":"tcp://127.0.0.1:5223","socket_type":"PULL"}' |
| 138 |
--filter '+MyApp::MessageWorker' |
| 139 |
--output STDOUT |
| 140 |
|
| 141 |
You log messages into the distributor as per the above simple example, |
| 142 |
and you can run multiple worker processes.. |
| 143 |
|
| 144 |
Less trivial setups could/would emit messages on error, or maybe re-emit |
| 145 |
the incoming message after transforming it in some way. |
| 146 |
|
| 147 |
SEE ALSO |
| 148 |
For more detailed information about ZeroMQ and how it works, please |
| 149 |
consult the ZeroMQ guide and the other links below: |
| 150 |
|
| 151 |
Message::Passing::Output::ZeroMQ |
| 152 |
Message::Passing::Input::ZeroMQ |
| 153 |
Message::Passing |
| 154 |
ZMQ::FFI |
| 155 |
<http://www.zeromq.org/> |
| 156 |
<http://zguide.zeromq.org/page:all> |
| 157 |
|
| 158 |
AUTHOR |
| 159 |
Tomas (t0m) Doran <[email protected]> |
| 160 |
|
| 161 |
SPONSORSHIP |
| 162 |
This module exists due to the wonderful people at Suretec Systems Ltd. |
| 163 |
<http://www.suretecsystems.com/> who sponsored its development for its |
| 164 |
VoIP division called SureVoIP <http://www.surevoip.co.uk/> for use with |
| 165 |
the SureVoIP API - |
| 166 |
<http://www.surevoip.co.uk/support/wiki/api_documentation> |
| 167 |
|
| 168 |
COPYRIGHT |
| 169 |
Copyright Suretec Systems 2012. |
| 170 |
|
| 171 |
LICENSE |
| 172 |
GNU Affero General Public License, Version 3 |
| 173 |
|
| 174 |
If you feel this is too restrictive to be able to use this software, |
| 175 |
please talk to us as we'd be willing to consider re-licensing under less |
| 176 |
restrictive terms. |
| 177 |
|