Debian Codegraph @

README

raw
1 NAME
2 Message::Passing::ZeroMQ - input and output messages to ZeroMQ.
3
4 SYNOPSIS
5 # Terminal 1:
6 $ message-passing --input STDIN --output ZeroMQ --output_options '{"connect":"tcp://127.0.0.1:5552"}'
7 {"data":{"some":"data"},"@metadata":"value"}
8
9 # Terminal 2:
10 $ message-passing --output STDOUT --input ZeroMQ --input_options '{"socket_bind":"tcp://*:5552"}'
11 {"data":{"some":"data"},"@metadata":"value"}
12
13 DESCRIPTION
14 A ZeroMQ transport for Message::Passing.
15
16 Designed for use as a log transport and aggregation mechanism for perl
17 applications, allowing you to aggregate structured and non-structured
18 log messages across the network in a non-blocking manor.
19
20 Clients (I.e. users of the Message::Passing::Output::ZeroMQ class)
21 connect to a server (I.e. a user of the Message::Passing::Input::ZeroMQ
22 class) via ZeroMQ's pub/sub sockets. These are setup to be lossy and
23 non-blocking, meaning that if the log-receiver process is down or slow,
24 then the application will queue a small (and configurable) amount of
25 logs on it's side, and after that log messages will be dropped.
26
27 Whilst throwing away log messages isn't a good thing to do, or something
28 that you want to happen regularly, in many (especially web application)
29 contexts, network logging being a single point of failure is not
30 acceptable from a reliability and graceful degradation standpoint.
31
32 The application grinding to a halt as a non-essential centralised
33 resource is unavailable (e.g. the log aggregation server) is
34 significantly less acceptable than the loss of non-essential logging
35 data.
36
37 HOW TO USE
38 In your application emitting messages, you can either use
39 Message::Passing::Output::ZeroMQ directly, or you can use it via
40 Log::Dispatch::Message::Passing.
41
42 use Log::Dispatch;
43 use Log::Dispatch::Message::Passing;
44 use Message::Passing::Output::ZeroMQ;
45 use Message::Passing::Filter::Encode::JSON;
46
47 my $log = Log::Dispatch->new;
48
49 $log->add(Log::Dispatch::Message::Passing->new(
50 name => 'myapp_aggregate_log',
51 min_level => 'debug',
52 output => Message::Passing::Filter::Encode::JSON->new(
53 output_to => Message::Passing::Output::ZeroMQ->new(
54 connect => 'tcp://192.168.0.1:5558',
55 )
56 ),
57 ));
58
59 $log->warn($_) for qw/ foo bar baz /;
60
61 On your log aggregation server, just run the message-passing utility:
62
63 message-passing --input ZeroMQ --input_options '{"socket_bind":"tcp://*:5222"}' \
64 --output File --output_options '{"filename":"/tmp/my_test.log"}'
65
66 SOCKET TYPES
67 ZeroMQ supports multiple socket types, the only ones used in
68 Message::Passing::ZeroMQ are:
69
70 PUB/SUB
71 Used for general message distribution - you can have either multiple
72 producers (PUB) which connect to one consumer (SUB), or multiple
73 consumers (SUB) which connect to one producer (PUB).
74
75 All consumers will get a copy of every message.
76
77 In Message::Passing terms, Message::Passing::Input::ZeroMQ is for SUB
78 sockets, and Message::Passing::Output::ZeroMQ is for PUB sockets.
79
80 PUSH/PULL
81 Used for message distribution. A sever (PUSH) distributes messages
82 between a number of connecting clients (PULL)
83
84 In Message::Passing terms, Message::Passing::Input::ZeroMQ is for PULL
85 sockets, and Message::Passing::Output::ZeroMQ is for PUSH sockets.
86
87 CONNECTION DIRECTION
88 Note that in ZeroMQ, the connection direction and the direction of
89 message flow can be entirely opposite. I.e. a client can connect to a
90 server and send messages to it, or receive messages from it (depending
91 on the direction of the socket types).
92
93 CONNECTION ATTRIBUTES
94 Both Message::Passing::Input::ZeroMQ and
95 Message::Passing::Output::ZeroMQ support either binding a server or
96 connecting to a remote host, due to the fact that ZeroMQ connections can
97 be in any direction, as noted above.
98
99 Therefore, each input or output should have one (but not both!) of the
100 following attributes:
101
102 connect
103 Connects to a remote server, e.g. "tcp://192.168.0.1:5222"
104
105 socket_bind
106 Binds a server and waits for connections from clients, e.g.
107 "tcp://*:5222"
108
109 socket_type
110 This defaults to "SUB" for Message::Passing::Input::ZeroMQ and "PUB" for
111 Message::Passing::Output::ZeroMQ, however you can override it to
112 "PUSH"/"PULL" as appropriate for your use case if desired.
113
114 MORE COMPLEX EXAMPLES
115 With this in mind, we can easily create a system which aggregates
116 messages from multiple publishers, and passes them out (in a round-robin
117 fashion) to a pool of workers.
118
119 # The message distributor:
120 message-passing --input ZeroMQ --input_options '{"socket_bind":"tcp://*:5222"}' \
121 --output ZeroMQ --output_options '{"socket_bind":"tcp://*:5223","socket_type":"PUSH"}'
122
123 # Workers
124 {
125 package MyApp::MessageWorker;
126 use Moo;
127
128 with 'Message::Passing::Role::Filter';
129
130 sub filter {
131 my ($self, $message) = @_;
132 # .... process the message in any way you want here
133 return undef; # Do not output the message..
134 }
135 }
136
137 message-passing --input ZeroMQ --input_options '{"connect":"tcp://127.0.0.1:5223","socket_type":"PULL"}'
138 --filter '+MyApp::MessageWorker'
139 --output STDOUT
140
141 You log messages into the distributor as per the above simple example,
142 and you can run multiple worker processes..
143
144 Less trivial setups could/would emit messages on error, or maybe re-emit
145 the incoming message after transforming it in some way.
146
147 SEE ALSO
148 For more detailed information about ZeroMQ and how it works, please
149 consult the ZeroMQ guide and the other links below:
150
151 Message::Passing::Output::ZeroMQ
152 Message::Passing::Input::ZeroMQ
153 Message::Passing
154 ZMQ::FFI
155 <http://www.zeromq.org/>
156 <http://zguide.zeromq.org/page:all>
157
158 AUTHOR
159 Tomas (t0m) Doran <[email protected]>
160
161 SPONSORSHIP
162 This module exists due to the wonderful people at Suretec Systems Ltd.
163 <http://www.suretecsystems.com/> who sponsored its development for its
164 VoIP division called SureVoIP <http://www.surevoip.co.uk/> for use with
165 the SureVoIP API -
166 <http://www.surevoip.co.uk/support/wiki/api_documentation>
167
168 COPYRIGHT
169 Copyright Suretec Systems 2012.
170
171 LICENSE
172 GNU Affero General Public License, Version 3
173
174 If you feel this is too restrictive to be able to use this software,
175 please talk to us as we'd be willing to consider re-licensing under less
176 restrictive terms.
177