Debian Codegraph @

vendor/clue/mq-react/README.md

raw
1 # clue/reactphp-mq
2
3 [![CI status](https://github.com/clue/reactphp-mq/actions/workflows/ci.yml/badge.svg)](https://github.com/clue/reactphp-mq/actions)
4 [![code coverage](https://img.shields.io/badge/code%20coverage-100%25-success)](#tests)
5 [![installs on Packagist](https://img.shields.io/packagist/dt/clue/mq-react?color=blue&label=installs%20on%20Packagist)](https://packagist.org/packages/clue/mq-react)
6
7 Mini Queue, the lightweight in-memory message queue to concurrently do many (but not too many) things at once,
8 built on top of [ReactPHP](https://reactphp.org/).
9
10 Let's say you crawl a page and find that you need to send 100 HTTP requests to
11 following pages which each takes `0.2s`. You can either send them all
12 sequentially (taking around `20s`) or you can use
13 [ReactPHP](https://reactphp.org/) to concurrently request all your pages at the
14 same time. This works perfectly fine for a small number of operations, but
15 sending an excessive number of requests can either take up all resources on your
16 side or may get you banned by the remote side as it sees an unreasonable number
17 of requests from your side.
18 Instead, you can use this library to effectively rate limit your operations and
19 queue excessives ones so that not too many operations are processed at once.
20 This library provides a simple API that is easy to use in order to manage any
21 kind of async operation without having to mess with most of the low-level details.
22 You can use this to throttle multiple HTTP requests, database queries or pretty
23 much any API that already uses Promises.
24
25 * **Async execution of operations** -
26 Process any number of async operations and choose how many should be handled
27 concurrently and how many operations can be queued in-memory. Process their
28 results as soon as responses come in.
29 The Promise-based design provides a *sane* interface to working with out of order results.
30 * **Lightweight, SOLID design** -
31 Provides a thin abstraction that is [*just good enough*](https://en.wikipedia.org/wiki/Principle_of_good_enough)
32 and does not get in your way.
33 Builds on top of well-tested components and well-established concepts instead of reinventing the wheel.
34 * **Good test coverage** -
35 Comes with an automated tests suite and is regularly tested in the *real world*.
36
37 **Table of contents**
38
39 * [Support us](#support-us)
40 * [Quickstart example](#quickstart-example)
41 * [Usage](#usage)
42 * [Queue](#queue)
43 * [Promises](#promises)
44 * [Cancellation](#cancellation)
45 * [Timeout](#timeout)
46 * [all()](#all)
47 * [any()](#any)
48 * [Blocking](#blocking)
49 * [Install](#install)
50 * [Tests](#tests)
51 * [License](#license)
52
53 ## Support us
54
55 We invest a lot of time developing, maintaining and updating our awesome
56 open-source projects. You can help us sustain this high-quality of our work by
57 [becoming a sponsor on GitHub](https://github.com/sponsors/clue). Sponsors get
58 numerous benefits in return, see our [sponsoring page](https://github.com/sponsors/clue)
59 for details.
60
61 Let's take these projects to the next level together! 🚀
62
63 ## Quickstart example
64
65 Once [installed](#install), you can use the following code to access an
66 HTTP webserver and send a large number of HTTP GET requests:
67
68 ```php
69 <?php
70
71 require __DIR__ . '/vendor/autoload.php';
72
73 $browser = new React\Http\Browser();
74
75 // load a huge array of URLs to fetch
76 $urls = file('urls.txt');
77
78 // each job should use the browser to GET a certain URL
79 // limit number of concurrent jobs here
80 $q = new Clue\React\Mq\Queue(3, null, function ($url) use ($browser) {
81 return $browser->get($url);
82 });
83
84 foreach ($urls as $url) {
85 $q($url)->then(function (Psr\Http\Message\ResponseInterface $response) use ($url) {
86 echo $url . ': ' . $response->getBody()->getSize() . ' bytes' . PHP_EOL;
87 }, function (Exception $e) {
88 echo 'Error: ' . $e->getMessage() . PHP_EOL;
89 });
90 }
91
92 ```
93
94 See also the [examples](examples/).
95
96 ## Usage
97
98 ### Queue
99
100 The `Queue` is responsible for managing your operations and ensuring not too
101 many operations are executed at once. It's a very simple and lightweight
102 in-memory implementation of the
103 [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket#As_a_queue) algorithm.
104
105 This means that you control how many operations can be executed concurrently.
106 If you add a job to the queue and it still below the limit, it will be executed
107 immediately. If you keep adding new jobs to the queue and its concurrency limit
108 is reached, it will not start a new operation and instead queue this for future
109 execution. Once one of the pending operations complete, it will pick the next
110 job from the queue and execute this operation.
111
112 The `new Queue(int $concurrency, ?int $limit, callable(mixed):PromiseInterface<T> $handler)` call
113 can be used to create a new queue instance.
114 You can create any number of queues, for example when you want to apply
115 different limits to different kinds of operations.
116
117 The `$concurrency` parameter sets a new soft limit for the maximum number
118 of jobs to handle concurrently. Finding a good concurrency limit depends
119 on your particular use case. It's common to limit concurrency to a rather
120 small value, as doing more than a dozen of things at once may easily
121 overwhelm the receiving side.
122
123 The `$limit` parameter sets a new hard limit on how many jobs may be
124 outstanding (kept in memory) at once. Depending on your particular use
125 case, it's usually safe to keep a few hundreds or thousands of jobs in
126 memory. If you do not want to apply an upper limit, you can pass a `null`
127 value which is semantically more meaningful than passing a big number.
128
129 ```php
130 // handle up to 10 jobs concurrently, but keep no more than 1000 in memory
131 $q = new Queue(10, 1000, $handler);
132 ```
133
134 ```php
135 // handle up to 10 jobs concurrently, do not limit queue size
136 $q = new Queue(10, null, $handler);
137 ```
138
139 ```php
140 // handle up to 10 jobs concurrently, reject all further jobs
141 $q = new Queue(10, 10, $handler);
142 ```
143
144 The `$handler` parameter must be a valid callable that accepts your job
145 parameters, invokes the appropriate operation and returns a Promise as a
146 placeholder for its future result.
147
148 ```php
149 // using a Closure as handler is usually recommended
150 $q = new Queue(10, null, function ($url) use ($browser) {
151 return $browser->get($url);
152 });
153 ```
154
155 ```php
156 // accepts any callable, so PHP's array notation is also supported
157 $q = new Queue(10, null, array($browser, 'get'));
158 ```
159
160 #### Promises
161
162 This library works under the assumption that you want to concurrently handle
163 async operations that use a [Promise](https://github.com/reactphp/promise)-based API.
164
165 The demonstration purposes, the examples in this documentation use
166 [ReactPHP's async HTTP client](https://github.com/reactphp/http#client-usage), but you
167 may use any Promise-based API with this project. Its API can be used like this:
168
169 ```php
170 $browser = new React\Http\Browser();
171
172 $promise = $browser->get($url);
173 ```
174
175 If you wrap this in a `Queue` instance as given above, this code will look
176 like this:
177
178 ```php
179 $browser = new React\Http\Browser();
180
181 $q = new Queue(10, null, function ($url) use ($browser) {
182 return $browser->get($url);
183 });
184
185 $promise = $q($url);
186 ```
187
188 The `$q` instance is invokable, so that invoking `$q(...$args)` will
189 actually be forwarded as `$browser->get(...$args)` as given in the
190 `$handler` argument when concurrency is still below limits.
191
192 Each operation is expected to be async (non-blocking), so you may actually
193 invoke multiple operations concurrently (send multiple requests in parallel).
194 The `$handler` is responsible for responding to each request with a resolution
195 value, the order is not guaranteed.
196 These operations use a [Promise](https://github.com/reactphp/promise)-based
197 interface that makes it easy to react to when an operation is completed (i.e.
198 either successfully fulfilled or rejected with an error):
199
200 ```php
201 $promise->then(
202 function ($result) {
203 var_dump('Result received', $result);
204 },
205 function (Exception $error) {
206 var_dump('There was an error', $error->getMessage());
207 }
208 );
209 ```
210
211 Each operation may take some time to complete, but due to its async nature you
212 can actually start any number of (queued) operations. Once the concurrency limit
213 is reached, this invocation will simply be queued and this will return a pending
214 promise which will start the actual operation once another operation is
215 completed. This means that this is handled entirely transparently and you do not
216 need to worry about this concurrency limit yourself.
217
218 If this looks strange to you, you can also use the more traditional
219 [blocking API](#blocking).
220
221 #### Cancellation
222
223 The returned Promise is implemented in such a way that it can be cancelled
224 when it is still pending.
225 Cancelling a pending operation will invoke its cancellation handler which is
226 responsible for rejecting its value with an Exception and cleaning up any
227 underlying resources.
228
229 ```php
230 $promise = $q($url);
231
232 Loop::addTimer(2.0, function () use ($promise) {
233 $promise->cancel();
234 });
235 ```
236
237 Similarly, cancelling an operation that is queued and has not yet been started
238 will be rejected without ever starting the operation.
239
240 #### Timeout
241
242 By default, this library does not limit how long a single operation can take,
243 so that the resulting promise may stay pending for a long time.
244 Many use cases involve some kind of "timeout" logic so that an operation is
245 cancelled after a certain threshold is reached.
246
247 You can simply use [cancellation](#cancellation) as in the previous chapter or
248 you may want to look into using [react/promise-timer](https://github.com/reactphp/promise-timer)
249 which helps taking care of this through a simple API.
250
251 The resulting code with timeouts applied look something like this:
252
253 ```php
254 use React\Promise\Timer;
255
256 $q = new Queue(10, null, function ($uri) use ($browser) {
257 return Timer\timeout($browser->get($uri), 2.0);
258 });
259
260 $promise = $q($uri);
261 ```
262
263 The resulting promise can be consumed as usual and the above code will ensure
264 that execution of this operation can not take longer than the given timeout
265 (i.e. after it is actually started).
266 In particular, note how this differs from applying a timeout to the resulting
267 promise. The following code will ensure that the total time for queuing and
268 executing this operation can not take longer than the given timeout:
269
270 ```php
271 // usually not recommended
272 $promise = Timer\timeout($q($url), 2.0);
273 ```
274
275 Please refer to [react/promise-timer](https://github.com/reactphp/promise-timer)
276 for more details.
277
278 #### all()
279
280 The static `all(int $concurrency, array<TKey,TIn> $jobs, callable(TIn):PromiseInterface<TOut> $handler): PromiseInterface<array<TKey,TOut>>` method can be used to
281 concurrently process all given jobs through the given `$handler`.
282
283 This is a convenience method which uses the `Queue` internally to
284 schedule all jobs while limiting concurrency to ensure no more than
285 `$concurrency` jobs ever run at once. It will return a promise which
286 resolves with the results of all jobs on success.
287
288 ```php
289 $browser = new React\Http\Browser();
290
291 $promise = Queue::all(3, $urls, function ($url) use ($browser) {
292 return $browser->get($url);
293 });
294
295 $promise->then(function (array $responses) {
296 echo 'All ' . count($responses) . ' successful!' . PHP_EOL;
297 }, function (Exception $e) {
298 echo 'Error: ' . $e->getMessage() . PHP_EOL;
299 });
300 ```
301
302 If either of the jobs fail, it will reject the resulting promise and will
303 try to cancel all outstanding jobs. Similarly, calling `cancel()` on the
304 resulting promise will try to cancel all outstanding jobs. See
305 [promises](#promises) and [cancellation](#cancellation) for details.
306
307 The `$concurrency` parameter sets a new soft limit for the maximum number
308 of jobs to handle concurrently. Finding a good concurrency limit depends
309 on your particular use case. It's common to limit concurrency to a rather
310 small value, as doing more than a dozen of things at once may easily
311 overwhelm the receiving side. Using a `1` value will ensure that all jobs
312 are processed one after another, effectively creating a "waterfall" of
313 jobs. Using a value less than 1 will reject with an
314 `InvalidArgumentException` without processing any jobs.
315
316 ```php
317 // handle up to 10 jobs concurrently
318 $promise = Queue::all(10, $jobs, $handler);
319 ```
320
321 ```php
322 // handle each job after another without concurrency (waterfall)
323 $promise = Queue::all(1, $jobs, $handler);
324 ```
325
326 The `$jobs` parameter must be an array with all jobs to process. Each
327 value in this array will be passed to the `$handler` to start one job.
328 The array keys will be preserved in the resulting array, while the array
329 values will be replaced with the job results as returned by the
330 `$handler`. If this array is empty, this method will resolve with an
331 empty array without processing any jobs.
332
333 The `$handler` parameter must be a valid callable that accepts your job
334 parameters, invokes the appropriate operation and returns a Promise as a
335 placeholder for its future result. If the given argument is not a valid
336 callable, this method will reject with an `InvalidArgumentException`
337 without processing any jobs.
338
339 ```php
340 // using a Closure as handler is usually recommended
341 $promise = Queue::all(10, $jobs, function ($url) use ($browser) {
342 return $browser->get($url);
343 });
344 ```
345
346 ```php
347 // accepts any callable, so PHP's array notation is also supported
348 $promise = Queue::all(10, $jobs, array($browser, 'get'));
349 ```
350
351 > Keep in mind that returning an array of response messages means that
352 the whole response body has to be kept in memory.
353
354 #### any()
355
356 The static `any(int $concurrency, array<TKey,TIn> $jobs, callable(TIn):Promise<TOut> $handler): PromiseInterface<TOut>` method can be used to
357 concurrently process the given jobs through the given `$handler` and
358 resolve with first resolution value.
359
360 This is a convenience method which uses the `Queue` internally to
361 schedule all jobs while limiting concurrency to ensure no more than
362 `$concurrency` jobs ever run at once. It will return a promise which
363 resolves with the result of the first job on success and will then try
364 to `cancel()` all outstanding jobs.
365
366 ```php
367 $browser = new React\Http\Browser();
368
369 $promise = Queue::any(3, $urls, function ($url) use ($browser) {
370 return $browser->get($url);
371 });
372
373 $promise->then(function (ResponseInterface $response) {
374 echo 'First response: ' . $response->getBody() . PHP_EOL;
375 }, function (Exception $e) {
376 echo 'Error: ' . $e->getMessage() . PHP_EOL;
377 });
378 ```
379
380 If all of the jobs fail, it will reject the resulting promise. Similarly,
381 calling `cancel()` on the resulting promise will try to cancel all
382 outstanding jobs. See [promises](#promises) and
383 [cancellation](#cancellation) for details.
384
385 The `$concurrency` parameter sets a new soft limit for the maximum number
386 of jobs to handle concurrently. Finding a good concurrency limit depends
387 on your particular use case. It's common to limit concurrency to a rather
388 small value, as doing more than a dozen of things at once may easily
389 overwhelm the receiving side. Using a `1` value will ensure that all jobs
390 are processed one after another, effectively creating a "waterfall" of
391 jobs. Using a value less than 1 will reject with an
392 `InvalidArgumentException` without processing any jobs.
393
394 ```php
395 // handle up to 10 jobs concurrently
396 $promise = Queue::any(10, $jobs, $handler);
397 ```
398
399 ```php
400 // handle each job after another without concurrency (waterfall)
401 $promise = Queue::any(1, $jobs, $handler);
402 ```
403
404 The `$jobs` parameter must be an array with all jobs to process. Each
405 value in this array will be passed to the `$handler` to start one job.
406 The array keys have no effect, the promise will simply resolve with the
407 job results of the first successful job as returned by the `$handler`.
408 If this array is empty, this method will reject without processing any
409 jobs.
410
411 The `$handler` parameter must be a valid callable that accepts your job
412 parameters, invokes the appropriate operation and returns a Promise as a
413 placeholder for its future result. If the given argument is not a valid
414 callable, this method will reject with an `InvalidArgumentExceptionn`
415 without processing any jobs.
416
417 ```php
418 // using a Closure as handler is usually recommended
419 $promise = Queue::any(10, $jobs, function ($url) use ($browser) {
420 return $browser->get($url);
421 });
422 ```
423
424 ```php
425 // accepts any callable, so PHP's array notation is also supported
426 $promise = Queue::any(10, $jobs, array($browser, 'get'));
427 ```
428
429 #### Blocking
430
431 As stated above, this library provides you a powerful, async API by default.
432
433 You can also integrate this into your traditional, blocking environment by using
434 [reactphp/async](https://github.com/reactphp/async). This allows you to simply
435 await async HTTP requests like this:
436
437 ```php
438 use function React\Async\await;
439
440 $browser = new React\Http\Browser();
441
442 $promise = Queue::all(3, $urls, function ($url) use ($browser) {
443 return $browser->get($url);
444 });
445
446 try {
447 $responses = await($promise);
448 // responses successfully received
449 } catch (Exception $e) {
450 // an error occurred while performing the requests
451 }
452 ```
453
454 Similarly, you can also wrap this in a function to provide a simple API and hide
455 all the async details from the outside:
456
457 ```php
458 use function React\Async\await;
459
460 /**
461 * Concurrently downloads all the given URIs
462 *
463 * @param string[] $uris list of URIs to download
464 * @return ResponseInterface[] map with a response object for each URI
465 * @throws Exception if any of the URIs can not be downloaded
466 */
467 function download(array $uris)
468 {
469 $browser = new React\Http\Browser();
470
471 $promise = Queue::all(3, $uris, function ($uri) use ($browser) {
472 return $browser->get($uri);
473 });
474
475 return await($promise);
476 }
477 ```
478
479 This is made possible thanks to fibers available in PHP 8.1+ and our
480 compatibility API that also works on all supported PHP versions.
481 Please refer to [reactphp/async](https://github.com/reactphp/async#readme) for more details.
482
483 > Keep in mind that returning an array of response messages means that the whole
484 response body has to be kept in memory.
485
486 ## Install
487
488 The recommended way to install this library is [through Composer](https://getcomposer.org/).
489 [New to Composer?](https://getcomposer.org/doc/00-intro.md)
490
491 This project follows [SemVer](https://semver.org/).
492 This will install the latest supported version:
493
494 ```bash
495 composer require clue/mq-react:^1.7
496 ```
497
498 See also the [CHANGELOG](CHANGELOG.md) for details about version upgrades.
499
500 This project aims to run on any platform and thus does not require any PHP
501 extensions and supports running on legacy PHP 5.3 through current PHP 8+.
502 It's *highly recommended to use the latest supported PHP version* for this project.
503
504 ## Tests
505
506 To run the test suite, you first need to clone this repo and then install all
507 dependencies [through Composer](https://getcomposer.org/):
508
509 ```bash
510 composer install
511 ```
512
513 To run the test suite, go to the project root and run:
514
515 ```bash
516 vendor/bin/phpunit
517 ```
518
519 The test suite is set up to always ensure 100% code coverage across all
520 supported environments. If you have the Xdebug extension installed, you can also
521 generate a code coverage report locally like this:
522
523 ```bash
524 XDEBUG_MODE=coverage vendor/bin/phpunit --coverage-text
525 ```
526
527 ## License
528
529 This project is released under the permissive [MIT license](LICENSE).
530
531 I'd like to thank [Bergfreunde GmbH](https://www.bergfreunde.de/), a German
532 online retailer for Outdoor Gear & Clothing, for sponsoring the first release! 🎉
533 Thanks to sponsors like this, who understand the importance of open source
534 development, I can justify spending time and focus on open source development
535 instead of traditional paid work.
536
537 > Did you know that I offer custom development services and issuing invoices for
538 sponsorships of releases and for contributions? Contact me (@clue) for details.