Debian Codegraph @

man/getURIAsynchronous.Rd

raw
1 \name{getURIAsynchronous}
2 \alias{getURIAsynchronous}
3 \alias{getURLAsynchronous}
4 \title{Download multiple URIs concurrently, with inter-leaved downloads}
5 \description{
6 This function allows the caller to specify multiple URIs to download
7 at the same time. All the requests are submitted and then the replies
8 are processed as data becomes available on each connection.
9 In this way, the responses are processed in an inter-leaved fashion,
10 with a chunk from one response from one request being processed and
11 then followed by a chunk from a different request.
12
13
14 Downloading documents asynchronously involves some trade-offs.
15 The switching between different streams, detecting when input is
16 available on any of them involves a little more processing
17 and so increases the consumption of CPU cycles.
18 On the other hand, there is a potentially large saving of
19 time when one considers total time to download.
20 See \url{https://aitap.grebedoc.dev/RCurl/concurrent.html}
21 for more details. This is a common trade-off that arises in
22 concurrent/parallel/asynchronous computing.
23
24
25 \code{\link{getURI}} calls this function if more than one
26 URI is specified and \code{async} is \code{TRUE}, the default in this case.
27 One can also download the (contents of the) multiple URIs
28 serially, i.e. one after the other using \code{\link{getURI}}
29 with a value of \code{FALSE} for \code{async}.
30 }
31 \usage{
32 getURIAsynchronous(url, ..., .opts = list(), write = NULL,
33 curl = getCurlHandle(),
34 multiHandle = getCurlMultiHandle(), perform = Inf,
35 .encoding = integer(), binary = rep(NA, length(url)))
36 }
37 \arguments{
38 \item{url}{a character vector identifying the URIs to download.}
39 \item{\dots}{named arguments to be passed to \code{\link{curlSetOpt}}
40 when creating each of the different \code{curlHandle} objects.}
41 \item{.opts}{a named list or \code{CURLOptions} object identifying the
42 curl options for the handle. This is merged with the values of \dots
43 to create the actual options for the curl handle in the request.}
44 \item{write}{an object giving the functions or routines that are to be
45 called when input is waiting on the different HTTP response
46 streams.
47 By default, a separate callback function is associated with each
48 input stream. This is necessary for the results to be meaningful
49 as if we use a single reader, it will be called for all streams
50 in a haphazard order and the content interleaved.
51 One can do interesting things however using a single object.
52 }
53 \item{curl}{the prototypical curlHandle that is duplicated and used in
54 in }
55 \item{multiHandle}{
56 this is a curl handle for performing asynchronous requests.
57 }
58 \item{perform}{a number which specifies the maximum number of calls to
59 \code{\link{curlMultiPerform}} that are to be made in this
60 function call. This is typically either 0 for no calls
61 or \code{Inf} meaning process the requests until completion.
62 One may find alternative values useful, such as 1 to ensure that
63 the requests are dispatched.
64 }
65 \item{.encoding}{an integer or a string that explicitly identifies the
66 encoding of the content that is returned by the HTTP server in its
67 response to our query. The possible strings are
68 \sQuote{UTF-8} or \sQuote{ISO-8859-1}
69 and the integers should be specified symbolically
70 as \code{CE_UTF8} and \code{CE_LATIN1}.
71 Note that, by default, the package attempts to process the header of
72 the HTTP response to determine the encoding. This argument is used
73 when such information is erroneous and the caller knows the correct
74 encoding.
75 }
76 \item{binary}{a logical vector identifying whether each URI has binary
77 content or simple text.}
78 }
79 \details{
80 This uses \code{\link{curlMultiPerform}}
81 and the multi/asynchronous interface for libcurl.
82 }
83 \value{
84 The return value depends on the run-time characteristics
85 of the call.
86 If the call merely specifies the URIs to be downloaded, the result
87 is a named character vector. The names identify the URIs
88 and the elements of the vector are the contents of the corresponding
89 URI.
90
91 If the requests are not performed or completed
92 (i.e. \code{perform} is zero or too small a value to process all the chunks)
93 a list with 2 elements is returned.
94 These elements are:
95 \item{multiHandle}{the curl multi-handle, of class
96 \code{\link{MultiCURLHandle-class}}. This can be used
97 in further calls to \code{\link{curlMultiPerform}}}
98 \item{write}{the \code{write} argument (after it was potentially
99 expanded to a list). This can then be used to fetch the results
100 of the requests when the requests are completed in the future.
101 }
102
103 }
104 \references{Curl homepage \url{https://curl.se/}}
105 \author{Duncan Temple Lang <[email protected]>}
106
107 \seealso{
108 \code{\link{getURL}}
109 \code{\link{getCurlMultiHandle}}
110 \code{\link{curlMultiPerform}}
111 }
112 \examples{
113 uris = c("https://r-project.org",
114 "https://aitap.grebedoc.dev/RCurl/philosophy.xml")
115 tryCatch(withAutoprint({
116 txt = getURIAsynchronous(uris, maxredirs = 20)
117 names(txt)
118 nchar(txt)
119 }), GenericCurlError = identity)
120 }
121 \keyword{IO}
122 \concept{Web}
123 \concept{URI}
124 \concept{Web services}
125
126