| 1 |
\name{getURIAsynchronous} |
| 2 |
\alias{getURIAsynchronous} |
| 3 |
\alias{getURLAsynchronous} |
| 4 |
\title{Download multiple URIs concurrently, with inter-leaved downloads} |
| 5 |
\description{ |
| 6 |
This function allows the caller to specify multiple URIs to download |
| 7 |
at the same time. All the requests are submitted and then the replies |
| 8 |
are processed as data becomes available on each connection. |
| 9 |
In this way, the responses are processed in an inter-leaved fashion, |
| 10 |
with a chunk from one response from one request being processed and |
| 11 |
then followed by a chunk from a different request. |
| 12 |
|
| 13 |
|
| 14 |
Downloading documents asynchronously involves some trade-offs. |
| 15 |
The switching between different streams, detecting when input is |
| 16 |
available on any of them involves a little more processing |
| 17 |
and so increases the consumption of CPU cycles. |
| 18 |
On the other hand, there is a potentially large saving of |
| 19 |
time when one considers total time to download. |
| 20 |
See \url{https://aitap.grebedoc.dev/RCurl/concurrent.html} |
| 21 |
for more details. This is a common trade-off that arises in |
| 22 |
concurrent/parallel/asynchronous computing. |
| 23 |
|
| 24 |
|
| 25 |
\code{\link{getURI}} calls this function if more than one |
| 26 |
URI is specified and \code{async} is \code{TRUE}, the default in this case. |
| 27 |
One can also download the (contents of the) multiple URIs |
| 28 |
serially, i.e. one after the other using \code{\link{getURI}} |
| 29 |
with a value of \code{FALSE} for \code{async}. |
| 30 |
} |
| 31 |
\usage{ |
| 32 |
getURIAsynchronous(url, ..., .opts = list(), write = NULL, |
| 33 |
curl = getCurlHandle(), |
| 34 |
multiHandle = getCurlMultiHandle(), perform = Inf, |
| 35 |
.encoding = integer(), binary = rep(NA, length(url))) |
| 36 |
} |
| 37 |
\arguments{ |
| 38 |
\item{url}{a character vector identifying the URIs to download.} |
| 39 |
\item{\dots}{named arguments to be passed to \code{\link{curlSetOpt}} |
| 40 |
when creating each of the different \code{curlHandle} objects.} |
| 41 |
\item{.opts}{a named list or \code{CURLOptions} object identifying the |
| 42 |
curl options for the handle. This is merged with the values of \dots |
| 43 |
to create the actual options for the curl handle in the request.} |
| 44 |
\item{write}{an object giving the functions or routines that are to be |
| 45 |
called when input is waiting on the different HTTP response |
| 46 |
streams. |
| 47 |
By default, a separate callback function is associated with each |
| 48 |
input stream. This is necessary for the results to be meaningful |
| 49 |
as if we use a single reader, it will be called for all streams |
| 50 |
in a haphazard order and the content interleaved. |
| 51 |
One can do interesting things however using a single object. |
| 52 |
} |
| 53 |
\item{curl}{the prototypical curlHandle that is duplicated and used in |
| 54 |
in } |
| 55 |
\item{multiHandle}{ |
| 56 |
this is a curl handle for performing asynchronous requests. |
| 57 |
} |
| 58 |
\item{perform}{a number which specifies the maximum number of calls to |
| 59 |
\code{\link{curlMultiPerform}} that are to be made in this |
| 60 |
function call. This is typically either 0 for no calls |
| 61 |
or \code{Inf} meaning process the requests until completion. |
| 62 |
One may find alternative values useful, such as 1 to ensure that |
| 63 |
the requests are dispatched. |
| 64 |
} |
| 65 |
\item{.encoding}{an integer or a string that explicitly identifies the |
| 66 |
encoding of the content that is returned by the HTTP server in its |
| 67 |
response to our query. The possible strings are |
| 68 |
\sQuote{UTF-8} or \sQuote{ISO-8859-1} |
| 69 |
and the integers should be specified symbolically |
| 70 |
as \code{CE_UTF8} and \code{CE_LATIN1}. |
| 71 |
Note that, by default, the package attempts to process the header of |
| 72 |
the HTTP response to determine the encoding. This argument is used |
| 73 |
when such information is erroneous and the caller knows the correct |
| 74 |
encoding. |
| 75 |
} |
| 76 |
\item{binary}{a logical vector identifying whether each URI has binary |
| 77 |
content or simple text.} |
| 78 |
} |
| 79 |
\details{ |
| 80 |
This uses \code{\link{curlMultiPerform}} |
| 81 |
and the multi/asynchronous interface for libcurl. |
| 82 |
} |
| 83 |
\value{ |
| 84 |
The return value depends on the run-time characteristics |
| 85 |
of the call. |
| 86 |
If the call merely specifies the URIs to be downloaded, the result |
| 87 |
is a named character vector. The names identify the URIs |
| 88 |
and the elements of the vector are the contents of the corresponding |
| 89 |
URI. |
| 90 |
|
| 91 |
If the requests are not performed or completed |
| 92 |
(i.e. \code{perform} is zero or too small a value to process all the chunks) |
| 93 |
a list with 2 elements is returned. |
| 94 |
These elements are: |
| 95 |
\item{multiHandle}{the curl multi-handle, of class |
| 96 |
\code{\link{MultiCURLHandle-class}}. This can be used |
| 97 |
in further calls to \code{\link{curlMultiPerform}}} |
| 98 |
\item{write}{the \code{write} argument (after it was potentially |
| 99 |
expanded to a list). This can then be used to fetch the results |
| 100 |
of the requests when the requests are completed in the future. |
| 101 |
} |
| 102 |
|
| 103 |
} |
| 104 |
\references{Curl homepage \url{https://curl.se/}} |
| 105 |
\author{Duncan Temple Lang <[email protected]>} |
| 106 |
|
| 107 |
\seealso{ |
| 108 |
\code{\link{getURL}} |
| 109 |
\code{\link{getCurlMultiHandle}} |
| 110 |
\code{\link{curlMultiPerform}} |
| 111 |
} |
| 112 |
\examples{ |
| 113 |
uris = c("https://r-project.org", |
| 114 |
"https://aitap.grebedoc.dev/RCurl/philosophy.xml") |
| 115 |
tryCatch(withAutoprint({ |
| 116 |
txt = getURIAsynchronous(uris, maxredirs = 20) |
| 117 |
names(txt) |
| 118 |
nchar(txt) |
| 119 |
}), GenericCurlError = identity) |
| 120 |
} |
| 121 |
\keyword{IO} |
| 122 |
\concept{Web} |
| 123 |
\concept{URI} |
| 124 |
\concept{Web services} |
| 125 |
|
| 126 |
|