| 1 |
\name{basicTextGatherer} |
| 2 |
\alias{basicTextGatherer} |
| 3 |
\alias{multiTextGatherer} |
| 4 |
\alias{debugGatherer} |
| 5 |
\title{Cumulate text across callbacks (from an HTTP response)} |
| 6 |
\description{ |
| 7 |
These functions create callback functions that can be used |
| 8 |
to with the libcurl engine when it passes information to us |
| 9 |
when it is available as part of the HTTP response. |
| 10 |
|
| 11 |
\code{basicTextGatherer} is a generator function that returns a closure which is |
| 12 |
used to cumulate text provided in callbacks from the libcurl |
| 13 |
engine when it reads the response from an HTTP request. |
| 14 |
|
| 15 |
\code{debugGatherer} can be used with the \code{debugfunction} |
| 16 |
libcurl option in a call and the associated \code{update} |
| 17 |
function is called whenever libcurl has information |
| 18 |
about the header, data and general messages about the |
| 19 |
request. |
| 20 |
|
| 21 |
These functions return a list of functions. |
| 22 |
Each time one calls \code{basicTextGatherer} or |
| 23 |
\code{debugGatherer}, one gets a new, separate |
| 24 |
collection of functions. However, each |
| 25 |
collection of functions (or instance) shares |
| 26 |
the variables across the functions and across calls. |
| 27 |
This allows them to store data persistently across |
| 28 |
the calls without using a global variable. |
| 29 |
In this way, we can have multiple instances of the collection |
| 30 |
of functions, with each instance updating its own local state |
| 31 |
and not interfering with those of the others. |
| 32 |
|
| 33 |
We use an S3 class named \code{RCurlCallbackFunction} to indicate |
| 34 |
that the collection of funcions can be used as a callback. |
| 35 |
The \code{update} function is the one that is actually used |
| 36 |
as the callback function in the CURL option. |
| 37 |
The \code{value} function can be invoked to get the current |
| 38 |
state that has been accumulated by the |
| 39 |
\code{update} function. This is typically used |
| 40 |
when the request is complete. |
| 41 |
One can reuse the same collection of functions across |
| 42 |
different requests. The information will be cumulated. |
| 43 |
Sometimes it is convenient to reuse the object but |
| 44 |
reset the state to its original empty value, as it had |
| 45 |
been created afresh. The \code{reset} function in the collection |
| 46 |
permits this. |
| 47 |
|
| 48 |
\code{multiTextGatherer} is used when we are downloading multiple |
| 49 |
URIs concurrently in a single libcurl operation. This merely |
| 50 |
uses the tools of \code{basicTextGatherer} applied to each of |
| 51 |
several URIs. See \code{\link{getURIAsynchronous}}. |
| 52 |
} |
| 53 |
\usage{ |
| 54 |
basicTextGatherer(txt = character(), max = NA, value = NULL, |
| 55 |
.mapUnicode = TRUE) |
| 56 |
multiTextGatherer(uris, binary = rep(NA, length(uris))) |
| 57 |
debugGatherer() |
| 58 |
} |
| 59 |
%- maybe also 'usage' for other objects documented here. |
| 60 |
\arguments{ |
| 61 |
\item{txt}{an initial character vector to start things. |
| 62 |
We allow this to be specified so that one can initialize |
| 63 |
the content. |
| 64 |
} |
| 65 |
\item{max}{if specified as an integer this controls the total number |
| 66 |
of characters that will be read. If more are read, the function |
| 67 |
tells libcurl to stop!} |
| 68 |
\item{uris}{for \code{multiTextGatherer}, this is either the number |
| 69 |
or the names of the uris being downloaded and for which we |
| 70 |
need a separate writer function. |
| 71 |
} |
| 72 |
\item{value}{if specified, a function that is called when retrieving |
| 73 |
the text usually after the completion of the request and the |
| 74 |
processing of the response. This function can be used to convert the |
| 75 |
result into a different format, e.g. parse an XML document, |
| 76 |
read values from table in the text.} |
| 77 |
\item{.mapUnicode}{a logical value that controls whether the resulting |
| 78 |
text is processed to map components of the form \\uxxxx to their |
| 79 |
appropriate Unicode representation.} |
| 80 |
\item{binary}{a logical vector that indicates which URIs yield binary content} |
| 81 |
} |
| 82 |
\details{ |
| 83 |
This is called when the libcurl engine finds sufficient |
| 84 |
data on the stream from which it is reading the response. |
| 85 |
It cumulates these bytes and hands them to a C routine in |
| 86 |
this package which calls the actual gathering function (or a suitable |
| 87 |
replacement) returned as the \code{update} component from this function. |
| 88 |
|
| 89 |
} |
| 90 |
\value{ |
| 91 |
Both the \code{basicTextGatherer} and \code{debugGatherer} |
| 92 |
functions return an object of class |
| 93 |
\code{RCurlCallbackFunction}. |
| 94 |
\code{basicTextGatherer} extends this with the class |
| 95 |
\code{RCurlTextHandler} |
| 96 |
and |
| 97 |
\code{debugGatherer} extends this with the class |
| 98 |
\code{RCurlDebugHandler}. |
| 99 |
Each of these has the same basic structure, |
| 100 |
being a list of 3 functions. |
| 101 |
\item{update}{the function that is called with the text from the |
| 102 |
callback routine and which processes this text by accumulating it |
| 103 |
into a vector} |
| 104 |
\item{value}{a function that returns the text cumulated across the |
| 105 |
callbacks. This takes an argument \code{collapse} (and additional ones) |
| 106 |
that are handed to \code{\link[base]{paste}}. |
| 107 |
If the value of \code{collapse} is given as \code{NULL}, |
| 108 |
the vector of elements containing the different text for each |
| 109 |
callback is returned. This is convenient when debugging or if one |
| 110 |
knows something about the nature of the callbacks, e.g. the regular |
| 111 |
size that causes iit to identify records in a natural way. |
| 112 |
} |
| 113 |
\item{reset}{a function that resets the internal state to its |
| 114 |
original, empty value. This can be used to reuse the same object |
| 115 |
across requests but to avoid cumulating new input with the material from previous requests.} |
| 116 |
|
| 117 |
|
| 118 |
\code{multiTextGatherer} returns a list with an element corresponding |
| 119 |
to each URI. Each element is an object obtained by calling |
| 120 |
\code{basicTextGatherer}, i.e. a collection of 3 functions with |
| 121 |
shared state. |
| 122 |
|
| 123 |
} |
| 124 |
\references{Curl homepage \url{https://curl.se/}} |
| 125 |
\author{Duncan Temple Lang} |
| 126 |
|
| 127 |
|
| 128 |
\seealso{ |
| 129 |
\code{\link{getURL}} |
| 130 |
\code{\link{dynCurlReader}} |
| 131 |
} |
| 132 |
\examples{ |
| 133 |
tryCatch(withAutoprint({ |
| 134 |
txt = getURL("https://r-project.org", write = basicTextGatherer()) |
| 135 |
|
| 136 |
h = basicTextGatherer() |
| 137 |
txt = getURL("https://r-project.org", write = h$update) |
| 138 |
## Cumulate across pages. |
| 139 |
txt = getURL("https://aitap.grebedoc.dev/RCurl/concurrent.html", write = h$update) |
| 140 |
|
| 141 |
|
| 142 |
headers = basicTextGatherer() |
| 143 |
txt = getURL("https://aitap.grebedoc.dev/RCurl/concurrent.html", |
| 144 |
header = TRUE, headerfunction = headers$update) |
| 145 |
|
| 146 |
## Now read the headers. |
| 147 |
cat(headers$value()) |
| 148 |
headers$reset() |
| 149 |
|
| 150 |
|
| 151 |
## Debugging callback |
| 152 |
d = debugGatherer() |
| 153 |
x = getURL( |
| 154 |
"https://aitap.grebedoc.dev/RCurl/concurrent.html", |
| 155 |
debugfunction = d$update, verbose = TRUE |
| 156 |
) |
| 157 |
cat(names(d$value())) |
| 158 |
d$value()[["headerIn"]] |
| 159 |
|
| 160 |
|
| 161 |
## This hung on Solaris |
| 162 |
## 2022-02-08 philosophy.html is malformed UTF-8 |
| 163 |
uris = c("https://aitap.grebedoc.dev/RCurl/concurrent.html", |
| 164 |
"https://aitap.grebedoc.dev/RCurl/philosophy.html") |
| 165 |
\dontrun{ |
| 166 |
g = multiTextGatherer(uris) |
| 167 |
txt = tryCatch(getURIAsynchronous(uris, write = g), |
| 168 |
GenericCurlError = conditionMessage) |
| 169 |
names(txt) # no names this way |
| 170 |
nchar(txt) |
| 171 |
|
| 172 |
# Now don't use names for the gatherer elements. |
| 173 |
g = multiTextGatherer(length(uris)) |
| 174 |
txt = tryCatch(getURIAsynchronous(uris, write = g), |
| 175 |
GenericCurlError = conditionMessage) |
| 176 |
names(txt) |
| 177 |
nchar(txt) |
| 178 |
} |
| 179 |
}), |
| 180 |
GenericCurlError = identity) |
| 181 |
|
| 182 |
|
| 183 |
\dontrun{ |
| 184 |
Sys.setlocale(,"en_US.latin1") |
| 185 |
Sys.setlocale(,"en_US.UTF-8") |
| 186 |
uris = c("https://aitap.grebedoc.dev/RCurl/concurrent.html", |
| 187 |
"https://aitap.grebedoc.dev/RCurl/philosophy.html") |
| 188 |
g = multiTextGatherer(uris) |
| 189 |
txt = getURIAsynchronous(uris, write = g) |
| 190 |
} |
| 191 |
} |
| 192 |
\keyword{IO} |
| 193 |
|
| 194 |
|