Debian Codegraph @

man/basicTextGatherer.Rd

raw
1 \name{basicTextGatherer}
2 \alias{basicTextGatherer}
3 \alias{multiTextGatherer}
4 \alias{debugGatherer}
5 \title{Cumulate text across callbacks (from an HTTP response)}
6 \description{
7 These functions create callback functions that can be used
8 to with the libcurl engine when it passes information to us
9 when it is available as part of the HTTP response.
10
11 \code{basicTextGatherer} is a generator function that returns a closure which is
12 used to cumulate text provided in callbacks from the libcurl
13 engine when it reads the response from an HTTP request.
14
15 \code{debugGatherer} can be used with the \code{debugfunction}
16 libcurl option in a call and the associated \code{update}
17 function is called whenever libcurl has information
18 about the header, data and general messages about the
19 request.
20
21 These functions return a list of functions.
22 Each time one calls \code{basicTextGatherer} or
23 \code{debugGatherer}, one gets a new, separate
24 collection of functions. However, each
25 collection of functions (or instance) shares
26 the variables across the functions and across calls.
27 This allows them to store data persistently across
28 the calls without using a global variable.
29 In this way, we can have multiple instances of the collection
30 of functions, with each instance updating its own local state
31 and not interfering with those of the others.
32
33 We use an S3 class named \code{RCurlCallbackFunction} to indicate
34 that the collection of funcions can be used as a callback.
35 The \code{update} function is the one that is actually used
36 as the callback function in the CURL option.
37 The \code{value} function can be invoked to get the current
38 state that has been accumulated by the
39 \code{update} function. This is typically used
40 when the request is complete.
41 One can reuse the same collection of functions across
42 different requests. The information will be cumulated.
43 Sometimes it is convenient to reuse the object but
44 reset the state to its original empty value, as it had
45 been created afresh. The \code{reset} function in the collection
46 permits this.
47
48 \code{multiTextGatherer} is used when we are downloading multiple
49 URIs concurrently in a single libcurl operation. This merely
50 uses the tools of \code{basicTextGatherer} applied to each of
51 several URIs. See \code{\link{getURIAsynchronous}}.
52 }
53 \usage{
54 basicTextGatherer(txt = character(), max = NA, value = NULL,
55 .mapUnicode = TRUE)
56 multiTextGatherer(uris, binary = rep(NA, length(uris)))
57 debugGatherer()
58 }
59 %- maybe also 'usage' for other objects documented here.
60 \arguments{
61 \item{txt}{an initial character vector to start things.
62 We allow this to be specified so that one can initialize
63 the content.
64 }
65 \item{max}{if specified as an integer this controls the total number
66 of characters that will be read. If more are read, the function
67 tells libcurl to stop!}
68 \item{uris}{for \code{multiTextGatherer}, this is either the number
69 or the names of the uris being downloaded and for which we
70 need a separate writer function.
71 }
72 \item{value}{if specified, a function that is called when retrieving
73 the text usually after the completion of the request and the
74 processing of the response. This function can be used to convert the
75 result into a different format, e.g. parse an XML document,
76 read values from table in the text.}
77 \item{.mapUnicode}{a logical value that controls whether the resulting
78 text is processed to map components of the form \\uxxxx to their
79 appropriate Unicode representation.}
80 \item{binary}{a logical vector that indicates which URIs yield binary content}
81 }
82 \details{
83 This is called when the libcurl engine finds sufficient
84 data on the stream from which it is reading the response.
85 It cumulates these bytes and hands them to a C routine in
86 this package which calls the actual gathering function (or a suitable
87 replacement) returned as the \code{update} component from this function.
88
89 }
90 \value{
91 Both the \code{basicTextGatherer} and \code{debugGatherer}
92 functions return an object of class
93 \code{RCurlCallbackFunction}.
94 \code{basicTextGatherer} extends this with the class
95 \code{RCurlTextHandler}
96 and
97 \code{debugGatherer} extends this with the class
98 \code{RCurlDebugHandler}.
99 Each of these has the same basic structure,
100 being a list of 3 functions.
101 \item{update}{the function that is called with the text from the
102 callback routine and which processes this text by accumulating it
103 into a vector}
104 \item{value}{a function that returns the text cumulated across the
105 callbacks. This takes an argument \code{collapse} (and additional ones)
106 that are handed to \code{\link[base]{paste}}.
107 If the value of \code{collapse} is given as \code{NULL},
108 the vector of elements containing the different text for each
109 callback is returned. This is convenient when debugging or if one
110 knows something about the nature of the callbacks, e.g. the regular
111 size that causes iit to identify records in a natural way.
112 }
113 \item{reset}{a function that resets the internal state to its
114 original, empty value. This can be used to reuse the same object
115 across requests but to avoid cumulating new input with the material from previous requests.}
116
117
118 \code{multiTextGatherer} returns a list with an element corresponding
119 to each URI. Each element is an object obtained by calling
120 \code{basicTextGatherer}, i.e. a collection of 3 functions with
121 shared state.
122
123 }
124 \references{Curl homepage \url{https://curl.se/}}
125 \author{Duncan Temple Lang}
126
127
128 \seealso{
129 \code{\link{getURL}}
130 \code{\link{dynCurlReader}}
131 }
132 \examples{
133 tryCatch(withAutoprint({
134 txt = getURL("https://r-project.org", write = basicTextGatherer())
135
136 h = basicTextGatherer()
137 txt = getURL("https://r-project.org", write = h$update)
138 ## Cumulate across pages.
139 txt = getURL("https://aitap.grebedoc.dev/RCurl/concurrent.html", write = h$update)
140
141
142 headers = basicTextGatherer()
143 txt = getURL("https://aitap.grebedoc.dev/RCurl/concurrent.html",
144 header = TRUE, headerfunction = headers$update)
145
146 ## Now read the headers.
147 cat(headers$value())
148 headers$reset()
149
150
151 ## Debugging callback
152 d = debugGatherer()
153 x = getURL(
154 "https://aitap.grebedoc.dev/RCurl/concurrent.html",
155 debugfunction = d$update, verbose = TRUE
156 )
157 cat(names(d$value()))
158 d$value()[["headerIn"]]
159
160
161 ## This hung on Solaris
162 ## 2022-02-08 philosophy.html is malformed UTF-8
163 uris = c("https://aitap.grebedoc.dev/RCurl/concurrent.html",
164 "https://aitap.grebedoc.dev/RCurl/philosophy.html")
165 \dontrun{
166 g = multiTextGatherer(uris)
167 txt = tryCatch(getURIAsynchronous(uris, write = g),
168 GenericCurlError = conditionMessage)
169 names(txt) # no names this way
170 nchar(txt)
171
172 # Now don't use names for the gatherer elements.
173 g = multiTextGatherer(length(uris))
174 txt = tryCatch(getURIAsynchronous(uris, write = g),
175 GenericCurlError = conditionMessage)
176 names(txt)
177 nchar(txt)
178 }
179 }),
180 GenericCurlError = identity)
181
182
183 \dontrun{
184 Sys.setlocale(,"en_US.latin1")
185 Sys.setlocale(,"en_US.UTF-8")
186 uris = c("https://aitap.grebedoc.dev/RCurl/concurrent.html",
187 "https://aitap.grebedoc.dev/RCurl/philosophy.html")
188 g = multiTextGatherer(uris)
189 txt = getURIAsynchronous(uris, write = g)
190 }
191 }
192 \keyword{IO}
193
194