Debian Codegraph @

man/getURL.Rd

raw
1 \name{getURL}
2 \alias{getURL}
3 \alias{getURI}
4 \alias{getURLContent}
5 \title{Download a URI}
6 \description{
7 These functions download one or more URIs (a.k.a. URLs).
8 It uses libcurl under the hood to perform the request
9 and retrieve the response.
10 There are a myriad of options that can be specified using
11 the \dots mechanism to control the creation and submission
12 of the request and the processing of the response.
13
14 \code{getURLContent} has been added as a high-level function
15 like \code{getURL} and \code{getBinaryURL} but which
16 determines the type of the content being downloaded
17 by looking at the resulting HTTP header's Content-Type
18 field. It uses this to determine whether the bytes
19 are binary or "text".
20
21 The request supports any of the facilities within the
22 version of libcurl that was installed.
23 One can examine these via \code{\link{curlVersion}}.
24
25 \code{getURLContent} doesn't perform asynchronous or multiple
26 concurrent requests at present.
27 }
28 \usage{
29 getURL(url, ..., .opts = list(),
30 write = basicTextGatherer(.mapUnicode = .mapUnicode),
31 curl = getCurlHandle(), async = length(url) > 1,
32 .encoding = integer(), .mapUnicode = TRUE)
33 getURI(url, ..., .opts = list(),
34 write = basicTextGatherer(.mapUnicode = .mapUnicode),
35 curl = getCurlHandle(), async = length(url) > 1,
36 .encoding = integer(), .mapUnicode = TRUE)
37 getURLContent(url, ..., curl = getCurlHandle(.opts = .opts), .encoding = NA,
38 binary = NA, .opts = list(...),
39 header = dynCurlReader(curl, binary = binary,
40 baseURL = url, isHTTP = isHTTP,
41 encoding = .encoding),
42 isHTTP = length(grep('^[[:space:]]*http', url)) > 0)
43 }
44 \arguments{
45 \item{url}{a string giving the URI}
46 \item{\dots}{named values that are interpreted as CURL options
47 governing the HTTP request.}
48 \item{.opts}{a named list or \code{CURLOptions} object identifying the
49 curl options for the handle. This is merged with the values of \dots
50 to create the actual options for the curl handle in the request.}
51 \item{write}{if explicitly supplied, this is a function that is called
52 with a single argument each time the the HTTP response handler has
53 gathered sufficient text. The argument to the function is
54 a single string. The default argument provides
55 both a function for cumulating this text and is then used
56 to retrieve it as the return value for this function.
57 }
58 \item{curl}{the previously initialized CURL context/handle which can
59 be used for multiple requests.}
60 \item{async}{a logical value that determines whether the download
61 request should be done via asynchronous, concurrent downloading or a serial
62 download. This really only arises when we are trying to download
63 multiple URIs in a single call. There are trade-offs between
64 concurrent and serial downloads, essentially trading CPU cycles
65 for shorter elapsed times. Concurrent downloads reduce the overall
66 time waiting for \code{getURI}/\code{getURL} to return.
67 }
68 \item{.encoding}{an integer or a string that explicitly identifies the
69 encoding of the content that is returned by the HTTP server in its
70 response to our query. The possible strings are
71 \sQuote{UTF-8} or \sQuote{ISO-8859-1}
72 and the integers should be specified symbolically
73 as \code{CE_UTF8} and \code{CE_LATIN1}.
74 Note that, by default, the package attempts to process the header of
75 the HTTP response to determine the encoding. This argument is used
76 when such information is erroneous and the caller knows the correct
77 encoding.
78 The default value leaves the decision to this default mechanism.
79 This does however currently involve processing each line/chunk
80 of the header (with a call to an R function). As a result,
81 if one knows the encoding for the resulting response,
82 specifying this avoids this slight overhead which is probably
83 quite small relative to network latency and speed.
84 }
85 \item{.mapUnicode}{a logical value that controls whether the resulting
86 text is processed to map components of the form \\uxxxx to their
87 appropriate Unicode representation.}
88 \item{binary}{a logical value indicating whether the caller knows
89 whether the resulting content is binary (\code{TRUE}) or not
90 (\code{FALSE}) or unknown (\code{NA}).
91 }
92 \item{header}{this is made available as a parameter of the function
93 to allow callers to construct different readers for processing
94 the header and body of the (HTTP) response.
95 Callers specifying this will typically only adjust the
96 call to \code{dynCurlReader}, e.g. to specify a
97 function for its \code{value} parameter to
98 control how the body is post-processed.
99
100 The caller can specify a value of \code{TRUE} or \code{FALSE}
101 for this parameter. \code{TRUE} means that the header will be
102 returned along with the body; \code{FALSE} corresponds to the
103 default and only the body will be returned.
104 When returning the header, it is first parsed via
105 \code{\link{parseHTTPHeader}},
106 unless the value of \code{header} is of class \code{AsIs}.
107 So to get the raw header, pass the argument as \code{header = I(TRUE)}.
108 }
109 \item{isHTTP}{a logical value that indicates whether the request an
110 HTTP request. This is used when determining how to process the response.}
111 }
112 %\details{}
113 \value{
114 If no value is supplied for \code{write},
115 the result is the text that is the HTTP response.
116 (HTTP header information is included if the header option for CURL is
117 set to \code{TRUE} and no handler for headerfunction is supplied in
118 the CURL options.)
119
120 Alternatively, if a value is supplied for the \code{write} parameter,
121 this is returned. This allows the caller to create a handler within
122 the call and get it back. This avoids having to explicitly create
123 and assign it and then call \code{getURL} and then access the result.
124 Instead, the 3 steps can be inlined in a single call.
125 }
126 \references{Curl homepage \url{https://curl.se/}}
127 \author{Duncan Temple Lang}
128
129
130 \seealso{
131 \code{\link{getBinaryURL}}
132 \code{\link{curlPerform}}
133 \code{\link{curlOptions}}
134 }
135 \examples{
136 # Regular HTTP
137 if(requireNamespace("XML", quietly = TRUE)) tryCatch(withAutoprint({
138 txt = getURL("https://r-project.org/", maxredirs = 20)
139 ## Then we could parse the result.
140 XML::htmlTreeParse(txt, asText = TRUE)
141 }), GenericCurlError = identity)
142
143
144 # HTTPS. First check to see that we have support compiled into
145 # libcurl for ssl.
146 if(interactive() && ("ssl" \%in\% names(curlVersion()$features))
147 && url.exists("https://sourceforge.net/")) {
148 txt = tryCatch(getURL("https://sourceforge.net/"),
149 error = function(e) {
150 getURL("https://sourceforge.net/",
151 ssl.verifypeer = FALSE)
152 })
153
154 }
155
156
157 # Create a CURL handle that we will reuse.
158 if(interactive()) {
159 curl = getCurlHandle(maxredirs = 20)
160 pages = list()
161 for(u in c("https://r-project.org",
162 "https://aitap.grebedoc.dev/RCurl/concurrent.html")) {
163 pages[[u]] = tryCatch(getURL(u, curl = curl),
164 GenericCurlError = identity)
165 }
166 }
167
168
169 # Set additional fields in the header of the HTTP request.
170 # verbose option allows us to see that they were included.
171 tryCatch(withAutoprint(
172 getURL("https://r-project.org", httpheader = c(Accept = "text/html",
173 MyField = "Duncan"),
174 verbose = TRUE)
175 ), GenericCurlError = identity)
176
177
178
179 # Arrange to read the header of the response from the HTTP server as
180 # a separate "stream". Then we can break it into name-value
181 # pairs. (The first line is the HTTP/1.1 200 Ok or 301 Moved Permanently
182 # status line)
183 tryCatch(withAutoprint({
184 h = basicTextGatherer()
185 txt = getURL("https://www.r-project.org",
186 header= TRUE, headerfunction = h$update,
187 httpheader = c(Accept="text/html", Test=1), verbose = TRUE)
188 print(paste(h$value(NULL)[-1], collapse=""))
189 con <- textConnection(paste(h$value(NULL)[-1], collapse=""))
190 read.dcf(con)
191 close(con)
192 }), GenericCurlError = identity)
193
194
195
196 \dontrun{
197 # Test the passwords.
198 x = getURL("https://example.invalid/RCurl/testPassword/index.html", userpwd = "bob:duncantl")
199
200 # Catch an error because no authorization
201 # We catch the generic HTTPError, but we could catch the more specific "Unauthorized" error
202 # type.
203 x = tryCatch(getURLContent("https://example.invalid/RCurl/testPassword/index.html"),
204 HTTPError = function(e) {
205 cat("HTTP error: ", e$message, "\n")
206 })
207 }
208
209 \dontrun{
210 # Needs specific information from the cookie file on a per user basis
211 # with a registration to the NY times.
212 x = getURL("https://www.nytimes.com",
213 header = TRUE, verbose = TRUE,
214 cookiefile = "/home/duncan/Rcookies",
215 netrc = TRUE,
216 maxredirs = as.integer(20),
217 netrc.file = "/home2/duncan/.netrc1",
218 followlocation = TRUE)
219 }
220
221 if(interactive()) tryCatch(withAutoprint({
222 d = debugGatherer()
223 x = getURL("https://r-project.org", debugfunction = d$update, verbose = TRUE)
224 d$value()
225 }), GenericCurlError = identity)
226
227 \dontrun{
228 #############################################
229 # Using an option set in R
230
231 opts = curlOptions(header = TRUE, userpwd = "bob:duncantl", netrc = TRUE)
232 getURL("https://example.invalid/RCurl/testPassword/index.html", verbose = TRUE, .opts = opts)
233
234 # Using options in the CURL handle.
235 h = getCurlHandle(header = TRUE, userpwd = "bob:duncantl", netrc = TRUE)
236 getURL("https://example.invalid/RCurl/testPassword/index.html", verbose = TRUE, curl = h)
237 }
238
239
240 # Use a C routine as the reader. Currently gives a warning.
241 if(interactive()) tryCatch(withAutoprint({
242 routine = getNativeSymbolInfo("R_internalWriteTest", PACKAGE = "RCurl")$address
243 getURL("https://r-project.org", writefunction = routine)
244 }), GenericCurlError = identity)
245
246
247
248 # Example
249 if(interactive()) {
250 uris = c("https://r-project.org",
251 "https://aitap.grebedoc.dev/RCurl/philosophy.xml")
252 txt = tryCatch(getURI(uris), GenericCurlError = conditionMessage)
253 names(txt)
254 nchar(txt)
255
256 txt = tryCatch(getURI(uris, async = FALSE),
257 GenericCurlError = conditionMessage)
258 names(txt)
259 nchar(txt)
260
261
262 routine = getNativeSymbolInfo("R_internalWriteTest", PACKAGE = "RCurl")$address
263 txt = tryCatch(getURI(uris, write = routine, async = FALSE),
264 GenericCurlError = conditionMessage)
265 names(txt)
266 nchar(txt)
267
268
269 # getURLContent() for text and binary
270 x = tryCatch(getURLContent("https://r-project.org"),
271 GenericCurlError = identity)
272 class(x)
273
274 x = tryCatch(getURLContent("https://aitap.codeberg.page/RCurl/data.gz"),
275 GenericCurlError = identity)
276 class(x)
277 attr(x, "Content-Type")
278
279 x = tryCatch(getURLContent("https://www.r-project.org/Rlogo.png"),
280 GenericCurlError = identity)
281 class(x)
282 attr(x, "Content-Type")
283
284
285 curl = getCurlHandle()
286 dd = tryCatch(getURLContent("https://aitap.grebedoc.dev/RCurl/RJSONIO.pdf",
287 curl = curl,
288 header = dynCurlReader(curl, binary = TRUE,
289 value = function(x) {
290 print(attributes(x))
291 x})),
292 GenericCurlError = identity)
293 }
294
295
296
297 # FTP
298 # Download the files within a directory.
299 if(interactive() && url.exists('ftp://ftp.wcc.nrcs.usda.gov')) {
300
301 url = 'ftp://ftp.wcc.nrcs.usda.gov/data/snow/snow_course/table/history/idaho/'
302 filenames = getURL(url, ftp.use.epsv = FALSE, dirlistonly = TRUE)
303
304 # Deal with newlines as \n or \r\n. (BDR)
305 # Or alternatively, instruct libcurl to change \n's to \r\n's for us with crlf = TRUE
306 # filenames = getURL(url, ftp.use.epsv = FALSE, ftplistonly = TRUE, crlf = TRUE)
307 filenames = paste(url, strsplit(filenames, "\r*\n")[[1]], sep = "")
308 con = getCurlHandle( ftp.use.epsv = FALSE)
309
310 # there is a slight possibility that some of the files that are
311 # returned in the directory listing and in filenames will disappear
312 # when we go back to get them. So we use a try() in the call getURL.
313 contents = sapply(filenames[1:5], function(x) try(getURL(x, curl = con)))
314 names(contents) = filenames[1:length(contents)]
315 }
316
317 }
318 \keyword{IO}
319 \concept{Web}
320 \concept{HTTP}