| 1 |
<html> |
| 2 |
<head> |
| 3 |
<title>netrik hacker's manual: file loader</title> |
| 4 |
</head> |
| 5 |
<body> |
| 6 |
|
| 7 |
<h1 align="center">netrik hacker's manual<br />>========================<</h1> |
| 8 |
|
| 9 |
<p> |
| 10 |
[This file contains a description of the file loading module. See hacking.txt or |
| 11 |
<a href="hacking.html">hacking.html</a> for an overview of the manual.] |
| 12 |
</p> |
| 13 |
|
| 14 |
<h2>load.c</h2> |
| 15 |
|
| 16 |
<p> |
| 17 |
Every open input resource has a descriptor associated with it. The descriptor |
| 18 |
is created when opening the with init_load(). |
| 19 |
</p> |
| 20 |
|
| 21 |
<p> |
| 22 |
The struct for this descriptor is declared in load.h, and contains: |
| 23 |
<ul> <li> |
| 24 |
The URL of the resource is stored by "url", which points to a "struct Url" |
| 25 |
containing the (split up) effective URL |
| 26 |
</li><li> |
| 27 |
"type" is an enum of type "Res_type", describing what kind of resource it is. |
| 28 |
(RES_STDIN for stdin, RES_FILE for local files, RES_HTTP for HTTP connections |
| 29 |
handled by builtin HTTP code, or RES_PIPE for piping data from wget; it may |
| 30 |
also be set to RES_FAIL during the loading process, indicating that the |
| 31 |
loading failed) |
| 32 |
</li><li> |
| 33 |
The union "handle" contains data necessary to handle the resource itself, |
| 34 |
e.g. a file handle. |
| 35 |
</li><li> |
| 36 |
"buf" is a pointer to a string, where every loaded data block is stored for |
| 37 |
processing. |
| 38 |
</li><li> |
| 39 |
"buf_end" contains the current end of the data in the buffer. (Not always the |
| 40 |
whole buffer is filled, escpecially at EOF.) |
| 41 |
</li><li> |
| 42 |
"buf_ptr" points to the current read position inside the buffer. |
| 43 |
</li><li> |
| 44 |
The "user_break" flag indicates when loading was interrupted by a user break |
| 45 |
and the document wasn't loaded completely. |
| 46 |
</li></ul> |
| 47 |
</p> |
| 48 |
|
| 49 |
<a name="initLoad" id="initLoad"> |
| 50 |
|
| 51 |
<h3>init_load()</h3> |
| 52 |
|
| 53 |
<p> |
| 54 |
init_load() takes two arguments: a string containing the target URL to be |
| 55 |
loaded, and a "struct Url" containing a base URL. (Normally the URL of the file |
| 56 |
visible in the pager up to now.) Besides of creating the descriptor it also |
| 57 |
allocs the "buf[]" and prepares the resource for reading data. |
| 58 |
</p> |
| 59 |
|
| 60 |
<p> |
| 61 |
An additional "form" argument can be given to submit the data of an HTML form |
| 62 |
along with the HTTP request. See |
| 63 |
<a href="hacking-links.html#submit">Submitting</a> in |
| 64 |
hacking-links.* for details on this. |
| 65 |
</p> |
| 66 |
|
| 67 |
<p> |
| 68 |
Calling this function with only a base URL and no target means that a page is |
| 69 |
to be reloaded from history. |
| 70 |
</p> |
| 71 |
|
| 72 |
<p> |
| 73 |
For all other target URLs, first the effective URL is determined. This is done |
| 74 |
by <a href="#mergeUrls">merge_urls()</a>, which first splits the |
| 75 |
URL up into its components using |
| 76 |
<a href="#splitUrl">split_url()</a>, and then merges it with the |
| 77 |
base URL components. A pointer to the resulting merged URL ist stored in |
| 78 |
"res->url". |
| 79 |
</p> |
| 80 |
|
| 81 |
<p> |
| 82 |
If merging the URLs failed for some reason, the resource type is set to |
| 83 |
"RES_FAIL", and init_load() returns normally. Thus, there is no special error |
| 84 |
handling necessary in the calling function; all necessary handling is done in |
| 85 |
<a href="#load">load()</a>. |
| 86 |
</p> |
| 87 |
|
| 88 |
<p> |
| 89 |
If the URL is "-", meaning load from stdin, "res->url->proto.type" is set to |
| 90 |
PT_INTERNAL (meaning no relative links are allowed, and the page is not to be |
| 91 |
kept in history). |
| 92 |
</p> |
| 93 |
|
| 94 |
<p> |
| 95 |
No URL merging is done if a page from history is reloaded. The split URL passed |
| 96 |
as the base is already the destination URL. |
| 97 |
</p> |
| 98 |
|
| 99 |
<p> |
| 100 |
Now having the destination URL, the resource is opened appropriately, depending |
| 101 |
on what resource type is set in the URL. |
| 102 |
</p> |
| 103 |
|
| 104 |
<p> |
| 105 |
If it is a HTTP URL, by default |
| 106 |
<a href="#initHttp">http_init_load()</a> is used to open a connection |
| 107 |
to the server and read the HTTP headers. "res->type" is set to HTTP. |
| 108 |
</p> |
| 109 |
|
| 110 |
<p> |
| 111 |
Alternatively, "cfg.wget" may be set (using --no-builtin-http), in which case |
| 112 |
"wget" is used instead of the builtin HTTP code. |
| 113 |
This is done by init_wget(), which starts wget with the correct |
| 114 |
filename, and initiates reading its standard output through a pipe. The popen() |
| 115 |
function returns our pipe end as a stream; we use this as the input. |
| 116 |
"res->type" is set to "RES_PIPE". |
| 117 |
</p> |
| 118 |
|
| 119 |
<p> |
| 120 |
FTP URLs are always fetched using wget. |
| 121 |
</p> |
| 122 |
|
| 123 |
<p> |
| 124 |
Local files are simply opened as the input stream, and "res->type" is set to |
| 125 |
"RES_FILE". |
| 126 |
</p> |
| 127 |
|
| 128 |
<p> |
| 129 |
Things are more complicated however as local files are allowed to be compressed |
| 130 |
with gzip or bzip2, which is handled transparently, i.e. the extension ".gz" or |
| 131 |
".bz2" needn't be present in the given URL. Thus we have to check for the |
| 132 |
alternative file names in a loop, and if one of them exists memorize it. |
| 133 |
</p> |
| 134 |
|
| 135 |
<p> |
| 136 |
If we have a compressed file name (presently recognized only by the file |
| 137 |
extension) -- regardless whether the complete file name was given or |
| 138 |
transparently extended as described above -- we do not read the file directly, |
| 139 |
but pipe it from gunzip/bunzip2, just es FTP is piped from wget. The check for |
| 140 |
compressed files is done in a second loop, independent of the file name loop. |
| 141 |
</p> |
| 142 |
|
| 143 |
<p> |
| 144 |
Internal URLs presently always mean loading from stdin. (Error pages also use |
| 145 |
"PT_INTERNAL", but init_load() returns immediataly after setting it, so this |
| 146 |
needn't be handled in the switch.) The standard input is reopened as a stream, |
| 147 |
and this one is used as the input. As we need some way to read user commands |
| 148 |
from the terminal, "stderr" is reopend in place of "stdin". (Some programs |
| 149 |
reopen "/dev/tty" instead -- no idea which one is better...) This way the |
| 150 |
normal "stdin" descriptor points to a terminal again, while the input file is |
| 151 |
read from the pipe that is (hopefully) connected to the original standard |
| 152 |
input. "res->type" is set to "RES_STDIN". |
| 153 |
</p> |
| 154 |
|
| 155 |
<p> |
| 156 |
Things get a bit more complicated if there was no protocol specification in the |
| 157 |
URL, and no base was supplied (meaning the URL has to be treated as an absolute |
| 158 |
URL in any case). In this case we have to guess whether it is a local file or |
| 159 |
an HTTP URL. |
| 160 |
</p> |
| 161 |
|
| 162 |
<p> |
| 163 |
First we try to open a local file with the path returned by merge_urls(). |
| 164 |
(Should be identical to the given URL.) This is achieved by handling PT_UNKNOWN |
| 165 |
together with PT_FILE. |
| 166 |
</p> |
| 167 |
|
| 168 |
<p> |
| 169 |
If there is no local file of the given name (also not a compressed one, s.a.), |
| 170 |
and the URL could be an HTTP URL without the protocol specification (which is |
| 171 |
the case if the string doesn't start with '/'), we try to load it as HTTP. As |
| 172 |
these URLs are split in another way, we have to prepend the "http://" to the |
| 173 |
URL string and call merge_urls() again. Afterwards, we proceed just as with any |
| 174 |
other HTTP URL. (Fallthrough to PT_HTTP.) |
| 175 |
</p> |
| 176 |
|
| 177 |
<p> |
| 178 |
After opening an HTTP connection (regardless whether it was PT_HTTP or |
| 179 |
PT_UNKNOWN recognized as HTTP), we need to check for HTTP redirects. If a |
| 180 |
redirect is set (a "Location:" header exists), the loading has to be repeated |
| 181 |
with the redirect URL. The new "main_url" is taken from the "Location:" value; |
| 182 |
the "base_url" is set to "res->url", i.e. the original absolute target URL, so |
| 183 |
the redirect will be relative to that; the "form" is also cleared, as redirects |
| 184 |
always use a simple GET. With this new URL(s), the whole process is repeated. |
| 185 |
In this reiteration the protocol type always has to be PT_HTTP; redirections to |
| 186 |
other resource types are not allowed. All this can happen up to five times; |
| 187 |
when more, it's probably a redirection loop, and we abort. |
| 188 |
</p> |
| 189 |
|
| 190 |
<p> |
| 191 |
If opening a file fails, init_load() returns immediately, only setting |
| 192 |
"res->type" to "RES_FAIL" and "res->url->proto" to "PT_INTERNAL". "RES_FAIL" is |
| 193 |
then handled appropriately in load(); this way, the caller needn't bother about |
| 194 |
it. (Except an additional error message and setting an error code in |
| 195 |
<a href="hacking-layout.html#parseSyntax">parse_syntax()</a>; see |
| 196 |
<a href="hacking-layout.html#warn">Warning Mesages</a> in |
| 197 |
hacking-layout.* for that.) "PT_INTERNAL" means that the page isn't to be kept |
| 198 |
in history, that following relative URLs isn't allowed etc. Thus, the link/page |
| 199 |
history handling functions also do not need special handling for the error |
| 200 |
pages. |
| 201 |
</p> |
| 202 |
|
| 203 |
<p> |
| 204 |
HTTP loading errors inside http_init_load() are handled the same way. |
| 205 |
</p> |
| 206 |
|
| 207 |
</a> <!--initLoad--> |
| 208 |
|
| 209 |
<a name="load" id="load"> |
| 210 |
|
| 211 |
<h3>load()</h3> |
| 212 |
|
| 213 |
<p> |
| 214 |
Data is read by calling load() with the resource descriptor returned by |
| 215 |
init_load(). |
| 216 |
</p> |
| 217 |
|
| 218 |
<p> |
| 219 |
Every call to load() reads one data block (of size BUF_SIZE) into a buffer. The |
| 220 |
reading function |
| 221 |
(<a href="hacking-layout.html#parseSyntax">parse_syntax()</a> or |
| 222 |
<a href="#parseHeader">parse_header()</a>) then processes the data, keeping track of the current read |
| 223 |
position inside the buffer by "res->buf_ptr"; when it reaches the end of the |
| 224 |
data block, load() is called again to read the next block. |
| 225 |
</p> |
| 226 |
|
| 227 |
<p> |
| 228 |
If "res->type" is "RES_HTTP", read() is used to read a data block from the |
| 229 |
socket; otherwise, fread() is used to read data from the input stream. (It |
| 230 |
doesn't matter if this stream is a normal file (RES_FILE), stdin (RES_STDIN), |
| 231 |
or a pipe (RES_PIPE).) |
| 232 |
</p> |
| 233 |
|
| 234 |
<p> |
| 235 |
"RES_FAIL" means that opening the input resource failed for some reason, or an |
| 236 |
error emerged in a previous load() call. In this case, no data is read; an |
| 237 |
empty buffer is returned, which normally would mean EOF. This causes |
| 238 |
<a href="hacking-layout.html#parseSyntax">parse_syntax()</a> to |
| 239 |
generate an empty page, or stops parsing at this point if some data already has |
| 240 |
been read before the error occured. For the latter case, some (little) |
| 241 |
additional handling is necessary in parse_syntax(), to ensure that an |
| 242 |
appropriate error message is printed and an error code returned to main(). |
| 243 |
(Causing a keywait before starting the pager.) |
| 244 |
</p> |
| 245 |
|
| 246 |
<p> |
| 247 |
The data block is stored in the "buf[]" referenced by the descriptor, and |
| 248 |
"buf_end" is set to the end of the data inside the buffer; "buf_ptr" is set to |
| 249 |
the beginning of the buffer. |
| 250 |
</p> |
| 251 |
|
| 252 |
</a> <!--load--> |
| 253 |
|
| 254 |
<a name="uninitLoad" id="uninitLoad"> |
| 255 |
|
| 256 |
<h3>uninit_load()</h3> |
| 257 |
|
| 258 |
<p> |
| 259 |
uninit_load() is used to close the input stream, and free the memory used to |
| 260 |
read the file. |
| 261 |
</p> |
| 262 |
|
| 263 |
<p> |
| 264 |
If the input was a pipe created by popen() (RES_PIPE), it needs to be closed |
| 265 |
with pclose() instead of fclose(). This function also passes the exit code from |
| 266 |
wget, which is necessary to decide whether the load via wget was sucessfull. |
| 267 |
</p> |
| 268 |
|
| 269 |
<p> |
| 270 |
If the input was a HTTP socket, close() is used instead of fclose(). |
| 271 |
</p> |
| 272 |
|
| 273 |
<p> |
| 274 |
After closing the stream, the input buffer and the "res" struct are freed. |
| 275 |
</p> |
| 276 |
|
| 277 |
</a> <!--uninitLoad--> |
| 278 |
|
| 279 |
<h2>url.c</h2> |
| 280 |
|
| 281 |
<p> |
| 282 |
This file contains a couple of functions for handling of URLs, which are used |
| 283 |
chiefly by the file loading module. |
| 284 |
</p> |
| 285 |
|
| 286 |
<a name="splitUrl" id="splitUrl"> |
| 287 |
|
| 288 |
<h3>split_url()</h3> |
| 289 |
|
| 290 |
<p> |
| 291 |
To allow operating on the URLs and loading the addressed files, the URL string |
| 292 |
given by the user or a link needs to be split up into components. split_url() |
| 293 |
parses the URL string, and returns the components by a pointer to a newly |
| 294 |
allocated "struct Url". |
| 295 |
</p> |
| 296 |
|
| 297 |
<p> |
| 298 |
The Url struct is used for all following processing steps. It contains the |
| 299 |
following data: |
| 300 |
<ul><li> |
| 301 |
The protocol (resource type) specification string in "proto.str" |
| 302 |
</li><li> |
| 303 |
The protocol type as an "enum Protocol" in "proto.type" |
| 304 |
</li><li> |
| 305 |
The host name of a HTTP or FTP URL "host" |
| 306 |
</li><li> |
| 307 |
The port number (HTTP/FTP) "port" |
| 308 |
</li><li> |
| 309 |
The complete directory name "dir" (path without file name) |
| 310 |
</li><li> |
| 311 |
The file name in "name" |
| 312 |
</li><li> |
| 313 |
The parameters for CGI scripts in "params" |
| 314 |
</li><li> |
| 315 |
The fragment identifier (anchor) in "frag" |
| 316 |
</li><li> |
| 317 |
"full_url", which stores a complete URL string (but for the fragment |
| 318 |
identifier) |
| 319 |
</li><li> |
| 320 |
"path", which stores a part of the complete URL, consisting of directory, file |
| 321 |
name, and CGI parameters |
| 322 |
</li><li> |
| 323 |
for merged URLs, the "absolute" flag tells whether the URL was generated from |
| 324 |
an absolute URL, or was relative to a previous page (necessary to decide |
| 325 |
the document was from the same site when moving through page history) |
| 326 |
</li><li> |
| 327 |
for merged URLs, the "local" flag tells whether the URL points to a local |
| 328 |
anchor (in the same document) |
| 329 |
</li></ul> |
| 330 |
|
| 331 |
Before the URL is actually parsed, various special characters that are not |
| 332 |
allowed directly in a URL are escaped. (Note: While escaping some invalid |
| 333 |
characters in a URL we get is merely convenient, various control char *have* to |
| 334 |
be escaped to prevent security problems!) |
| 335 |
</p> |
| 336 |
|
| 337 |
<p> |
| 338 |
The escaping is done by building a new URL string in a loop, storing for each |
| 339 |
character of the original string either the character itself if it is OK, or |
| 340 |
the %hh representation (three chars) if it needs to be escaped. Storing is done |
| 341 |
using snprintf, which is overkill in case of unescaped characters, but used for |
| 342 |
both cases for consistency. The "size" parameter for snprintf is determined by |
| 343 |
the string length minus the already stored length. (We use snprintf(), although |
| 344 |
theoretically no overflow is possible here -- but isn't this always the case |
| 345 |
with buffer overflows?...) |
| 346 |
</p> |
| 347 |
|
| 348 |
<p> |
| 349 |
We need to escape '\'' although it is a valid URL char, as it would end the |
| 350 |
shell quoting when loading with wget/gunzip/bunzip2 otherwise. |
| 351 |
</p> |
| 352 |
|
| 353 |
<p> |
| 354 |
On the other hand, we do not escape the characters demarked as "national" in |
| 355 |
the URI standard -- while the standard discourages using them directly in a URL |
| 356 |
(as it may cause trouble when passing URLs in non-digital form), there is no |
| 357 |
reason not to process them digitally once we get them. |
| 358 |
</p> |
| 359 |
|
| 360 |
<p> |
| 361 |
The parser is very similar to the HTML parser in |
| 362 |
<a href="hacking-layout.html#parseSyntax">parse_syntax.c</a> (see |
| 363 |
<a href="hacking-layout.html#parsing">Parsing</a> in |
| 364 |
hacking-layout.*): The URL string is processed char by char in a loop. In every |
| 365 |
iteration, one char is examined, and action is taken (in a switch statement) |
| 366 |
depending on what character it is, and in what mode the parser currently is. |
| 367 |
</p> |
| 368 |
|
| 369 |
<p> |
| 370 |
There is a parsing mode for every URL component. (There are also some |
| 371 |
additional ones for constructs like the "://" after the protocol |
| 372 |
specification.) Every time we encounter some special character seperating |
| 373 |
different components, the mode is switched to the one fitting the next |
| 374 |
component, and its beginning position (normaly the char after the one causing |
| 375 |
the mode change) is stored in "word_start". Everything between the previous |
| 376 |
"word_start" and the current char is stored to the respective split URL field |
| 377 |
of the component parsed up to now, using store_component(). |
| 378 |
</p> |
| 379 |
|
| 380 |
<p><pre> |
| 381 |
url: "http://domain:80/dir1/dir2/name.ext?params#fragment" |
| 382 |
^ ^url_char |
| 383 |
word_start |
| 384 |
parse_mode: PM_HOST |
| 385 |
</pre></p> |
| 386 |
|
| 387 |
<p><pre> |
| 388 |
url: "http://domain:80/dir1/dir2/name.ext?params#fragment" |
| 389 |
url_char^^ |
| 390 |
word_start |
| 391 |
parse_mode: PM_PORT |
| 392 |
components->host: "domain" |
| 393 |
</pre></p> |
| 394 |
|
| 395 |
<p> |
| 396 |
If the separating char is not the one that would introduce the component |
| 397 |
normally following now, that means that component is missing, and we |
| 398 |
immediately have to proceed with the next one. |
| 399 |
</p> |
| 400 |
|
| 401 |
<p><pre> |
| 402 |
url: "http://domain/dir1/dir2/name.ext#fragment" |
| 403 |
url_char^^ |
| 404 |
word_start |
| 405 |
parse_mode: PM_PARAMS |
| 406 |
</pre></p> |
| 407 |
|
| 408 |
<p> |
| 409 |
This is done by setting "recycle", meaning that the current character is to be |
| 410 |
parsed again. In this new iteration the parser will see the separating char |
| 411 |
again, thus introducing a second mode change. store_component() will store a |
| 412 |
NULL string for that component, and the parser will go on with parsing the next |
| 413 |
one. |
| 414 |
</p> |
| 415 |
|
| 416 |
<p><pre> |
| 417 |
url: "http://domain/dir1/dir2/name.ext#fragment" |
| 418 |
url_char^^ |
| 419 |
word_start |
| 420 |
parse_mode: PM_FRAG |
| 421 |
components->params: NULL |
| 422 |
</pre></p> |
| 423 |
|
| 424 |
<p> |
| 425 |
The first mode change is a bit more tricky: At the beginning of the URL, we do |
| 426 |
not know if it is a full qualified one (starting with a protocol |
| 427 |
specification), or a relative URL without a protocol. At first we assume that |
| 428 |
it starts with the protocol. If a ":" follows the first word, our guess was |
| 429 |
right, and we proceed normally with the host. |
| 430 |
</p> |
| 431 |
|
| 432 |
<p><pre> |
| 433 |
url: "http://domain:80/dir1/dir2/name.ext?params#fragment" |
| 434 |
url_char^^ |
| 435 |
word_start |
| 436 |
parse_mode: PM_PROTO_END1 |
| 437 |
components->proto.str: "http" |
| 438 |
</pre></p> |
| 439 |
|
| 440 |
<p> |
| 441 |
If any other separating char occurs instead, we have to skip protocol, host, |
| 442 |
and port, and switch immediately to path parsing. We also do a "recycle" then, |
| 443 |
as the current char needs to be parsed in "PM_PATH" mode. |
| 444 |
</p> |
| 445 |
|
| 446 |
<p><pre> |
| 447 |
url: "dir1/dir2/name.ext?params#fragment" |
| 448 |
^url_char |
| 449 |
^word_start |
| 450 |
parse_mode: PM_PATH |
| 451 |
</pre></p> |
| 452 |
|
| 453 |
<p> |
| 454 |
Path parsing is also a bit more complicated, as the directory name and the file |
| 455 |
name are stored separately. For that purpose, while parsing the path we keep |
| 456 |
track where the last '/' was (in "name_start"); everything before it |
| 457 |
(inclusive) belongs to the directory, and what follows it is the file name. |
| 458 |
</p> |
| 459 |
|
| 460 |
<p><pre> |
| 461 |
url: "http://domain:80/dir1/dir2/name.ext?params#fragment" |
| 462 |
^ ^ ^url_char |
| 463 |
word_start name_start |
| 464 |
|<- dir ->| |
| 465 |
|<name>| |
| 466 |
parse_mode: PM_PARA |
| 467 |
components->dir: "/dir1/dir2/" |
| 468 |
components->name: "name.ext" |
| 469 |
</pre></p> |
| 470 |
|
| 471 |
<p> |
| 472 |
There is one exception to this: If the last part is either "." or "..", then it |
| 473 |
is *not* a file name, but also part of the directory name. The whole path is |
| 474 |
stored as the directory name (and file name is left empty), after appending a |
| 475 |
'/' character, to prevent problems in further processing steps. (Which rely on |
| 476 |
the path ending in '/'.) |
| 477 |
</p> |
| 478 |
|
| 479 |
<p> |
| 480 |
"full_url" and "path" aren't filled in split_url(), as this URL won't be used |
| 481 |
directly; they are only necessary for the final URL created in |
| 482 |
<a href="#mergeUrls">merge_urls()</a>. The "absolute" and |
| 483 |
"local" flags are also set only in merge_urls(). |
| 484 |
</p> |
| 485 |
|
| 486 |
<p> |
| 487 |
If an error occurs during URL parsing (either the protocol specification |
| 488 |
contains an unknown protocol type, or an unexpected character is encountered), |
| 489 |
split_url() sets "proto.type" to "PT_INTERNAL" and immediately returns. As |
| 490 |
"PT_INTERNAL" normally can't ever be generated in split_url(), the caller knows |
| 491 |
it was an error by this. Misusing "PT_INTERNAL" for this is surely a bit |
| 492 |
confusing; it has a big advantage, though: The created (empty) pages are |
| 493 |
correctly handled as temporary by all the page history functions without |
| 494 |
needing any exception handling. |
| 495 |
</p> |
| 496 |
|
| 497 |
</a> <!--splitUrl--> |
| 498 |
|
| 499 |
<a name="mergeUrls" id="mergeUrls"> |
| 500 |
|
| 501 |
<h3>merge_urls()</h3> |
| 502 |
|
| 503 |
<p> |
| 504 |
When loading a relative URL using the ":e" command, or when following a link, |
| 505 |
the absolute URL of the target needs to be determined by combining the URL of |
| 506 |
the current page with the given relative URL; and for both relative and |
| 507 |
absolute URLs, fields not given in the URL(s) need to be set to default values. |
| 508 |
All that is done by merge_urls(). |
| 509 |
</p> |
| 510 |
|
| 511 |
<p> |
| 512 |
This function takes a base URL supplied as a "Url" struct (already split up), |
| 513 |
and a main url given as string and split up from within merge_urls(); it |
| 514 |
returns a newly allocated "Url" struct pointer. Base being NULL means to treat |
| 515 |
the main URL as an absolute one. |
| 516 |
</p> |
| 517 |
|
| 518 |
<p> |
| 519 |
merge_urls() also takes an optional "form" parameter. This string contains the |
| 520 |
(URL encoded) data of a form to be submitted to the HTTP server as part of the |
| 521 |
URL. ("GET" method.) If present, it is stored in the resulting URL in place of |
| 522 |
any other CGI parameters. |
| 523 |
</p> |
| 524 |
|
| 525 |
<p> |
| 526 |
But for a few exeptions, merging is done component by component. If a |
| 527 |
component is present in the main URL, it is taken from there; if it's not, |
| 528 |
either it is taken from "base_url", or a default value is used if no "base_url" |
| 529 |
is given. After the first component was taken from "main_url", "base_url" is no |
| 530 |
longer used; all following components have to be specified, or default values |
| 531 |
are taken. This is achieved by setting "base_url" to NULL. |
| 532 |
</p> |
| 533 |
|
| 534 |
<p> |
| 535 |
The "port" component has no test on its own -- it is always taken from where |
| 536 |
the "host" is taken. |
| 537 |
</p> |
| 538 |
|
| 539 |
<p> |
| 540 |
The handling of "dir" is a bit more complicated: If "main_url" contains a |
| 541 |
relative "dir" (not starting with '/'), and a "base_url" is given, the new |
| 542 |
"dir" has to be created by joining both. (Concatenating the one in "main_url" |
| 543 |
to the one in "base_url".) |
| 544 |
</p> |
| 545 |
|
| 546 |
<p> |
| 547 |
A relative path starting with "." and/or ".." needs addtional handling: The "." |
| 548 |
is just a dummy and can be simply skipped. |
| 549 |
</p> |
| 550 |
|
| 551 |
<p><pre> |
| 552 |
base: netrik/test/ |
| 553 |
merge: ./../test/forms.html |
| 554 |
^^skip |
| 555 |
|
| 556 |
base: netrik/test/ |
| 557 |
merge: ../test/forms.html |
| 558 |
</pre></p> |
| 559 |
|
| 560 |
<p> |
| 561 |
The ".." is also skipped, but additionally requires discarding the last |
| 562 |
component of the base path: |
| 563 |
</p> |
| 564 |
|
| 565 |
<p><pre> |
| 566 |
vvvvv discard |
| 567 |
base: netrik/test/ |
| 568 |
merge: ../test/forms.html |
| 569 |
^^^skip |
| 570 |
|
| 571 |
base: netrik/ |
| 572 |
merge: test/forms.html |
| 573 |
</pre></p> |
| 574 |
|
| 575 |
<p> |
| 576 |
This procedure needs to be repeated if the merge path has more than one |
| 577 |
occurence of "..". It is only stopped when there are no more components to |
| 578 |
discard in the base path; the remaining ".." occurences are kept in this case. |
| 579 |
</p> |
| 580 |
|
| 581 |
<p> |
| 582 |
(The standard actually requires handling "." and ".." occuring *anywhere* |
| 583 |
inside the merged URL; however, we do not see any point in adding a |
| 584 |
considerable amount of extra code for handling such nonsensical cases. |
| 585 |
Especially as most servers can cope with passed "." and ".." compoenents |
| 586 |
anyways -- we didn't handle them at all for a long time, and seldom there were |
| 587 |
problems.) |
| 588 |
</p> |
| 589 |
|
| 590 |
<p> |
| 591 |
There is also an exeption about the default value if |
| 592 |
nothing was supplied: For local (or unknown) URLs there is no default dir (the |
| 593 |
current directory is used), while HTTP and other use the root dir ('/') as |
| 594 |
default. |
| 595 |
</p> |
| 596 |
|
| 597 |
<p> |
| 598 |
Handling of CGI parameters works as the other components, except that if "form" |
| 599 |
is present, this is stored as "url->params", and both the params in the main |
| 600 |
and base URLs are ignored. The form data string is stored just as it were part |
| 601 |
of the main URL -- the string is copied and "base_url" is set to NULL. This |
| 602 |
ensures that form submits are always handled as new documents ("local" flag not |
| 603 |
set), even if the submit URL is otherwise identical to the current URL, and |
| 604 |
that any fragment identifier from the old URL is discarded. |
| 605 |
</p> |
| 606 |
|
| 607 |
<p> |
| 608 |
After merging the URLs, the "full_url" and "path" components have to be set. |
| 609 |
The "full_url" is created by concatenating all components (and separators), |
| 610 |
except for the fragment identifier. (Concatenating is done by the str_append() |
| 611 |
function created for that. Maybe we should try to use asprintf() or something |
| 612 |
instead?) "path" is simply a pointer to the starting position of the "dir" |
| 613 |
(and/or following) component(s) inside the "full_url" string. Thus it contains |
| 614 |
the directory, file name, and CGI paramters. |
| 615 |
</p> |
| 616 |
|
| 617 |
<p> |
| 618 |
The "absolute" flag is determined after merging the protocol specification. If |
| 619 |
"base_url" is NULL at this point, we know we have an absolute URL: Either it |
| 620 |
was NULL from the beginning (meaning absolute in any case), or it was reset |
| 621 |
when merging the protocol, because "main_url" contains a protocol specification |
| 622 |
-- also meaning an absolute URL. |
| 623 |
</p> |
| 624 |
|
| 625 |
<p> |
| 626 |
Setting the "local" flag is very similar: If, after all components except the |
| 627 |
fragment identifier have been merged, "base_url" is still not NULL, we know |
| 628 |
that "main_url" didn't contain any components up to now; it *can* only consist |
| 629 |
of a fragment identifier, meaning it references a local anchor. |
| 630 |
</p> |
| 631 |
|
| 632 |
<p> |
| 633 |
If an error occured during URL splitting, merge_urls() only prints an |
| 634 |
additional error message, and proceeds normally. The result is that the |
| 635 |
"PT_INTERNAL" indicating the error is stored in the merged URL, and can be |
| 636 |
handled by the caller, just as for split_url() itself. |
| 637 |
</p> |
| 638 |
|
| 639 |
</a> <!-- mergeUrls --> |
| 640 |
|
| 641 |
<a name="freeUrl" id="freeUrl"> |
| 642 |
|
| 643 |
<h3>free_url()</h3> |
| 644 |
|
| 645 |
<p> |
| 646 |
This function destroys a split URL structure by freeing the memory used by all |
| 647 |
the component strings, and afterwards the struct itself. |
| 648 |
</p> |
| 649 |
|
| 650 |
</a> |
| 651 |
|
| 652 |
<h2>http.c</h2> |
| 653 |
|
| 654 |
<p> |
| 655 |
The functions for handling HTTP resources are a bit more complicated, and |
| 656 |
reside in a source file on their own. |
| 657 |
</p> |
| 658 |
|
| 659 |
<a name="initHttp" id="initHttp"> |
| 660 |
|
| 661 |
<p> |
| 662 |
http_init_load() is called from init_load(). It prepares the handle, and opens |
| 663 |
the HTTP connection. This includes looking up the IP address, opening a socket, |
| 664 |
connecting to the desired server, creating and sending a HTTP request for the |
| 665 |
desired page, and reading/parsing the HTTP header of the file returned by the |
| 666 |
server. |
| 667 |
</p> |
| 668 |
|
| 669 |
<p> |
| 670 |
Most of the work is done in get_http_socket(), which is called from |
| 671 |
http_init_load() after creating the HTTP handle. |
| 672 |
</p> |
| 673 |
|
| 674 |
<p> |
| 675 |
First, this function checks for a possible proxy to send the request to. |
| 676 |
</p> |
| 677 |
|
| 678 |
<p> |
| 679 |
Having this, it creates the connect structure "sap", and then looks up the IP |
| 680 |
address of the connect server -- which is the proxy if one is present, or the |
| 681 |
target host otherwise. This, among other data, is stored in "sap". |
| 682 |
</p> |
| 683 |
|
| 684 |
<p> |
| 685 |
The next step is opening a socket using socket(), and then establishing a TCP |
| 686 |
connection to the desired server using the "sap" structure prepared before. |
| 687 |
</p> |
| 688 |
|
| 689 |
<p> |
| 690 |
Finally, the HTTP request is constructed in get_http_cmd(), and submitted over |
| 691 |
the socket. |
| 692 |
</p> |
| 693 |
|
| 694 |
<p> |
| 695 |
get_http_cmd() normally just puts together a very simple HTTP request, which |
| 696 |
consists only of the request line with the path (or, for proxies, the full |
| 697 |
URL), and a "Host:" header containing the target host. (This is required in |
| 698 |
HTTP/1.1, to allow multiple hosts on one IP.) |
| 699 |
</p> |
| 700 |
|
| 701 |
<p> |
| 702 |
Things get slightly more interesting when some form data is to be POSTed. (GET |
| 703 |
needn't be handled here, as the form data is already encoded into the URL |
| 704 |
before calling http_init_load() in this case.) Besides of adding the |
| 705 |
"Content-Type:" and "Content-Encoding:" header fields, the form data has to be |
| 706 |
submitted inside the body of the request. For that purpose, the form data is |
| 707 |
encoded using |
| 708 |
<a href="hacking-links.html#mimeEncode">mime_encode()</a> or |
| 709 |
<a href="hacking-links.html#urlEncode">url_encode()</a> (see |
| 710 |
<a href="hacking-links.html">hacking-links.*</a>) -- dependending on |
| 711 |
the desired encoding stored in the "method" field of the form item -- and |
| 712 |
stored in "form_data", which is a string submitted at the end of the request. |
| 713 |
</p> |
| 714 |
|
| 715 |
<p> |
| 716 |
During the whole exection time of get_http_socket(), user breaks are enabled, |
| 717 |
so that all the slow functions (DNS lookup, connecting to server, submitting |
| 718 |
request), which are often even waiting for a timeout on failure, can be |
| 719 |
interrupted. |
| 720 |
</p> |
| 721 |
|
| 722 |
<p> |
| 723 |
After the connection has been established with get_http_socket(), |
| 724 |
parse_header() is used to read and parse the HTTP headers. |
| 725 |
</p> |
| 726 |
|
| 727 |
<p> |
| 728 |
Errors in HTTP loading are handled by setting "RES_FAIL" and "PT_INTERNAL", |
| 729 |
just like file loading errors are handled in init_load(). |
| 730 |
</p> |
| 731 |
|
| 732 |
<a name="parseHeader" id="parseHeader"> |
| 733 |
|
| 734 |
<h3>parse_header()</h3> |
| 735 |
|
| 736 |
<p> |
| 737 |
The parser is very similar to the one in |
| 738 |
<a href="hacking-layout.html#parseSyntax">parse_syntax()</a> (see |
| 739 |
<a href="hacking-layout.html#parsing">Parsing</a> in |
| 740 |
<a href="hacking-layout.html">hacking-layout.*</a>): The input is |
| 741 |
processed character-wise by an FSM parser, where each char is read from a |
| 742 |
buffer which is refilled (more or less) transparently each time all chars have |
| 743 |
been processed. |
| 744 |
</p> |
| 745 |
|
| 746 |
<p> |
| 747 |
The parser first skips the status line (we do not care about the return code, |
| 748 |
for now...), and afterwards parses all header lines, extracting the name and |
| 749 |
value of each, and storing them inside the "headers" structure of the HTTP |
| 750 |
handle so they can be used later. (Presently, the only one used is the |
| 751 |
"Location:" header, which is cheked in |
| 752 |
<a href="#initLoad">init_load()</a> to handle redirections.) |
| 753 |
</p> |
| 754 |
|
| 755 |
<p> |
| 756 |
The parser tries to be as tolerant as possible about broken (or unknown...) |
| 757 |
syntax. For that matter, '\r' characters are completely ignored, so both the |
| 758 |
DOS-like '\r\n' linefeeds and unix-like '\n' do work. Illegal characters in the |
| 759 |
header name are just ignored, as well as a missing space after the ':' |
| 760 |
separating header name and value, and spaces at the beginning of a line which |
| 761 |
can't be a folding. |
| 762 |
</p> |
| 763 |
|
| 764 |
<p> |
| 765 |
There are a couple of both correct and broken example files in the test/ |
| 766 |
directory. Use them by piping the contents to tcplisten and pointing netrik to |
| 767 |
the listening port. (You can also concatenate some HTML file to avoid "No data" |
| 768 |
errors.) |
| 769 |
</p> |
| 770 |
|
| 771 |
<p> |
| 772 |
The data (from the TCP socket) ending before the whole header was parsed is |
| 773 |
treated as an HTTP loading error, except when a user break was performed, in |
| 774 |
which case parse_header() simply returns without taking any more action. |
| 775 |
</p> |
| 776 |
|
| 777 |
</a> <!-- parseHeader --> |
| 778 |
|
| 779 |
</a> <!-- initHTTP --> |
| 780 |
|
| 781 |
<h2>User Breaks</h2> |
| 782 |
|
| 783 |
<p> |
| 784 |
The loading of a file or HTTP page can be interrupted by sending SIGINT. |
| 785 |
interrupt.c contains a couple of functions to faciliate the SIGINT handling. |
| 786 |
Those are called from various places in load.c and http.c; main() calls |
| 787 |
init_int() at startup, which sets some constants used in the other functions. |
| 788 |
</p> |
| 789 |
|
| 790 |
<p> |
| 791 |
At the beginning of init_load(), hold_int() is called. This function uses |
| 792 |
sigprocmask() to block SIGINT. The signal is put on hold, i.e. it doesn't have |
| 793 |
it's usual effect of aborting the program anymore, but it's also not discarded; |
| 794 |
instead, it's stored, and awaits its release. |
| 795 |
</p> |
| 796 |
|
| 797 |
<p> |
| 798 |
The signal is released using enable_load(), which is called in two places: |
| 799 |
during read() or fread() in load(), and during get_http_socket() in |
| 800 |
http_init_load(). (These are the functions which may take fairly long, and need |
| 801 |
a way to be interrupted directly.) |
| 802 |
</p> |
| 803 |
|
| 804 |
<p> |
| 805 |
enable_load() sets int_handler() as the handler for SIGINT (using sigaction()), |
| 806 |
and then unblocks the signal with procsigmask(). However, not only arriving new |
| 807 |
signals will invoke int_handler() now; if some signal was sent between the |
| 808 |
hold_int() and enable_int(), it will be deliverd after enable_int(), too. This |
| 809 |
way, every SIGINT during the whole loading process will cause a break, not only |
| 810 |
if it was sent during the periods where it can be handled. |
| 811 |
</p> |
| 812 |
|
| 813 |
<p> |
| 814 |
Before calling enable_load(), a return point is set using setjmp(), and stored |
| 815 |
in "label_int". int_handler() does nothing else but immediately jumping to that |
| 816 |
return point with longjmp(). That means, when an interrupt occurs before or |
| 817 |
during the read()/fread() in load(), the signal handler jumps to another |
| 818 |
position in load(). Here, "res->user_break" is set to indicate the interrupt, |
| 819 |
and load() returns, instead of continuing the read(). Likewise in |
| 820 |
http_init_load(). |
| 821 |
</p> |
| 822 |
|
| 823 |
<p> |
| 824 |
At the end of load() and http_init_load(), hold_int() is called, so SIGINT will |
| 825 |
be put on hold again till the next call of load(). |
| 826 |
</p> |
| 827 |
|
| 828 |
<p> |
| 829 |
uninit_load() calls disable_int(). This function used sigaction() to set the |
| 830 |
SIGINT handler to SIG_IGN, and then unblocks the signal. Thus, any SIGINTs |
| 831 |
arriving after file loading has finished are discarded. They are only enabled |
| 832 |
again in init_load(), when the next file load beginns. |
| 833 |
</p> |
| 834 |
|
| 835 |
</body> |
| 836 |
</html> |