Debian Codegraph @

doc/hacking-load.html

raw
1 <html>
2 <head>
3 <title>netrik hacker's manual: file loader</title>
4 </head>
5 <body>
6
7 <h1 align="center">netrik hacker's manual<br />>========================&lt;</h1>
8
9 <p>
10 [This file contains a description of the file loading module. See hacking.txt or
11 <a href="hacking.html">hacking.html</a> for an overview of the manual.]
12 </p>
13
14 <h2>load.c</h2>
15
16 <p>
17 Every open input resource has a descriptor associated with it. The descriptor
18 is created when opening the with init_load().
19 </p>
20
21 <p>
22 The struct for this descriptor is declared in load.h, and contains:
23 <ul> <li>
24 The URL of the resource is stored by "url", which points to a "struct Url"
25 containing the (split up) effective URL
26 </li><li>
27 "type" is an enum of type "Res_type", describing what kind of resource it is.
28 (RES_STDIN for stdin, RES_FILE for local files, RES_HTTP for HTTP connections
29 handled by builtin HTTP code, or RES_PIPE for piping data from wget; it may
30 also be set to RES_FAIL during the loading process, indicating that the
31 loading failed)
32 </li><li>
33 The union "handle" contains data necessary to handle the resource itself,
34 e.g. a file handle.
35 </li><li>
36 "buf" is a pointer to a string, where every loaded data block is stored for
37 processing.
38 </li><li>
39 "buf_end" contains the current end of the data in the buffer. (Not always the
40 whole buffer is filled, escpecially at EOF.)
41 </li><li>
42 "buf_ptr" points to the current read position inside the buffer.
43 </li><li>
44 The "user_break" flag indicates when loading was interrupted by a user break
45 and the document wasn't loaded completely.
46 </li></ul>
47 </p>
48
49 <a name="initLoad" id="initLoad">
50
51 <h3>init_load()</h3>
52
53 <p>
54 init_load() takes two arguments: a string containing the target URL to be
55 loaded, and a "struct Url" containing a base URL. (Normally the URL of the file
56 visible in the pager up to now.) Besides of creating the descriptor it also
57 allocs the "buf[]" and prepares the resource for reading data.
58 </p>
59
60 <p>
61 An additional "form" argument can be given to submit the data of an HTML form
62 along with the HTTP request. See
63 <a href="hacking-links.html#submit">Submitting</a> in
64 hacking-links.* for details on this.
65 </p>
66
67 <p>
68 Calling this function with only a base URL and no target means that a page is
69 to be reloaded from history.
70 </p>
71
72 <p>
73 For all other target URLs, first the effective URL is determined. This is done
74 by <a href="#mergeUrls">merge_urls()</a>, which first splits the
75 URL up into its components using
76 <a href="#splitUrl">split_url()</a>, and then merges it with the
77 base URL components. A pointer to the resulting merged URL ist stored in
78 "res->url".
79 </p>
80
81 <p>
82 If merging the URLs failed for some reason, the resource type is set to
83 "RES_FAIL", and init_load() returns normally. Thus, there is no special error
84 handling necessary in the calling function; all necessary handling is done in
85 <a href="#load">load()</a>.
86 </p>
87
88 <p>
89 If the URL is "-", meaning load from stdin, "res->url->proto.type" is set to
90 PT_INTERNAL (meaning no relative links are allowed, and the page is not to be
91 kept in history).
92 </p>
93
94 <p>
95 No URL merging is done if a page from history is reloaded. The split URL passed
96 as the base is already the destination URL.
97 </p>
98
99 <p>
100 Now having the destination URL, the resource is opened appropriately, depending
101 on what resource type is set in the URL.
102 </p>
103
104 <p>
105 If it is a HTTP URL, by default
106 <a href="#initHttp">http_init_load()</a> is used to open a connection
107 to the server and read the HTTP headers. "res->type" is set to HTTP.
108 </p>
109
110 <p>
111 Alternatively, "cfg.wget" may be set (using --no-builtin-http), in which case
112 "wget" is used instead of the builtin HTTP code.
113 This is done by init_wget(), which starts wget with the correct
114 filename, and initiates reading its standard output through a pipe. The popen()
115 function returns our pipe end as a stream; we use this as the input.
116 "res->type" is set to "RES_PIPE".
117 </p>
118
119 <p>
120 FTP URLs are always fetched using wget.
121 </p>
122
123 <p>
124 Local files are simply opened as the input stream, and "res->type" is set to
125 "RES_FILE".
126 </p>
127
128 <p>
129 Things are more complicated however as local files are allowed to be compressed
130 with gzip or bzip2, which is handled transparently, i.e. the extension ".gz" or
131 ".bz2" needn't be present in the given URL. Thus we have to check for the
132 alternative file names in a loop, and if one of them exists memorize it.
133 </p>
134
135 <p>
136 If we have a compressed file name (presently recognized only by the file
137 extension) -- regardless whether the complete file name was given or
138 transparently extended as described above -- we do not read the file directly,
139 but pipe it from gunzip/bunzip2, just es FTP is piped from wget. The check for
140 compressed files is done in a second loop, independent of the file name loop.
141 </p>
142
143 <p>
144 Internal URLs presently always mean loading from stdin. (Error pages also use
145 "PT_INTERNAL", but init_load() returns immediataly after setting it, so this
146 needn't be handled in the switch.) The standard input is reopened as a stream,
147 and this one is used as the input. As we need some way to read user commands
148 from the terminal, "stderr" is reopend in place of "stdin". (Some programs
149 reopen "/dev/tty" instead -- no idea which one is better...) This way the
150 normal "stdin" descriptor points to a terminal again, while the input file is
151 read from the pipe that is (hopefully) connected to the original standard
152 input. "res->type" is set to "RES_STDIN".
153 </p>
154
155 <p>
156 Things get a bit more complicated if there was no protocol specification in the
157 URL, and no base was supplied (meaning the URL has to be treated as an absolute
158 URL in any case). In this case we have to guess whether it is a local file or
159 an HTTP URL.
160 </p>
161
162 <p>
163 First we try to open a local file with the path returned by merge_urls().
164 (Should be identical to the given URL.) This is achieved by handling PT_UNKNOWN
165 together with PT_FILE.
166 </p>
167
168 <p>
169 If there is no local file of the given name (also not a compressed one, s.a.),
170 and the URL could be an HTTP URL without the protocol specification (which is
171 the case if the string doesn't start with '/'), we try to load it as HTTP. As
172 these URLs are split in another way, we have to prepend the "http://" to the
173 URL string and call merge_urls() again. Afterwards, we proceed just as with any
174 other HTTP URL. (Fallthrough to PT_HTTP.)
175 </p>
176
177 <p>
178 After opening an HTTP connection (regardless whether it was PT_HTTP or
179 PT_UNKNOWN recognized as HTTP), we need to check for HTTP redirects. If a
180 redirect is set (a "Location:" header exists), the loading has to be repeated
181 with the redirect URL. The new "main_url" is taken from the "Location:" value;
182 the "base_url" is set to "res->url", i.e. the original absolute target URL, so
183 the redirect will be relative to that; the "form" is also cleared, as redirects
184 always use a simple GET. With this new URL(s), the whole process is repeated.
185 In this reiteration the protocol type always has to be PT_HTTP; redirections to
186 other resource types are not allowed. All this can happen up to five times;
187 when more, it's probably a redirection loop, and we abort.
188 </p>
189
190 <p>
191 If opening a file fails, init_load() returns immediately, only setting
192 "res->type" to "RES_FAIL" and "res->url->proto" to "PT_INTERNAL". "RES_FAIL" is
193 then handled appropriately in load(); this way, the caller needn't bother about
194 it. (Except an additional error message and setting an error code in
195 <a href="hacking-layout.html#parseSyntax">parse_syntax()</a>; see
196 <a href="hacking-layout.html#warn">Warning Mesages</a> in
197 hacking-layout.* for that.) "PT_INTERNAL" means that the page isn't to be kept
198 in history, that following relative URLs isn't allowed etc. Thus, the link/page
199 history handling functions also do not need special handling for the error
200 pages.
201 </p>
202
203 <p>
204 HTTP loading errors inside http_init_load() are handled the same way.
205 </p>
206
207 </a> <!--initLoad-->
208
209 <a name="load" id="load">
210
211 <h3>load()</h3>
212
213 <p>
214 Data is read by calling load() with the resource descriptor returned by
215 init_load().
216 </p>
217
218 <p>
219 Every call to load() reads one data block (of size BUF_SIZE) into a buffer. The
220 reading function
221 (<a href="hacking-layout.html#parseSyntax">parse_syntax()</a> or
222 <a href="#parseHeader">parse_header()</a>) then processes the data, keeping track of the current read
223 position inside the buffer by "res->buf_ptr"; when it reaches the end of the
224 data block, load() is called again to read the next block.
225 </p>
226
227 <p>
228 If "res->type" is "RES_HTTP", read() is used to read a data block from the
229 socket; otherwise, fread() is used to read data from the input stream. (It
230 doesn't matter if this stream is a normal file (RES_FILE), stdin (RES_STDIN),
231 or a pipe (RES_PIPE).)
232 </p>
233
234 <p>
235 "RES_FAIL" means that opening the input resource failed for some reason, or an
236 error emerged in a previous load() call. In this case, no data is read; an
237 empty buffer is returned, which normally would mean EOF. This causes
238 <a href="hacking-layout.html#parseSyntax">parse_syntax()</a> to
239 generate an empty page, or stops parsing at this point if some data already has
240 been read before the error occured. For the latter case, some (little)
241 additional handling is necessary in parse_syntax(), to ensure that an
242 appropriate error message is printed and an error code returned to main().
243 (Causing a keywait before starting the pager.)
244 </p>
245
246 <p>
247 The data block is stored in the "buf[]" referenced by the descriptor, and
248 "buf_end" is set to the end of the data inside the buffer; "buf_ptr" is set to
249 the beginning of the buffer.
250 </p>
251
252 </a> <!--load-->
253
254 <a name="uninitLoad" id="uninitLoad">
255
256 <h3>uninit_load()</h3>
257
258 <p>
259 uninit_load() is used to close the input stream, and free the memory used to
260 read the file.
261 </p>
262
263 <p>
264 If the input was a pipe created by popen() (RES_PIPE), it needs to be closed
265 with pclose() instead of fclose(). This function also passes the exit code from
266 wget, which is necessary to decide whether the load via wget was sucessfull.
267 </p>
268
269 <p>
270 If the input was a HTTP socket, close() is used instead of fclose().
271 </p>
272
273 <p>
274 After closing the stream, the input buffer and the "res" struct are freed.
275 </p>
276
277 </a> <!--uninitLoad-->
278
279 <h2>url.c</h2>
280
281 <p>
282 This file contains a couple of functions for handling of URLs, which are used
283 chiefly by the file loading module.
284 </p>
285
286 <a name="splitUrl" id="splitUrl">
287
288 <h3>split_url()</h3>
289
290 <p>
291 To allow operating on the URLs and loading the addressed files, the URL string
292 given by the user or a link needs to be split up into components. split_url()
293 parses the URL string, and returns the components by a pointer to a newly
294 allocated "struct Url".
295 </p>
296
297 <p>
298 The Url struct is used for all following processing steps. It contains the
299 following data:
300 <ul><li>
301 The protocol (resource type) specification string in "proto.str"
302 </li><li>
303 The protocol type as an "enum Protocol" in "proto.type"
304 </li><li>
305 The host name of a HTTP or FTP URL "host"
306 </li><li>
307 The port number (HTTP/FTP) "port"
308 </li><li>
309 The complete directory name "dir" (path without file name)
310 </li><li>
311 The file name in "name"
312 </li><li>
313 The parameters for CGI scripts in "params"
314 </li><li>
315 The fragment identifier (anchor) in "frag"
316 </li><li>
317 "full_url", which stores a complete URL string (but for the fragment
318 identifier)
319 </li><li>
320 "path", which stores a part of the complete URL, consisting of directory, file
321 name, and CGI parameters
322 </li><li>
323 for merged URLs, the "absolute" flag tells whether the URL was generated from
324 an absolute URL, or was relative to a previous page (necessary to decide
325 the document was from the same site when moving through page history)
326 </li><li>
327 for merged URLs, the "local" flag tells whether the URL points to a local
328 anchor (in the same document)
329 </li></ul>
330
331 Before the URL is actually parsed, various special characters that are not
332 allowed directly in a URL are escaped. (Note: While escaping some invalid
333 characters in a URL we get is merely convenient, various control char *have* to
334 be escaped to prevent security problems!)
335 </p>
336
337 <p>
338 The escaping is done by building a new URL string in a loop, storing for each
339 character of the original string either the character itself if it is OK, or
340 the %hh representation (three chars) if it needs to be escaped. Storing is done
341 using snprintf, which is overkill in case of unescaped characters, but used for
342 both cases for consistency. The "size" parameter for snprintf is determined by
343 the string length minus the already stored length. (We use snprintf(), although
344 theoretically no overflow is possible here -- but isn't this always the case
345 with buffer overflows?...)
346 </p>
347
348 <p>
349 We need to escape '\'' although it is a valid URL char, as it would end the
350 shell quoting when loading with wget/gunzip/bunzip2 otherwise.
351 </p>
352
353 <p>
354 On the other hand, we do not escape the characters demarked as "national" in
355 the URI standard -- while the standard discourages using them directly in a URL
356 (as it may cause trouble when passing URLs in non-digital form), there is no
357 reason not to process them digitally once we get them.
358 </p>
359
360 <p>
361 The parser is very similar to the HTML parser in
362 <a href="hacking-layout.html#parseSyntax">parse_syntax.c</a> (see
363 <a href="hacking-layout.html#parsing">Parsing</a> in
364 hacking-layout.*): The URL string is processed char by char in a loop. In every
365 iteration, one char is examined, and action is taken (in a switch statement)
366 depending on what character it is, and in what mode the parser currently is.
367 </p>
368
369 <p>
370 There is a parsing mode for every URL component. (There are also some
371 additional ones for constructs like the "://" after the protocol
372 specification.) Every time we encounter some special character seperating
373 different components, the mode is switched to the one fitting the next
374 component, and its beginning position (normaly the char after the one causing
375 the mode change) is stored in "word_start". Everything between the previous
376 "word_start" and the current char is stored to the respective split URL field
377 of the component parsed up to now, using store_component().
378 </p>
379
380 <p><pre>
381 url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
382 ^ ^url_char
383 word_start
384 parse_mode: PM_HOST
385 </pre></p>
386
387 <p><pre>
388 url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
389 url_char^^
390 word_start
391 parse_mode: PM_PORT
392 components->host: "domain"
393 </pre></p>
394
395 <p>
396 If the separating char is not the one that would introduce the component
397 normally following now, that means that component is missing, and we
398 immediately have to proceed with the next one.
399 </p>
400
401 <p><pre>
402 url: "http://domain/dir1/dir2/name.ext#fragment"
403 url_char^^
404 word_start
405 parse_mode: PM_PARAMS
406 </pre></p>
407
408 <p>
409 This is done by setting "recycle", meaning that the current character is to be
410 parsed again. In this new iteration the parser will see the separating char
411 again, thus introducing a second mode change. store_component() will store a
412 NULL string for that component, and the parser will go on with parsing the next
413 one.
414 </p>
415
416 <p><pre>
417 url: "http://domain/dir1/dir2/name.ext#fragment"
418 url_char^^
419 word_start
420 parse_mode: PM_FRAG
421 components->params: NULL
422 </pre></p>
423
424 <p>
425 The first mode change is a bit more tricky: At the beginning of the URL, we do
426 not know if it is a full qualified one (starting with a protocol
427 specification), or a relative URL without a protocol. At first we assume that
428 it starts with the protocol. If a ":" follows the first word, our guess was
429 right, and we proceed normally with the host.
430 </p>
431
432 <p><pre>
433 url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
434 url_char^^
435 word_start
436 parse_mode: PM_PROTO_END1
437 components->proto.str: "http"
438 </pre></p>
439
440 <p>
441 If any other separating char occurs instead, we have to skip protocol, host,
442 and port, and switch immediately to path parsing. We also do a "recycle" then,
443 as the current char needs to be parsed in "PM_PATH" mode.
444 </p>
445
446 <p><pre>
447 url: "dir1/dir2/name.ext?params#fragment"
448 ^url_char
449 ^word_start
450 parse_mode: PM_PATH
451 </pre></p>
452
453 <p>
454 Path parsing is also a bit more complicated, as the directory name and the file
455 name are stored separately. For that purpose, while parsing the path we keep
456 track where the last '/' was (in "name_start"); everything before it
457 (inclusive) belongs to the directory, and what follows it is the file name.
458 </p>
459
460 <p><pre>
461 url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
462 ^ ^ ^url_char
463 word_start name_start
464 |&lt;- dir ->|
465 |&lt;name>|
466 parse_mode: PM_PARA
467 components->dir: "/dir1/dir2/"
468 components->name: "name.ext"
469 </pre></p>
470
471 <p>
472 There is one exception to this: If the last part is either "." or "..", then it
473 is *not* a file name, but also part of the directory name. The whole path is
474 stored as the directory name (and file name is left empty), after appending a
475 '/' character, to prevent problems in further processing steps. (Which rely on
476 the path ending in '/'.)
477 </p>
478
479 <p>
480 "full_url" and "path" aren't filled in split_url(), as this URL won't be used
481 directly; they are only necessary for the final URL created in
482 <a href="#mergeUrls">merge_urls()</a>. The "absolute" and
483 "local" flags are also set only in merge_urls().
484 </p>
485
486 <p>
487 If an error occurs during URL parsing (either the protocol specification
488 contains an unknown protocol type, or an unexpected character is encountered),
489 split_url() sets "proto.type" to "PT_INTERNAL" and immediately returns. As
490 "PT_INTERNAL" normally can't ever be generated in split_url(), the caller knows
491 it was an error by this. Misusing "PT_INTERNAL" for this is surely a bit
492 confusing; it has a big advantage, though: The created (empty) pages are
493 correctly handled as temporary by all the page history functions without
494 needing any exception handling.
495 </p>
496
497 </a> <!--splitUrl-->
498
499 <a name="mergeUrls" id="mergeUrls">
500
501 <h3>merge_urls()</h3>
502
503 <p>
504 When loading a relative URL using the ":e" command, or when following a link,
505 the absolute URL of the target needs to be determined by combining the URL of
506 the current page with the given relative URL; and for both relative and
507 absolute URLs, fields not given in the URL(s) need to be set to default values.
508 All that is done by merge_urls().
509 </p>
510
511 <p>
512 This function takes a base URL supplied as a "Url" struct (already split up),
513 and a main url given as string and split up from within merge_urls(); it
514 returns a newly allocated "Url" struct pointer. Base being NULL means to treat
515 the main URL as an absolute one.
516 </p>
517
518 <p>
519 merge_urls() also takes an optional "form" parameter. This string contains the
520 (URL encoded) data of a form to be submitted to the HTTP server as part of the
521 URL. ("GET" method.) If present, it is stored in the resulting URL in place of
522 any other CGI parameters.
523 </p>
524
525 <p>
526 But for a few exeptions, merging is done component by component. If a
527 component is present in the main URL, it is taken from there; if it's not,
528 either it is taken from "base_url", or a default value is used if no "base_url"
529 is given. After the first component was taken from "main_url", "base_url" is no
530 longer used; all following components have to be specified, or default values
531 are taken. This is achieved by setting "base_url" to NULL.
532 </p>
533
534 <p>
535 The "port" component has no test on its own -- it is always taken from where
536 the "host" is taken.
537 </p>
538
539 <p>
540 The handling of "dir" is a bit more complicated: If "main_url" contains a
541 relative "dir" (not starting with '/'), and a "base_url" is given, the new
542 "dir" has to be created by joining both. (Concatenating the one in "main_url"
543 to the one in "base_url".)
544 </p>
545
546 <p>
547 A relative path starting with "." and/or ".." needs addtional handling: The "."
548 is just a dummy and can be simply skipped.
549 </p>
550
551 <p><pre>
552 base: netrik/test/
553 merge: ./../test/forms.html
554 ^^skip
555
556 base: netrik/test/
557 merge: ../test/forms.html
558 </pre></p>
559
560 <p>
561 The ".." is also skipped, but additionally requires discarding the last
562 component of the base path:
563 </p>
564
565 <p><pre>
566 vvvvv discard
567 base: netrik/test/
568 merge: ../test/forms.html
569 ^^^skip
570
571 base: netrik/
572 merge: test/forms.html
573 </pre></p>
574
575 <p>
576 This procedure needs to be repeated if the merge path has more than one
577 occurence of "..". It is only stopped when there are no more components to
578 discard in the base path; the remaining ".." occurences are kept in this case.
579 </p>
580
581 <p>
582 (The standard actually requires handling "." and ".." occuring *anywhere*
583 inside the merged URL; however, we do not see any point in adding a
584 considerable amount of extra code for handling such nonsensical cases.
585 Especially as most servers can cope with passed "." and ".." compoenents
586 anyways -- we didn't handle them at all for a long time, and seldom there were
587 problems.)
588 </p>
589
590 <p>
591 There is also an exeption about the default value if
592 nothing was supplied: For local (or unknown) URLs there is no default dir (the
593 current directory is used), while HTTP and other use the root dir ('/') as
594 default.
595 </p>
596
597 <p>
598 Handling of CGI parameters works as the other components, except that if "form"
599 is present, this is stored as "url->params", and both the params in the main
600 and base URLs are ignored. The form data string is stored just as it were part
601 of the main URL -- the string is copied and "base_url" is set to NULL. This
602 ensures that form submits are always handled as new documents ("local" flag not
603 set), even if the submit URL is otherwise identical to the current URL, and
604 that any fragment identifier from the old URL is discarded.
605 </p>
606
607 <p>
608 After merging the URLs, the "full_url" and "path" components have to be set.
609 The "full_url" is created by concatenating all components (and separators),
610 except for the fragment identifier. (Concatenating is done by the str_append()
611 function created for that. Maybe we should try to use asprintf() or something
612 instead?) "path" is simply a pointer to the starting position of the "dir"
613 (and/or following) component(s) inside the "full_url" string. Thus it contains
614 the directory, file name, and CGI paramters.
615 </p>
616
617 <p>
618 The "absolute" flag is determined after merging the protocol specification. If
619 "base_url" is NULL at this point, we know we have an absolute URL: Either it
620 was NULL from the beginning (meaning absolute in any case), or it was reset
621 when merging the protocol, because "main_url" contains a protocol specification
622 -- also meaning an absolute URL.
623 </p>
624
625 <p>
626 Setting the "local" flag is very similar: If, after all components except the
627 fragment identifier have been merged, "base_url" is still not NULL, we know
628 that "main_url" didn't contain any components up to now; it *can* only consist
629 of a fragment identifier, meaning it references a local anchor.
630 </p>
631
632 <p>
633 If an error occured during URL splitting, merge_urls() only prints an
634 additional error message, and proceeds normally. The result is that the
635 "PT_INTERNAL" indicating the error is stored in the merged URL, and can be
636 handled by the caller, just as for split_url() itself.
637 </p>
638
639 </a> <!-- mergeUrls -->
640
641 <a name="freeUrl" id="freeUrl">
642
643 <h3>free_url()</h3>
644
645 <p>
646 This function destroys a split URL structure by freeing the memory used by all
647 the component strings, and afterwards the struct itself.
648 </p>
649
650 </a>
651
652 <h2>http.c</h2>
653
654 <p>
655 The functions for handling HTTP resources are a bit more complicated, and
656 reside in a source file on their own.
657 </p>
658
659 <a name="initHttp" id="initHttp">
660
661 <p>
662 http_init_load() is called from init_load(). It prepares the handle, and opens
663 the HTTP connection. This includes looking up the IP address, opening a socket,
664 connecting to the desired server, creating and sending a HTTP request for the
665 desired page, and reading/parsing the HTTP header of the file returned by the
666 server.
667 </p>
668
669 <p>
670 Most of the work is done in get_http_socket(), which is called from
671 http_init_load() after creating the HTTP handle.
672 </p>
673
674 <p>
675 First, this function checks for a possible proxy to send the request to.
676 </p>
677
678 <p>
679 Having this, it creates the connect structure "sap", and then looks up the IP
680 address of the connect server -- which is the proxy if one is present, or the
681 target host otherwise. This, among other data, is stored in "sap".
682 </p>
683
684 <p>
685 The next step is opening a socket using socket(), and then establishing a TCP
686 connection to the desired server using the "sap" structure prepared before.
687 </p>
688
689 <p>
690 Finally, the HTTP request is constructed in get_http_cmd(), and submitted over
691 the socket.
692 </p>
693
694 <p>
695 get_http_cmd() normally just puts together a very simple HTTP request, which
696 consists only of the request line with the path (or, for proxies, the full
697 URL), and a "Host:" header containing the target host. (This is required in
698 HTTP/1.1, to allow multiple hosts on one IP.)
699 </p>
700
701 <p>
702 Things get slightly more interesting when some form data is to be POSTed. (GET
703 needn't be handled here, as the form data is already encoded into the URL
704 before calling http_init_load() in this case.) Besides of adding the
705 "Content-Type:" and "Content-Encoding:" header fields, the form data has to be
706 submitted inside the body of the request. For that purpose, the form data is
707 encoded using
708 <a href="hacking-links.html#mimeEncode">mime_encode()</a> or
709 <a href="hacking-links.html#urlEncode">url_encode()</a> (see
710 <a href="hacking-links.html">hacking-links.*</a>) -- dependending on
711 the desired encoding stored in the "method" field of the form item -- and
712 stored in "form_data", which is a string submitted at the end of the request.
713 </p>
714
715 <p>
716 During the whole exection time of get_http_socket(), user breaks are enabled,
717 so that all the slow functions (DNS lookup, connecting to server, submitting
718 request), which are often even waiting for a timeout on failure, can be
719 interrupted.
720 </p>
721
722 <p>
723 After the connection has been established with get_http_socket(),
724 parse_header() is used to read and parse the HTTP headers.
725 </p>
726
727 <p>
728 Errors in HTTP loading are handled by setting "RES_FAIL" and "PT_INTERNAL",
729 just like file loading errors are handled in init_load().
730 </p>
731
732 <a name="parseHeader" id="parseHeader">
733
734 <h3>parse_header()</h3>
735
736 <p>
737 The parser is very similar to the one in
738 <a href="hacking-layout.html#parseSyntax">parse_syntax()</a> (see
739 <a href="hacking-layout.html#parsing">Parsing</a> in
740 <a href="hacking-layout.html">hacking-layout.*</a>): The input is
741 processed character-wise by an FSM parser, where each char is read from a
742 buffer which is refilled (more or less) transparently each time all chars have
743 been processed.
744 </p>
745
746 <p>
747 The parser first skips the status line (we do not care about the return code,
748 for now...), and afterwards parses all header lines, extracting the name and
749 value of each, and storing them inside the "headers" structure of the HTTP
750 handle so they can be used later. (Presently, the only one used is the
751 "Location:" header, which is cheked in
752 <a href="#initLoad">init_load()</a> to handle redirections.)
753 </p>
754
755 <p>
756 The parser tries to be as tolerant as possible about broken (or unknown...)
757 syntax. For that matter, '\r' characters are completely ignored, so both the
758 DOS-like '\r\n' linefeeds and unix-like '\n' do work. Illegal characters in the
759 header name are just ignored, as well as a missing space after the ':'
760 separating header name and value, and spaces at the beginning of a line which
761 can't be a folding.
762 </p>
763
764 <p>
765 There are a couple of both correct and broken example files in the test/
766 directory. Use them by piping the contents to tcplisten and pointing netrik to
767 the listening port. (You can also concatenate some HTML file to avoid "No data"
768 errors.)
769 </p>
770
771 <p>
772 The data (from the TCP socket) ending before the whole header was parsed is
773 treated as an HTTP loading error, except when a user break was performed, in
774 which case parse_header() simply returns without taking any more action.
775 </p>
776
777 </a> <!-- parseHeader -->
778
779 </a> <!-- initHTTP -->
780
781 <h2>User Breaks</h2>
782
783 <p>
784 The loading of a file or HTTP page can be interrupted by sending SIGINT.
785 interrupt.c contains a couple of functions to faciliate the SIGINT handling.
786 Those are called from various places in load.c and http.c; main() calls
787 init_int() at startup, which sets some constants used in the other functions.
788 </p>
789
790 <p>
791 At the beginning of init_load(), hold_int() is called. This function uses
792 sigprocmask() to block SIGINT. The signal is put on hold, i.e. it doesn't have
793 it's usual effect of aborting the program anymore, but it's also not discarded;
794 instead, it's stored, and awaits its release.
795 </p>
796
797 <p>
798 The signal is released using enable_load(), which is called in two places:
799 during read() or fread() in load(), and during get_http_socket() in
800 http_init_load(). (These are the functions which may take fairly long, and need
801 a way to be interrupted directly.)
802 </p>
803
804 <p>
805 enable_load() sets int_handler() as the handler for SIGINT (using sigaction()),
806 and then unblocks the signal with procsigmask(). However, not only arriving new
807 signals will invoke int_handler() now; if some signal was sent between the
808 hold_int() and enable_int(), it will be deliverd after enable_int(), too. This
809 way, every SIGINT during the whole loading process will cause a break, not only
810 if it was sent during the periods where it can be handled.
811 </p>
812
813 <p>
814 Before calling enable_load(), a return point is set using setjmp(), and stored
815 in "label_int". int_handler() does nothing else but immediately jumping to that
816 return point with longjmp(). That means, when an interrupt occurs before or
817 during the read()/fread() in load(), the signal handler jumps to another
818 position in load(). Here, "res->user_break" is set to indicate the interrupt,
819 and load() returns, instead of continuing the read(). Likewise in
820 http_init_load().
821 </p>
822
823 <p>
824 At the end of load() and http_init_load(), hold_int() is called, so SIGINT will
825 be put on hold again till the next call of load().
826 </p>
827
828 <p>
829 uninit_load() calls disable_int(). This function used sigaction() to set the
830 SIGINT handler to SIG_IGN, and then unblocks the signal. Thus, any SIGINTs
831 arriving after file loading has finished are discarded. They are only enabled
832 again in init_load(), when the next file load beginns.
833 </p>
834
835 </body>
836 </html>