Debian Codegraph @

doc/hacking.html

raw ◀ ▶
Source reconstructed at patch state 0 of 1 — not the installed (fully patched) source.
1 Debian patch touch this file
1 <html>
2 <head>
3 <title>netrik hacker's manual</title>
4 </head>
5 <body>
6
7 <h1 align="center">netrik hacker's manual<br />>========================&lt;</h1>
8
9 <p>
10 The manual contains detailed documetation on the netrik code. It should be of
11 interest to you if you want to hack on netrik, but also if you are only curious
12 how it works.
13 </p>
14
15 <p>
16 The manual includes:
17 <ul><li>
18 hacking.* (this file): general notes, module overview
19 </li><li>
20 <a href="hacking-load.html">hacking-load.*</a>: loading HTML files from local disk or HTTP connections
21 </li><li>
22 <a href="hacking-page.html">hacking-page.*</a>: page (history) handling
23 </li><li>
24 <a href="hacking-layout.html">hacking-layout.*</a>: the layouting/rendering module
25 </li><li>
26 <a href="hacking-pager.html">hacking-pager.*</a>: the interactive viewing module
27 </li><li>
28 <a href="hacking-links.html">hacking-links.*</a>: hyperlink handling
29 </li><li>
30 <a href="hacking-search.html">hacking-links.*</a>: text search
31 </li><li>
32 <a href="hacking-terminal.html">hacking-terminal.*</a>: terminal input/output and color handling
33 </li></ul>
34 ("*" means txt or html -- choose the one you prefer...)
35 </p>
36
37 <a name="notes" id="notes">
38
39 <h2>0. General notes</h2>
40
41 <p>
42 The program (except for the layout engine) presently doesn't obey any specific
43 structure. It can be roughly divided into a copule of big modules: layout
44 engine, file loading, and pager; the page handling, which is another part of
45 the loading mechanism, may also be considered a module of its own. However,
46 these modules aren't seperated clearly; they intersect at many places. The
47 loading is tightly coupled to the layouting -- the page handling part invokes
48 certain layouting passes, while the file loader offers it's service to feed the
49 data into the layout engine. However, these two parts of the loading mechanism
50 need to exchange much information (URLs, error conditions etc.), which can only
51 be passed through the layout engine. The pager naturally has to make use of the
52 layout engine to get the screen content; but it also uses some of it's data
53 structures and helper functions for link handling. The main programm basically
54 makes calls to both the pager and the page handling (which in turn invokes the
55 layouting), but also has to use data structures and functions directly;
56 actually, it contains part of the page handlig. Finally, the loading module
57 affects and depends on some data used by the pager.
58 </p>
59
60 <p>
61 A problem arises from the fact that all control passing is done by direct
62 function calls. This was perfectly OK at the time when netrik wasn't much more
63 than a layouting engine; with all the link and anchor handling, various loading
64 mechanisms (including error handling, user breaks etc.), form handling, and
65 other functionality, this has gotten quite chaotic -- it's very hard to retrace
66 the various intermingled calls between the modules or the data dependencies.
67 </p>
68
69 <p>
70 This probably could be helped somewhat by using a kind of object-oriented
71 approach, i.e. trying to put all operations on a given data structure in one
72 place. (Encapsulation.) However, I don't think it would be possible to make
73 great progress without much hassle and considerable efficiency loss.
74 </p>
75
76 <p>
77 I've devised a much nicer, clearer solution instead (well, at least it looks
78 nice in my imagination ;-) ), which involves a fundamental change in program
79 structure. Basically it's about passing flow control to a main dispatcher,
80 using state handles and an event queue.
81 </p>
82
83 <p>
84 We will probably implement that concept in one of the next major releases. It
85 is inevitable for multitasking (displaying of partially loaded pages,
86 multiwindowing) anyways, as we want to stick to explicit multitasking (real
87 multithreading/multiprocessing is much more complicated, and terribly
88 inefficient), and only this concept allows resuming and a fine grained
89 scheduling. But again, it should also simplify and clearify the overall program
90 structure, so we will probably implement it even before it's absolutely
91 necessary.
92 </p>
93
94 <p>
95 As mentioned before, the layout engine, contrary to the other modules, has a
96 fairly clear concept from the beginning. The main idea was to split HTML
97 processing into a couple of independant passes. As the individual passes are
98 quite simple, this makes understanding and altering the code quite easy.
99 </p>
100
101 <p>
102 On the other hand, this isn't terribly efficient the way it is done now. We
103 think however that in the early stage netrik is now, it's more important to
104 keep the code as simple as possible to faciliate fast developement.
105 </p>
106
107 <p>
108 Also, splitting up the processing is necessary to allow rendering of not
109 completely loaded pages. As this is to be one of the major features of netrik
110 (it's not implemented yet), we pay much attention to that.
111 </p>
112
113 <p>
114 However, the splitting into passes isn't presently optimal by any means. It's
115 more or less the first which came to mind. In the new parser planned for the
116 next major release the segmentation will be somewhat different.
117 </p>
118
119 <p>
120 Most of the processing steps of the layouting are working recursively due to
121 the nature of SGML/XML. They arn't implemented recursively, though. This makes
122 understanding a tick harder, but there is a couple of reasons for that.
123 Efficiency is one of them, but not that important presently. The main reason is
124 that a real recursive implementation wouldn't work with the planned
125 multitasking system (s.a.) as it doesn't allow resuming.
126 </p>
127
128 </a> <!-- notes -->
129
130 <h2>1. Module Overview</h2>
131
132 <p>
133 Here is an overview how the different modules work together. Detailed
134 descripitons of the modules are located in the various hacking-... files.
135 </p>
136
137 <a name="main" id="main">
138
139 <h3>main.c</h3>
140
141 <p>
142 The main program is responsible for two things: Program initialization, and the
143 main (pager) loop.
144 </p>
145
146 <p>
147 Initialization presently mostly comprises reading command line and config file
148 options.
149 </p>
150
151 <p>
152 This is somewhat tricky, but not too complicated when you get the idea: The
153 config file (~/.netrikrc) just contains options identical to the command line
154 options, one per line. They are simply prepended to the command line options,
155 and processed with getopt_long() together.
156 </p>
157
158 <p>
159 For that, netrik first reads the file. This is done by determining the file
160 name (using the HOME environment variable), then using stat() to get the file
161 size; this also tells us whether the file exists at all. Then it is opened and
162 read into a memory buffer, all at once. The next step is the intersting one:
163 The file contents is scanned, to extract the single options. Every encounterd
164 newline is replaced by a '\0' (string end), and a pointer to the beginning of
165 that line (option) is stored in the "argv_all" array . "argv_all" and
166 "argc_all" are identical to "argv" and "argc", only we also store the config
167 file options here. argv_all[0] is not filled, as argv[0] is always the command
168 that started the programm; the real arguments start with 1. After extracting
169 all config file options, the real command line options from "argv" are appended
170 to "argv_all". This way, defaults can be set in the config file, but still
171 overridden by command line options.
172 </p>
173
174 <p>
175 Having this, the arguments are scanned with getopt_long() in config_cmdln(),
176 where options are extracted and stored in the global "cfg" struct, and
177 non-option arguments singled out. (They are moved to the last argv-entries by
178 getopt_long().) The last argument is used as the resource name for the document
179 to load on startup; the other ones are ignored. (This allows setting a "home
180 page" in the config file, which will be overwritten if some URL is specified on
181 the command line.)
182 </p>
183
184 <p>
185 After scannig the arguments, the color map is initialized to the desired color
186 scheme. (See <a href="hacking-terminal.html#colorSchemes">Color
187 Schemes</a> in hacking-terminal.* for details.)
188 </p>
189
190 <p>
191 Now the first document is loaded and, unless netrik was invoked with the
192 "--dump" option, the pager loop is entered.
193 </p>
194
195 <p>
196 At the beginning of each iteration,
197 <a href="hacking-pager.html#display">display()</a> is called to start
198 the pager. This one performs all pure viewing operations, including scrolling
199 and link selection; it returns only when some page handling and/or leaving
200 fullscreen mode is required. main() then performs the desired action (indicated
201 by the return value of display()), before restarting the pager in the next
202 iteration.
203 </p>
204
205 <p>
206 Most of those actions involve loading some new page -- actually, the main loop
207 can be considered part of the <a href="#page">Page Loading</a>
208 mechanism.
209 </p>
210
211 <p>
212 The new page (as well es the startup page, before entering the loop) is always
213 loaded by calling
214 <a href="hacking-page.html#loadPage">load_page()</a> in one mode or
215 the other. The mode varies depending on the exact command: Following a link (or
216 a form submit button), opening a new URL on the command prompt (s.b.), history
217 commands. The precise actions are described in detail in
218 <a href="hacking-load.html">hacking-load.*</a>.
219 </p>
220
221 <p>
222 There are also situations that do not require loading a new page; in this case,
223 the viewer is simply restarted with the same page in the next iteration.
224 </p>
225
226 <p>
227 For one, this includes handling of non-submit form items. This is described
228 under <a href="hacking-links.html#manipulating">Manipulating</a>
229 in hacking-links.*. Note that all form elements, including submit buttons, and
230 also normal links are handled by one pager return value; they are distinguished
231 by the type of the active link item, which is handled in another switch by
232 main().
233 </p>
234
235 <p>
236 The second group of commands that do not involve loading a new page are the URL
237 displaying commands ('u', 'U', 'c'), which just print some information on the
238 screen before returning to the pager. Besides of that, they are also related to
239 the page handling, and thus described in
240 <a href="hacking-page.html">hacking-page.*</a>.
241 </p>
242
243 <p>
244 The command prompt is somewhat special also, as it doesn't cause loading some
245 new page immediately. readline() is used to get a command line from the user
246 instead, and the action taken depends on that command. However, presently only
247 the ":e" and ":E" commands are implemented, which also load an new page; they
248 are described in <a href="hacking-page.html">hacking-page.*</a> also.
249 </p>
250
251 <p>
252 Finally, if the pager returned because the terminal size changed (SIGWINCH
253 received), main() calls
254 <a href="hacking-layout.html#resize">resize()</a> (described in
255 hacking-layout.*) to prepare the page for the new screen width, and then
256 restarts the pager as usual. This is detailed under
257 a href=hacking-terminal.in#winch">SIGWINCH Handling</a> in
258 hacking-terminal.*.
259 </p>
260
261 <p>
262 A somewhat unobvious aspect is switching between curses fullscreen mode and
263 normal scrolling mode. The viewer always operates in fullscreen mode, while
264 everything in main() is done in scrolling mode.
265 </p>
266
267 <p>
268 Curses fullscreen mode is initialized once, just before entering the main loop.
269 The program doesn't switch to fullscreen mode immediately, however; this is
270 done only on the first refresh, which is induced by the getch() inside
271 display() -- after rendering the screen content.
272 </p>
273
274 <p>
275 Fullscreen mode is turned off using endwin() before display() returns, and will
276 be reactivated only when getch() is called again from display() in the next
277 iteration.
278 </p>
279
280 <p>
281 Some commands need to display some information to the user before returning to
282 the pager. Particularily the URL show commands, but also loading if some error
283 occurs or if in --debug mode. This is accomplised by the "pager_wait" flag,
284 which is set by any command that needs to display something. If the flag was
285 set during an iteration, main() waits for a keypress before proceeding with the
286 next one -- which will reactivate the pager.
287 </p>
288
289 <p>
290 Quitting netrik is also handled in the main loop: When the user typed 'q' in
291 the pager, display() returns a special value, and the loop terminates. A few
292 cleanup actions follow, whereafter main() returns.
293 </p>
294
295 </a> <!-- main -->
296
297 <a name="loading" id="loading">
298
299 <h2>File Loading</h2>
300
301 <p>
302 The loading module consists of two parts: One is responsible for actually
303 loading an HTML document from a local file or from a HTTP server. It consists
304 of the files load.[ch], http.[ch], http-parse-header.[ch], the url handling
305 functions in url.[ch], and interrupt.[ch] for user break handling. Some
306 functions from forms.c are necessary also, when submitting HTML forms. This
307 part is invoked from
308 <a href="hacking-layout.html#layout">layout()</a>, as part of the <a
309 href="#layout">Layouting</a> process.
310 </p>
311
312 <p>
313 The other part of the loading mechanism is the <a href="#page">Page
314 Loading</a> system.
315 </p>
316
317 <p>
318 The main file of the file loader is load.c, which contains the init_load() and
319 load() functions.
320 </p>
321
322 <p>
323 The loading is intialized by a call to
324 <a href="hacking-load.html#initLoad">init_load()</a> with the desired
325 URL as argument. A base URL is also passed, which is merged with the target URL
326 to create the effective URL, if the target URL is a relative one. (Following
327 links etc.) init_load() then decides whether it is a local file or an HTTP URL,
328 and initializes the loading. (Opens file or establishes HTTP connection.)
329 </p>
330
331 <p>
332 The loading itself is done by calling
333 <a href="hacking-load.html#load">load()</a>. This function fills a
334 buffer, which then can be processed. After ther buffer is processed, load()
335 has to be called again, loading the next chunk.
336 </p>
337
338 <p>
339 The loading module is described in detail in
340 <a href="hacking-load.html">hacking-load.*</a>.
341 </p>
342
343 </a> <!-- loading -->
344
345 <a name="page" id="page">
346
347 <h3>Page Loading</h3>
348
349 <p>
350 The second part of the loading mechanism is the page handling system. This one
351 is responsible for getting a new document upon request, and preparing it so it
352 can be displayed in the <a href="#viewer">Viewer</a>. It's also
353 responsible for history handling, and everything else connected with handling
354 the <a href="hacking-page.html#pageList">Page List</a> (see
355 hacking-page.*).
356 </p>
357
358 <p>
359 The page loading mechanism only has the page.c and page.h files for itself;
360 however, part of it is also located in <a href="#main">main()</a>.
361 </p>
362
363 <p>
364 The main function is
365 <a href="hacking-page.html#loadPage">load_page()</a>, with the basic
366 functionality of loading a new document and preparing it so it can (and will)
367 be displayed in the <a href="#viewer">Viewer</a>, as soon as
368 <a href="hacking-pager.html#display">display()</a> (see
369 hacking-pager.*) is called from <a href="#main">main()</a>. For
370 this, a new page list entry is created, and
371 <a href="hacking-layout.html#layout">layout()</a> (see
372 hacking-layout.*) is used to load the page and prepare it for rendering.
373 </p>
374
375 <p>
376 load_page() also has other modes, depending on what kind of page is to be
377 displayed: If the new page uses the same HTML document, and the page only has
378 to jump to an anchor, the document isn't reloaded; a new page descriptor is
379 created, but the layout date is taken from the old page.
380 </p>
381
382 <p>
383 If a page from the page history is to be reloaded, no new page descriptor is
384 created; the old one is simply reactivated. The document is reloaded using
385 layout(), if necessary.
386 </p>
387
388 <p>
389 load_page() is called from <a href="#main">main()</a> to load the
390 start page given on the command line, and then in various modes from the main
391 loop, when the user requests various functions in the
392 <a href="#viewer">Viewer</a> (loading new URL, following link,
393 going back/forward in page history).
394 </p>
395
396 <p>
397 The page loading mechanism is described in detail in
398 <a href="hacking-page.html">hacking-page.*</a>.
399 </p>
400
401 </a> <!-- page -->
402
403 <a name="layout" id="layout">
404
405 <h3>Layouting</h3>
406
407 <p>
408 The layout engine consists of the files parse-syntax.c, syntax.h, facilities.c,
409 dump-tree.c parse_elements.c, sgml.c, parse-struct.c, items.c, items.h, pre-render.c,
410 render.c, render.h, layout.c, and layout.h.
411 </p>
412
413 <p>
414 As mentioned before, layouting is done in several passes.
415 </p>
416
417 <p>
418 The first passes are parsing syntax
419 (<a href="hacking-layout.html#parseSyntax">parse_syntax()</a>),
420 looking up the element and attribute names
421 (<a href="hacking-layout.html#parseElements">parse_elements()</a>),
422 (optionally) fixing a broken tree created by SGML documents
423 (<a href="hacking-layout.html#sgmlRework">sgml_rework()</a>,
424 interpreting the elements
425 (<a href="hacking-layout.html#parseStruct">parse_struct()</a>),
426 and assigning positions and sizes to all items of the output page
427 (<a href="hacking-layout.html#preRender">pre_render()</a>).
428 </p>
429
430 <p>
431 make_link_list() and make_anchor_list() are also called after parse_struct(),
432 creating the <a href="hacking-links.html#linkList">link list</a> and
433 <a href="hacking-links.html#anchorList">anchor list</a> data
434 structures necessary for link handling. (See
435 <a href="hacking-links.html">hacking-links.*</a>.)
436 </p>
437
438 <p>
439 All these passes are applied from layout(), immediately after the file is
440 opened; <a href="#loading">load()</a> is called from within
441 parse_syntax().
442 </p>
443
444 <p>
445 After these processing steps, the page is ready for rendering. The actual rendering is
446 done just in time by <a href="hacking-layout.html#render">render()</a>,
447 which is called from the viewer, each time some new region of the output page
448 needs to be displayed.
449 </p>
450
451 <p>
452 The layout engine is described in detail in
453 <a href="hacking-layout.html">hacking-layout.*</a>.
454 </p>
455
456 </a> <!-- layout -->
457
458 <a name="viewer" id="viewer">
459
460 <h3>Viewer</h3>
461
462 <p>
463 The viewer module consists of the files pager.c and pager.h.
464 </p>
465
466 <p>
467 The pager uses curses to display the layouted page in an interactive manner.
468 </p>
469
470 <p>
471 Every time the visible output page region changes,
472 <a href="hacking-layout.html#render">render()</a> is called to display
473 the new region.
474 </p>
475
476 <p>
477 The viewer module is described in detail in
478 <a href="hacking-pager.html">hacking-pager.*</a>.
479 </p>
480
481 </a> <!-- viewer -->
482
483 <h3>Link Handling</h3>
484
485 <p>
486 Hyperlink (and anchor) handling is not a module in a classical sense; there is
487 no central place, no source files specific for that. There are the links.c and
488 links.h files, but they contain only some helper functions; most of the code
489 necessary for handling links is distributed among almost all of the other
490 modules.
491 </p>
492
493 <p>
494 The layout engine needs to extract all links and anchors while parsing the
495 page, and assign coordinates to them while pre-rendering. The pager needs to
496 highlight the selected link; provide commands for selecting and following
497 links; and inform the main program about the link following. The main program
498 needs to initiate loading of the link. The file loader needs to construct the
499 target URL from the current page URL and the link URL.
500 </p>
501
502 <p>
503 All necessary steps are described in
504 <a href="hacking-links.html">hacking-links.*</a>, or else pointers to
505 the specific module documentation are given.
506 </p>
507
508 <h3>Form Handling</h3>
509
510 <p>
511 HTML forms are very similar to links, and mostly they are handled together. Some
512 special handling is required at certain places, though. These are mentioned in
513 <a href="hacking-links.html">hacking-links.*</a>. Some additional
514 functions necessary for form handling reside in forms.c and form-file.c; these are also covered
515 in hacking-links.*.
516 </p>
517
518 <a name="search" id="search">
519
520 <h3>Text Search</h3>
521
522 <p>
523 The search command presently doesn't reside in a file of its own; search.c only
524 defines the global "search" struct, which is used to pass information between
525 the search handling inside the pager and in main(), as well as keep information
526 between multiple searches. Searching itself is performed inside main().
527 </p>
528
529 <p>
530 The search framework is very similar to the command prompt (see
531 <a href="#main">main.c</a>). When typing the search character,
532 display() quits with a special status. main() than requests a search string
533 (via readline) and performs the search, resulting in a new cursor position;
534 display() is then restarted in the next main loop iteration, and knows to jump
535 to the match postion by the state stored in "search.type".
536 </p>
537
538 <p>
539 Details on how the search itself works can be found in
540 <a href="hacking-search.html">hacking-search.*</a>.
541 </p>
542
543 </a> <!-- search -->
544
545 <h3>Terminal Handling</h3>
546
547 <p>
548 netrik uses the curses library for all terminal handling.
549 </p>
550
551 <p>
552 There are two distinct modes in which netrik operates: Full screen mode (where
553 all output is managed by curses), and scrolling mode, where curses (more
554 exactly, the terminfo handling functions of curses) is used to get some control
555 strings (color setting etc.) and for setting text attributes like bold, but
556 output is printed to the screen directly.
557 </p>
558
559 <p>
560 Other terminal related issues like keyboard input and signal handling are also
561 done using curses.
562 </p>
563
564 <p>
565 Most input/output tasks (reading a single character from keyboard, printing a
566 text string etc.) are trivial, and the curses functions are called directly
567 from the functions needing them (mostly in pager.c and render.c). There are
568 however a number of more complicated tasks like initialising the terminal or
569 setting colors, which can't be done in a single function call. Those are
570 handled by helper functions, which are located in
571 <a href="hacking-terminal.html#screenC">screen.c</a> (documented
572 in <a href="hacking-terminal.html">hacking-terminal.*</a>).
573 </p>
574
575 <p>
576 Another issue related to output (though only indirectly to terminal handling)
577 is handling of text colors inside netrik. This is also described in
578 hacking-terminal.*, under <a href="hacking-terminal.html#color">Color
579 Handling</a>.
580 </p>
581
582 <p>
583 Last but not least, handling of SIGWINCH is an important issue of terminal
584 handling -- reacting to terminal size changes. Handling the signals itself is
585 controlled by the helper functions in winch.c, called from pager.c. The real
586 resizing action necessary to adopt to the new screen size is handled by
587 <a href="#main">main.c</a>, which in turn uses the
588 <a href="hacking-layout.html#resize">resize()</a> function from
589 pre-render.c. (Described in
590 <a href="hacking-pager.html">hacking-pager.*</a>.)
591 </p>
592
593 <p>
594 SIGWINCH handling is described in more detail under
595 <a href="hacking-terminal.html#winch">SIGWINCH Handling</a>,
596 again in hacking-terminal.*.
597 </p>
598
599 </body>
600 </html>