Debian Codegraph @

docs/CRAWL.announce

raw
1 The TRAVERSAL code from old versions of Lynx has been upgraded by David
2 Mathog ([email protected]) so that it works again, can be
3 implemented via a command line switch (-traversal) instead of via a
4 compilation symbol for creating a separate Lynx executable as in those
5 previous versions, and can be used in conjunction with a -crawl switch
6 to make Lynx a front end for a Web Crawler.
7
8
9 Usage:
10
11 lynx [-traversal] [-realm] [-crawl] ["startpage"]
12
13
14 Added switches are:
15
16 -traversal Follow all http links derived from startpage that are
17 on the same server as startpage. If startpage isn't
18 specified then the traversal begins with the default
19 startfile or WWW_HOME.
20
21 -realm Further restrict http links to ones in the same realm
22 (having a matching base URI) as the startpage (e.g.,
23 http://host/~user/ will restrict the traversal to that
24 user's public html tree).
25
26 -crawl With [-traversal] outputs each unique hypertext page
27 as an lnk###########.dat file in the format specified
28 below. With [-dump] outputs only the startpage, in
29 the same format, to stdout.
30
31
32 Note on startpage:
33
34 If a startpage is specified and contains any uppercase
35 characters, on VMS it should be enclosed in double-quotes.
36 The code that extracts the access and host fields from
37 startpage for comparisons with links to ensure they are
38 not on another server, and the comparisons with already
39 traversed links, are case sensitive, and the startpage
40 will go to all lowercase on VMS if no double-quotes are
41 supplied, such that it might be treated as a new link if
42 encountered with uppercase letters.
43
44
45 Files created and/or used with the -traversal switch, based on definitions
46 in userdefs.h:
47
48 TRAVERSE_FILE (traverse.dat):
49 Contains a list of all URLs that were traversed. Note
50 that if a URL appears in this file it will not be
51 traversed again (important if runs are started and
52 stopped). Placing an entry in this file BEFORE the
53 run will block traversal of that URL. Unlike reject.dat
54 a final * has no effect (see below). Note that Lynx
55 internal client-side image MAP URLs will be included in
56 this file (e.g., LYNXIMGMAP:http://server/foo.html#map1),
57 in addition to the "real" (external) http URLs.
58
59 TRAVERSE_FOUND_FILE (traverse2.dat):
60 Contains a list of all URLs that were traversed, in the
61 order encountered or re-encountered (but not re-traversed)
62 during a traversal run, and the TITLEs of the documents
63 (separated from the URLs by TABs) A URL and TITLE may be
64 present in this list many times. To simplify the list,
65 on VMS use: sort/nodups traverse2.dat;1 ;2
66 Note that the URLs and TITLEs of the Lynx internal
67 client-side image MAP pseudo-documents will not be
68 included in this file, though "traversed", but only the
69 http URLs and TITLEs derived from the MAP's AREA tag
70 HREFs that were traversed.
71
72 TRAVERSE_REJECT_FILE (reject.dat):
73 Contains a list of URLs that have been rejected from the
74 traversal. Once a URL has been entered in this list, it
75 will not be traversed. URLs that end in a * will cause
76 rejection of all URLs that match up to the character before
77 the *. So for instance, to reject all htbin references on a
78 site put this line in the reject.dat file BEFORE starting
79 the run: http://www.site.wherever:8000/htbin*
80
81 TRAVERSE_ERRORS (traverse.errors):
82 A list of links that could not be accessed or had an
83 unknown status returned by the http server. If the
84 owner of the document containing the link is know via
85 a LINK REV="made" HREF="mailto:foo" in it and the
86 MAIL_SYSTEM_ERROR_LOGGING was set true in userdefs.h
87 or lynx.cfg (not recommended!!!), a message about the
88 problem will be mailed to the owner as well.
89
90
91 Files created during traversals if the -crawl switch is included with the
92 -traversal switch:
93
94 lnk########.dat Numbered output files containing the contents of traversed
95 hypertext documents in text format. All hypertext links
96 within the document have been stripped, and the URL and
97 TITLE of the document are recorded as the first two lines,
98 e.g., for the seqaxp.bio.caltech.edu home page the first
99 two lines will be:
100
101 THE_URL:http://seqaxp.bio.caltech.edu:8000/
102 THE_TITLE:SAF Web server home page
103
104 The VMSIndex software is being adapted to use this
105 information to extract the corresponding URL and TITLE
106 for use in indexing the lnk########.dat files, e.g.:
107
108 $ build_index -
109 /url=(text="THE_URL:") -
110 /topic=(text="THE_TITLE:",EXCLUDE) -
111 /output=INDEX_NAME -
112 lnk*.dat
113
114 A clever person should be able to figure out a way to
115 index the lnk########.dat files on Unix as well.
116
117 If you want the hypertext links in the document to be
118 numbered, include the -number_links switch. By default,
119 this will cause the list of References (URLs for the
120 numbered links) to be appended as well. If you want
121 numbered links but not the References list, include the
122 -nolist switch as well.
123
124 Note that any client-side image MAP pseudo documents
125 that were "traversed" will not have lnk########.dat
126 output files created for them, but output files will
127 be created for "real" documents that were traversed
128 based on the HREFs of the MAP's AREA tags.
129
130 This functionality is still under development. Feedback and suggestions
131 are welcome.