| 1 |
The TRAVERSAL code from old versions of Lynx has been upgraded by David |
| 2 |
Mathog ([email protected]) so that it works again, can be |
| 3 |
implemented via a command line switch (-traversal) instead of via a |
| 4 |
compilation symbol for creating a separate Lynx executable as in those |
| 5 |
previous versions, and can be used in conjunction with a -crawl switch |
| 6 |
to make Lynx a front end for a Web Crawler. |
| 7 |
|
| 8 |
|
| 9 |
Usage: |
| 10 |
|
| 11 |
lynx [-traversal] [-realm] [-crawl] ["startpage"] |
| 12 |
|
| 13 |
|
| 14 |
Added switches are: |
| 15 |
|
| 16 |
-traversal Follow all http links derived from startpage that are |
| 17 |
on the same server as startpage. If startpage isn't |
| 18 |
specified then the traversal begins with the default |
| 19 |
startfile or WWW_HOME. |
| 20 |
|
| 21 |
-realm Further restrict http links to ones in the same realm |
| 22 |
(having a matching base URI) as the startpage (e.g., |
| 23 |
http://host/~user/ will restrict the traversal to that |
| 24 |
user's public html tree). |
| 25 |
|
| 26 |
-crawl With [-traversal] outputs each unique hypertext page |
| 27 |
as an lnk###########.dat file in the format specified |
| 28 |
below. With [-dump] outputs only the startpage, in |
| 29 |
the same format, to stdout. |
| 30 |
|
| 31 |
|
| 32 |
Note on startpage: |
| 33 |
|
| 34 |
If a startpage is specified and contains any uppercase |
| 35 |
characters, on VMS it should be enclosed in double-quotes. |
| 36 |
The code that extracts the access and host fields from |
| 37 |
startpage for comparisons with links to ensure they are |
| 38 |
not on another server, and the comparisons with already |
| 39 |
traversed links, are case sensitive, and the startpage |
| 40 |
will go to all lowercase on VMS if no double-quotes are |
| 41 |
supplied, such that it might be treated as a new link if |
| 42 |
encountered with uppercase letters. |
| 43 |
|
| 44 |
|
| 45 |
Files created and/or used with the -traversal switch, based on definitions |
| 46 |
in userdefs.h: |
| 47 |
|
| 48 |
TRAVERSE_FILE (traverse.dat): |
| 49 |
Contains a list of all URLs that were traversed. Note |
| 50 |
that if a URL appears in this file it will not be |
| 51 |
traversed again (important if runs are started and |
| 52 |
stopped). Placing an entry in this file BEFORE the |
| 53 |
run will block traversal of that URL. Unlike reject.dat |
| 54 |
a final * has no effect (see below). Note that Lynx |
| 55 |
internal client-side image MAP URLs will be included in |
| 56 |
this file (e.g., LYNXIMGMAP:http://server/foo.html#map1), |
| 57 |
in addition to the "real" (external) http URLs. |
| 58 |
|
| 59 |
TRAVERSE_FOUND_FILE (traverse2.dat): |
| 60 |
Contains a list of all URLs that were traversed, in the |
| 61 |
order encountered or re-encountered (but not re-traversed) |
| 62 |
during a traversal run, and the TITLEs of the documents |
| 63 |
(separated from the URLs by TABs) A URL and TITLE may be |
| 64 |
present in this list many times. To simplify the list, |
| 65 |
on VMS use: sort/nodups traverse2.dat;1 ;2 |
| 66 |
Note that the URLs and TITLEs of the Lynx internal |
| 67 |
client-side image MAP pseudo-documents will not be |
| 68 |
included in this file, though "traversed", but only the |
| 69 |
http URLs and TITLEs derived from the MAP's AREA tag |
| 70 |
HREFs that were traversed. |
| 71 |
|
| 72 |
TRAVERSE_REJECT_FILE (reject.dat): |
| 73 |
Contains a list of URLs that have been rejected from the |
| 74 |
traversal. Once a URL has been entered in this list, it |
| 75 |
will not be traversed. URLs that end in a * will cause |
| 76 |
rejection of all URLs that match up to the character before |
| 77 |
the *. So for instance, to reject all htbin references on a |
| 78 |
site put this line in the reject.dat file BEFORE starting |
| 79 |
the run: http://www.site.wherever:8000/htbin* |
| 80 |
|
| 81 |
TRAVERSE_ERRORS (traverse.errors): |
| 82 |
A list of links that could not be accessed or had an |
| 83 |
unknown status returned by the http server. If the |
| 84 |
owner of the document containing the link is know via |
| 85 |
a LINK REV="made" HREF="mailto:foo" in it and the |
| 86 |
MAIL_SYSTEM_ERROR_LOGGING was set true in userdefs.h |
| 87 |
or lynx.cfg (not recommended!!!), a message about the |
| 88 |
problem will be mailed to the owner as well. |
| 89 |
|
| 90 |
|
| 91 |
Files created during traversals if the -crawl switch is included with the |
| 92 |
-traversal switch: |
| 93 |
|
| 94 |
lnk########.dat Numbered output files containing the contents of traversed |
| 95 |
hypertext documents in text format. All hypertext links |
| 96 |
within the document have been stripped, and the URL and |
| 97 |
TITLE of the document are recorded as the first two lines, |
| 98 |
e.g., for the seqaxp.bio.caltech.edu home page the first |
| 99 |
two lines will be: |
| 100 |
|
| 101 |
THE_URL:http://seqaxp.bio.caltech.edu:8000/ |
| 102 |
THE_TITLE:SAF Web server home page |
| 103 |
|
| 104 |
The VMSIndex software is being adapted to use this |
| 105 |
information to extract the corresponding URL and TITLE |
| 106 |
for use in indexing the lnk########.dat files, e.g.: |
| 107 |
|
| 108 |
$ build_index - |
| 109 |
/url=(text="THE_URL:") - |
| 110 |
/topic=(text="THE_TITLE:",EXCLUDE) - |
| 111 |
/output=INDEX_NAME - |
| 112 |
lnk*.dat |
| 113 |
|
| 114 |
A clever person should be able to figure out a way to |
| 115 |
index the lnk########.dat files on Unix as well. |
| 116 |
|
| 117 |
If you want the hypertext links in the document to be |
| 118 |
numbered, include the -number_links switch. By default, |
| 119 |
this will cause the list of References (URLs for the |
| 120 |
numbered links) to be appended as well. If you want |
| 121 |
numbered links but not the References list, include the |
| 122 |
-nolist switch as well. |
| 123 |
|
| 124 |
Note that any client-side image MAP pseudo documents |
| 125 |
that were "traversed" will not have lnk########.dat |
| 126 |
output files created for them, but output files will |
| 127 |
be created for "real" documents that were traversed |
| 128 |
based on the HREFs of the MAP's AREA tags. |
| 129 |
|
| 130 |
This functionality is still under development. Feedback and suggestions |
| 131 |
are welcome. |