Debian Codegraph @

rfc2517.html

raw
1 <pre>Network Working Group R. Moats
2 Request for Comments: 2517 R. Huber
3 Category: Informational AT&amp;T
4 February 1999
5
6
7 <span class="h1">Building Directories from DNS: Experiences from WWWSeeker</span>
8
9 Status of this Memo
10
11 This memo provides information for the Internet community. It does
12 not specify an Internet standard of any kind. Distribution of this
13 memo is unlimited.
14
15 Copyright Notice
16
17 Copyright (C) The Internet Society (1999). All Rights Reserved.
18
19 Abstract
20
21 There has been much discussion and several documents written about
22 the need for an Internet Directory. Recently, this discussion has
23 focused on ways to discover an organization's domain name without
24 relying on use of DNS as a directory service. This memo discusses
25 lessons that were learned during InterNIC Directory and Database
26 Services' development and operation of WWWSeeker, an application that
27 finds a web site given information about the name and location of an
28 organization. The back end database that drives this application was
29 built from information obtained from domain registries via WHOIS and
30 other protocols. We present this information to help future
31 implementors avoid some of the blind alleys that we have already
32 explored. This work builds on the Netfind system that was created by
33 Mike Schwartz and his team at the University of Colorado at Boulder
34 [<a href="#ref-1" title="&quot;Applying an Information Gathering Architecture to Netfind: A White Pages Tool for a Changing and Growing Internet&quot;">1</a>].
35
36 <span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span>
37
38 Over time, there have been several RFCs [<a href="#ref-2" title="&quot;Plan for Internet Directory Services&quot;">2</a>, <a href="#ref-3" title="&quot;A Strategic Plan for Deploying an Internet X.500 Directory Service&quot;">3</a>, <a href="#ref-4" title="&quot;White Pages Meeting Report&quot;">4</a>] about approaches
39 for providing Internet Directories. Many of the earlier documents
40 discussed white pages directories that supply mappings from a
41 person's name to their telephone number, email address, etc.
42
43 More recently, there has been discussion of directories that map from
44 a company name to a domain name or web site. Many people are using
45 DNS as a directory today to find this type of information about a
46 given company. Typically when DNS is used, users guess the domain
47 name of the company they are looking for and then prepend "www.".
48 This makes it highly desirable for a company to have an easily
49
50
51
52 <span class="grey">Moats &amp; Huber Informational [Page 1]</span></pre>
53 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span>
54 <span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span>
55
56
57 guessable name.
58
59 There are two major problems here. As the number of assigned names
60 increases, it becomes more difficult to get an easily guessable name.
61 Also, the TLD must be guessed as well as the name. While many users
62 just guess ".COM" as the "default" TLD today, there are many two-
63 letter country code top-level domains in current use as well as other
64 gTLDs (.NET, .ORG, and possibly .EDU) with the prospect of additional
65 gTLDs in the future. As the number of TLDs in general use increases,
66 guessing gets more difficult.
67
68 Between July 1996 and our shutdown in March 1998, the InterNIC
69 Directory and Database Services project maintained the Netfind search
70 engine [<a href="#ref-1" title="&quot;Applying an Information Gathering Architecture to Netfind: A White Pages Tool for a Changing and Growing Internet&quot;">1</a>] and the associated database that maps organization
71 information to domain names. This database thus acted as the type of
72 Internet directory that associates company names with domain names.
73 We also built WWWSeeker, a system that used the Netfind database to
74 find web sites associated with a given organization. The experienced
75 gained from maintaining and growing this database provides valuable
76 insight into the issues of providing a directory service. We present
77 it here to allow future implementors to avoid some of the blind
78 alleys that we have already explored.
79
80 <span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Directory Population</span>
81
82 <span class="h3"><a class="selflink" id="section-2.1" href="#section-2.1">2.1</a> What to do?</span>
83
84 There are two issues in populating a directory: finding all the
85 domain names (building the skeleton) and associating those domains
86 with entities (adding the meat). These two issues are discussed
87 below.
88
89 <span class="h3"><a class="selflink" id="section-2.2" href="#section-2.2">2.2</a> Building the skeleton</span>
90
91 In "building the skeleton", it is popular to suggest using a variant
92 of a "tree walk" to determine the domains that need to be added to
93 the directory. Our experience is that this is neither a reasonable
94 nor an efficient proposal for maintaining such a directory. Except
95 for some infrequent and long-standing DNS surveys [<a href="#ref-5" title="&quot;Network Wizards Internet Domain Survey&quot;">5</a>], DNS "tree
96 walks" tend to be discouraged by the Internet community, especially
97 given that the frequency of DNS changes would require a new tree walk
98 monthly (if not more often). Instead, our experience has shown that
99 data on allocated DNS domains can usually be retrieved in bulk
100 fashion with FTP, HTTP, or Gopher (we have used each of these for
101 particular TLDs). This has the added advantage of both "building the
102 skeleton" and "adding the meat" at the same time. Our favorite
103 method for finding a server that has allocated DNS domain information
104 is to start with the list maintained at
105
106
107
108 <span class="grey">Moats &amp; Huber Informational [Page 2]</span></pre>
109 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span>
110 <span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span>
111
112
113 <a href="http://www.alldomains.com/countryindex.html">http://www.alldomains.com/countryindex.html</a> and go from there.
114 Before this was available, it was necessary to hunt for a registry
115 using trial and error.
116
117 When maintaining the database, existing domains may be verified via
118 direct DNS lookups rather than a "tree walk." "Tree walks" should
119 therefore be the choice of last resort for directory population, and
120 bulk retrieval should be used whenever possible.
121
122 <span class="h3"><a class="selflink" id="section-2.3" href="#section-2.3">2.3</a> Adding the meat</span>
123
124 A possibility for populating a directory ("adding the meat") is to
125 use an automated system that makes repeated queries using the WHOIS
126 protocol to gather information about the organization that owns a
127 domain. The queries would be made against a WHOIS server located
128 with the above method. At the conclusion of the InterNIC Directory
129 and Database Services project, our backend database contained about
130 2.9 million records built from data that could be retrieved via
131 WHOIS. The entire database contained 3.25 million records, with the
132 additional records coming from sources other than WHOIS.
133
134 In our experience this information contains many factual and
135 typographical errors and requires further examination and processing
136 to improve its quality. Further, TLD registrars that support WHOIS
137 typically only support WHOIS information for second level domains
138 (i.e. ne.us) as opposed to lower level domains (i.e.
139 windrose.omaha.ne.us). Also, there are TLDs without registrars, TLDs
140 without WHOIS support, and still other TLDs that use other methods
141 (HTTP, FTP, gopher) for providing organizational information. Based
142 on our experience, an implementor of an internet directory needs to
143 support multiple protocols for directory population. An automated
144 WHOIS search tool is necessary, but isn't enough.
145
146 <span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. Directory Updating: Full Rebuilds vs Incremental Updates</span>
147
148 Given the size of our database in April 1998 when it was last
149 generated, a complete rebuild of the database that is available from
150 WHOIS lookups would require between 134.2 to 167.8 days just for
151 WHOIS lookups from a Sun SPARCstation 20. This estimate does not
152 include other considerations (for example, inverting the token tree
153 required about 24 hours processing time on a Sun SPARCstation 20)
154 that would increase the amount of time to rebuild the entire
155 database.
156
157 Whether this is feasible depends on the frequency of database updates
158 provided. Because of the rate of growth of allocated domain names
159 (150K-200K new allocated domains per month in early 1998), we
160 provided monthly updates of the database. To rebuild the database
161
162
163
164 <span class="grey">Moats &amp; Huber Informational [Page 3]</span></pre>
165 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span>
166 <span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span>
167
168
169 each month (based on the above time estimate) would require between 3
170 and 5 machines to be dedicated full time (independent of machine
171 architecture). Instead, we checkpointed the allocated domain list
172 and rebuild on an incremental basis during one weekend of the month.
173 This allowed us to complete the update on between 1 and 4 machines (3
174 Sun SPARCstation 20s and a dual-processor Sparcserver 690) without
175 full dedication over a couple of days. Further, by coupling
176 incremental updates with periodic refresh of existing data (which can
177 be done during another part of the month and doesn't require full
178 dedication of machine hardware), older records would be periodically
179 updated when the underlying information changes. The tradeoff is
180 timeliness and accuracy of data (some data in the database may be
181 old) against hardware and processing costs.
182
183 <span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Directory Presentation: Distributed vs Monolithic</span>
184
185 While a distributed directory is a desirable goal, we maintained our
186 database as a monolithic structure. Given past growth, it is not
187 clear at what point migrating to a distributed directory becomes
188 actually necessary to support customer queries. Our last database
189 contained over 3.25 million records in a flat ASCII file. Searching
190 was done via a PERL script of an inverted tree (also produced by a
191 PERL script). While admittedly primitive, this configuration
192 supported over 200,000 database queries per month from our production
193 servers.
194
195 Increasing the database size only requires more disk space to hold
196 the database and inverted tree. Of course, using database technology
197 would probably improve performance and scalability, but we had not
198 reached the point where this technology was required.
199
200 <span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Security Considerations</span>
201
202 The underlying data for the type of directory discussed in this
203 document is already generally available through WHOIS, DNS, and other
204 standard interfaces. No new information is made available by using
205 these techniques though many types of search become much easier. To
206 the extent that easier access to this data makes it easier to find
207 specific sites or machines to attack, security may be decreased.
208
209 The protocols discussed here do not have built-in security features.
210 If one source machine is spoofed while the directory data is being
211 gathered, substantial amounts of incorrect and misleading data could
212 be pulled in to the directory and be spread to a wider audience.
213
214
215
216
217
218
219
220 <span class="grey">Moats &amp; Huber Informational [Page 4]</span></pre>
221 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span>
222 <span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span>
223
224
225 In general, building a directory from registry data will not open any
226 new security holes since the data is already available to the public.
227 Existing security and accuracy problems with the data sources are
228 likely to be amplified.
229
230 <span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. Acknowledgments</span>
231
232 This work described in this document was partially supported by the
233 National Science Foundation under Cooperative Agreement NCR-9218179.
234
235 <span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. References</span>
236
237 [<a id="ref-1">1</a>] M. F. Schwartz, C. Pu. "Applying an Information
238 Gathering Architecture to Netfind: A White Pages Tool for a
239 Changing and Growing Internet", University of Colorado Technical
240 Report CU-CS-656-93. December 1993, revised July 1994.
241
242 URL:ftp://ftp.cs.colorado.edu/pub/cs/techreports/schwartz/Netfind
243
244 [<a id="ref-2">2</a>] Sollins, K., "Plan for Internet Directory Services", <a href="./rfc1107">RFC 1107</a>,
245 July 1989.
246
247 [<a id="ref-3">3</a>] Hardcastle-Kille, S., Huizer, E., Cerf, V., Hobby, R. and S.
248 Kent, "A Strategic Plan for Deploying an Internet X.500 Directory
249 Service", <a href="./rfc1430">RFC 1430</a>, February 1993.
250
251 [<a id="ref-4">4</a>] Postel, J. and C. Anderson, "White Pages Meeting Report", <a href="./rfc1588">RFC</a>
252 <a href="./rfc1588">1588</a>, February 1994.
253
254 [<a id="ref-5">5</a>] M. Lottor, "Network Wizards Internet Domain Survey", available
255 from <a href="http://www.nw.com/zone/WWW/top.html">http://www.nw.com/zone/WWW/top.html</a>
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276 <span class="grey">Moats &amp; Huber Informational [Page 5]</span></pre>
277 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span>
278 <span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span>
279
280
281 <span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. Authors' Addresses</span>
282
283 Ryan Moats
284 AT&amp;T
285 15621 Drexel Circle
286 Omaha, NE 68135-2358
287 USA
288
289 EMail: [email protected]
290
291
292 Rick Huber
293 AT&amp;T
294 Room C3-3B30, 200 Laurel Ave. South
295 Middletown, NJ 07748
296 USA
297
298 EMail: [email protected]
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332 <span class="grey">Moats &amp; Huber Informational [Page 6]</span></pre>
333 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span>
334 <span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span>
335
336
337 <span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. Full Copyright Statement</span>
338
339 Copyright (C) The Internet Society (1999). All Rights Reserved.
340
341 This document and translations of it may be copied and furnished to
342 others, and derivative works that comment on or otherwise explain it
343 or assist in its implementation may be prepared, copied, published
344 and distributed, in whole or in part, without restriction of any
345 kind, provided that the above copyright notice and this paragraph are
346 included on all such copies and derivative works. However, this
347 document itself may not be modified in any way, such as by removing
348 the copyright notice or references to the Internet Society or other
349 Internet organizations, except as needed for the purpose of
350 developing Internet standards in which case the procedures for
351 copyrights defined in the Internet Standards process must be
352 followed, or as required to translate it into languages other than
353 English.
354
355 The limited permissions granted above are perpetual and will not be
356 revoked by the Internet Society or its successors or assigns.
357
358 This document and the information contained herein is provided on an
359 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
360 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
361 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
362 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
363 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388 Moats &amp; Huber Informational [Page 7]
389 </pre>