| 1 |
<pre>Network Working Group R. Moats |
| 2 |
Request for Comments: 2517 R. Huber |
| 3 |
Category: Informational AT&T |
| 4 |
February 1999 |
| 5 |
|
| 6 |
|
| 7 |
<span class="h1">Building Directories from DNS: Experiences from WWWSeeker</span> |
| 8 |
|
| 9 |
Status of this Memo |
| 10 |
|
| 11 |
This memo provides information for the Internet community. It does |
| 12 |
not specify an Internet standard of any kind. Distribution of this |
| 13 |
memo is unlimited. |
| 14 |
|
| 15 |
Copyright Notice |
| 16 |
|
| 17 |
Copyright (C) The Internet Society (1999). All Rights Reserved. |
| 18 |
|
| 19 |
Abstract |
| 20 |
|
| 21 |
There has been much discussion and several documents written about |
| 22 |
the need for an Internet Directory. Recently, this discussion has |
| 23 |
focused on ways to discover an organization's domain name without |
| 24 |
relying on use of DNS as a directory service. This memo discusses |
| 25 |
lessons that were learned during InterNIC Directory and Database |
| 26 |
Services' development and operation of WWWSeeker, an application that |
| 27 |
finds a web site given information about the name and location of an |
| 28 |
organization. The back end database that drives this application was |
| 29 |
built from information obtained from domain registries via WHOIS and |
| 30 |
other protocols. We present this information to help future |
| 31 |
implementors avoid some of the blind alleys that we have already |
| 32 |
explored. This work builds on the Netfind system that was created by |
| 33 |
Mike Schwartz and his team at the University of Colorado at Boulder |
| 34 |
[<a href="#ref-1" title=""Applying an Information Gathering Architecture to Netfind: A White Pages Tool for a Changing and Growing Internet"">1</a>]. |
| 35 |
|
| 36 |
<span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span> |
| 37 |
|
| 38 |
Over time, there have been several RFCs [<a href="#ref-2" title=""Plan for Internet Directory Services"">2</a>, <a href="#ref-3" title=""A Strategic Plan for Deploying an Internet X.500 Directory Service"">3</a>, <a href="#ref-4" title=""White Pages Meeting Report"">4</a>] about approaches |
| 39 |
for providing Internet Directories. Many of the earlier documents |
| 40 |
discussed white pages directories that supply mappings from a |
| 41 |
person's name to their telephone number, email address, etc. |
| 42 |
|
| 43 |
More recently, there has been discussion of directories that map from |
| 44 |
a company name to a domain name or web site. Many people are using |
| 45 |
DNS as a directory today to find this type of information about a |
| 46 |
given company. Typically when DNS is used, users guess the domain |
| 47 |
name of the company they are looking for and then prepend "www.". |
| 48 |
This makes it highly desirable for a company to have an easily |
| 49 |
|
| 50 |
|
| 51 |
|
| 52 |
<span class="grey">Moats & Huber Informational [Page 1]</span></pre> |
| 53 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span> |
| 54 |
<span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span> |
| 55 |
|
| 56 |
|
| 57 |
guessable name. |
| 58 |
|
| 59 |
There are two major problems here. As the number of assigned names |
| 60 |
increases, it becomes more difficult to get an easily guessable name. |
| 61 |
Also, the TLD must be guessed as well as the name. While many users |
| 62 |
just guess ".COM" as the "default" TLD today, there are many two- |
| 63 |
letter country code top-level domains in current use as well as other |
| 64 |
gTLDs (.NET, .ORG, and possibly .EDU) with the prospect of additional |
| 65 |
gTLDs in the future. As the number of TLDs in general use increases, |
| 66 |
guessing gets more difficult. |
| 67 |
|
| 68 |
Between July 1996 and our shutdown in March 1998, the InterNIC |
| 69 |
Directory and Database Services project maintained the Netfind search |
| 70 |
engine [<a href="#ref-1" title=""Applying an Information Gathering Architecture to Netfind: A White Pages Tool for a Changing and Growing Internet"">1</a>] and the associated database that maps organization |
| 71 |
information to domain names. This database thus acted as the type of |
| 72 |
Internet directory that associates company names with domain names. |
| 73 |
We also built WWWSeeker, a system that used the Netfind database to |
| 74 |
find web sites associated with a given organization. The experienced |
| 75 |
gained from maintaining and growing this database provides valuable |
| 76 |
insight into the issues of providing a directory service. We present |
| 77 |
it here to allow future implementors to avoid some of the blind |
| 78 |
alleys that we have already explored. |
| 79 |
|
| 80 |
<span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Directory Population</span> |
| 81 |
|
| 82 |
<span class="h3"><a class="selflink" id="section-2.1" href="#section-2.1">2.1</a> What to do?</span> |
| 83 |
|
| 84 |
There are two issues in populating a directory: finding all the |
| 85 |
domain names (building the skeleton) and associating those domains |
| 86 |
with entities (adding the meat). These two issues are discussed |
| 87 |
below. |
| 88 |
|
| 89 |
<span class="h3"><a class="selflink" id="section-2.2" href="#section-2.2">2.2</a> Building the skeleton</span> |
| 90 |
|
| 91 |
In "building the skeleton", it is popular to suggest using a variant |
| 92 |
of a "tree walk" to determine the domains that need to be added to |
| 93 |
the directory. Our experience is that this is neither a reasonable |
| 94 |
nor an efficient proposal for maintaining such a directory. Except |
| 95 |
for some infrequent and long-standing DNS surveys [<a href="#ref-5" title=""Network Wizards Internet Domain Survey"">5</a>], DNS "tree |
| 96 |
walks" tend to be discouraged by the Internet community, especially |
| 97 |
given that the frequency of DNS changes would require a new tree walk |
| 98 |
monthly (if not more often). Instead, our experience has shown that |
| 99 |
data on allocated DNS domains can usually be retrieved in bulk |
| 100 |
fashion with FTP, HTTP, or Gopher (we have used each of these for |
| 101 |
particular TLDs). This has the added advantage of both "building the |
| 102 |
skeleton" and "adding the meat" at the same time. Our favorite |
| 103 |
method for finding a server that has allocated DNS domain information |
| 104 |
is to start with the list maintained at |
| 105 |
|
| 106 |
|
| 107 |
|
| 108 |
<span class="grey">Moats & Huber Informational [Page 2]</span></pre> |
| 109 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span> |
| 110 |
<span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span> |
| 111 |
|
| 112 |
|
| 113 |
<a href="http://www.alldomains.com/countryindex.html">http://www.alldomains.com/countryindex.html</a> and go from there. |
| 114 |
Before this was available, it was necessary to hunt for a registry |
| 115 |
using trial and error. |
| 116 |
|
| 117 |
When maintaining the database, existing domains may be verified via |
| 118 |
direct DNS lookups rather than a "tree walk." "Tree walks" should |
| 119 |
therefore be the choice of last resort for directory population, and |
| 120 |
bulk retrieval should be used whenever possible. |
| 121 |
|
| 122 |
<span class="h3"><a class="selflink" id="section-2.3" href="#section-2.3">2.3</a> Adding the meat</span> |
| 123 |
|
| 124 |
A possibility for populating a directory ("adding the meat") is to |
| 125 |
use an automated system that makes repeated queries using the WHOIS |
| 126 |
protocol to gather information about the organization that owns a |
| 127 |
domain. The queries would be made against a WHOIS server located |
| 128 |
with the above method. At the conclusion of the InterNIC Directory |
| 129 |
and Database Services project, our backend database contained about |
| 130 |
2.9 million records built from data that could be retrieved via |
| 131 |
WHOIS. The entire database contained 3.25 million records, with the |
| 132 |
additional records coming from sources other than WHOIS. |
| 133 |
|
| 134 |
In our experience this information contains many factual and |
| 135 |
typographical errors and requires further examination and processing |
| 136 |
to improve its quality. Further, TLD registrars that support WHOIS |
| 137 |
typically only support WHOIS information for second level domains |
| 138 |
(i.e. ne.us) as opposed to lower level domains (i.e. |
| 139 |
windrose.omaha.ne.us). Also, there are TLDs without registrars, TLDs |
| 140 |
without WHOIS support, and still other TLDs that use other methods |
| 141 |
(HTTP, FTP, gopher) for providing organizational information. Based |
| 142 |
on our experience, an implementor of an internet directory needs to |
| 143 |
support multiple protocols for directory population. An automated |
| 144 |
WHOIS search tool is necessary, but isn't enough. |
| 145 |
|
| 146 |
<span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. Directory Updating: Full Rebuilds vs Incremental Updates</span> |
| 147 |
|
| 148 |
Given the size of our database in April 1998 when it was last |
| 149 |
generated, a complete rebuild of the database that is available from |
| 150 |
WHOIS lookups would require between 134.2 to 167.8 days just for |
| 151 |
WHOIS lookups from a Sun SPARCstation 20. This estimate does not |
| 152 |
include other considerations (for example, inverting the token tree |
| 153 |
required about 24 hours processing time on a Sun SPARCstation 20) |
| 154 |
that would increase the amount of time to rebuild the entire |
| 155 |
database. |
| 156 |
|
| 157 |
Whether this is feasible depends on the frequency of database updates |
| 158 |
provided. Because of the rate of growth of allocated domain names |
| 159 |
(150K-200K new allocated domains per month in early 1998), we |
| 160 |
provided monthly updates of the database. To rebuild the database |
| 161 |
|
| 162 |
|
| 163 |
|
| 164 |
<span class="grey">Moats & Huber Informational [Page 3]</span></pre> |
| 165 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span> |
| 166 |
<span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span> |
| 167 |
|
| 168 |
|
| 169 |
each month (based on the above time estimate) would require between 3 |
| 170 |
and 5 machines to be dedicated full time (independent of machine |
| 171 |
architecture). Instead, we checkpointed the allocated domain list |
| 172 |
and rebuild on an incremental basis during one weekend of the month. |
| 173 |
This allowed us to complete the update on between 1 and 4 machines (3 |
| 174 |
Sun SPARCstation 20s and a dual-processor Sparcserver 690) without |
| 175 |
full dedication over a couple of days. Further, by coupling |
| 176 |
incremental updates with periodic refresh of existing data (which can |
| 177 |
be done during another part of the month and doesn't require full |
| 178 |
dedication of machine hardware), older records would be periodically |
| 179 |
updated when the underlying information changes. The tradeoff is |
| 180 |
timeliness and accuracy of data (some data in the database may be |
| 181 |
old) against hardware and processing costs. |
| 182 |
|
| 183 |
<span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Directory Presentation: Distributed vs Monolithic</span> |
| 184 |
|
| 185 |
While a distributed directory is a desirable goal, we maintained our |
| 186 |
database as a monolithic structure. Given past growth, it is not |
| 187 |
clear at what point migrating to a distributed directory becomes |
| 188 |
actually necessary to support customer queries. Our last database |
| 189 |
contained over 3.25 million records in a flat ASCII file. Searching |
| 190 |
was done via a PERL script of an inverted tree (also produced by a |
| 191 |
PERL script). While admittedly primitive, this configuration |
| 192 |
supported over 200,000 database queries per month from our production |
| 193 |
servers. |
| 194 |
|
| 195 |
Increasing the database size only requires more disk space to hold |
| 196 |
the database and inverted tree. Of course, using database technology |
| 197 |
would probably improve performance and scalability, but we had not |
| 198 |
reached the point where this technology was required. |
| 199 |
|
| 200 |
<span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Security Considerations</span> |
| 201 |
|
| 202 |
The underlying data for the type of directory discussed in this |
| 203 |
document is already generally available through WHOIS, DNS, and other |
| 204 |
standard interfaces. No new information is made available by using |
| 205 |
these techniques though many types of search become much easier. To |
| 206 |
the extent that easier access to this data makes it easier to find |
| 207 |
specific sites or machines to attack, security may be decreased. |
| 208 |
|
| 209 |
The protocols discussed here do not have built-in security features. |
| 210 |
If one source machine is spoofed while the directory data is being |
| 211 |
gathered, substantial amounts of incorrect and misleading data could |
| 212 |
be pulled in to the directory and be spread to a wider audience. |
| 213 |
|
| 214 |
|
| 215 |
|
| 216 |
|
| 217 |
|
| 218 |
|
| 219 |
|
| 220 |
<span class="grey">Moats & Huber Informational [Page 4]</span></pre> |
| 221 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span> |
| 222 |
<span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span> |
| 223 |
|
| 224 |
|
| 225 |
In general, building a directory from registry data will not open any |
| 226 |
new security holes since the data is already available to the public. |
| 227 |
Existing security and accuracy problems with the data sources are |
| 228 |
likely to be amplified. |
| 229 |
|
| 230 |
<span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. Acknowledgments</span> |
| 231 |
|
| 232 |
This work described in this document was partially supported by the |
| 233 |
National Science Foundation under Cooperative Agreement NCR-9218179. |
| 234 |
|
| 235 |
<span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. References</span> |
| 236 |
|
| 237 |
[<a id="ref-1">1</a>] M. F. Schwartz, C. Pu. "Applying an Information |
| 238 |
Gathering Architecture to Netfind: A White Pages Tool for a |
| 239 |
Changing and Growing Internet", University of Colorado Technical |
| 240 |
Report CU-CS-656-93. December 1993, revised July 1994. |
| 241 |
|
| 242 |
URL:ftp://ftp.cs.colorado.edu/pub/cs/techreports/schwartz/Netfind |
| 243 |
|
| 244 |
[<a id="ref-2">2</a>] Sollins, K., "Plan for Internet Directory Services", <a href="./rfc1107">RFC 1107</a>, |
| 245 |
July 1989. |
| 246 |
|
| 247 |
[<a id="ref-3">3</a>] Hardcastle-Kille, S., Huizer, E., Cerf, V., Hobby, R. and S. |
| 248 |
Kent, "A Strategic Plan for Deploying an Internet X.500 Directory |
| 249 |
Service", <a href="./rfc1430">RFC 1430</a>, February 1993. |
| 250 |
|
| 251 |
[<a id="ref-4">4</a>] Postel, J. and C. Anderson, "White Pages Meeting Report", <a href="./rfc1588">RFC</a> |
| 252 |
<a href="./rfc1588">1588</a>, February 1994. |
| 253 |
|
| 254 |
[<a id="ref-5">5</a>] M. Lottor, "Network Wizards Internet Domain Survey", available |
| 255 |
from <a href="http://www.nw.com/zone/WWW/top.html">http://www.nw.com/zone/WWW/top.html</a> |
| 256 |
|
| 257 |
|
| 258 |
|
| 259 |
|
| 260 |
|
| 261 |
|
| 262 |
|
| 263 |
|
| 264 |
|
| 265 |
|
| 266 |
|
| 267 |
|
| 268 |
|
| 269 |
|
| 270 |
|
| 271 |
|
| 272 |
|
| 273 |
|
| 274 |
|
| 275 |
|
| 276 |
<span class="grey">Moats & Huber Informational [Page 5]</span></pre> |
| 277 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span> |
| 278 |
<span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span> |
| 279 |
|
| 280 |
|
| 281 |
<span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. Authors' Addresses</span> |
| 282 |
|
| 283 |
Ryan Moats |
| 284 |
AT&T |
| 285 |
15621 Drexel Circle |
| 286 |
Omaha, NE 68135-2358 |
| 287 |
USA |
| 288 |
|
| 289 |
EMail: [email protected] |
| 290 |
|
| 291 |
|
| 292 |
Rick Huber |
| 293 |
AT&T |
| 294 |
Room C3-3B30, 200 Laurel Ave. South |
| 295 |
Middletown, NJ 07748 |
| 296 |
USA |
| 297 |
|
| 298 |
EMail: [email protected] |
| 299 |
|
| 300 |
|
| 301 |
|
| 302 |
|
| 303 |
|
| 304 |
|
| 305 |
|
| 306 |
|
| 307 |
|
| 308 |
|
| 309 |
|
| 310 |
|
| 311 |
|
| 312 |
|
| 313 |
|
| 314 |
|
| 315 |
|
| 316 |
|
| 317 |
|
| 318 |
|
| 319 |
|
| 320 |
|
| 321 |
|
| 322 |
|
| 323 |
|
| 324 |
|
| 325 |
|
| 326 |
|
| 327 |
|
| 328 |
|
| 329 |
|
| 330 |
|
| 331 |
|
| 332 |
<span class="grey">Moats & Huber Informational [Page 6]</span></pre> |
| 333 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span> |
| 334 |
<span class="grey"><a href="./rfc2517">RFC 2517</a> Building Directories from DNS February 1999</span> |
| 335 |
|
| 336 |
|
| 337 |
<span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. Full Copyright Statement</span> |
| 338 |
|
| 339 |
Copyright (C) The Internet Society (1999). All Rights Reserved. |
| 340 |
|
| 341 |
This document and translations of it may be copied and furnished to |
| 342 |
others, and derivative works that comment on or otherwise explain it |
| 343 |
or assist in its implementation may be prepared, copied, published |
| 344 |
and distributed, in whole or in part, without restriction of any |
| 345 |
kind, provided that the above copyright notice and this paragraph are |
| 346 |
included on all such copies and derivative works. However, this |
| 347 |
document itself may not be modified in any way, such as by removing |
| 348 |
the copyright notice or references to the Internet Society or other |
| 349 |
Internet organizations, except as needed for the purpose of |
| 350 |
developing Internet standards in which case the procedures for |
| 351 |
copyrights defined in the Internet Standards process must be |
| 352 |
followed, or as required to translate it into languages other than |
| 353 |
English. |
| 354 |
|
| 355 |
The limited permissions granted above are perpetual and will not be |
| 356 |
revoked by the Internet Society or its successors or assigns. |
| 357 |
|
| 358 |
This document and the information contained herein is provided on an |
| 359 |
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING |
| 360 |
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING |
| 361 |
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION |
| 362 |
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF |
| 363 |
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. |
| 364 |
|
| 365 |
|
| 366 |
|
| 367 |
|
| 368 |
|
| 369 |
|
| 370 |
|
| 371 |
|
| 372 |
|
| 373 |
|
| 374 |
|
| 375 |
|
| 376 |
|
| 377 |
|
| 378 |
|
| 379 |
|
| 380 |
|
| 381 |
|
| 382 |
|
| 383 |
|
| 384 |
|
| 385 |
|
| 386 |
|
| 387 |
|
| 388 |
Moats & Huber Informational [Page 7] |
| 389 |
</pre> |