Debian Codegraph @

tools/private_domains_checker/README.md

raw
1 # PSL Private Section Domains WHOIS Checker
2
3 ## Overview
4
5 The `PSLPrivateDomainsProcessor` is a Python script designed to fetch data from the Public Suffix List (PSL) and check the domain status, expiry dates, and `_psl` TXT records of the private section domains.
6
7 It performs WHOIS checks on these domains and saves the results into CSV files for manual review.
8
9 ## Requirements
10
11 - Python 3.x
12 - `requests`
13 - `pandas`
14 - `whoisdomain`
15
16 You can install the required packages using pip:
17
18 ```sh
19 pip install -r requirements.txt
20 ```
21
22 Ensure that `whois` is installed on your operating system.
23
24 ```sh
25 sudo apt install whois # Debian/Ubuntu
26 sudo yum install whois # Fedora/Centos/Rocky
27 ```
28
29 ## Usage
30
31 `PSLPrivateDomainsProcessor.py`: The main script containing the `PSLPrivateDomainsProcessor` class and functions for DNS and WHOIS checks.
32
33 Run the script using Python:
34
35 ```sh
36 cd private_domains_checker
37 mkdir data
38 python PSLPrivateDomainsProcessor.py
39 ```
40
41 ## Main Components
42
43 ### Functions
44
45 - `make_dns_request(domain, record_type)`: Makes DNS requests to both Google and Cloudflare DNS APIs.
46 - `check_dns_status(domain)`: Checks the DNS status of a domain using Google and Cloudflare DNS APIs.
47 - `get_whois_data(domain)`: Retrieves WHOIS data for a domain using the whoisdomain package.
48 - `check_psl_txt_record(domain)`: Checks the `_psl` TXT record for a domain using Google and Cloudflare DNS APIs.
49
50 ### Class
51
52 #### PSLPrivateDomainsProcessor
53
54 - `fetch_psl_data()`: Fetches the PSL data from the specified URL.
55 - `parse_domain(domain)`: Parses and normalizes a domain.
56 - `parse_psl_data(psl_data)`: Parses the fetched PSL data and separates ICANN and private domains.
57 - `process_domains(raw_domains, domains)`: Processes each domain, performing DNS, WHOIS, and `_psl` TXT record checks.
58 - `save_results()`: Saves all processed domain data to `data/all.csv`.
59 - `save_invalid_results()`: Saves domains with invalid DNS or expired WHOIS data to `data/nxdomain.csv` and `data/expired.csv`.
60 - `save_hold_results()`: Saves domains with WHOIS status containing any form of "hold" to `data/hold.csv`.
61 - `save_missing_psl_txt_results()`: Saves domains with invalid `_psl` TXT records to `data/missing_psl_txt.csv`.
62 - `save_expiry_less_than_2yrs_results()`: Saves domains with WHOIS expiry date less than 2 years from now to `data/expiry_less_than_2yrs.csv`.
63 - `run()`: Executes the entire processing pipeline.
64
65 ## Output
66
67 The script generates the following CSV files in the `data` directory:
68
69 - `all.csv`: Contains all processed domain data.
70 - `nxdomain.csv`: Contains domains that could not be resolved (`NXDOMAIN`).
71 - `expired.csv`: Contains domains with expired WHOIS records.
72 - `hold.csv`: Contains domains with WHOIS status indicating any kind of "hold".
73 - `missing_psl_txt.csv`: Contains domains with invalid `_psl` TXT records.
74 - `expiry_less_than_2yrs.csv`: Contains domains with WHOIS expiry date less than 2 years from now.
75
76 ## Example
77
78 An example CSV entry:
79
80 | psl_entry | top_level_domain | dns_status | whois_status | whois_domain_expiry_date | whois_domain_status | psl_txt_status | expiry_check_status |
81 | -------------- | ---------------- | ---------- | ------------ | ----------------------- | ---------------------------- | -------------- | ------------------- |
82 | example.com | example.com | ok | ok | 2024-12-31 | "clientTransferProhibited" | "valid" | ok |
83
84 ## Publicly Registrable Namespace Determination
85
86 The script determines the publicly registrable namespace from private domains by using the ICANN section.
87
88 Here's how it works:
89
90 1. **ICANN Domains Set**: ICANN domains are stored in a set for quick lookup.
91 2. **Domain Parsing**: For each private domain, the script splits the domain into parts. It then checks if any suffix of these parts exists in the ICANN domains set.
92 3. **Normalization**: The private domain is normalized to its publicly registrable form using the ICANN domains set.
93
94 Examples:
95
96 - **Input**: PSL private domain entry `"*.example.com"`
97 - **Process**:
98 - Remove leading `'*.'`: `"example.com"`
99 - Check `"com"` against the ICANN domains set: Found
100 - **Output**: `"example.com"`
101
102 - **Input**: PSL private domain entry `"sub.example.co.uk"`
103 - **Process**:
104 - Check `"example.co.uk"` against the ICANN domains set: Not found
105 - Check `"co.uk"` against the ICANN domains set: Found
106 - **Output**: `"example.co.uk"`
107
108 The output is then used for checking WHOIS data.
109
110 ## License
111
112 This tool is licensed under the MIT License.