| 1 |
# PSL Private Section Domains WHOIS Checker |
| 2 |
|
| 3 |
## Overview |
| 4 |
|
| 5 |
The `PSLPrivateDomainsProcessor` is a Python script designed to fetch data from the Public Suffix List (PSL) and check the domain status, expiry dates, and `_psl` TXT records of the private section domains. |
| 6 |
|
| 7 |
It performs WHOIS checks on these domains and saves the results into CSV files for manual review. |
| 8 |
|
| 9 |
## Requirements |
| 10 |
|
| 11 |
- Python 3.x |
| 12 |
- `requests` |
| 13 |
- `pandas` |
| 14 |
- `whoisdomain` |
| 15 |
|
| 16 |
You can install the required packages using pip: |
| 17 |
|
| 18 |
```sh |
| 19 |
pip install -r requirements.txt |
| 20 |
``` |
| 21 |
|
| 22 |
Ensure that `whois` is installed on your operating system. |
| 23 |
|
| 24 |
```sh |
| 25 |
sudo apt install whois # Debian/Ubuntu |
| 26 |
sudo yum install whois # Fedora/Centos/Rocky |
| 27 |
``` |
| 28 |
|
| 29 |
## Usage |
| 30 |
|
| 31 |
`PSLPrivateDomainsProcessor.py`: The main script containing the `PSLPrivateDomainsProcessor` class and functions for DNS and WHOIS checks. |
| 32 |
|
| 33 |
Run the script using Python: |
| 34 |
|
| 35 |
```sh |
| 36 |
cd private_domains_checker |
| 37 |
mkdir data |
| 38 |
python PSLPrivateDomainsProcessor.py |
| 39 |
``` |
| 40 |
|
| 41 |
## Main Components |
| 42 |
|
| 43 |
### Functions |
| 44 |
|
| 45 |
- `make_dns_request(domain, record_type)`: Makes DNS requests to both Google and Cloudflare DNS APIs. |
| 46 |
- `check_dns_status(domain)`: Checks the DNS status of a domain using Google and Cloudflare DNS APIs. |
| 47 |
- `get_whois_data(domain)`: Retrieves WHOIS data for a domain using the whoisdomain package. |
| 48 |
- `check_psl_txt_record(domain)`: Checks the `_psl` TXT record for a domain using Google and Cloudflare DNS APIs. |
| 49 |
|
| 50 |
### Class |
| 51 |
|
| 52 |
#### PSLPrivateDomainsProcessor |
| 53 |
|
| 54 |
- `fetch_psl_data()`: Fetches the PSL data from the specified URL. |
| 55 |
- `parse_domain(domain)`: Parses and normalizes a domain. |
| 56 |
- `parse_psl_data(psl_data)`: Parses the fetched PSL data and separates ICANN and private domains. |
| 57 |
- `process_domains(raw_domains, domains)`: Processes each domain, performing DNS, WHOIS, and `_psl` TXT record checks. |
| 58 |
- `save_results()`: Saves all processed domain data to `data/all.csv`. |
| 59 |
- `save_invalid_results()`: Saves domains with invalid DNS or expired WHOIS data to `data/nxdomain.csv` and `data/expired.csv`. |
| 60 |
- `save_hold_results()`: Saves domains with WHOIS status containing any form of "hold" to `data/hold.csv`. |
| 61 |
- `save_missing_psl_txt_results()`: Saves domains with invalid `_psl` TXT records to `data/missing_psl_txt.csv`. |
| 62 |
- `save_expiry_less_than_2yrs_results()`: Saves domains with WHOIS expiry date less than 2 years from now to `data/expiry_less_than_2yrs.csv`. |
| 63 |
- `run()`: Executes the entire processing pipeline. |
| 64 |
|
| 65 |
## Output |
| 66 |
|
| 67 |
The script generates the following CSV files in the `data` directory: |
| 68 |
|
| 69 |
- `all.csv`: Contains all processed domain data. |
| 70 |
- `nxdomain.csv`: Contains domains that could not be resolved (`NXDOMAIN`). |
| 71 |
- `expired.csv`: Contains domains with expired WHOIS records. |
| 72 |
- `hold.csv`: Contains domains with WHOIS status indicating any kind of "hold". |
| 73 |
- `missing_psl_txt.csv`: Contains domains with invalid `_psl` TXT records. |
| 74 |
- `expiry_less_than_2yrs.csv`: Contains domains with WHOIS expiry date less than 2 years from now. |
| 75 |
|
| 76 |
## Example |
| 77 |
|
| 78 |
An example CSV entry: |
| 79 |
|
| 80 |
| psl_entry | top_level_domain | dns_status | whois_status | whois_domain_expiry_date | whois_domain_status | psl_txt_status | expiry_check_status | |
| 81 |
| -------------- | ---------------- | ---------- | ------------ | ----------------------- | ---------------------------- | -------------- | ------------------- | |
| 82 |
| example.com | example.com | ok | ok | 2024-12-31 | "clientTransferProhibited" | "valid" | ok | |
| 83 |
|
| 84 |
## Publicly Registrable Namespace Determination |
| 85 |
|
| 86 |
The script determines the publicly registrable namespace from private domains by using the ICANN section. |
| 87 |
|
| 88 |
Here's how it works: |
| 89 |
|
| 90 |
1. **ICANN Domains Set**: ICANN domains are stored in a set for quick lookup. |
| 91 |
2. **Domain Parsing**: For each private domain, the script splits the domain into parts. It then checks if any suffix of these parts exists in the ICANN domains set. |
| 92 |
3. **Normalization**: The private domain is normalized to its publicly registrable form using the ICANN domains set. |
| 93 |
|
| 94 |
Examples: |
| 95 |
|
| 96 |
- **Input**: PSL private domain entry `"*.example.com"` |
| 97 |
- **Process**: |
| 98 |
- Remove leading `'*.'`: `"example.com"` |
| 99 |
- Check `"com"` against the ICANN domains set: Found |
| 100 |
- **Output**: `"example.com"` |
| 101 |
|
| 102 |
- **Input**: PSL private domain entry `"sub.example.co.uk"` |
| 103 |
- **Process**: |
| 104 |
- Check `"example.co.uk"` against the ICANN domains set: Not found |
| 105 |
- Check `"co.uk"` against the ICANN domains set: Found |
| 106 |
- **Output**: `"example.co.uk"` |
| 107 |
|
| 108 |
The output is then used for checking WHOIS data. |
| 109 |
|
| 110 |
## License |
| 111 |
|
| 112 |
This tool is licensed under the MIT License. |