# Expired Domain Scraper

A Node.js script that scrapes the list of expired domains from expireddomains.net at configurable intervals.

## Features

- ✅ Scrapes expired domains every N minutes (configurable)
- ✅ Extracts domain name and additional metadata (backlinks, domain pop, birth date, etc.)
- ✅ Saves results to JSON file
- ✅ Easy configuration via external config file
- ✅ Runs continuously with configurable intervals
- ✅ Graceful shutdown handling

## Installation

1. Install dependencies:
```bash
npm install
```

## Configuration

Edit the `config.json` file to customize the scraper behavior:

```json
{
  "intervalMinutes": 1,        // How often to scrape (in minutes)
  "saveToFile": true,           // Whether to save results to a file
  "outputFile": "domains.json", // Output filename
  "onScrapeComplete": null,     // Optional callback function name
  "debug": false,               // Enable debug logging
  "saveRawHtml": false          // Save raw HTML response for debugging
}
```

### Configuration Options

- **intervalMinutes**: Number of minutes between each scrape (default: 1)
- **saveToFile**: Set to `true` to save results to a file, `false` to only log to console
- **outputFile**: Name of the output JSON file (default: "domains.json")
- **onScrapeComplete**: Name of a custom callback function (advanced usage)
- **debug**: Set to `true` to enable verbose logging for troubleshooting
- **saveRawHtml**: Set to `true` to save the raw HTML response to a file for inspection

## Usage

### Start the scraper:
```bash
npm start
```

Or directly with Node:
```bash
node domain-scraper.js
```

### Stop the scraper:
Press `Ctrl+C` to gracefully shutdown the scraper.

## Output Format

The script saves data in the following JSON format:

```json
{
  "scrapedAt": "2025-11-01T12:00:00.000Z",
  "totalDomains": 50,
  "domains": [
    {
      "domain": "example.com",
      "bl": "150",
      "domainPop": "200",
      "birth": "2010-01-01",
      "entries": "5",
      "scrapedAt": "2025-11-01T12:00:00.000Z"
    }
    // ... more domains
  ]
}
```

## Example Configurations

### Scrape every 5 minutes:
```json
{
  "intervalMinutes": 5,
  "saveToFile": true,
  "outputFile": "domains.json"
}
```

### Scrape every 30 seconds (0.5 minutes):
```json
{
  "intervalMinutes": 0.5,
  "saveToFile": true,
  "outputFile": "domains.json"
}
```

### Scrape every hour, don't save to file:
```json
{
  "intervalMinutes": 60,
  "saveToFile": false,
  "outputFile": "domains.json"
}
```

## Notes

- The script includes a User-Agent header to mimic a real browser
- First scrape happens immediately when you start the script
- Subsequent scrapes happen at the configured interval
- The website structure may change over time, requiring updates to the scraper
- Be respectful of the target website's resources and don't set intervals too low
- Consider the website's robots.txt and terms of service

## Troubleshooting

### Enable Debug Mode
If the scraper isn't finding domains, enable debug mode in `config.json`:
```json
{
  "debug": true,
  "saveRawHtml": true
}
```

This will:
- Show detailed logging of what links are being found
- Save the raw HTML response to `raw-response.html` for inspection
- Display the first few domains being processed

### "Error loading config.json"
- Make sure `config.json` exists in the same directory as the script
- Check that the JSON syntax is valid

### No domains found
- The website structure may have changed
- Check your internet connection
- The website may be blocking automated requests

### Network errors
- Check your internet connection
- The website may be temporarily down
- Try increasing the timeout in the script

## License

MIT