# Leadership Message Board Crawler

This project is a Python-based web crawler designed to scrape data from the leadership message board on the People.cn website. The crawler can extract thread lists, detailed contents, and answers, and it provides functionalities for saving and processing the collected data.

## Files Description

- `main.py`: The main entry point of the application. It provides a command-line interface for different operations such as scraping data, processing the spider list, generating statistics, and outputting data.
- `spider.py`: Contains the core functionality for fetching thread lists and details from the website.
- `spider_list.py`: Handles the processing of the spider list.
- `statistics.py`: Generates statistics based on the collected data.

## How to Use

### Prerequisites

- Python 3.x
- Required Python packages: `requests`, `BeautifulSoup4`

You can install the necessary packages using pip:
```sh
pip install requests beautifulsoup4
```

### Running the Application

1. Ensure all files (`main.py`, `spider.py`, `spider_list.py`, `statistics.py`) are in the same directory.

2. Open a terminal or command prompt and navigate to the directory containing these files.

3. Run the main script:
```sh
python main.py
```

4. Follow the on-screen prompts to select an operation:
   - `(t) 爬取领导留言板 (Scrape leadership message board)`
   - `(b) 处理爬虫列表 (Process spider list)`
   - `(s) 统计 (Statistics)`
   - `(o) 输出 (Output)`
   - `(e) 退出 (Exit)`

5. If you choose to scrape the leadership message board (option `t`), you will be prompted to enter the leadership message board ID (`fid`). For example, enter `540` and press Enter to start scraping the data for that specific ID.

## Example

Below is an example of how to run the application and scrape data for a specific ID:

```sh
$ python main.py
操作：(t) 爬取领导留言板 (b) 处理爬虫列表 (s) 统计 (o) 输出 (e) 退出
t
请输入领导留言板 ID (fid):
540
```

## Notes

- Ensure you have a stable internet connection while running the crawler.
- The data will be saved in a directory named `data` followed by the leadership ID.

## License

This project is licensed under the MIT License.