This is the first exercise to learn parallel and non-parallel programming in web crawling. The implementations might not be correct, and could be a source of shame next year :D
Next year: This exercise also includes a test of Scrapy, a popular Python library for web scraping.
I scraped a website that lists the salaries of Spanish politicians. If you're interested in knowing the salaries of these individuals, this can provide a good first approximation. However, please note that this only includes a small portion of their total income and some of them. We have so much, if you need to import politicians the spaniards are a good option.
First, open a terminal and clone from github.com 'git clone https://github.com/
To use the project, follow these steps:
- Install the required Python packages:
pip install -r requirements.txt - Run the scripts:
python sueldos_asincyo.py,python sueldos_partidos.pyandpython sueldos_sincronos.py - If you want to run the Scrapy spider go inside sueldos_scrapy folder and run
scrapy crawl sueldos -o output.csvadapting the output to your prefered format.
This project is licensed under the MIT License. See the LICENSE file for details.
Contributions are welcome! If you have any ideas or improvements, feel free to open an issue or submit a pull request.
If you have any questions or need further assistance, you can reach out to us at egoitzabilleira@protonmail.com.