Dark Souls III Weapon Scraper

A web scraping tool that extracts weapon statistics from the Dark Souls III Fandom wiki. The scraper automatically opens Chromium, navigates to the weapon list page, collects all weapon URLs, and then parses each weapon's scaling grades and stat requirements.

Overview

This tool automates the process of collecting weapon data from the Dark Souls III wiki. It:

1. Launches Chromium browser (for session management)
2. Parses the weapon list page to find all individual weapon pages
3. Extracts scaling grades (Strength, Dexterity, Intelligence, Faith) from each weapon page
4. Extracts stat requirements (Strength, Dexterity, Intelligence, Faith) from each weapon page
5. Outputs all collected data to a CSV file for further analysis

Requirements

• C++26
• libixwebsocket (WebSocket client library)
• OpenSSL (SSL/TLS support)
• libcurl (HTTP client)
• zlib (compression)
• Chromium browser (must be installed and accessible via chromium command)

Install dependencies on Debian/Ubuntu:

sudo apt install g++ libixwebsocket-dev libssl-dev libcurl4-openssl-dev zlib1g-dev chromium

Build

Run the build script:

./build.sh

The script compiles the program with all required dependencies:

g++ main.cpp base64.cpp -std=c++26 -lixwebsocket -lssl -lcrypto -lz -lcurl

Usage

Simply run the compiled executable:

./a.out

The program will:

1. Open Chromium browser automatically
2. Wait for the browser to initialize
3. Begin scraping the Dark Souls III weapon data
4. Display extracted data in the console
5. Save all data to out.csv

Output

The scraper produces out.csv with the following columns:

Column	Description
Strength (req)	Required Strength stat
Dexterity (req)	Required Dexterity stat
Intelligence (req)	Required Intelligence stat
Faith (req)	Required Faith stat
Strength (scale)	Strength scaling grade (A-E)
Dexterity (scale)	Dexterity scaling grade (A-E)
Intelligence (scale)	Intelligence scaling grade (A-E)
Faith (scale)	Faith scaling grade (A-E)
URL	Weapon page URL

Example Output

Site to parse: 50
A E - -    16 18 0 0    https://darksouls.fandom.com/ru/wiki/Longsword
B C - -    10 10 0 0    https://darksouls.fandom.com/ru/wiki/Shortsword
...

Architecture

Parser Class

The Parser class orchestrates the entire scraping process:

1. Domain extraction: Extracts the base domain from URLs
2. URL resolution: Converts relative URLs to absolute URLs
3. Weapon list parsing: Finds all weapon page links from the main weapons page
4. Weapon data parsing: For each weapon page, extracts scaling grades and stat requirements
5. CSV export: Writes all collected data to out.csv

Scrambler Class

Handles browser automation and HTML retrieval through Chromium. The Scrambler class provides:

• Browser initialization and control
• HTML page retrieval from URLs
• Session management

Logger Class

Provides logging functionality with tag-based filtering for debugging.

CSV Output Format

The generated CSV file has the following structure:

Requirements,,,,Scale,,,,URL
Strength,Dexterity,Intelligence,Faith,Strength,Dexterity,Intelligence,Faith,http
16,18,0,0,A,E,-,-,https://darksouls.fandom.com/ru/wiki/Longsword
10,10,0,0,B,C,-,-,https://darksouls.fandom.com/ru/wiki/Shortsword
...

Notes

• The scraper is specifically designed for the Russian Dark Souls III Fandom wiki structure
• Russian scaling characters (А, В, С, Е) are automatically converted to English equivalents (A, B, C, E)
• The program requires an active internet connection
• Chromium must be installed and accessible in the system PATH
• The scraper includes a 1-second delay after launching Chromium to ensure the browser is ready
• The tool uses WebSocket communication for browser control via ixwebsocket
• SSL/TLS is handled through OpenSSL
• HTTP requests are processed via libcurl

Dependencies Overview

• ixwebsocket: WebSocket client for real-time browser communication
• OpenSSL: SSL/TLS encryption for secure connections
• libcurl: HTTP client for fetching web pages
• zlib: Data compression support

License

MIT