LinkedIn Metadata Extractor

Local, human-supervised CLI tool for extracting structured metadata from a batch of LinkedIn profile URLs.

This project is intentionally conservative:

• It runs locally on your machine.
• It tries a public-access pass before a logged-in pass.
• It does not ask for your LinkedIn password inside the app.
• It stores audit artifacts per URL so failures can be reviewed later.
• It is designed for low-volume, personal workflows rather than aggressive automation.

Current Scope

This first implementation pass includes:

• project scaffold
• Pydantic data models
• Typer CLI skeleton
• CSV input loading
• JSON and CSV export
• SQLite job ledger for resume support
• Playwright session initialization
• public-pass navigation and page classification
• HTML snapshot parsing with offline test fixtures

The authenticated flow is now split into two steps:

• capture a logged-in session once from a normal user-controlled Chrome window
• run the authenticated scraping pass in an independent Playwright browser using the saved session state

This keeps the batch run out of your normal Chrome tabs and avoids repeated Chrome crash or recovery dialogs.

Setup

1. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

2. Install dependencies

pip install -r requirements.txt
python -m playwright install chromium

If you prefer not to activate the environment, the equivalent direct commands are:

.venv/bin/pip install -r requirements.txt
.venv/bin/python -m playwright install chromium

3. Prepare an input CSV

Expected columns:

• profile_url
• candidate_id optional
• notes optional

Example:

profile_url,candidate_id,notes
https://www.linkedin.com/in/example-1,cand_001,referral batch
https://www.linkedin.com/in/example-2,cand_002,

Commands

Primary auth flow: capture from normal Chrome, scrape in background

Start a normal Chrome window with a dedicated profile directory and remote debugging enabled:

python app.py open-chrome

This launches system Chrome with a dedicated local profile and prints the attach URL, which defaults to http://127.0.0.1:9222.

In that Chrome window:

• log in to LinkedIn manually
• verify you can open your LinkedIn home feed or profile
• leave Chrome running

After you finish logging in, capture the authenticated session into a reusable storage-state file:

python app.py attach-session

By default this saves .session/storage_state.json. It reuses your existing Chrome session and does not need to open a new LinkedIn tab just to capture the session.

Fallback: initialize Playwright storage state

This opens a visible browser window using Playwright's bundled Chromium or system Chrome. Log in manually, then wait for the save step to complete.

python app.py init-session

By default the saved session is written to .session/storage_state.json.

Legacy profile bootstrap flow

This opens a Chromium-based browser with a specific user data directory so you can log in manually and later reuse that same local profile. The attach-to-running-Chrome flow above is now the preferred option.

python app.py init-session --browser-profile-dir "/path/to/local/chrome-user-data"

This is useful when you want the scraper to launch a persistent local browser profile instead of loading a Playwright storage_state.json.

Run public pass first

python app.py scrape-public input.csv --output-dir outputs/run_001

Useful safety flags:

python app.py scrape-public input.csv \
  --output-dir outputs/run_001 \
  --max-profiles 25 \
  --delay-min 3 \
  --delay-max 7 \
  --pause-every 10

Run logged-in pass second

This only targets rows already marked requires_login in the existing run state.

python app.py scrape-auth input.csv --output-dir outputs/run_001 --resume

Default background mode using the captured storage state:

python app.py scrape-auth input.csv \
  --output-dir outputs/run_001 \
  --storage-state .session/storage_state.json

This launches an independent Playwright browser and runs the authenticated batch in the background by default, so it does not open profile tabs in your normal Chrome window.

If you want to watch the authenticated pass while debugging selectors:

python app.py scrape-auth input.csv \
  --output-dir outputs/run_001 \
  --storage-state .session/storage_state.json \
  --show-browser

Legacy persistent-profile mode still exists, but it is no longer the preferred batch path:

python app.py scrape-auth input.csv \
  --output-dir outputs/run_001 \
  --browser-profile-dir "/path/to/local/chrome-user-data"

When using --browser-profile-dir, keep the profile local and user-controlled. In practice it is safest to close other Chrome windows using the same profile before running the tool.

Re-export merged results

python app.py export outputs/run_001

Validate output

python app.py validate outputs/run_001/results.json

Output Layout

Each run writes to a dedicated output directory:

outputs/run_001/
  job_state.sqlite
  results.json
  results.csv
  results_en.json
  results_en.csv
  run_summary.json
  logs/run.log
  artifacts/screenshots/
  artifacts/raw_html/
  artifacts/debug_json/

How Resume Works

The SQLite ledger stores:

• input profile URL
• phase
• access mode
• status
• failure reason
• retry count
• result payload path

Using --resume skips URLs already attempted in the same phase.

How Failures Are Recorded

For each processed URL, the tool keeps audit details including:

• requested URL
• final redirected URL
• page title
• page classification
• HTML snapshot path
• screenshot path when applicable
• debug text snippet

This makes it easier to inspect login walls, unavailable pages, verification pages, and parser failures manually.

Known Limitations

• Selector coverage is still heuristic-based and should still be validated on more saved real-world samples.
• The preferred authenticated workflow assumes you launch Chrome once for manual login capture, then reuse the saved storage state for batch scraping.
• The legacy --attach-browser batch flow is intentionally no longer supported because it interferes with normal Chrome tab usage.
• The parser is DOM-first with lightweight heuristics; it does not yet fully handle all LinkedIn layout variants.
• English export writes results_en.json and results_en.csv, but it requires access to an online translation service at export time.
• Date normalization is conservative and preserves raw strings when parsing is uncertain.
• No CAPTCHA solving, proxy rotation, fingerprint spoofing, or credential capture is implemented.

Compliance Notice

You are responsible for ensuring your use of this tool complies with applicable law, website terms, platform policies, and your organization's internal policies.