Getting Started

Getting Started

A 10-minute path from "I cloned the repo" to "I have a working library and can add papers."

Prerequisites

What	Why
Python 3.11+	Server, ingest pipeline, CLI
Node.js	Only for the arXiv fetch script (tools/fetch_arxiv.js) — Python doesn't replace it because the script runs on your host machine, while ingest runs in a sandbox in some workflows.
A modern browser	Chrome, Firefox, Safari, or Edge — anything with ES modules + WebAssembly.

Install

git clone https://github.com/pquarterman17/ScientificLitterScoop.git
cd ScientificLitterScoop
pip install -e .
scq init             # creates data/scientific_litter_scoop.db

pip install -e . puts the scq console script on your PATH and makes python -m scq work.

Launch the server

# Windows: double-click START.bat (or from a terminal:)
python -m scq serve

# macOS / Linux: double-click START.command, or:
python -m scq serve

Then open http://localhost:8080/paper_database.html. The scraper page is at /paper_scraper.html.

SCQ_NO_BROWSER=1 python -m scq serve skips opening tabs (useful for headless dev or scripting). The server recognises false/0/no/off as "still open" — don't be fooled by your shell's truthiness expectations.

Add a paper

Two-step pipeline:

# Step 1 — fetch metadata + PDF from arXiv (uses Node.js)
bash tools/fetch.sh 2401.12345          # macOS / Linux
tools\fetch.bat 2401.12345              # Windows

# Step 2 — ingest into the database (uses Python)
scq process 2401.12345 --note "optional inline note"

What step 2 does:

1. Reads inbox/<arxiv_id>_meta.json written by step 1
2. Extracts figures + captions from the PDF (scq.ingest.extract)
3. Generates BibTeX + Physical-Review plain-text citations (scq.ingest.process)
4. Auto-tags from arXiv categories + keyword matching (the auto-tag-rules config domain)
5. Inserts into SQLite + appends to references.bib / references.txt

Reload the database page to see the paper.

For batch imports (a folder of PDFs, a Mendeley .bib), see scq inbox --help and scq mendeley --help.

Configure your library

Three layers of configuration, none of which you're forced to touch:

Layer	File	When you'd edit
Bootstrap paths	data/user_config/paths.toml	Your DB / papers / inbox dirs are somewhere unusual (OneDrive, NAS, project-specific).
Domain config	data/user_config/<domain>.json	You want digests on Wednesdays, or PRB enabled by default, or APA citations.
Secrets	OS keyring	SMTP password for the daily digest emails: scq config set-secret email_app_password

Inspect what's actually in effect:

scq config show              # all 9 domains as JSON
scq config show digest       # just one
scq config get digest maxPapers
scq config paths             # resolved filesystem paths
scq config validate          # schema-check every domain (exit 1 on errors)

Each domain ships a <domain>.json.example in data/user_config/. Copy, edit, drop the .example suffix.

Daily arXiv digest (optional)

A GitHub Actions workflow (.github/workflows/arxiv_digest.yml) emails a daily summary of new papers in your arxivCategories. To turn it on:

1. Add recipients to data/user_config/digest.json:

   { "recipients": [{"email": "you@example.com", "frequency": "daily"}] }

2. Set the SMTP From address in data/user_config/email.json and the App Password as a GitHub Secret named SCQ_EMAIL_APP_PASSWORD.
3. (Optional) Adjust the cron line:

   scq schedule update --cadence weekly --day mon --time 07:00

4. Commit + push.

Test the pipeline locally without sending mail: scq digest --test --no-email.

Move your library to a new machine

# On the old machine
scq config export ~/scq-setup.zip --include-paths

# Transfer the zip + your data/scientific_litter_scoop.db however you like

# On the new machine
git clone <repo>; cd ScientificLitterScoop; pip install -e .
scq config import ~/scq-setup.zip
# Then drop the .db file into data/scientific_litter_scoop.db
# Re-set secrets:
scq config set-secret email_app_password

Secrets are intentionally never bundled — they're in the OS keyring and have to be re-entered on the destination.

Where to go next

• Anything failing? Check scq config validate first — most "why isn't this working?" sessions end in a typo'd JSON file.
• Curious how it works? Architecture for the layered structure, or docs/architecture.md in-tree.
• Want to add features? Adding Features has recipes.