LitFlow

LitFlow is a minimal project skeleton for automated literature extraction and knowledge management.

中文使用与维护入口文档见：

• docs/LITFLOW_MANUAL.md

Project Structure

.
├── backend/
│   ├── app/
│   │   └── main.py
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── App.vue
│   │   ├── main.ts
│   │   └── style.css
│   ├── index.html
│   ├── package.json
│   ├── postcss.config.js
│   ├── tailwind.config.js
│   ├── tsconfig.json
│   └── vite.config.ts
├── storage/
├── docs/
├── CODEX.md
├── README.md
└── .gitignore

Windows One-Click Start

From the project root, double-click:

start-litflow.bat

Or run in PowerShell:

.\start-litflow.bat

The script opens separate terminal windows for the backend and frontend.

• Backend docs: http://127.0.0.1:8000/docs
• Frontend app: http://localhost:5173

To stop LitFlow, close the two terminal windows or run:

stop-litflow.bat

The stop script prints safe stop instructions and does not kill ports automatically.

Start Backend

cd backend
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
uvicorn app.main:app --reload

Backend URLs:

• http://127.0.0.1:8000/
• http://127.0.0.1:8000/health

Environment variables:

Copy .env.example to backend/.env or to a project-root .env, then fill in your own keys:

copy .env.example backend\.env

or:

copy .env.example .env

Do not commit a real .env. It is already listed in .gitignore.

Semantic Scholar search API:

curl "http://127.0.0.1:8000/search/semantic-scholar?query=flashattention&limit=5"

OpenAlex search API:

curl "http://127.0.0.1:8000/search/openalex?query=flashattention&limit=5"

arXiv search API:

curl "http://127.0.0.1:8000/search/arxiv?query=flashattention&limit=5"

All sources search API:

curl "http://127.0.0.1:8000/search/all?query=flashattention&limit=10"

Unified search results include Scoring Engine v3 fields:

• relevance_score
• authority_score
• foundation_score
• implementation_score
• survey_value_score
• frontier_score
• accessibility_score
• final_score
• venue_normalized
• venue_type
• publication_type
• publication_status
• rank_source
• rank_value
• rank_note
• venue_rank

final_score is computed as:

0.30 * relevance_score
+ 0.25 * authority_score
+ 0.15 * foundation_score
+ 0.15 * implementation_score
+ 0.10 * frontier_score
+ 0.05 * accessibility_score

foundation_score highlights highly cited, older, rank-strong, or core-method papers. implementation_score highlights systems and engineering papers with implementation keywords, systems venues, citations, and accessible PDFs. survey_value_score highlights surveys, benchmarks, evaluations, reviews, taxonomies, and dataset-style papers. The frontier_score weight is lower in v3 so low-citation new preprints do not dominate the ranking solely because they are recent.

quality_score is still returned for compatibility and is set to final_score. venue_rank is also kept for compatibility and is set to rank_value.

Conference rank uses CORE-style A*, A, B, C, Unranked, and Unknown categories. LitFlow first checks the local CSV file:

storage/rankings/core_conference_rankings.csv

The CSV columns are:

acronym,name,rank,source,year,note

The checked-in file is a small development seed, not a complete official list. You can replace it with a complete CSV prepared from the CORE / ICORE portal. When a venue is not found in the CSV, LitFlow falls back to its small built-in mapping with rank_source = LitFlow-fallback.

Journal rank uses local quartile data from:

storage/rankings/journal_rankings.csv

The journal CSV columns are:

journal_title,aliases,rank_source,quartile,year,category,note

This checked-in journal file is a small seed, not a complete official SCImago/JCR database. You can replace it with licensed SCImago or JCR export data using the same columns. The seed maps venues such as Nature, Science, Nature Machine Intelligence, Journal of Machine Learning Research / JMLR, PNAS, Cell, Patterns, and Scientific Reports to local journal quartiles such as Q1 or Q2. When a journal-like venue is not found in the local journal CSV, LitFlow marks it Unknown with rank_source = SCImago/JCR-ready.

Preprint-only papers are marked Unpublished. The system no longer uses S or Journal as rank values; conferences use CORE-style ranks and journals use Q1, Q2, Q3, Q4, or Q-Unknown.

Manual paper overrides are stored in:

storage/rankings/manual_paper_overrides.csv

This CSV is for high-impact papers whose metadata or venue is unstable across search sources, or whose influence is not captured by ordinary CORE/JCR rules. It is a curated personal-library override, not an official ranking source. Use it sparingly and keep notes clear, especially for workshop, technical report, or classic production-ML papers. Overrides are matched by normalized title pattern before normal venue classification. rank_value = Influential is not treated as CORE A*; it only adds a modest authority/foundation signal while final score still depends on relevance and the other scoring components.

Search all sources and save results:

curl -X POST "http://127.0.0.1:8000/search/all/save" ^
  -H "Content-Type: application/json" ^
  -d "{\"query\":\"flashattention\",\"limit\":10}"

Search Campaign / curated query expansion:

Search Campaign runs a predefined set of high-signal seed queries for a research direction, merges duplicate results, and sorts candidates by LitFlow scoring. It does not replace normal single-query search. It is not full fuzzy search or an embedding retriever, but it is better suited for building a research library from curated seed papers and related query variants.

Available endpoints:

curl "http://127.0.0.1:8000/search/campaigns"
curl -X POST "http://127.0.0.1:8000/search/campaigns/run" ^
  -H "Content-Type: application/json" ^
  -d "{\"campaign_name\":\"AI Systems / Inference Systems\",\"limit_per_query\":8,\"save\":false}"

Built-in campaigns:

• AI Systems / Inference Systems
• Trustworthy AI Systems
• Mechanistic Interpretability
• Scientific ML / AI for Science
• Embodied AI / World Models

Campaign runs return candidates only by default. The Dashboard Search Campaign panel lets you review results and use Save Selected; it does not automatically save all results.

Legal open-access PDF resolver and downloader:

The system only resolves and downloads legal open-access PDF URLs from existing paper PDF URLs, arXiv PDF links, or Unpaywall. It does not bypass paywalls, logins, or CAPTCHA.

curl -X POST "http://127.0.0.1:8000/papers/1/resolve-pdf"
curl -X POST "http://127.0.0.1:8000/papers/1/download-pdf"

Manual PDF URL:

If automatic PDF resolution fails but you know a legal open-access PDF URL, set the paper metadata pdf_url and then run download-pdf, parse-pdf, or process as usual. Use only legal open-access sources such as arXiv, OpenReview, official proceedings, an author homepage, or an institutional repository. LitFlow does not bypass paywalls, logins, or CAPTCHA.

The Dashboard paper detail panel includes a Manual PDF URL section near PDF Preview. The same metadata update is available through the existing PATCH /papers/{paper_id} endpoint:

curl -X PATCH "http://127.0.0.1:8000/papers/1" ^
  -H "Content-Type: application/json" ^
  -d "{\"pdf_url\":\"https://arxiv.org/pdf/1706.03762\"}"

To clear a saved PDF URL:

curl -X PATCH "http://127.0.0.1:8000/papers/1" ^
  -H "Content-Type: application/json" ^
  -d "{\"pdf_url\":null}"

Downloaded PDFs are saved with readable filenames when enough metadata is available:

{year}-{short-title}-{venue}-{rank}-id{paper_id}.pdf

Older files such as storage/pdfs/1.pdf remain valid and are not migrated.

Enriched paper display API:

These endpoints read saved database papers and compute display-only score and venue rank fields at request time. They do not modify the database.

curl "http://127.0.0.1:8000/papers/enriched"
curl "http://127.0.0.1:8000/papers/1/enriched"

The enriched responses include final_score, foundation_score, implementation_score, survey_value_score, venue_normalized, venue_type, publication_type, publication_status, rank_source, rank_value, rank_note, and compatibility fields such as venue_rank. These fields are computed at read time for Dashboard display.

Refresh ranking and scoring display:

When local venue aliases, journal rankings, or scoring rules change, saved papers can be refreshed without changing the database schema. LitFlow computes the latest enrichment at read time, so these endpoints return the current venue/rank/score view and the Dashboard reloads /papers/enriched.

curl -X POST "http://127.0.0.1:8000/papers/1/refresh-enrichment"
curl -X POST "http://127.0.0.1:8000/papers/refresh-enrichment-batch" ^
  -H "Content-Type: application/json" ^
  -d "{\"paper_ids\":null}"

In the Dashboard, use Refresh Rank & Score in the paper detail panel for one paper, or Refresh All Rankings in the top tool bar after updating ranking CSVs or scoring code.

LitFlow also saves paper authors from search results. Authors are stored in separate author/link records and deduplicated with a simple normalized-name rule: lowercase, remove punctuation, and collapse whitespace. LitFlow does not perform institution recognition or complex author disambiguation.

Single-paper export:

curl "http://127.0.0.1:8000/papers/1/export/markdown"
curl "http://127.0.0.1:8000/papers/1/export/bibtex"

Markdown export includes frontmatter metadata, authors, venue rank fields, abstract, and the latest LLM extraction when available. If no extraction exists yet, LitFlow still exports a basic literature-note template with "No extraction available yet." in the structured summary sections. BibTeX export includes an author field and produces a simple @inproceedings, @article, or @misc entry based on the detected venue type. The Dashboard also provides Export Markdown and Export BibTeX buttons in the paper detail panel.

In the paper detail panel, Preview Markdown fetches the Markdown export and shows the Markdown text inside the page without downloading a file. Export Markdown downloads the .md file and uses the filename from the backend Content-Disposition header.

Research Topics / Tags:

LitFlow supports lightweight research topics for organizing papers by research direction. A paper can belong to multiple topics. Topics are manual labels only; LitFlow does not run automatic classification, embeddings, or a knowledge graph for this feature.

Default topics can be created with:

curl -X POST "http://127.0.0.1:8000/topics/seed-defaults"

The default set is organized into main research directions and functional tags.

Main directions:

• AI Systems / Inference Systems
• Trustworthy AI Systems
• Scientific ML / AI for Science
• Embodied AI / World Models
• Mechanistic Interpretability
• ML Theory / Generalization
• Causal ML / Decision Systems

Functional tags:

• Systems Optimization
• MLOps / Production ML
• Evaluation / Red Teaming
• Alignment / Safety
• Calibration / Uncertainty
• OOD / Robustness
• Interpretability / Circuits
• World Models / Agents
• Physics-informed ML
• Foundation Models
• Survey / Benchmark
• General Foundation

List topics and assign topics to a paper:

curl "http://127.0.0.1:8000/topics"
curl -X PUT "http://127.0.0.1:8000/papers/1/topics" ^
  -H "Content-Type: application/json" ^
  -d "{\"topic_names\":[\"AI Systems / Inference Systems\",\"Systems Optimization\"]}"
curl "http://127.0.0.1:8000/papers/1/topics"

Assign topics to multiple selected papers:

curl -X POST "http://127.0.0.1:8000/topics/apply-to-papers" ^
  -H "Content-Type: application/json" ^
  -d "{\"paper_ids\":[1,2,3],\"topic_names\":[\"AI Systems / Inference Systems\",\"Systems Optimization\"],\"mode\":\"add\"}"

Use mode: "add" to keep existing paper topics and add the requested topics. Use mode: "replace" to replace each paper's topics with topic_names.

The Dashboard loads topics at startup. Use the sidebar Topic filter to show papers assigned to a selected topic. Use the paper detail panel Topics section to set one paper with topic chips, an available-topic checkbox list, and custom comma-separated topics. In Batch Mode, use Batch Topic Assignment to add or replace topics for all currently selected papers.

Markdown export includes topics in YAML frontmatter and in the Metadata section. This keeps saved notes ready for future cross-paper RAG filtering by topic.

There are two export modes:

• Export Markdown / Export BibTeX: downloads files through the browser to the browser's default download directory.
• Save to Library: writes a local literature workspace inside the project under storage/library/{paper_id}-{year}-{short-title}/.

The local workspace contains:

storage/library/{paper_id}-{year}-{short-title}/
├── paper.pdf
├── note.md
├── citation.bib
├── metadata.json
└── assets/

storage/assets/{paper_id}/ is LitFlow's intermediate asset directory for extracted page images, captions, and tables. Save to Library keeps that directory intact and copies its contents into the literature package at storage/library/{paper_id}-{year}-{short-title}/assets/.

The generated note.md uses package-internal relative image paths such as assets/page-001.png, so the whole literature package can be moved together without breaking embedded images.

Workspace files are generated artifacts and are ignored by git.

Workspace API:

curl -X POST "http://127.0.0.1:8000/papers/1/save-workspace"
curl "http://127.0.0.1:8000/papers/1/workspace"

Exported filenames use readable metadata:

{year}-{short-title}-{venue}-{rank}-id{paper_id}.md
{year}-{short-title}-{venue}-{rank}-id{paper_id}.bib

Example:

2023-flashattention-2-iclr-a-star-id1.md

PDF text parsing and RAG-ready chunks:

Run download-pdf first so Paper.local_pdf_path points to a local PDF file, then parse the PDF and inspect the saved text/chunks.

curl -X POST "http://127.0.0.1:8000/papers/1/download-pdf"
curl -X POST "http://127.0.0.1:8000/papers/1/parse-pdf"
curl "http://127.0.0.1:8000/papers/1/pdf-text"
curl "http://127.0.0.1:8000/papers/1/chunks"

PDF figures, tables, and unstructured assets MVP:

Run download-pdf first so Paper.local_pdf_path points to a local PDF file, then extract page images and simple captions.

curl -X POST "http://127.0.0.1:8000/papers/1/extract-assets"
curl "http://127.0.0.1:8000/papers/1/assets"

Extracted assets are saved under:

storage/assets/{paper_id}/

Page images are served from:

http://127.0.0.1:8000/static/assets/{paper_id}/page-001.png

This is an MVP. It supports page image rendering and simple Figure, Fig., and Table caption detection. It does not support OCR, precise figure/table cropping, or multimodal chart understanding. Markdown export includes a Figures and Tables section with extracted page images, figure captions, table captions, and raw table text when available.

LLM structured extraction:

Run download-pdf and parse-pdf first so the paper has saved chunks. Use mock mode to test the database flow without calling an external API.

curl -X POST "http://127.0.0.1:8000/papers/1/extract" ^
  -H "Content-Type: application/json" ^
  -d "{\"mode\":\"mock\",\"user_topic\":\"LLM inference systems\",\"max_chunks\":8}"

curl "http://127.0.0.1:8000/papers/1/extractions"
curl "http://127.0.0.1:8000/papers/1/latest-extraction"

For OpenAI extraction, set OPENAI_API_KEY in .env. You can optionally set LLM_MODEL; it defaults to gpt-4.1-mini.

OPENAI_API_KEY=your_openai_api_key
LLM_MODEL=gpt-4.1-mini

curl -X POST "http://127.0.0.1:8000/papers/1/extract" ^
  -H "Content-Type: application/json" ^
  -d "{\"mode\":\"openai\",\"user_topic\":\"LLM inference systems\",\"max_chunks\":8}"

Single-paper RAG question answering:

Run parse-pdf first so the paper has saved PaperChunk records. The RAG endpoint uses simple keyword retrieval only; it does not use embeddings or a vector database.

Mock mode does not call any external API:

curl -X POST "http://127.0.0.1:8000/papers/1/ask" ^
  -H "Content-Type: application/json" ^
  -d "{\"question\":\"What is the main systems optimization in this paper?\",\"mode\":\"mock\",\"top_k\":5}"

OpenAI mode uses OPENAI_API_KEY and optional LLM_MODEL from .env:

curl -X POST "http://127.0.0.1:8000/papers/1/ask" ^
  -H "Content-Type: application/json" ^
  -d "{\"question\":\"What is the main systems optimization in this paper?\",\"mode\":\"openai\",\"top_k\":5}"

Cross-paper library RAG:

LitFlow also supports asking questions across the whole saved paper library:

• Endpoint: POST /ask/library
• Retrieval: BM25-like scoring over saved PaperChunk records
• Topic filter: optional, using Research Topics / Tags
• Modes: mock for local workflow testing, openai for real answer synthesis

This is not embedding RAG and does not use a vector database. It only searches existing parsed chunks in the database.

Mock mode:

curl -X POST "http://127.0.0.1:8000/ask/library" ^
  -H "Content-Type: application/json" ^
  -d "{\"question\":\"What are the main bottlenecks in LLM inference systems?\",\"mode\":\"mock\",\"top_k\":10,\"topic\":\"AI Systems / Inference Systems\"}"

OpenAI mode:

curl -X POST "http://127.0.0.1:8000/ask/library" ^
  -H "Content-Type: application/json" ^
  -d "{\"question\":\"What are the main bottlenecks in LLM inference systems?\",\"mode\":\"openai\",\"top_k\":10,\"topic\":\"AI Systems / Inference Systems\"}"

Omit topic or set it to null to search all papers:

curl -X POST "http://127.0.0.1:8000/ask/library" ^
  -H "Content-Type: application/json" ^
  -d "{\"question\":\"What limitations are common across these papers?\",\"mode\":\"mock\",\"top_k\":10,\"topic\":null}"

The Dashboard includes an Ask Library panel below search. This is cross-paper RAG. The right-side Ask Paper panel remains the single-paper RAG interface.

One-click paper processing workflow:

The workflow endpoint can resolve, download, parse, and extract a paper in one request. Mock extraction is useful when OpenAI quota is unavailable.

Endpoint:

• POST /papers/{paper_id}/process

curl -X POST "http://127.0.0.1:8000/papers/1/process" ^
  -H "Content-Type: application/json" ^
  -d "{\"resolve_pdf\":true,\"download_pdf\":true,\"parse_pdf\":true,\"extract\":true,\"extract_mode\":\"mock\",\"user_topic\":\"LLM inference systems\",\"max_chunks\":8}"

If you want to use OpenAI extraction, set extract_mode to openai. If OpenAI quota or API access is unavailable, keep extract_mode as mock.

Batch processing selected papers:

The Dashboard paper table supports selecting multiple saved papers with the row checkboxes. Use the header checkbox to select all currently visible rows. The Batch Process panel above the table shows the selected count and runs the same workflow for the selected papers.

Endpoint:

• POST /papers/process-batch

curl -X POST "http://127.0.0.1:8000/papers/process-batch" ^
  -H "Content-Type: application/json" ^
  -d "{\"paper_ids\":[1,2],\"resolve_pdf\":true,\"download_pdf\":true,\"parse_pdf\":true,\"extract\":true,\"extract_assets\":false,\"save_workspace\":false,\"extract_mode\":\"mock\",\"max_chunks\":8}"

By default, batch processing resolves PDFs, downloads PDFs, parses PDFs, and runs extraction. It does not extract assets or save local workspaces unless you enable those options. openai mode may consume API quota. A single batch is limited to 20 papers to avoid accidental large runs. Each response includes a per-paper list of workflow steps and their success, skipped, or failed status.

Open API docs:

• http://127.0.0.1:8000/docs

Maintenance

Reset local test library data:

Use this only when you want to clear development or test papers before starting a fresh literature collection. The script deletes LitFlow runtime library data from the SQLite database and clears generated local files under storage/pdfs/, storage/assets/, and storage/library/.

It does not delete source code, .env, README files, or storage/rankings/ CSV files. Research topics are preserved by default; paper-topic links are cleared.

Preview what would be removed:

python scripts/reset_litflow_library.py --dry-run

Run the reset:

python scripts/reset_litflow_library.py

The script requires typing RESET before it deletes anything.

Start Frontend

cd frontend
npm install
npm run dev

Frontend URL:

• http://127.0.0.1:5173/

Paper Dashboard

Start the backend first:

cd backend
.venv\Scripts\activate
uvicorn app.main:app --reload

Then start the frontend:

cd frontend
npm install
npm run dev

Open:

• http://localhost:5173

You can search papers directly from the Dashboard with the top Search panel. For example:

• query: flashattention
• limit: 10

The frontend first calls /search/all and shows results without saving them. Select the papers you want, then click Save Selected to call /search/save-selected. Save All Results is still available as a secondary action and calls /search/all/save. Successful saves refresh the Dashboard automatically.

You can also call the backend endpoint manually:

curl -X POST "http://127.0.0.1:8000/search/all/save" ^
  -H "Content-Type: application/json" ^
  -d "{\"query\":\"flashattention\",\"limit\":10}"

The dashboard supports Search Only, Select Results, Save Selected, Save All Results, paper filtering, PDF resolve/download/parse, mock extraction, one-click Process Paper, loading the latest extraction, and mock RAG ask.

The frontend also includes a Literature Quality Board for reading triage. Each paper is assigned a client-side reading priority: Must Read, High Priority, Frontier Watch, Skim, or Archive. These labels combine rank, relevance, authority, frontier signal, publication status, year, and final score. They are an assistant for organizing reading work, not an absolute academic evaluation.

The paper detail Actions area includes an Extraction Mode selector with openai and mock. Run Extraction and Process Paper use the currently selected mode.

The paper detail panel also provides a structured reading workspace:

• latest extraction displayed as structured summary sections
• PDF preview for downloaded files under /static/pdfs/{file}
• Markdown preview without downloading
• Markdown and BibTeX export buttons