Quick Start

1. Edit config.yaml
2. Run：

python fetch_openalex.py --config config.yaml --outdir data --mailto you@example.com

Run only a specific label:

python fetch_openalex.py --config config.yaml --outdir data --mailto you@example.com --label medicine_topic_x

output structure

• data/<label>/works.jsonl
• data/<label>/state.json
• data/<label>/ids.sqlite
• data/index.csv
• data/<label>/works.parquet(optional)

Field Definitions

Light version（field_set: light）always outputs the following fields:

• openalex_id
• doi
• title
• abstract(null if missing)
• publication_year
• primary_source
• topics（id + display_name）
• concepts（id + display_name）
• cited_by_count
• author_count or first_author or authorships（depends on authorships_mode）

Full version（field_set: full）additionally includes:

• publication_date
• type
• language
• open_access

Sampling Modes

• sample_mode: sequential：fetch sequentially until reaching target_n
• sample_mode: random：approximate random sampling based on reservoir sampling
  • Uses sample_seed for reproducibility
  • oversample_factor controls the scan limit (default: 3× the target)

Resume / Checkpointing

state.json records fetched count, scanned count, and RNG state. Re-running will continue from the last checkpoint and de-duplicate via ids.sqlite.

Topic / Concept Filter Probing

If you’re unsure whether to use topics.id or concepts.id, you can configure:

probe_topic_or_concept_id: Txxxx

The script will try topics.id and concepts.id; on success, it automatically appends the correct one to the original filter.

If you only know a name (e.g., “Cardiology” / “Machine Learning”), you can resolve via search:

topic_search: Cardiology

Or use concept search:

concept_search: Bioinformatics

The script will query the OpenAlex API to resolve the corresponding id and append it to the filter. If the topic search yields no match, it automatically falls back to concept search.

Example Configuration

defaults:
  target_n: 10000
  per_page: 200
  field_set: light
  authorships_mode: count
  sample_mode: sequential
  sample_seed: "42"
  oversample_factor: 3.0
  export_parquet: false
  retry_max: 5
  backoff_base: 1.0
  backoff_max: 30.0
  sleep_min: 0.2
  sleep_max: 0.5

tasks:
  - label: medicine_topic_x
    filter: from_publication_date:2010-01-01,topics.id:Txxxx
    target_n: 10000

Embeddings and Geometric Analysis (PoC)

1) Set the API Key

Prefer environment variables:

export GEMINI_API_KEY="your_key"

Or：

export GOOGLE_API_KEY="your_key"

2) Small-sample embeddings (500 per class)

python embed_gemini.py --data_dir data --out_dir emb --max_per_subtopic 500 --mode title_only

3) Full-scale embeddings (100,000 per class)

python embed_gemini.py --data_dir data --out_dir emb --max_per_subtopic 100000 --mode title_only

Parallel acceleration example (parallelize by label):

python embed_gemini.py --data_dir data --out_dir emb --max_per_subtopic 10000 --mode title_only --label_workers 3

4) Analysis and Visualization

python analyze_viz.py --emb_dir emb --out_dir reports

Only output 3D plots:

python analyze_viz.py --emb_dir emb --out_dir reports --plot_mode 3d

Interactive 3D (mouse drag to rotate; generates HTML):

python analyze_viz.py --emb_dir emb --out_dir reports --plot_mode 3d --interactive3d true

Disable cross-discipline red highlighting:

python analyze_viz.py --emb_dir emb --out_dir reports --plot_mode 3d --interactive3d true --highlight_interdisciplinary false

time / influence 的 3D HTML (draggable):

python analyze_viz.py --emb_dir emb --out_dir reports --plot_mode 3d --interactive3d true

PCA Plots Colored by Time

After running analyze_viz.py, it will additionally generate 2D/3D plots colored by time:

• within_medicine_points_pca_time.png
• within_cs_points_pca_time.png
• within_medicine_points_pca3d_1_3_time.png
• within_cs_points_pca3d_1_3_time.png
• cross_theme_points_pca_time.png
• cross_theme_points_pca3d_1_3_time.png

Colored by influence（log(cited_by_count+1)）：

• within_medicine_points_pca_influence.png
• within_cs_points_pca_influence.png
• within_medicine_points_pca3d_1_3_influence.png
• within_cs_points_pca3d_1_3_influence.png
• cross_theme_points_pca_influence.png
• cross_theme_points_pca3d_1_3_influence.png

Interdisciplinarity annotation (top 20% by concept diversity as “interdisciplinary”):

• In cross-theme time/influence plots, marked with red hollow circles

Plot Only from Existing CSV (Without Disrupting Progress)

python analyze_viz.py --plot_from_csv true --csv_dir reports --out_dir reports

Embedding vs Graph Distance

Compare correlation between embedding distance and shortest-path distance on a kNN graph:

python analyze_graph_distance.py --emb_dir emb --out_dir reports/graph --n_components 256 --k 10 --metric cosine

Output：

• reports/graph/corr_table.csv
• reports/graph/fail_pairs_top.csv
• reports/graph/scatter_medicine.png
• reports/graph/scatter_cs.png

Continuous-variable Analysis (Time / Influence / Interdisciplinarity)

analyze_viz.py reads from data/<label>/works.jsonl：

• Time：publication_year
• Influence：log(cited_by_count + 1)
• Interdisciplinarity：diversity of the concepts list (using log(#unique concepts))

And outputs:

• reports/time_direction.csv
• reports/influence_direction.csv
• reports/entropy_direction.csv
• reports/within_*_time_direction_cosine.csv
• reports/within_*_influence_direction_cosine.csv
• reports/within_*_entropy_direction_cosine.csv

5) Output Locations

• emb/<label>/X.npy
• emb/<label>/meta.jsonl
• emb/<label>/state.json
• reports/summary_k95.csv
• reports/within_*_centroids_pca.png
• reports/within_*_centroid_distance.png
• reports/cross_theme_points_pca.png
• reports/bridge_candidates.csv