DergiPark Article Scraper and Sentence Aligner

This repository contains tools for scraping academic articles from DergiPark and creating aligned sentence pairs from bilingual Turkish-English content.

Requirements

Install the required dependencies:

pip install requests beautifulsoup4 numpy scikit-learn transformers nltk torch

Usage

The toolkit consists of two main components:

1. scraper.py - Scrapes article data from DergiPark
2. sentence_aligner.py - Creates aligned sentence pairs from the scraped articles

1. Scraping Articles from DergiPark

The scraper.py script extracts article data from DergiPark in both Turkish and English, including titles, abstracts, authors, DOI, and references.

Basic Usage:

python scraper.py https://dergipark.org.tr/tr/pub/jesd/archive

Options:

python scraper.py [OPTIONS] [ARCHIVE_URL]

• -o, --output FILENAME - Output JSON file (default: dergipark_articles.json)
• -s, --single URL - Scrape a single article URL instead of the whole archive
• --max-issues N - Limit the number of issues to scrape (for testing)
• --max-articles N - Limit the number of articles per issue (for testing)

Examples:

# Scrape a single article
python scraper.py -s https://dergipark.org.tr/tr/pub/jesd/issue/90843/1511442

# Scrape with limited scope (for testing)
python scraper.py --max-issues 2 --max-articles 3 https://dergipark.org.tr/tr/pub/jesd/archive

# Specify a custom output file
python scraper.py -o my_articles.json https://dergipark.org.tr/tr/pub/jesd/archive

2. Creating Aligned Sentence Pairs

The sentence_aligner.py script processes the scraped articles and creates aligned sentence pairs using the LaBSE model.

Basic Usage:

python sentence_aligner.py --input dergipark_articles.json --output parallel_corpus.json

Options:

python sentence_aligner.py [OPTIONS]

• -i, --input FILENAME - Input JSON file with articles (default: dergipark_articles.json)
• -o, --output FILENAME - Output JSON file for aligned sentences (default: parallel_corpus.json)
• -t, --threshold FLOAT - Minimum similarity score (0-1) to consider a match (default: 0.7)
• -m, --model MODEL_NAME - Pre-trained model to use (default: sentence-transformers/LaBSE)
• -l, --limit N - Limit the number of articles to process

Examples:

# Process only the first 5 articles with a higher similarity threshold
python sentence_aligner.py -i dergipark_articles.json -o parallel_corpus.json -l 5 -t 0.8

# Use a different sentence embedding model
python sentence_aligner.py --model "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"

Output Format

Scraper Output (JSON)

The scraper produces a JSON file with article data in the following format:

[
  {
    "id": "1511442",
    "title_tr": "BİLİNMEYEN MARKOV ATLAMALI SİSTEMLERİN MODELLEMESİ...",
    "title_en": "MODELLING OF UNKNOWN MARKOV JUMP SYSTEMS...",
    "abstract_tr": "Markov atlama sistemlerinin...",
    "abstract_en": "Markov jump systems are difficult...",
    "keywords_tr": ["Markov Atlamalı Sistemler", "..."],
    "keywords_en": ["Markov Jump Systems", "..."],
    "doi": "https://doi.org/10.21923/jesd.1511442",
    "authors": ["Author1", "Author2"],
    "url": "https://dergipark.org.tr/tr/pub/jesd/issue/90843/1511442",
    "references": ["Reference1", "Reference2", "..."]
  }
]

Sentence Aligner Output (JSON)

The sentence aligner produces a JSON file with aligned sentence pairs:

[
  {
    "tr": "Turkish sentence text",
    "en": "English sentence text",
    "score": 0.82,
    "article_id": "1511442",
    "source": "abstract"
  }
]

Notes

• The NLTK tokenizer requires language resources. The script will download them automatically on the first run.
• For better results in sentence alignment, use a threshold of 0.7-0.8.
• Processing large archives may take a long time due to semantic similarity calculations.