Skip to content

feat: NER, NEL, change-stream worker, and classify API #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Hamz-Horizon wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: NER, NEL, change-stream worker, and classify API #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
__pycache__
*.pyc
.git
.env
*.egg-info
.venv
venv
benchmark_notebooks
docs
test_*.py
*.ipynb
*.json
!app/**
!inference/**
!data/**
!requirements.txt
18 changes: 18 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# ---------------------------------------------------------------
# Classifier Service — Environment Variables
# ---------------------------------------------------------------
#
# 1. Copy this file: cp .env.example .env
# 2. Fill in the values below.
# 3. NEVER commit the .env file — it contains secrets.
# ---------------------------------------------------------------

# REQUIRED — HuggingFace access token for the NER model.
# Request access to tabiya/roberta-base-job-ner, then create a
# read token at https://huggingface.co/settings/tokens
HF_TOKEN=

# OPTIONAL — MongoDB connection (only needed for the change-stream-worker).
# Leave blank if you only need the classify API.
# APPLICATION_MONGODB_URI=mongodb+srv://<user>:<password>@<cluster>.mongodb.net/?retryWrites=true&w=majority
# APPLICATION_DATABASE_NAME=horizon-scraper-dev
395 changes: 0 additions & 395 deletions DATA_LICENSE
View file
Open in desktop

This file was deleted.

13 changes: 13 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && \
python -c "import nltk; nltk.download('punkt', quiet=True); nltk.download('punkt_tab', quiet=True)"

COPY . .

EXPOSE 5001 5002 5003

CMD ["uvicorn", "app.server.classify_server:app", "--host", "0.0.0.0", "--port", "5001"]
314 changes: 147 additions & 167 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,196 +1,176 @@
# Tabiya Livelihoods Classifier

The Tabiya Livelihoods Classifier provides an easy-to-use implementation of the entity-linking paradigm to support job description heuristics. Using state-of-the-art transformer neural networks, this tool can extract five entity types: Occupation, Skill, Qualification, Experience, and Domain. For the Occupations and Skills, ESCO-related entries are retrieved. The procedure consists of two discrete steps: entity extraction and similarity vector search.
Extracts **occupations**, **skills**, **qualifications**, **experience**, and **domain** entities from job text and links them to the [ESCO taxonomy](https://esco.ec.europa.eu/).

## Table of Contents
---

- **[Installation](#installation)**
- **[Use the model](inference/README.md)**: Instructions on how to use the inference pipeline.
- **[Job Analysis Application](app/README.md)**: A web application for analyzing job descriptions and extracting and linking relevant entities.
- **[Training](train/README.md)**: Details on how to train the model.
- **[Model's Architecture](#models-architecture)**
- **[Datasets](#datasets)**
- **[License](#license)**
- **[Bibliography](#bibliography)**
## Prerequisites

## Installation
- [Docker Desktop](https://www.docker.com/products/docker-desktop/)
- [Git](https://git-scm.com/) (v2.37+) with [Git LFS](https://git-lfs.com/)
- **HuggingFace Token**

### Prerequisites
---

- A recent version of [git](https://git-scm.com/) (e.g. ^2.37 )
- [Python 3.10 or higher](https://www.python.org/downloads/)
- [Poerty 1.8 or higher](https://python-poetry.org/)
> Note: to install Poetry consult the [Poetry documentation](https://python-poetry.org/docs/#installing-with-the-official-installer)

> Note: Install poetry system-wide (not in a virtualenv).
- [Git LFS](https://git-lfs.github.com/)
## Setup

### Using Git LFS

This repository uses Git LFS for handling large files. Before you can use this repository, you need to install and set up Git LFS on your local machine.
See https://git-lfs.com/ for installation instructions.
```bash
git lfs install
git clone https://github.com/tabiya-tech/tabiya-livelihoods-classifier.git
cd tabiya-livelihoods-classifier

After Git LFS is set up, follow these steps to clone the repository:
cp .env.example .env
# Edit .env and set: HF_TOKEN=hf_your_token_here

```shell
git clone https://github.com/tabiya-tech/tabiya-livelihoods-classifier.git
docker compose up -d
```

If you already cloned the repository without Git LFS, run:
First startup takes several minutes (image build + ~500 MB model download). Monitor with `docker compose logs -f`.

Once ready, verify:

```bash
curl http://localhost:5001/v1/health
```

```shell
git lfs pull
```json
{
"status": "healthy",
"service": "classify-api",
"dependencies": {
"ner-api": "healthy",
"nel-api": "healthy"
}
}
```

### Install the dependencies
---

#### Set up virtualenv
In the **root directory** of the backend project (so, the same directory as this README file), run the following commands:
## Usage

```shell
# create a virtual environment
python3 -m venv venv
### Classify a job

# activate the virtual environment
source venv/bin/activate
```bash
curl -X POST http://localhost:5001/v1/classify \
-H "Content-Type: application/json" \
-d '{
"title": "Head Chef",
"description": "We are looking for an experienced Head Chef to plan menus, manage kitchen staff, and ensure food quality."
}'
```

```shell
# Use the version of the dependencies specified in the lock file
poetry lock --no-update
# Install missing and remove unreferenced packages
poetry install --sync
You can also pass a single `"text"` field instead of `title` + `description`.

### Options

| Option | Default | Description |
|--------|---------|-------------|
| `top_k` | 5 | ESCO matches per entity |
| `min_similarity` | 0.0 | Minimum cosine similarity (0.0–1.0) |
| `extract_entities` | all | Filter entity types, e.g. `["occupation", "skill"]` |

### Batch

```bash
curl -X POST http://localhost:5001/v1/classify/batch \
-H "Content-Type: application/json" \
-d '{
"jobs": [
{"job_id": "001", "title": "Nurse", "description": "Provide patient care."},
{"job_id": "002", "text": "Electrician. Install and maintain electrical systems."}
]
}'
```

> Note:
> Install the dependencies for the training using:
> ```shell
> # Use the version of the dependencies specified in the lock file
> poetry lock --no-update
> # Install missing and remove unreferenced packages
> poetry install --sync --with train
> ```

> Note:
> Before running any tasks, activate the virtual
> environment so that the installed dependencies are available:
> ```shell
> # activate the virtual environment
> source venv/bin/activate
> ```
> To deactivate the virtual environment, run:
> ```shell
> # deactivate the virtual environment
> deactivate
> ```

Activate Python and download the NLTK punctuation package to use the sentence tokenizer. You only need to download `punkt` once.

```shell
python <<EOF
import nltk
nltk.download('punkt')
EOF
Returns a `batch_id` — poll `/v1/batch/{id}/status`, fetch results at `/v1/batch/{id}/results`.

### Response format

```json
{
"classification": {
"entities": [
{
"entity_type": "occupation",
"surface_form": "Head Chef",
"span": {"start": 0, "end": 9},
"linked_entities": [
{
"label": "head chef",
"code": "3434.2",
"uri": "http://data.europa.eu/esco/occupation/...",
"similarity_score": 0.91,
"taxonomy": "esco"
}
]
},
{
"entity_type": "skill",
"surface_form": "plan menus",
"span": {"start": 52, "end": 62},
"linked_entities": [
{
"label": "plan menus",
"similarity_score": 0.88,
"taxonomy": "esco"
}
]
}
],
"entity_counts": {"occupation": 1, "skill": 3}
},
"metadata": {
"classifier_version": "1.0.0",
"model_name": "tabiya/roberta-base-job-ner",
"linker_model": "all-MiniLM-L6-v2",
"processing_time_ms": 245.3,
"input_text_hash": "abc123..."
}
}
```

### Environment Variable & Configuration
Entity types: `occupation`, `skill`, `qualification`, `experience`, `domain`. Only occupation, skill, and qualification get `linked_entities`.

---

The repo uses the following environment variable:
## API endpoints

- `HF_TOKEN`: To use the project, you need access to the HuggingFace 🤗 entity extraction model. Contact the administrators via [tabiya@benisis.de]. From there, you need to create a read access token to use the model. Find or create your read access token [here](https://huggingface.co/settings/tokens).
The backend supports the use of a `.env` file to set the environment variable. Create a `.env` file in the root
directory of the backend project and set the environment variables as follows:
| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/v1/classify` | Classify a single job |
| `POST` | `/v1/classify/batch` | Submit a batch (async) |
| `GET` | `/v1/batch/{id}/status` | Poll batch progress |
| `GET` | `/v1/batch/{id}/results` | Fetch batch results |
| `GET` | `/v1/health` | Health check |
| `GET` | `/v1/version` | Version info |

```dotenv
# .env file
HF_TOKEN=<YOUR_HF_TOKEN>
Interactive docs at `http://localhost:5001/docs`.

---

## Manual setup (without Docker)

Requires **Python 3.10+** and [Poetry](https://python-poetry.org/).

```bash
python3 -m venv venv && source venv/bin/activate
poetry install --sync
python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab')"
export HF_TOKEN=hf_your_token_here
```

> ATTENTION: The .env file should be kept secure and not shared with others as it contains sensitive information.
> It should not be committed to the repository.

- **[Use the model](inference/README.md)**: Instructions on how to use the inference tool.
- **[Use the API](app/README.md)**: Instructions on how to use the API.
- **[Training](train/README.md)**: Details on how to train the model.

## Model's Architecture

![Model Architecture](./pics/entity_linker.png)

## License

The code and model weights are licensed under the MIT License. See the [LICENSE](./LICENSE) file for details.

The [datasets](#Datasets) are licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0). See the [DATA_LICENSE](./DATA_LICENSE) file for details.

## Datasets

### Occupations
- **Location**: [inference/files/occupations_augmented.csv](inference/files/occupations_augmented.csv)
- **Source**: [ESCO dataset - v1.1.1](https://esco.ec.europa.eu/en/use-esco/download)
- **Description**: ESCO (European Skills, Competences, Qualifications and Occupations) is the European multilingual classification of Skills, Competences, and Occupations. This dataset includes information relevant to the occupations.
- **License**: Creative Commons Attribution 4.0 International see [DATA_LICENSE](./DATA_LICENSE) for details.
- **Modifications**: The columns retained are `alt_label`, `preferred_label`, `esco_code`, and `uuid`. Each alternative label has been separated into individual rows.

### Skills
- **Location**: [inference/files/skills.csv](inference/files/skills.csv)
- **Source**: [ESCO dataset - v1.1.1](https://esco.ec.europa.eu/en/use-esco/download)
- **Description**: ESCO (European Skills, Competences, Qualifications and Occupations) is the European multilingual classification of Skills, Competences and Occupations. This dataset includes information relevant to the skills.
- **License**: Creative Commons Attribution 4.0 International see [Data License](./DATA_LICENSE) for details.
- **Modifications**: The columns retained are `preferred_label` and `uuid`.

### Qualifications
- **Location**: [inference/files/qualifications.csv](inference/files/qualifications.csv)
- **Source**: [Official European Union EQF comparison website](https://europass.europa.eu/en/compare-qualifications)
- **Description**: This dataset contains EQF (European Qualifications Framework) relevant information extracted from the official EQF comparison website. It includes data strings, country information, and EQF levels. Non-English text was ignored.
- **License**: Please refer to the original source for [license information](https://europass.europa.eu/en/node/2161).
- **Modifications**: Non-English text was removed, and the remaining information was formatted into a structured database.

### Hahu Test
- **Location**: [inference/files/eval/redacted_hahu_test_with_id.csv](inference/files/eval/redacted_hahu_test_with_id.csv)
- **Source**: [hahu_test](https://huggingface.co/datasets/tabiya/hahu_test)
- **Description**: This dataset consists of 542 entries chosen at random from the 11 general classification system of the Ethiopian hahu jobs platform. 50 entries were selected from each class to create the final dataset.
- **License**: Creative Commons Attribution 4.0 International see [Data License](./DATA_LICENSE) for details.
- **Modifications**: No modifications were made to the selected entries.

### House and Tech
- **Location**:
- [inference/files/eval/house_test_annotations.csv](inference/files/eval/house_test_annotations.csv)
- [inference/files/eval/house_validation_annotations.csv](inference/files/eval/house_validation_annotations.csv)
- [inference/files/eval/tech_test_annotations.csv](inference/files/eval/tech_test_annotations.csv)
- [inference/files/eval/tech_validation_annotations.csv](inference/files/eval/tech_validation_annotations.csv)
- **Source**: Provided by [Decorte et al.](https://arxiv.org/abs/2209.05987)
- **Description**: The dataset includes the HOUSE and TECH extensions of the SkillSpan Dataset. In the original work by Decorte et al., the test and development entities of the SkillSpan Dataset were annotated into the ESCO model.
- **License**: MIT, Please refer to the original source.
- **Modifications**: The datasets were used as provided without further modifications.

### Qualification Mapping
- **Location**: [inference/files/eval/qualification_mapping.csv](inference/files/eval/qualification_mapping.csv)
- **Source**: Extended from the [Green Benchmark](https://github.com/acp19tag/skill-extraction-dataset) Qualifications
- **Description**: This dataset maps the Green Benchmark Qualifications to the appropriate EQF levels. Two annotators tagged the qualifications, resulting in a Cohen's Kappa agreement of 0.45, indicating moderate agreement.
- **License**: Creative Commons Attribution 4.0 International see [Data License](./DATA_LICENSE) for details.
- **Modifications**: Extended the dataset to include EQF level mappings, and the annotations were verified by two annotators.

### Access and Usage

To use these datasets, ensure you comply with the original dataset's license and terms of use. Any modifications made should be documented and attributed appropriately in your project.

For datasets requiring access tokens, such as those from HuggingFace 🤗, please contact the maintainers to obtain a read access token.

## Bibliography

A list on interesting and relevant material for reading:
* **GPT NER** [GPT-NER: Named Entity Recognition via Large Language Models](https://arxiv.org/pdf/2304.10428) (Shuhe Wang)
* **Skill Extraction with LLMs** [Rethinking Skill Extraction in the Job Market Domain using Large Language Models](https://arxiv.org/pdf/2402.03832) (Mike Zhang)
* **NER annotation with LLM** [LLMs Accelerate Annotation for Medical Information Extraction](https://proceedings.mlr.press/v225/goel23a)
* **Skills Entity Linking** Zhang, Mike, Rob van der Goot, and Barbara Plank. "Entity Linking in the Job Market Domain." arXiv preprint arXiv:2401.17979 (2024).
* **Skills-ML** is an open-source Python library for developing and analyzing skills and competencies from unstructured text. (link: http://dataatwork.org/skills-ml/)
* **SkillSpan**: Hard and Soft Skill Extraction from English Job Postings https://arxiv.org/abs/2204.12811 (Mike Zhang)
* **work2vec**: Using the full text of data from 200 million online job postings, we train and evaluate a natural language processing (NLP) model to learn the language of jobs. We analyze how jobs have changed in the past decade, and show how different words in the posting denote different occupations. We use this approach to create novel indexes of jobs, such as work-from-home ability. In ongoing work, we quantify the return to various skills.

https://digitaleconomy.stanford.edu/research/job2vec/
https://digitaleconomy.stanford.edu/people/sarah-h-bana/
* **Data Science and ESCO** Insights into how ESCO is leveraging data-science techniques. https://esco.ec.europa.eu/en/about-esco/data-science-and-esco
* **Machine Learning Assisted Mapping of Multilingual Occupational Data to ESCO**: A report that discusses the multilingual mapping
approach that the ESCO team established to support the maintenance of ESCO. https://esco.ec.europa.eu/en/about-esco/publications/publication/machine-learning-assisted-mapping-multilingual-occupational
* **ESCO Publications**: Artificial intelligence & machine learning. https://esco.ec.europa.eu/en/about-esco/publications?f%5B0%5D=theme%3A109860&page=0
Start each in a separate terminal:

```bash
uvicorn app.server.ner_server:app --host 0.0.0.0 --port 5002
uvicorn app.server.nel_server:app --host 0.0.0.0 --port 5003
uvicorn app.server.classify_server:app --host 0.0.0.0 --port 5001
```

---

## Stopping

```bash
docker compose down
```
Loading