A privacy-preserving NLP pipeline for clinical records — DPDP Act compliant.
This project provides a comprehensive solution for de-identifying Protected Health Information (PHI) from clinical text and FHIR resources. It combines transformer-based NER with regex pattern matching to achieve high accuracy in PHI detection and removal.
- Transformer-based NER: Uses Bio_ClinicalBERT model for accurate PHI detection
- Regex Pattern Matching: Complementary rule-based detection for structured PHI
- Reversible Tokenization: Replace PHI with secure tokens that can be re-linked for authorized access
- FHIR R4 Support: Process entire FHIR resources (Patient, Observation, Condition, etc.)
- Audit Logging: Complete audit trail for compliance with healthcare regulations
- DPDP Act Compliant: Designed to meet India's Digital Personal Data Protection Act requirements
- Streamlit UI: Interactive demo interface for testing and visualization
┌─────────────────────────────────────────────────────────────┐
│ DeidentPipeline │
├─────────────────────────────────────────────────────────────┤
│ 1. Bio_ClinicalBERT NER (transformer) │
│ 2. Regex rules (pattern matching) │
│ 3. Ensemble merger with confidence threshold │
│ 4. Reversible tokenization │
│ 5. Audit logging │
└─────────────────────────────────────────────────────────────┘
phi-deid-pipeline/
├── app.py # Streamlit demo application
├── requirements.txt # Python dependencies
├── .env.example # Environment variables template
├── models/ # Trained models
│ └── bio_clinicalbert_deid/ # Bio_ClinicalBERT de-identification model
├── src/
│ ├── pipeline/
│ │ ├── deid_pipeline.py # Main de-identification pipeline
│ │ ├── fhir_processor.py # FHIR resource processing
│ │ ├── regex_rules.py # Regex-based PHI detection
│ │ └── reversible_tokenizer.py # Reversible tokenization
│ ├── audit/
│ │ └── audit_logger.py # Audit logging functionality
│ ├── data_prep/
│ │ ├── i2b2_parser.py # i2b2 dataset parser
│ │ └── synthetic_generator.py # Synthetic data generator
│ ├── evaluation/
│ │ └── metrics.py # Evaluation metrics
│ └── model/
│ └── config.py # Model configuration
├── notebooks/
│ └── colab_train.py # Google Colab training script
└── data/
├── raw/ # Raw input data
├── processed/ # Processed output data
├── synthetic/ # Synthetic clinical notes
└── audit/ # Audit database
- Python 3.9+
- CUDA-capable GPU (recommended for faster inference)
-
Clone the repository
git clone https://github.com/supritkumar007/phi-deid-pipeline.git cd phi-deid-pipeline -
Create a virtual environment
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
-
Install dependencies
pip install -r requirements.txt
-
Set up environment variables
cp .env.example .env # Edit .env with your configuration
streamlit run app.pyThe demo provides:
- Free-text clinical note de-identification
- FHIR resource processing
- Audit log viewer
- Re-link functionality for authorized token recovery
from src.pipeline.deid_pipeline import DeidentPipeline
# Initialize pipeline
pipeline = DeidentPipeline(
model_path="models/bio_clinicalbert_deid",
confidence_threshold=0.80,
)
# De-identify text
result = pipeline.process(
text="Patient John Doe (MRN: 12345) was admitted on 01/15/2024.",
document_id="DOC001",
operator_id="demo_user"
)
print(result.deidentified_text)
# Output: Patient <NAME-1> (MRN: <ID-1>) was admitted on <DATE-1>.The pipeline detects and handles the following PHI categories:
| Category | Description | Example |
|---|---|---|
| NAME | Patient names | John Doe, Priya Sharma |
| DATE | Dates and timestamps | 01/15/2024, 12:30 PM |
| AGE | Patient age | 46 years old |
| LOCATION | Addresses, cities | Bengaluru, Karnataka |
| CONTACT | Phone, email | 9876543210 |
| ID | MRN, SSN, account numbers | MH238471 |
| PROFESSION | Occupation, employer | Engineer, Teacher |
Supported FHIR R4 resource types:
- Patient
- Observation
- Condition
- DiagnosticReport
- DocumentReference
- Practitioner
- Encounter
from src.pipeline.fhir_processor import deidentify_fhir, load_fhir_bundle
# Load FHIR bundle
resources = load_fhir_bundle("patient_fhir.json")
# De-identify each resource
for resource in resources:
result = deidentify_fhir(resource, pipeline)
print(f"Removed {result['total_phi_removed']} PHI elements")The pipeline maintains a complete audit trail including:
- Document ID
- Operator ID
- Timestamp
- Original text (hashed)
- De-identified text
- Token mappings
- PHI spans detected
View audit logs via the Streamlit UI or directly query the SQLite database at data/audit/audit.db.
PHI elements are replaced with secure tokens that can be re-linked for authorized access:
# Re-link tokens to original values
from src.pipeline.reversible_tokenizer import ReversibleTokenizer
tokenizer = ReversibleTokenizer()
original = tokenizer.decode(token_map, deidentified_text)MODEL_PATH=models/bio_clinicalbert_deid
CONFIDENCE_THRESHOLD=0.80
SECRET_KEY=your-secret-key-here
AUDIT_DB_PATH=data/audit/audit.db
TOKEN_DB_PATH=data/audit/token_store.db| Parameter | Default | Description |
|---|---|---|
| model_path | models/bio_clinicalbert_deid | Path to trained model |
| confidence_threshold | 0.80 | Minimum confidence for PHI detection |
| device | auto | Device for inference (cpu/cuda/auto) |
- Inference Speed: ~50-100 sentences/second on GPU
- PHI Detection F1 Score: ~95% on i2b2 test set
- Supported Languages: English clinical text
- Bio_ClinicalBERT - Pre-trained clinical NLP model
- i2b2 - Dataset for PHI de-identification research
- Microsoft Presidio - PII detection framework