Skip to content

feat: add weighted confidence scoring for extraction validation#424

Open
Arijit429 wants to merge 9 commits into
fireform-core:mainfrom
Arijit429:weighted-confidence-scoring
Open

feat: add weighted confidence scoring for extraction validation#424
Arijit429 wants to merge 9 commits into
fireform-core:mainfrom
Arijit429:weighted-confidence-scoring

Conversation

@Arijit429

@Arijit429 Arijit429 commented Apr 12, 2026

Copy link
Copy Markdown
Contributor

Closes #60
Closes #450

🚀 Summary

This PR enhances the extraction validation workflow by introducing field-level weighted confidence scoring instead of using a flat deduction model.

The goal is to make confidence evaluation more representative of real-world data importance and improve the reliability of the requires_review decision.


✨ What Changed

Updated the ExtractionValidator logic to use weighted importance for each field:

FIELD_WEIGHTS = {
    "location": 30,
    "time": 20,
    "severity": 30,
    "description": 20
}

Each missing field now reduces confidence based on its relative importance instead of applying a uniform deduction.


💡 Why This Helps

In the previous approach, all missing fields contributed equally to confidence reduction.

This could under-represent critical missing information.

For example:

  • missing location should have higher impact
  • missing severity should strongly affect review decision
  • missing description may be comparatively less critical

The new weighted model makes confidence scoring more realistic and production-aligned.


🔍 Example

Previous behavior

all fields = equal impact

New behavior

location → 30
severity → 30
time → 20
description → 20

This allows the validator to make smarter review decisions.


🧪 Testing

Added dedicated unit test coverage for weighted confidence scoring.

Validated scenarios include:

  • multiple high-weight missing fields
  • partial extraction completeness
  • review threshold correctness
  • confidence floor protection

Executed locally using:

PYTHONPATH=. pytest test/test_extraction_validator.py -q

and verified successful execution.


🎯 Impact

This improves:

  • confidence score realism
  • human review routing accuracy
  • downstream PDF fill reliability
  • production readiness of extraction workflow

@Arijit429

Copy link
Copy Markdown
Contributor Author

Hi maintainers, this update improves the review workflow by making confidence scoring field-aware and more aligned with real extraction importance—happy to refine the weighting logic based on project needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants