Skip to content

feat(engine): add term name validation and configurable deny-list for concept/entity extraction - #24

Draft
pcfreak30 wants to merge 1 commit into
swarmclawai:mainfrom
pcfreak30:feat/term-name-validation
Draft

feat(engine): add term name validation and configurable deny-list for concept/entity extraction#24
pcfreak30 wants to merge 1 commit into
swarmclawai:mainfrom
pcfreak30:feat/term-name-validation

Conversation

@pcfreak30

Copy link
Copy Markdown

Add post-extraction validation pipeline for concept and entity names to prevent NLP artifacts and generic stopwords from polluting the knowledge graph.

Three layers of defense:

  • sanitizeEntityName(): strip trailing punctuation artifacts (?, :, ,, ;, smart quotes, periods) that compromise leaves on entity names
  • isValidTermName(): structural rejection of parsing fragments (bracket prefixes, unmatched parens, ellipsis patterns, all-closed- class POS), plus English stopword filtering via ENGLISH_STOPWORDS
  • validatedTerms(): centralized quality gate applied across all extraction paths (heuristic, provider, vision, merge) with optional user-configurable termDenyList

New config surface:

  • analysis.termDenyList in swamvault.config.json for domain-specific stop words that the NLP layer cannot detect on its own
  • Zod-validated via analysisConfigSchema in config.ts
  • Wired through compileVault -> analyzeSource -> all extraction paths

Compound-name first-word denial: entries in termDenyList automatically reject multi-word terms starting with a deny-listed word (catches NLP artifacts like "Transcript If", "Transcript What's").

ANALYSIS_FORMAT_VERSION bumped 8 -> 9 to invalidate cached analyses.


@waydelyle I am sharing this in hopes you are interested in it as is, a starting point or can take it as a reference. I have been using this for over a month for my own needs and would like to see this upstreamed in some way.

All the effort is from AI and this is not my domain area as a whole, but it works for me for what I needed.

… concept/entity extraction

Add post-extraction validation pipeline for concept and entity names
to prevent NLP artifacts and generic stopwords from polluting the
knowledge graph.

Three layers of defense:
- sanitizeEntityName(): strip trailing punctuation artifacts (?, :,
  ,, ;, smart quotes, periods) that compromise leaves on entity names
- isValidTermName(): structural rejection of parsing fragments
  (bracket prefixes, unmatched parens, ellipsis patterns, all-closed-
  class POS), plus English stopword filtering via ENGLISH_STOPWORDS
- validatedTerms(): centralized quality gate applied across all
  extraction paths (heuristic, provider, vision, merge) with optional
  user-configurable termDenyList

New config surface:
- analysis.termDenyList in swamvault.config.json for domain-specific
  stop words that the NLP layer cannot detect on its own
- Zod-validated via analysisConfigSchema in config.ts
- Wired through compileVault -> analyzeSource -> all extraction paths

Compound-name first-word denial: entries in termDenyList automatically
reject multi-word terms starting with a deny-listed word (catches NLP
artifacts like "Transcript If", "Transcript What's").

ANALYSIS_FORMAT_VERSION bumped 8 -> 9 to invalidate cached analyses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant