feat(engine): add term name validation and configurable deny-list for concept/entity extraction - #24
Draft
pcfreak30 wants to merge 1 commit into
Draft
Conversation
… concept/entity extraction Add post-extraction validation pipeline for concept and entity names to prevent NLP artifacts and generic stopwords from polluting the knowledge graph. Three layers of defense: - sanitizeEntityName(): strip trailing punctuation artifacts (?, :, ,, ;, smart quotes, periods) that compromise leaves on entity names - isValidTermName(): structural rejection of parsing fragments (bracket prefixes, unmatched parens, ellipsis patterns, all-closed- class POS), plus English stopword filtering via ENGLISH_STOPWORDS - validatedTerms(): centralized quality gate applied across all extraction paths (heuristic, provider, vision, merge) with optional user-configurable termDenyList New config surface: - analysis.termDenyList in swamvault.config.json for domain-specific stop words that the NLP layer cannot detect on its own - Zod-validated via analysisConfigSchema in config.ts - Wired through compileVault -> analyzeSource -> all extraction paths Compound-name first-word denial: entries in termDenyList automatically reject multi-word terms starting with a deny-listed word (catches NLP artifacts like "Transcript If", "Transcript What's"). ANALYSIS_FORMAT_VERSION bumped 8 -> 9 to invalidate cached analyses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add post-extraction validation pipeline for concept and entity names to prevent NLP artifacts and generic stopwords from polluting the knowledge graph.
Three layers of defense:
New config surface:
Compound-name first-word denial: entries in termDenyList automatically reject multi-word terms starting with a deny-listed word (catches NLP artifacts like "Transcript If", "Transcript What's").
ANALYSIS_FORMAT_VERSION bumped 8 -> 9 to invalidate cached analyses.
@waydelyle I am sharing this in hopes you are interested in it as is, a starting point or can take it as a reference. I have been using this for over a month for my own needs and would like to see this upstreamed in some way.
All the effort is from AI and this is not my domain area as a whole, but it works for me for what I needed.