Add DOE chapter text extraction for entity analysis - #22
Open
kgdunn wants to merge 4 commits into
Open
Conversation
kgdunn
commented
Dec 25, 2025
kgdunn
commented
Owner
- Created Python script to extract text from Design of Experiments .rst files
- Filtered out reStructuredText comments while preserving content
- Extracted 38,009 words from 24 files for knowledge graph research
- Created Python script to extract text from Design of Experiments .rst files - Filtered out reStructuredText comments while preserving content - Extracted 38,009 words from 24 files for knowledge graph research
- Created comprehensive entity extraction using spaCy NLP - Extracted 4,609 entities and 132 relationship types (original) - Created refined extraction focused on DOE domain concepts - Refined output: 1,189 entities and 38 relationship types - Output format ready for Neo4j graph database import - Includes metadata: file location, line numbers, context Files: - extract_entities_relationships.py: Original extraction script - extract_entities_refined.py: Refined domain-focused extraction - entities.json, relationships.json: Original comprehensive output - entities_refined.json, relationships_refined.json: Clean domain output
- Created script to export entities and relationships as CSV - entities_list.csv: 1,189 entities with occurrence counts - relationships_list.csv: 38 relationships with examples - Includes 'Keep' column for manual filtering - Enables fine-tuning of knowledge graph content before Neo4j import
Implements strict canonical naming rules for knowledge graph: - Canonical term mappings (e.g., "analysis of variance" → "ANOVA") - Greek letters converted to words (α → "alpha") - Stopwords removed from entity names - Confidence scoring (0.0-1.0) for each extraction Extraction results: - 564 entity extractions from 51 unique canonical entities - Average confidence: 0.80 - 16 structural relationships (IS_VARIANT_OF, ABBREVIATION_OF) Top entities: main effect, experiment, factor, DOE, response variable, interaction, least squares model, full factorial design Files: - extract_canonical_entities.py: Canonical entity extraction script - extract_canonical_relationships.py: Relationship extraction script - entities_canonical.json: JSON array of canonical entities - relationships_canonical.json: JSON array of relationships Ready for Neo4j graph database import.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.