Repository navigation
dict phrases with option to concat the words into one literal #434
Description
It seems that there should be a way for specifying phrases in the dictionary that don't end up reduced or can be concatenated into a literal of the phrase words. @amnonma, what do you think?
Activity
amnonma commentedon May 21, 2023
Sounds good in principle.
If it means modifying the input buffer, that can get expensive and tricky.
NLP++ could create a new type of "glommed literal", I suppose that could work internally.
Would want to flag with a "glommed" attribute, etc.
I imagine you'd want to get of any whitespace in the concatenated resulting token, and adjust the start and end offset of the literal accordingly.
@dehilsterlexis
amnonma commentedon May 21, 2023
@dehilsterlexis
Since we're speaking of such things, the dictionary or phrasal dictionary might be a good place to split words.
Eg,
resultsspeed => results speed
Do that in a phrasal dict and then you can look up the split words in a follow-on dictionary.
Ordering these "pre tokenizer" items in this way would simplify the rules-based analyzer that follows.
Could be powerful with this type of ordering:
amnonma commentedon May 21, 2023
@dehilsterlexis
It's actually easy to concatenate with an NLP++ rule.
If you use nodetreetext(node) and nodestreetext(node1,nodeN), these know to look for an attribute called "text" and NOT TRAVERSE beneath nodes that have that attribute. You really should try these instead of $text, $treetext, and so on.
They should become built-in functions.
@post
S("text") = "hotrod";
single();
@rules
_hotrod <- _hot _rod @@
dehilsterlexis commentedon May 22, 2023
@amnonma we could even try to implement a word splitter into the mix somehow as part of the tokenizer.
Merge pull request #786 from VisualText/feature/dict-shape-433
ddehilster commentedon Oct 15, 2026
Done in #786 (4.2.30):
hot - rod join=1turns the phrase into one token,hotrod, spanning the same text (whitespace and punctuation dropped). The joined token is then looked up itself, sohotrod s=vehicleapplies. Docs: VisualText/visualtext-files#242.