From ae5dbd38fef6a4047ba85177b541de9387ad15dc Mon Sep 17 00:00:00 2001 From: David de Hilster Date: Tue, 13 Oct 2026 12:59:29 -0400 Subject: [PATCH] Help: rmchars, $os, negative strpiece, split into characters, Unicode num, names, dict case (engine 4.2.15) - New pages: rmchars (nlp-engine#428) and $os (#168), added to the index, Table_of_String_Functions and Special_Variables. - strpiece: negative positions count from the end; positions are characters (#264). - split: an empty delimiter returns the characters (#345); fixes the syntax line, which read str(...) instead of split(...). - num: Unicode decimal digits (full-width, Arabic-Indic) are read (#209). - Tokens / Pass_Files: nonliteral names may contain digits and underscores (#396). - About_the_analyzer_sequence: how dicttok matches dictionary entries by case (#386). Co-Authored-By: Claude Opus 5.5 --- Help/markdown/$os.md | 44 ++++++++++++++++++ Help/markdown/NLP_PP_Stuff/Pass_Files.md | 2 +- .../NLP_PP_Stuff/Special_Variables.md | 1 + Help/markdown/NLP_PP_Stuff/Tokens.md | 2 +- Help/markdown/Table_of_String_Functions.md | 1 + .../About_the_analyzer_sequence.md | 2 + Help/markdown/index.md | 4 +- Help/markdown/num.md | 2 + Help/markdown/rmchars.md | 46 +++++++++++++++++++ Help/markdown/split.md | 6 ++- Help/markdown/strpiece.md | 4 ++ 11 files changed, 109 insertions(+), 5 deletions(-) create mode 100644 Help/markdown/$os.md create mode 100644 Help/markdown/rmchars.md diff --git a/Help/markdown/$os.md b/Help/markdown/$os.md new file mode 100644 index 0000000..74ef795 --- /dev/null +++ b/Help/markdown/$os.md @@ -0,0 +1,44 @@ +[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md) + +# $os + +## Purpose + +Find out which operating system the engine is running on. + +## Syntax + +``` +returnedString = variableType("$os") +``` + +``` +returnedString - type: str + +variableType - type: G +``` + +## Returns + +`windows`, `linux` or `macos`. + +## Remarks + +Use it where an analyzer has to do something differently per platform, such as building a file path or choosing the command for [system](system.md). + +**NEW 4.2.15** + +## Example + +``` +@CODE +if (G("$os") == "windows") + system("dir > listing.txt"); +else + system("ls > listing.txt"); +@@CODE +``` + +## See Also + +[$dev]($dev.md), [system](system.md), [Special Variables](NLP_PP_Stuff/Special_Variables.md#table) diff --git a/Help/markdown/NLP_PP_Stuff/Pass_Files.md b/Help/markdown/NLP_PP_Stuff/Pass_Files.md index 942ca38..93f4bd2 100644 --- a/Help/markdown/NLP_PP_Stuff/Pass_Files.md +++ b/Help/markdown/NLP_PP_Stuff/Pass_Files.md @@ -146,7 +146,7 @@ Optionally, each element can be followed by a list of key=values *pairs* enclose | **#** | Comment. Text till the end of the line is ignored. | | **/\* \*/** | Block comment. Everything between `/*` and the next `*/` is ignored, including line breaks. Block comments do not nest: the first `*/` closes the comment. Delimiters inside a string or inside a `#` comment are ordinary text. | | **AAA or NNN** | Literal tokens. Completely alphabetic or completely numeric. | -| **_AAA** | Nonliteral tokens must begin with underscore. All the remaining letters MUST be alphabetic. (May expand to underscore,num,dash someday.) | +| **_AAA** | Nonliteral tokens must begin with underscore, followed by letters, digits and underscores: `_noun`, `_B2`, `_phone_num`. Underscores after the first are **NEW 4.2.15**. Dashes are not allowed. | | **"XXX"** | Strings get double quoted. Everything within string taken literally (except backslashed characters). | | **\X** | Backslash any nonalphanumeric char you want taken literally. | | **\\** | Literal backslash character. | diff --git a/Help/markdown/NLP_PP_Stuff/Special_Variables.md b/Help/markdown/NLP_PP_Stuff/Special_Variables.md index d8fc8bc..0e7a8e7 100644 --- a/Help/markdown/NLP_PP_Stuff/Special_Variables.md +++ b/Help/markdown/NLP_PP_Stuff/Special_Variables.md @@ -50,6 +50,7 @@ A consolidated table of NLP++ special variables is provided below. Each item in | **$passnum** | G | Get current pass number. **NEW 2.0.2.4** | | **$rulenum** | G | Get current rule number within current pass. **NEW 2.0.2.4** | | [**$dev**](../$dev.md) | G | Check if the engine was run with **-DEV**. Returns 1 under -DEV, else 0. Use it to gate an analyzer's own diagnostic passes on what the caller asked for. **NEW 3.8.4** | +| [**$os**](../$os.md) | G | Operating system the engine runs on: `windows`, `linux` or `macos`. **NEW 4.2.15** | | [**$silent**](../$silent.md) | G | Check if the engine was run with **-SILENT**. Returns 1 under -SILENT, else 0. Note -SILENT also suppresses the analyzer's own output files. **NEW 3.8.4** | | [**$isdirrun**](../$isdirrun.md) | G | Check if the engine is processing a directory of files instead of an individual file. Returns 1 if it is analyzing a directory, else 0. **NEW 1.26.0** | | [**$isfirstfile**](../$isfirstfile.md) | G | Check if is first file processed in a directory. Returns 1 if it is the first file, else 0. **NEW 1.24.0** | diff --git a/Help/markdown/NLP_PP_Stuff/Tokens.md b/Help/markdown/NLP_PP_Stuff/Tokens.md index 22734ed..1492568 100644 --- a/Help/markdown/NLP_PP_Stuff/Tokens.md +++ b/Help/markdown/NLP_PP_Stuff/Tokens.md @@ -9,7 +9,7 @@ Following are some token types that occur in NLP++ code. | **#** | Comment. Text till the end of the line is ignored. | | **/\* \*/** | Block comment. Everything between `/*` and the next `*/` is ignored, including line breaks. Block comments do not nest: the first `*/` closes the comment. Delimiters inside a string or inside a `#` comment are ordinary text. | | **AAA or NNN** | Literal tokens. Completely alphabetic or completely numeric. | -| **_AAA** | Nonliteral tokens must begin with underscore. All the remaining letters MUST be alphabetic. Currently, only alphabetics are allowed after the initial underscore. | +| **_AAA** | Nonliteral tokens must begin with underscore, followed by letters, digits and underscores: `_noun`, `_B2`, `_phone_num`. Underscores after the first are **NEW 4.2.15**. | | **"XXX"** | Strings get double quoted. Everything within string taken literally (except backslashed characters). | | **\X** | Backslash any nonalphanumeric char you want taken literally. If X = one of a b n r f t v, these are special escapes analogous to C/C++. | | **\\** | Literal backslash character. | diff --git a/Help/markdown/Table_of_String_Functions.md b/Help/markdown/Table_of_String_Functions.md index 8499a78..21973e8 100644 --- a/Help/markdown/Table_of_String_Functions.md +++ b/Help/markdown/Table_of_String_Functions.md @@ -16,6 +16,7 @@ The return types include:  **STR** (string), **STR ARRAY** (string array), **IN | [**refind(str, pattern_str [, flags_str])**](refind.md) | **STR ARRAY** | Regular expression match: element 0 is the whole match, 1..n the capture groups. | | [**rematch(str, pattern_str [, flags_str])**](rematch.md) | **BOOL** | True if the regular expression matches the string. | | [**resubst(str, pattern_str, replacement_str [, flags_str])**](resubst.md) | **STR** | Replace every regular expression match, with `$1` backreferences. | +| [**rmchars(str, chars_str)**](rmchars.md) | **STR** | Remove every character of chars_str from str. **NEW 4.2.15** | | [**split(str, char_str)**](split.md) | **STR ARRAY** | Split a string using the given char as separator. | | [**stem(str)**](stem.md) | **STR** | Compute stem of noun and verb. Return lowercase string. | | [**strchar(str, num)**](strchar.md) | **STR** | Index to numth char of string | diff --git a/Help/markdown/VisualText_Basics/About_the_analyzer_sequence.md b/Help/markdown/VisualText_Basics/About_the_analyzer_sequence.md index 239829c..d68c36e 100644 --- a/Help/markdown/VisualText_Basics/About_the_analyzer_sequence.md +++ b/Help/markdown/VisualText_Basics/About_the_analyzer_sequence.md @@ -18,6 +18,8 @@ User passes employ either the **pattern** (or **pat**) and the **recursive** (or If you need a customized tokenization pass to handle OCR output (already exists, commercially available in conjunction with http://www.XIEO.info)), convert HTML to plain text, or another specialized input format, TAI can help.  Contact us at support@textanalysis.com, 1-877-235-6259 USA toll free. +**Dictionary matching and case.** **dicttok** looks each word up as written, then in lowercase, then Capitalized (first letter uppercase). So a **.dict** entry written in lowercase (`jumps`) or Capitalized (`Jumps`) matches the word in any capitalization: `jumps`, `Jumps`, `JUMPS`. An all-caps entry (`US`) matches only that spelling, which keeps it apart from `us`. Multi-word phrases are matched in lowercase, so write phrase entries in lowercase. Capitalized entries matching other capitalizations is **NEW 4.2.15**. + **dicttok** also puts a **CTRL** attribute on control characters to help manage control characters (e.g., using feature-based matching), as in the code snippet below. **@PRE** <2,2> **vareq**(**"CTRL"**,128);  # Match ASCII 128 = €** @POST** "output.txt" << "Found ctrl = " << **N**("$text",2) << "\n";** @RULES**** _xNIL** <- â** _xCTRL**** _xALPHA**** @@** diff --git a/Help/markdown/index.md b/Help/markdown/index.md index 25a56bf..3b6d657 100644 --- a/Help/markdown/index.md +++ b/Help/markdown/index.md @@ -96,6 +96,7 @@ to a topic. - [$rulenum]($rulenum.md) - **Run Mode Variables** - [$dev]($dev.md) + - [$os]($os.md) - [$silent]($silent.md) - **Test Variables** - [$end]($end.md) @@ -464,6 +465,7 @@ to a topic. - [refind](refind.md) - [rematch](rematch.md) - [resubst](resubst.md) + - [rmchars](rmchars.md) - [split](split.md) - [sqlstr](sqlstr.md) - [stem](stem.md) @@ -602,7 +604,7 @@ above. They are grouped here so that every page remains reachable from this inde ### Run Mode Variables -[$dev]($dev.md)  |  [$silent]($silent.md) +[$dev]($dev.md)  |  [$os]($os.md)  |  [$silent]($silent.md) ### Control Flow Keywords diff --git a/Help/markdown/num.md b/Help/markdown/num.md index 42be021..3a35ef9 100644 --- a/Help/markdown/num.md +++ b/Help/markdown/num.md @@ -26,6 +26,8 @@ Returns the integer value of the string if it can be converted; a float argument This function also accepts a num arg. +Digits from other scripts count too: `num("１２")` (full-width) is 12 and `num("٣")` (Arabic-Indic) is 3. Any Unicode decimal digit is read as its value. **NEW 4.2.15** + ## Example ``` diff --git a/Help/markdown/rmchars.md b/Help/markdown/rmchars.md new file mode 100644 index 0000000..f2c212e --- /dev/null +++ b/Help/markdown/rmchars.md @@ -0,0 +1,46 @@ +[← Help Contents](index.md) | [📘 NLP++ Textbook](NLP++_Textbook.md) + +# rmchars + +## Purpose + +Remove every occurrence of a set of characters from a string. + +## Syntax + +``` +returnedString = rmchars(string, charsString) +``` + +``` +returnedString - type: str + +string - type: str + +charsString - type: str +``` + +## Returns + +*string* with every character that appears in *charsString* removed. If *charsString* is empty, *string* is returned unchanged. + +## Remarks + +Each character of *charsString* is removed wherever it occurs; order doesn't matter. Characters are whole characters, not bytes: `rmchars("héllo", "é")` is `hllo`. + +To remove a multi-character substring instead, use [strsubst](strsubst.md) with an empty replacement. + +**NEW 4.2.15** + +## Example + +``` +@CODE +"output.txt" << rmchars("(555) 867-5309", "()- ") << "\n"; +# Writes 5558675309 +@@CODE +``` + +## See Also + +[strsubst](strsubst.md), [split](split.md), [strclean](strclean.md), [String Functions](Table_of_String_Functions.md) diff --git a/Help/markdown/split.md b/Help/markdown/split.md index ddf0a06..9ddc7a8 100644 --- a/Help/markdown/split.md +++ b/Help/markdown/split.md @@ -9,7 +9,7 @@ Split a string into pieces using a separator character. ## Syntax ``` -returnedArray = str(str, charString) +returnedArray = split(str, charString) ``` ``` @@ -30,7 +30,9 @@ An array of strings, each of which is a component of the given string. ## Remarks -Similar to the Perl *split* function. +Similar to the Perl *split* function. Each character in *charString* is a delimiter. + +With an empty *charString*, `split` returns the string's characters: `split("héllo", "")` is `h`, `é`, `l`, `l`, `o`. Characters are whole characters, not bytes, so an accented letter or emoji stays one element. **NEW 4.2.15** (it used to be an error). ## Example diff --git a/Help/markdown/strpiece.md b/Help/markdown/strpiece.md index 359b602..721a49d 100644 --- a/Help/markdown/strpiece.md +++ b/Help/markdown/strpiece.md @@ -28,6 +28,10 @@ The substring of `string` from the zero-based index startNumber to endNumber, in ## Remarks +Positions count characters, not bytes, so accented letters and emoji count as one each. + +A negative position counts from the end: `-1` is the last character, `-2` the one before it. `strpiece("hello", 1, -1)` is `ello`, and `strpiece("hello", 0, -2)` is `hell`. **NEW 4.2.15** + ## Example ```