docs(ai): add learned summary 767-tokenizer-no-escape

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: thomas-mangin

ai/LEARNED-INDEX.md

@@ -47,6 +47,7 @@ Buffer-first, zero-copy, attribute pools, UPDATE building, NLRI parsing.
 - [722](plan/learned/722-spec-bgp-4-aspa-policy.md) -- ASPA policy enforcement: override ordering (ASPA reject wins over origin accept), re-validation via validateCh, origin policy is hardcoded not configurable
 - [764](plan/learned/764-attr-flags-json.md) -- Attribute flags in Ze native JSON: includeFlags parameter on shared formatter, static flags for pool RIB, unwrap at extractRoutes for LG consumers
 - [765](plan/learned/765-gc-pressure-reduction.md) -- GC pressure reduction: [256]bool for attr codes, inline FNV-1a, NextHopAddrs inline struct, LargeCommunities dedup fast path, clear() map reuse; stack arrays that escape via closure/interface/return are NOT optimizations
+- [767](plan/learned/767-tokenizer-no-escape.md) -- Command tokenizer: removed backslash escape handling, backslash is a normal character; no per-byte escape scan on every command
 
 ## Plugin System
 

plan/learned/.counter

@@ -1 +1 @@
-766
+767

plan/learned/767-tokenizer-no-escape.md

@@ -0,0 +1,37 @@
+# 767 — Command Tokenizer: No Escape Sequences
+
+## Context
+
+Both command tokenizers (`plugin/server/command.go:tokenize` and
+`cli/model_commands.go:tokenizeCommand`) scanned every byte/rune for
+backslash escape sequences (`\"`, `\\`). Commands never contain
+backslashes in practice; the escape logic was defensive code with no
+real use case.
+
+## Key Decisions
+
+### Backslash is a normal character
+
+Rather than adding a fast path (`IndexByte` check to skip escape logic),
+we removed escape handling entirely. Backslash is treated as a literal
+character, the same as any other. This removes two branches per rune
+from the hot loop and simplifies the tokenizer to: split on whitespace,
+respect quote delimiters.
+
+### joinTokensWithQuotes drops escape encoding
+
+The round-trip encoder `joinTokensWithQuotes` no longer escapes
+backslashes or quotes. Since `tokenizeCommand` uses `"` as a delimiter
+(never as content), no token from the tokenizer can contain `"`, so
+there is nothing to escape.
+
+### Web tokenizer was already correct
+
+`web/cli.go:tokenizeCommand` never had escape handling. The two
+"full-featured" tokenizers now match its simplicity.
+
+## Traps
+
+- Removing escape support means there is no way to embed a literal `"`
+  inside a quoted value. This is acceptable: Ze command values (peer
+  names, descriptions, paths) have no use case for embedded quotes.