feat!: statement-only tables, shallow expressions, trim expensive variants - #10
Merged
Conversation
- Add ExpressionStatement:0 as explicit statement candidate - Filter raw expressions from statement context (statement-only tables) - Each statement type gets ~1/32 probability instead of ~1/256 - Cap SwitchStatement at 2 cases (was 15), ForStatement to 3 variants - Default maxExprDepth changed to 1 (leaf-only expressions) - Add hasExportDefault guard, name dedup for vars/imports - Output now resembles a real JS module: imports, declarations, exports BREAKING CHANGE: new encoding format due to candidate pool changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Cap call/new/array/object/template counts to 0-4 (was 0-31) - Cap Arrow/FunctionExpression params to 0-3 (was 0-23) - Cap Sequence/TaggedTemplate counts - cosmeticPackageName uses cosmetic RNG (varies per seed) - Dedup exported function names vs import names Output now resembles real minified JS modules with varied imports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reflects new default maxExprDepth=1. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Scraper now collects packageImports: { pkg: [export names] } mapping
- Encoder picks a package, then uses that package's actual exports as
import specifier names
- Fallback to generic imported names if package has no tracked exports
- Default imports use the package's first export as local name
Before: import {RedFormat, ShaderMaterial} from "node:util" (wrong!)
After: import {escape, Minimatch, GLOBSTAR} from "minimatch" (real)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- nameFromHash: generate a/b/../z/aa/ab/../zz names (702-slot space) instead of _tjh style; exclude JS reserved words do/if/in - ObjectExpression: keys are now cosmetic short identifiers (cosmeticProp) rather than structural child expressions; decoder skips key - SwitchStatement: case tests are now cosmetic sequential integers (0,1,2) rather than structural expressions; decoder skips test - Fix pre-existing bug: ExportNamedDeclaration:variable was missing the collision-check loop present in VariableDeclaration; same fix applied to both encoder and decoder - Update snapshots for changed cosmetic output - Document 3 structural improvements in CONTEXT.md (variable/function reuse, imports region lock, per-statement depth budget)
Add hasLeftImportRegion boolean to EncodingContext. Once any non-import statement is selected at blockDepth === 0 (top level), the flag flips and ImportDeclaration is filtered out of all subsequent candidates. Encoder flips in buildTopLevelWithCandidate after table lookup; decoder flips in the stmt work item handler after write. Both sides are identical in timing so encoder/decoder stay in sync. Eliminates imports appearing randomly in the middle of generated code.
- Add stmtDepthFromHash(hash, globalMax) to context.ts: derives a per-statement expression depth cap from the post-selection hash. Distribution is 50%->1, 25%->2, 12.5%->3, 12.5%->globalMax. Only applies when globalMax > 1 (non-default maxExprDepth). - Encoder: save/restore ctx.maxExprDepth around each top-level statement. Applies the cap so expression children of that statement use the reduced depth budget, making the output visually varied. - Decoder: mirrors encoder with a new max-depth-restore work item pushed before pushStmtChildren. Restores maxExprDepth after all children of the top-level statement are processed (LIFO stack ensures ordering). - cosmeticIdent(): boost scope-reuse probability from 1/3 to 2/3. When in-scope variables exist, declared names now appear in identifier positions significantly more often, improving code naturalness. - analyze-corpus.ts: add identifier reference ratio computation. Outputs __identRefRatio__ to corpus-weights.json (ready for next scraper run), measuring what fraction of expression nodes are Identifier references — useful for calibrating the weight boost. - Fix decode.ts: new max-depth-restore WorkItem type + handler. - Test: update 'different depths produce different output' to use depths [1, 5, 10, 20, 50] — depth=1 always differs from depth>=3 due to different block-filtering at blockDepth=0.
- Add measureMaxExprDepth(js) helper that walks the parsed AST and returns the max expression nesting depth, mirroring the encoder's depth counting (0 = direct expression child of a statement, +1 per recursion into expression children). - Add 'output expression depth does not exceed maxExprDepth' test with 6 representative cases across depths 1–20. - Add depth assertion to 'lorem roundtrips with depth 64'. - Fix: add hard expression depth cap in filterCandidates. Previously, at exprDepth >= maxExprDepth the pool still contained non-leaf candidates (20/32 table slots), so the encoder could select a non-leaf and produce cosmetic (padLeaf) children at depth+1 — causing the AST to exceed maxExprDepth by one level. The new hard filter (exprDepth >= maxExprDepth && children.length > 0 → exclude) restricts the pool to leaves only at the cap, so the AST depth is bounded exactly by maxExprDepth. Decoder already stopped recursing at depth >= maxExprDepth, so this change is consistent and requires only snapshot updates.
- Both READMEs: replace 'expression children become cosmetic' with accurate description (only leaf expressions available at cap) - root README: 67 → 68 tests - packages/core README: test count already correct after previous commit - encode.ts: update stale comment on cosmeticChildren (now dead path) - CONTEXT.md: fix stale default (20 → 1) and cosmetic-children description
Replace the single Identifier:corpus candidate (which always occupies 1/N of leaf table slots regardless of weight) with dynamic Identifier:scope:i candidates — one per typedScope entry. As scope grows, scope-referencing candidates flood the table: with N=10 entries, P(scope ref) ~ 10/(base+10) vs the previous 1/(base+1). - context.ts: rename Identifier:0 to Identifier:corpus (variant=-1, filtered out when scope non-empty). Dynamically add Identifier:scope:i for each typedScope[i] with per-var weight = corpus_base / N (normalized). Depth scaling applied to scope variants identically to other leaves. Remove old flat +0.5*N boost (irrelevant due to bijective table mechanics). - encode.ts: Identifier:scope:i emits typedScope[i].name (deterministic). Identifier:corpus (variant=-1) picks from corpus idents via cosmeticCorpusIdent(). - decode.ts: Identifier node looks up name in typedScope via findIndex and returns Identifier:scope:i; if not found, returns Identifier:corpus. Use actual AST variable names (not nameFromHash) for typedScope entries so findIndex matches even after consistent identifier renaming. - scripts/analyze-corpus.ts: add per-bucket identRefRatio computation. - test: update randomize-all-names test to use consistent renaming (same old name maps to same new name everywhere), matching the invariant that consistently-renamed JS still decodes identically. Update snapshots.
…ifier roundtrips The decoder's scope-save for ArrowFunctionExpression, FunctionExpression, and FunctionDeclaration was using nameFromHash(hash, 900+i) for both scope (dedup tracking) and typedScope (findIndex lookup). When the generated JS is re-parsed after consistent identifier renaming, param names in the AST differ from the nameFromHash-generated names. findIndex then fails to locate them, returning Identifier:corpus instead of the correct Identifier:scope:i, corrupting bits. Fix: use actual AST param names (n.params.map(p => p.name)) for typedScope while keeping nameFromHash names for the scope array (dedup tracking only). This way findIndex correctly resolves scope-variant identifiers regardless of rename.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major output quality improvements. Each encoded message now resembles a real minified JS module with varied imports, declarations, and exports.
Statement-only tables
Shallow expression default
Trimmed expensive candidates
Other fixes
Sample output (seed 0, "attack at dawn")
```js
import select from "expo-random";
var _quq=((select++))(typeof (25),({}),(select>>>null));
switch(((select&=((select,false))))){}
import{__extends,NoBlending,convertToTexture,RedFormat}from "tslib";
import{toInt,_has}from "url";
import{constructNow,isArray,getDefaultOptions}from "minimatch";
...
```
Breaking change
Entirely new encoding format due to candidate pool changes.
Test plan
🤖 Generated with Claude Code