bloblang(v2): Add V2 language runtime and migration path#436
Open
Jeffail wants to merge 23 commits into
Open
Conversation
Adds the design specification for Bloblang V2 under internal/bloblang2/spec/, split across thirteen numbered chapters (overview, type system, expressions, control flow, maps, imports, execution model, error handling, special features, grammar, common patterns, implementation guide, standard library) plus a top-level PROPOSAL.md and README.md. The spec is the source of truth for the Go and TypeScript runtimes and the V1 -> V2 migrator that follow.
Adds the syntax half of the Go runtime under internal/bloblang2/go/pratt/syntax/: a hand-written scanner, a Pratt parser producing an AST, a name resolver, a post-parse optimizer pass, and a V2 pretty-printer used by the migrator's output stage. Includes parser and scanner unit tests plus go-fuzz harnesses (FuzzParse, FuzzScan) with a seed corpus.
Adds the runtime half of the Go implementation under internal/bloblang2/go/pratt/eval/: a tree-walking interpreter with a small opcode dispatch path for methods and functions, a variable stack with slot allocation handled by the resolver, message-context support, and the V2 standard library covering arithmetic, collections, string handling, lambdas, strftime, and message-coupled operations. Tested against the eval-side unit suite (interp, stdlib, strftime, argument folding). Spec conformance is exercised separately once the spectest runner and corpus land.
Adds internal/bloblang2/go/spectest/, a runner that reads the YAML spec test corpus, executes each case against a configurable interpreter, and produces structured pass/fail reports. Provides the schema for spec tests, typed-value support so YAML can carry V2-typed inputs/outputs, and a compare layer that distinguishes exact-equality assertions from error-shape assertions. The runner is shared by both the Go interpreter tests and (via the TypeScript port) the TS interpreter tests.
Adds internal/bloblang2/spec/tests/, the YAML corpus of conformance cases that anchors both runtimes to the V2 spec. Tests are organised by topic: access, case_studies, control_flow, edge_cases, error_handling, imports, input_output, lambdas, maps, operators, optimizations, stdlib, types, variables. Each case is executed via the spectest runner; both the Go and TS runtimes are required to pass the full corpus.
Adds the top-level internal/bloblang2 entry point: bloblang2.go exposes the runtime to other internal packages, benchmark_test.go provides a corpus-wide V1 vs V2 benchmark harness, and the Taskfile ties together Go, TypeScript, and tree-sitter tests behind a unified action-first surface (build, test, demo, clean). Also includes the PARITY.md V1 method tracking table, the package README, and REMAINING.md.
Adds an LSP server under internal/bloblang2/go/lsp/ (with a small bloblang2-lsp binary) that exposes the V2 parser/resolver as diagnostics and completions over the standard JSON-RPC protocol. Ships a matching Neovim plugin under internal/bloblang2/plugins/nvim/ that wires .blobl2 filetype detection, syntax highlighting via the tree-sitter grammar, and the LSP client.
Adds internal/bloblang2/tree-sitter/, a tree-sitter grammar for V2 syntax with corpus tests, plus the syntax highlighting query used by the Neovim plugin. Also adds internal/bloblang2/demo/, a small Go-served web playground with a Monaco-style editor, a case-study dropdown of real-world mappings, an engine selector for switching between server-side and browser-side execution (the latter via the TypeScript runtime), and syntax highlighting backed by tree-sitter.
Adds internal/bloblang2/ts/, a full TypeScript port of the V2 runtime: scanner, parser, resolver, optimizer, tree-walking interpreter, value system, and standard library. Bundles to a browser-loadable script that powers the demo's client-side engine. The TypeScript runtime passes the same spec conformance corpus (internal/bloblang2/spec/tests/) as the Go runtime, exercised via a small spectest harness layered over the same YAML schema.
Adds internal/bloblang2/migrator/bloblang_v1_spec.md, a reference specification for Bloblang V1 derived from the existing V1 implementation and tightened via adversarial review and a test-driven verification pass. This is the source-of-truth document the migrator's translator rules are written against. Adds internal/bloblang2/migrator/v1spec/, a V1 conformance test corpus organised by topic (access, case_studies, control_flow, edge_cases, error_handling, imports, input_output, lambdas, maps, operators, optimizations, stdlib, types, variables) together with a runner that exercises the V1 corpus against internal/impl/pure for typed-numeric coverage.
Adds internal/bloblang2/migrator/v1ast/, a hand-written scanner, parser, and AST for Bloblang V1, plus a printer used to round-trip V1 source through the migrator. The package preserves comments and blank-line trivia so the V1 -> V2 translation pipeline can emit V2 source that retains the V1 author's formatting intent. This is the V1 front-end consumed by the translator package that follows.
Adds internal/bloblang2/migrator/translator/, the core V1 -> V2 translation pipeline. The translator consumes V1 AST produced by v1ast, walks expressions and statements applying targeted rewrite rules (methods, imports, control flow, lambdas, mapping mode), and emits V2 source via the syntax printer while preserving trivia. The package layers its rules so behaviour-equivalent rewrites are applied directly and known V1 -> V2 semantic divergences are flagged as SemanticChange entries on the change report rather than silently rewritten. A corpus regression test plus per-rule, property, and spec-coverage tests pin behaviour.
Adds internal/bloblang2/migrator/benchmark/, a corpus-wide V1 -> V2 migration benchmark suite with a coverage probe and migration smoke test that quantifies translator coverage and runtime parity against a curated V1 corpus. Adds internal/bloblang2/migrator/demo/, a small Go-served web playground that wires the migrator behind a UI with real V1 case studies. Useful for eyeballing translator output and for showing the behaviour of flagged SemanticChange entries.
Adds public/bloblangv2/, the exposed Go surface for V2: Environment, Executor, MessageContext, plugin registration for methods and functions with parse-time argument folding, ParseError plumbing, parameter and spec types, and a View layer over the schema for external consumers. The package is the integration point used by public/service (and any downstream Benthos plugin) to register V2 plugins, parse mappings, and execute them against message contexts.
Adds public/bloblangv2/migrator/, the public-facing wrapper around the internal translator. It exposes a stable API for migrating a single Bloblang mapping (or a set of mappings sharing an import graph) from V1 to V2, surfaces structured Change entries (rewrites and flagged SemanticChange divergences), and lets callers register extra rules via the Options surface for plugin-specific rewrites. This is the layer consumed by the upcoming public/service config migrator, and by external tools that just need mapping-level translation without pulling in internal/.
Threads public/bloblangv2 through the Benthos framework so V2
mappings can be parsed, linted, and executed alongside V1 in
existing pipelines:
- internal/bundle, internal/manager: NewManagement now carries a
BloblV2Environment alongside the V1 BloblEnvironment, with a
matching OptSetBloblV2Environment option.
- internal/docs: LintBloblangV2Mapping lints a field as V2,
side-effect-free using the configured V2 environment.
- internal/cli, internal/cli/studio, internal/cli/test,
internal/config/schema, internal/stream/manager: V2 envs are
plumbed through CLI, studio sync, config schema enumeration,
and the stream manager.
- public/service: a bloblangv2 batch processor, a
config_bloblangv2 field type, schema and linter integration,
plus the corresponding additions to Environment, StreamBuilder,
ResourceBuilder, StreamConfigLinter, and ComponentConfigLinter.
Adds config/test/bloblang/ YAML fixtures covering the V2 batch
processor's golden path, filter behaviour, and metadata reset
semantics.
Adds internal/impl/pure/processor_bloblang_v2.go and processor_bloblang_v2_file.go, the runtime processors that execute V2 mappings inside a Benthos pipeline. The _v2 processor takes a mapping inline; the _file processor reads it from disk, which is how the public/service config migrator emits long mappings. Both register against the public/service registry and parse via the configured public/bloblangv2 environment, so they pick up plugin methods and functions registered elsewhere in the binary (including the V1 stdlib parity ports).
Adds V2 implementations of V1 stdlib methods and functions,
registered against the global public/bloblangv2 environment via
init() side-effects. Pure (deterministic) helpers live under
internal/impl/pure/bloblangv2_*.go; impure (random, time-based)
helpers live under internal/impl/io/bloblangv2_*.go.
Coverage:
- pure: arrays, crypto (AES, hashes), encoding (JSON, YAML, CSV,
base64, JSON schema, URLs), numbers (bitwise, log family, min,
max), objects, parsing (parse_json, format_json, escapes),
regex (replace, replace_many, find/find_all variants), string
(case, trim, replace, hash, filepath, uuid_v5), time
(deterministic timestamp formatting and arithmetic).
- io: ids (uuid_v7, ksuid, nanoid), time (timestamp_unix variants
that touch the wall clock).
Each port mirrors the V1 method or function signature so the V1 ->
V2 translator can rewrite call sites with no semantic change. The
internal/bloblang2/PARITY.md table tracks remaining gaps.
Adds public/service/migrator/, a config-level migrator that walks parsed Benthos YAML configurations, locates bloblang fields and the bloblang processor (including instances nested inside switch, branch, processor_resources, and cache_resources), and applies the public/bloblangv2 mapping migrator to each. Includes a lazy import resolver so transitively-imported V1 mapping files are translated once and rewritten in place, a from-only rule that guards against silently broken rewrites, support for emitting the bloblang_v2_file processor for file-backed mappings, a structured report API surfacing per-field outcomes, and pluggable rules for plugin-specific rewrites. Integration tests exercise multi-config migration, diamond imports, mixed V1/V2 configs, and partial-failure behaviour.
Adds the google/uuid dependency required by the V2 standard library (uuid_v5). Restructures taskfiles/test.yml so the default unit and unit-race tasks pass -short and skip long-running corpus and benchmark tests, keeping the per-PR loop under a minute. Adds a new unit-full task (alias ut-full) with a longer timeout that runs the full suite, including the migrator corpus, fuzz seeds, and benchmark smoke tests.
Contributor
|
That's cool! Curious to play with the language server. Will this also have a And, a small pet peeve: do you think you could shoehorn bitwise math operators in there? :) |
Collaborator
Author
I think instead of mutation I'm just going to work on a follow up suite of optimizations so that the user doesn't need to choose between the two modes. Most of the performance gain of mutation can be achieved with better COW mechanisms which we're already playing with under the hood in v2. The other stuff can also come later, not 100000% sure on the bitwise stuff but because we no longer have the pipe operator for coalescing it's possible. |
Contributor
|
Here's what my claude found: |
Contributor
|
Should this PR include the |
Migrator correctness:
- Surface Info-severity semantic-change notes by default; only benign idiom
rewrites stay gated behind Verbose.
- .slice byte-vs-codepoint is flagged only when the receiver may be a string;
a statically-known non-string (array/object) receiver is not flagged, and
string-returning chains (.reverse()/.slice()) still are.
- Drop the .contains "type-strict" flag: V2 numeric equality uses promotion
(1 == 1.0), so there is no divergence — the warning was factually wrong.
- Translate object .map_each faithfully. V1 binds each entry as {key, value}
and replaces the value; V2 has no drop-in equivalent, so an object-literal
receiver is rewritten to .map_entries with the entry rebound via .into
(bodies that drop entries via deleted() are flagged Unsupported). A
dynamic receiver emits .map with an accurate {key,value}-vs-value warning.
- Flag the |/.or coalesce rewrite duplicating a nondeterministic/side-
effecting fallback (evaluated twice on the error path). Impurity is derived
from the V1 function registry plus a curated set of pure-but-
nondeterministic builtins, recursing into lambda bodies and if/match/map
sub-expressions. An IsNondeterministicFunc hook lets a distribution declare
its own plugin functions (e.g. Connect's snowflake_id).
- Preserve V1 .map_each "nothing keeps the element" by synthesising an else
branch for an else-less body if (element for arrays, <param>.value for
objects), including the discard-param form.
- Treat V1 .sort(<comparator>) as Unsupported (V2 has no comparator sort):
emit a best-effort ascending .sort() instead of nonsense input.left/right.
- Functions with no V2 equivalent (error_source_*(), json()) are recorded
Unsupported rather than passed through as a Note that counts Exact — a
perfect coverage ratio must not hide V2 that will not compile.
- Fix coverage double-counts: a Rewritten/Unsupported flag that returns nil
to reuse the 1:1 shape no longer also tallies Exact.
Public API insulation & ergonomics:
- Replace the public alias types with wrapper structs converted at the
Migrate boundary, so internal translator struct changes don't break the API.
- Thread context.Context through Migrate and FileResolver (import resolution
honours cancellation); a nil context defaults to Background rather than
panicking.
- Register{,Method,Function}Rule return error (+ Must* variants).
- service/migrator Options.Environment overrides the component registry (a
zero-value Environment falls back to the global instead of panicking), and
Options.BloblangIsNondeterministicFunc is hoisted for discoverability.
Parser:
- Bound expression nesting (maxParseDepth) so pathological input fails fast;
on exceeding the limit the parse is marked fatal and drained to EOF so the
array/object element loops can't spin on unbalanced deep input.
- Memoize the per-'(' lambda-ahead verdict so the parenthesis lookahead is
amortized O(1): a valid deeply-parenthesised mapping now parses in linear
time instead of O(depth x tail) (~600KB input: 6.8s -> 0.07s).
Adds regression tests throughout (adversarial review across purity, parser,
coverage accounting, method semantics, the public boundary, and test integrity;
each surviving finding verified against both engines and the spec before fixing).
Collaborator
Author
|
Thanks for the reviews — pushed @squiidz: worked through the audit.
@JakeSCahill: the Known limitation: dynamic-receiver object |
V1 .fold(init, <query>) accepts the accumulator body in query form, where
`this` is the {tally, value} context (e.g. `.fold({}, this.tally.merge(...))`),
in addition to the explicit-lambda form. The migrator only handled the lambda
form; the query form fell through to a 1:1 translation that emitted
`.fold(init, <non-lambda expr>)`, which V2 rejects at exec ("fold() second
argument must be a lambda").
Refactor the fold-context rewrite into a shared walk with two matchers — the
named-param form and a new `this`-rooted query form — so `this.tally` /
`this.value` become the bare tally/value names the V2 (tally, value) lambda
binds. A body referencing the context as a whole object (bare `this` / bare
param, or a field other than .tally/.value) has no V2 equivalent and is now
recorded Unsupported rather than emitting a broken best-effort .fold.
Add a dual-engine regression suite (query_form_validity_test) covering the
query form of every lambda/query-accepting method: each must compile and run
on the real V2 environment and match the V1 engine, or be flagged Unsupported.
It validates against bloblangv2.GlobalEnvironment (what migrated mappings
actually run on), not a bare interpreter that registers only a subset of the
stdlib.
Contributor
|
…ne divergence sweep
Two batches of migrator/runtime work sitting on top of the earlier review fixes.
Review findings (correctness + runtime safety):
- mapping/mutation root semantics were inverted: V1 `mapping` (MapPart) starts
root empty, `mutation` (MapOnto) starts from the input, but the `output = input`
prelude was attached to the wrong mode — so migrated mappings could leak input
fields and migrated mutations could drop the doc. Prelude moved to ModeMutation,
with the doc comments and tests that encoded the wrong model corrected, and the
corpus/smoke harnesses aligned to ModeMapping. New mode_semantics_test.go asserts
the data behaviour (no leak / no drop).
- `match S {…}` now rebinds `this` -> subject in patterns and bodies (emits
`match S as __match {…}` and rewrites equality patterns), but only when a case
actually references `this`. New match_rebind_test.go.
- bounded `.repeat`/`range` output so a huge count can't OOM, and Run's recover now
converts any escaping panic to a message-level runtime error (defence in depth).
New stdlib_bounds_test.go.
- WithoutMethods/WithoutFunctions now honour the removal set at both parse and
runtime (plugins were being re-added after the delete). New without_plugins_test.go.
Dual-engine divergence sweep (differential V1-vs-V2 testing against the real V2
GlobalEnvironment, not the bare interpreter subset):
- `.trim(cutset)`: V2 was silently ignoring the arg — now rejects it (matching the
self-validating-arity convention of contains/char/range), and the migrator drops
the cutset and warns. `.trim()` no-arg is faithful.
- `.map(fn)` (whole-value) rewritten to V2 `.into(fn)`; `.format_json()` emits an
explicit 4-space indent to match V1's default. Both verified equivalent.
- flag genuine divergences the translator can't paper over: dot-path args to
get/exists/without, `.bool()`/`.number()` coercion + lost default fallback,
`.round()` half-away-from-zero vs banker's, `.all([])` vacuous truth,
`.contains()` on an object receiver, and `match` without a wildcard. All surface
at Warning severity in default (non-verbose) reports.
- new dual-engine harness + suites: dualengine_test.go (shared runner),
dual_language_test.go (~75 confirmed-equivalent cases), dual_divergence_test.go
(8 flagged-divergence assertions).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces Bloblang V2 — a redesigned mapping language — as a parallel runtime alongside the existing V1 implementation, together with a V1 → V2 migration path for both individual mappings and full Benthos YAML configs.
V2 is reached through opt-in surfaces (
bloblang_v2/bloblang_v2_fileprocessors, a separate V2 plugin registry, dedicated lint rules). V1 remains untouched: no V1 API is removed, no V1 processor is renamed, and no schema-level changes affect V1-only pipelines.The branch is organised into 20 topical commits — reviewers can navigate by subsystem rather than by chronological development.
What's included
internal/bloblang2/spec/, the source of truth for both runtimes.internal/bloblang2/go/pratt/.internal/bloblang2/ts/, passing the same spec conformance corpus as the Go runtime.SemanticChangeentries rather than silently rewriting, and a corpus-wide migration benchmark.public/bloblangv2(plugin registration, parsing, execution) andpublic/service/migrator(config-level migration with diamond-import handling, mixed V1/V2 configs, and partial-failure reporting).internal/impl/{pure,io}/bloblangv2_*.go.internal/manager,internal/bundle, the CLI, studio sync, the config schema, and the public/service surface (Environment, StreamBuilder, ResourceBuilder, lint, schema).Coexistence with V1
V2 and V1 maintain separate plugin registries; plugins registered against one are not visible to the other. Host components select a language per field — a
bloblangfield uses V1 and is linted via the V1 path; abloblang_v2field uses V2 and is linted viaLintBloblangV2Mapping. The two processor types operate side by side in the same pipeline.One known gap: interpolated string fields (the
${! ... }form) still dispatch through the V1 environment only. This and other deferred items (custom V2 lint surface, V1 ↔ V2 plugin bridge) are tracked ininternal/bloblang2/REMAINING.md.For documentation authors
The full V2 language specification lives at
internal/bloblang2/spec/, organised into 13 chapters covering lexical structure, type system, expressions, control flow, maps, imports, the execution model, error handling, special features, formal grammar, common patterns, an implementation guide, and the standard library reference. The spec README provides a guided table of contents and is the recommended entry point.Per-method V1 → V2 status, including any semantic shifts (e.g. variadic arguments folded into arrays, the new error-object shape, deferred batch-3 items), is tracked in
internal/bloblang2/PARITY.md.