This document is normative. Together with rules.proto, conformance.proto,
features.md, the reference bundle and its conformance suite, it is sufficient
to implement an engine without reading the Go code of the compiler.
Structural format version: 1.
| Enum value | Meaning |
|---|---|
VALUE_TYPE_STRING |
A possibly absent sequence of Unicode code points. |
VALUE_TYPE_INTEGER |
A checked signed 64 bit integer that may be indeterminate. |
VALUE_TYPE_BOOLEAN |
A predicate result. |
VALUE_TYPE_CANONICALIZATION_STEP |
A transformation applied to the current value. |
VALUE_TYPE_ASSERTION |
An ordered format check. |
VALUE_TYPE_CHECKSUM_OUTCOME |
A tri-state checksum result. |
VALUE_TYPE_UNSPECIFIED is refused at load time, like every other zero enum value.
A string view may be absent. Absence propagates: every string constructor
applied to an absent operand yields an absent result. Every predicate applied to
an absent operand yields false, except IS_ABSENT, which yields true.
Absence is never an error and never an exception.
An integer expression is indeterminate when it cannot be evaluated: an absent
or empty operand, a code point outside the mapping domain anywhere in the view,
even at a position no weight pairs with, an index outside a
remainder table, or a complement operand outside [0, modulus]. An indeterminate
integer propagates through every integer operation and makes the enclosing
checksum node evaluate to unsupported with unsupported_checksum. It never
produces invalid: refusing a valid identifier is the most serious defect of the
project.
A Program is a typed acyclic graph serialized in topological order. Every
Node.input_nodes entry references a strictly lower index of the same program.
root_node designates the result of the program.
| Program kind | Accepted root | Accepted operation categories |
|---|---|---|
PROGRAM_KIND_CANONICALIZATION |
CANONICALIZATION_OP_KIND_SEQUENCE |
string (except SUBJECT), predicate, canonicalization |
PROGRAM_KIND_FORMAT |
ASSERTION_OP_KIND_SEQUENCE |
string, predicate, assertion, CALL_OP_KIND_FORMAT |
PROGRAM_KIND_CHECKSUM |
any node producing a checksum outcome, never a WHEN branch |
string, predicate, integer, checksum, CALL_OP_KIND_CHECKSUM |
Program.subject_node, when present, produces subject() for a top level
invocation. A canonicalization program never declares a subject and never
declares a capture. Program.captures names nodes of a format program; a capture
reference is lowered to a direct node reference, so the list is metadata used for
diagnostics and for the per format capture limit.
Nodes are re-evaluated at every reference: no memoization is observable. This
matters inside a canonicalization program, where value() designates the value
current at the moment the enclosing step runs. Each node evaluation, each
canonicalization step application and each program invocation consumes one unit
of the evaluation budget of 100000 steps. In addition, an operation that
produces a string, and every canonicalization step, is billed one further
unit per started slice of 64 produced code points. That charge is what
bounds the memory a bundle can make an engine allocate: the total number
of code points materialized by one public operation can never exceed the
budget times 64. Exhausting the budget is an engine error.
The budget bounds an interpretation. An engine that compiles the rules into native code ahead of time does not count steps: the graph is acyclic and the call depth is bounded, so the emitted code terminates by construction, and the bounds below are what its generator enforces instead. An engine that walks the IR at validation time applies the budget as written. Section 2.4 of the specification binds observable results, never the strategy that produces them.
A generator that inlines an operand where the graph reads it more than once must bound what it emits, and the budget above is that bound: a generated program may hold at most 100000 operation instances, counted after inlining.
The count is what a generator emits, and three things follow from that.
It starts at the emission roots and follows operands, so a node no root reaches costs nothing: a generator does not emit dead code.
The roots are the program root, the subject_node when the program declares
one, and every capture no other root already reaches. A capture any root
reaches is not a second emission: it is emitted inside that root's
expression, and counting its subtree again charges it twice. This holds for
a capture reached by another capture, not only by the program root.
Check 14 counts from the roots that check 15 validates, so an engine may run
15 first. Both answer invalid_ruleset, so the order is not observable, and
stating it would constrain an implementation without changing an answer.
Take the captures from the highest index down. An operand always sits at a lower index than the node reading it, so a capture reached by another is seen after the one reaching it and one pass settles it. Walking the capture list in its own order makes the count depend on how the captures happen to be listed, which is not an observable property of the bundle.
Their costs are summed, because a generator emits all of them. Checking each root separately would let a program carry any number of roots just below the ceiling.
None of this is theoretical. Two engines counted every node rather than the reachable ones and answered differently on the same bundle; two others then summed every capture and reported 3204 instances for the published bundle where it holds 3069, because its 54 captures are all reached from their roots. No conformance case could see either divergence: both readings agree on every rule an author would write.
A CALL counts as one instance. The callee is a separate program, emitted
once and reached by a function call, so its own instances are bounded on
its own.
The arithmetic saturates rather than wrapping. A chain two hundred levels deep reaches 2^201 instances, and an accumulator that overflows lands on a small number that passes: the overflow is the shape of the attack, not an edge case.
The bound is not a formality. The node count of a program is bounded, but the graph is a DAG, and a DAG whose every node reads the previous one twice expands exponentially while passing every one of the 25 load checks. Without a bound, such a bundle is a denial of service against the generator rather than against the engine, and those 25 checks would not see it.
Choosing the budget rather than a new number is deliberate: a generated
program may not carry more instances than an interpreter would have taken
steps to run it once. A generator that shares a repeated operand instead of
inlining it stays free to do so, provided the sharing preserves the short
circuit of ALL, ANY and of the assertion sequence, which stop at the
first decisive operand.
CALL_OP_KIND_FORMAT runs a format program with the operand as its subject and
propagates the callee reason code and message key unchanged.
CALL_OP_KIND_CHECKSUM runs a checksum program with the operand as its subject
and propagates its outcome unchanged. In a called program, value() still
designates the canonical value of the identifier under validation.
The call graph must be acyclic, must only connect programs of the same kind, and its static depth must be at most 32.
Output: VALUE_TYPE_STRING.
Operands: none.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1).
Yields the constant text. Constants are UTF-8 and at most 4096 bytes long. A string literal in the surface language lowers to this operation.
Surface syntax: value().
Output: VALUE_TYPE_STRING.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1).
In a canonicalization program, yields the value current at the moment the enclosing step runs. In a format or checksum program, yields the canonical value of the identifier under validation. Never absent.
Surface syntax: subject().
Output: VALUE_TYPE_STRING.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1).
Yields the subject of the enclosing program: the caller supplied view for a called program, otherwise Program.subject_node, otherwise the canonical value. Forbidden in canonicalization programs. May be absent when the caller supplied an absent view.
Surface syntax: country_code().
Output: VALUE_TYPE_STRING.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), IDENTIFIER_DISPATCH_V1 (5).
Yields the canonical country code of the selected dispatch target, or the absent string for a GLOBAL target.
Surface syntax: slice(expr, start, end).
Output: VALUE_TYPE_STRING.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: start (required), end (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Yields the code points of expr in [start, end). Absent when expr is absent, when start > end or when end exceeds the length of expr.
Surface syntax: slice_from(expr, start).
Output: VALUE_TYPE_STRING.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: start (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Yields the code points of expr from start to the end. Absent when expr is absent or when start exceeds the length of expr.
Surface syntax: slice_to(expr, end).
Output: VALUE_TYPE_STRING.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: end (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Yields the code points of expr before end. Absent when expr is absent or when end exceeds the length of expr.
Surface syntax: before_first(expr, delimiter).
Output: VALUE_TYPE_STRING.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Yields the part of expr before the first occurrence of the non empty constant text. Absent when expr is absent or when text does not occur.
Surface syntax: after_first(expr, delimiter).
Output: VALUE_TYPE_STRING.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Yields the part of expr after the first occurrence of the non empty constant text. Absent when expr is absent or when text does not occur.
Surface syntax: strip_prefix(expr, prefix).
Output: VALUE_TYPE_STRING.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Yields expr without its exact leading text. Absent when expr is absent or does not start with text.
Surface syntax: concat(expr...).
Output: VALUE_TYPE_STRING.
Operands: then at least 1 and at most 256 repeated expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
Concatenates its operands in order. Absent when any operand is absent.
Surface syntax: digits_to_integer(expr).
Output: VALUE_TYPE_INTEGER.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Reads expr as a non negative decimal integer. Indeterminate when expr is absent, empty or contains a non ASCII digit. Accepted only when the compiler proves the maximum length of expr is at most 18 code points.
Surface syntax: mod_digits(expr, modulus).
Output: VALUE_TYPE_INTEGER.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: modulus (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Computes the remainder of expr modulo modulus, digit by digit, without any big integer conversion. Indeterminate when expr is absent, empty or contains a non ASCII digit. The result lies in [0, modulus).
Surface syntax: weighted_sum(expr, weights, alignment, mapping[, alphabet]).
Output: VALUE_TYPE_INTEGER.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: weights (required), alignment (required), mapping (required), alphabet (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30), CHECKSUM_WEIGHTED_V1 (33).
Sums mapping(expr[i]) * weight(i) over the paired positions. LEFT pairs position i with weights[i], RIGHT pairs the last position with the last weight, and CYCLE pairs position i with weights[i mod len(weights)]. LEFT and RIGHT only pair min(len(expr), len(weights)) positions; the remaining positions of expr contribute nothing. Indeterminate when expr is absent, empty, or contains a code point outside the mapping domain. CUSTOM_ALPHABET takes the value of a code point from its index in alphabet, which is required by that mapping and forbidden by the others. The alphabet holds between 1 and 256 code points and lists none of them twice: a repeated code point would carry two values, and which one an engine returned would depend on how it searched, which is how two conformant engines disagree without either being wrong.
Surface syntax: modulo(int_expr, modulus).
Output: VALUE_TYPE_INTEGER.
Operands: int_expr of type VALUE_TYPE_INTEGER.
Parameters: modulus (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Euclidean remainder of int_expr modulo modulus. The result always lies in [0, modulus). Indeterminate when the operand is indeterminate.
Surface syntax: complement(int_expr, modulus).
Output: VALUE_TYPE_INTEGER.
Operands: int_expr of type VALUE_TYPE_INTEGER.
Parameters: modulus (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Yields modulus - int_expr. Indeterminate when the operand is indeterminate or outside [0, modulus].
Surface syntax: remainder_map(int_expr, values).
Output: VALUE_TYPE_INTEGER.
Operands: int_expr of type VALUE_TYPE_INTEGER.
Parameters: remainder_values (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Yields remainder_values[int_expr]. Indeterminate when the operand is indeterminate or outside [0, len(remainder_values)).
Surface syntax: is_empty(expr).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and holds zero code points. False when expr is absent.
Surface syntax: is_absent(expr).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), STRING_VIEWS_V1 (10).
True when expr is absent. This is the only predicate that observes absence as true.
Surface syntax: equals(left, right).
Output: VALUE_TYPE_BOOLEAN.
Operands: left of type VALUE_TYPE_STRING, right of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when both operands are present and hold the same code point sequence. False when either operand is absent.
Surface syntax: length_eq(expr, n).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: length (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and holds exactly length code points.
Surface syntax: length_in(expr, [n...]).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: lengths (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and its code point length belongs to lengths. lengths is sorted, deduplicated and non empty.
Surface syntax: length_between(expr, min, max).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: min_length (required), max_length (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and its code point length lies in [min_length, max_length]. The compiler requires min_length <= max_length.
Surface syntax: ascii_digits(expr).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2).
True when expr is present, non empty and made only of U+0030..U+0039.
Surface syntax: ascii_upper_letters(expr).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2).
True when expr is present, non empty and made only of U+0041..U+005A.
Surface syntax: ascii_alphanumeric(expr).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2).
True when expr is present, non empty and made only of U+0030..U+0039 or U+0041..U+005A.
Surface syntax: ascii_charset(expr, chars).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2).
True when expr is present, non empty and every code point belongs to the non empty ASCII set text. text is deduplicated and sorted by code point by the compiler.
Surface syntax: starts_with(expr, prefix).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and starts with the non empty constant text.
Surface syntax: ends_with(expr, suffix).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and ends with the non empty constant text.
Surface syntax: prefix_in(expr, prefixes).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: values (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and starts with at least one element of values. values is non empty, sorted and deduplicated by the compiler, and every element is non empty. Every element has the same length, and a bundle mixing lengths is refused: over one sorted list of mixed lengths, a search for the greatest element not after expr answers wrongly rather than slowly, since ["AB", "ABA"] against "ABCD" finds ABA, which is not a prefix, while AB is. At one length, starting with an element is equalling its opening of that length, so the search is exact. Mixed lengths are written as one prefix_in per length under an any.
That refusal takes the evidence with it. Every prefix_in a bundle may now carry holds one length, so no conformance case can distinguish a search run per length from one run over the whole table: the shape that separates them is the shape the loader refuses. An engine MUST therefore pin the semantics below its loader, by a native test comparing its search against the definition transcribed literally -- some element is a prefix of the subject -- over tables of mixed lengths. This is the second rule the corpus cannot carry, alongside invalid_encoding, and for the same kind of reason: what makes a case expressible and what makes a rule worth stating are not the same thing.
The unit is bytes because the search is over bytes, and an engine working in another unit may group more finely without contradicting this: two elements of the same byte length can differ in code points, since PZ and é are both two bytes and are not both two code points. A finer grouping refuses nothing this accepts. No conformance case separates the two readings, because every element of the published bundle is ASCII, where they agree.
Surface syntax: char_at_in(expr, index, chars).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: index (required), text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present, index is a valid code point position and the code point at index belongs to the non empty ASCII set text.
Surface syntax: contains(expr, literal).
Output: VALUE_TYPE_BOOLEAN.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when expr is present and contains the non empty constant text.
Surface syntax: all(predicate...).
Output: VALUE_TYPE_BOOLEAN.
Operands: then at least 1 and unbounded repeated predicate of type VALUE_TYPE_BOOLEAN.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when every operand is true. Operands are evaluated in order and evaluation stops at the first false operand.
Surface syntax: any(predicate...).
Output: VALUE_TYPE_BOOLEAN.
Operands: then at least 1 and unbounded repeated predicate of type VALUE_TYPE_BOOLEAN.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
True when at least one operand is true. Operands are evaluated in order and evaluation stops at the first true operand.
Surface syntax: not(predicate).
Output: VALUE_TYPE_BOOLEAN.
Operands: predicate of type VALUE_TYPE_BOOLEAN.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
Negates its operand.
Surface syntax: profile_is(name).
Output: VALUE_TYPE_BOOLEAN.
Operands: none.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), PROFILES_V1 (21).
True when the effective validation profile equals text, which is either compatible or strict_current.
Surface syntax: integer_is(int_expr, constant).
Output: VALUE_TYPE_BOOLEAN.
Operands: int_expr of type VALUE_TYPE_INTEGER.
Parameters: constant (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30), CHECKSUM_INTEGER_PREDICATE_V1 (35).
True when int_expr equals the literal constant. It is the only predicate reading an integer, and exists so that a checksum can branch on the value of a remainder: several national registers recompute their weighted sum with a second set of weights when the first remainder reaches a given value. An indeterminate operand yields false, so the branch does not apply and the enclosing CHOOSE falls through.
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: then at least 0 and unbounded repeated step of type VALUE_TYPE_CANONICALIZATION_STEP.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Applies its operands in order to the current value. It is the only accepted root of a canonicalization program.
Surface syntax: trim_whitespace().
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2), CANONICALIZATION_BASIC_V1 (3).
Removes every leading and trailing code point of the frozen whitespace_v1 table.
Surface syntax: remove_whitespace().
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2), CANONICALIZATION_BASIC_V1 (3).
Removes every code point of the frozen whitespace_v1 table.
Surface syntax: uppercase_ascii().
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), ASCII_AND_WHITESPACE_V1 (2), CANONICALIZATION_BASIC_V1 (3).
Maps only a..z to A..Z. Every other code point is preserved.
Surface syntax: remove_chars(list).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Removes every code point belonging to the non empty set text. The compiler deduplicates and sorts text by code point.
Surface syntax: replace_prefix(from, to).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: text (required), replacement (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Replaces the exact leading text by replacement when present. text is non empty and differs from replacement.
Surface syntax: prepend(value).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Inserts the non empty constant text before the current value.
Surface syntax: append(value).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Appends the non empty constant text after the current value.
Surface syntax: insert(index, value).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: index (required), text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Inserts the non empty constant text at code point position index. When index is greater than the current length the step leaves the value unchanged.
Surface syntax: left_pad(length, char).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: length (required), text (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3).
Prepends copies of the single code point text until the value holds length code points. A longer value is never truncated. length is at least 1 and bounded like every other slice bound, so an engine that sizes a buffer from it has a static maximum.
Surface syntax: prepend_country_if_missing().
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: none.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_BASIC_V1 (3), IDENTIFIER_DISPATCH_V1 (5).
Leaves the value unchanged when it starts with one of the accepted_prefixes of the selected target; otherwise prepends the canonical_prefix of the target, or its country_code when no canonical prefix is declared. Forbidden in a pre-canonicalization program and in a canonicalizer referenced by a GLOBAL definition.
Surface syntax: when(predicate, step...).
Output: VALUE_TYPE_CANONICALIZATION_STEP.
Operands: predicate of type VALUE_TYPE_BOOLEAN, then at least 1 and unbounded repeated step of type VALUE_TYPE_CANONICALIZATION_STEP.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CANONICALIZATION_CONDITIONAL_V1 (4).
Evaluates the predicate against the value current at that point and, when true, applies the trailing steps in order.
Output: VALUE_TYPE_ASSERTION.
Operands: then at least 1 and unbounded repeated assertion of type VALUE_TYPE_ASSERTION.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
Evaluates its operands in order and stops at the first failure, whose reason code and message key become the result of the program. It is the only accepted root of a format program.
Surface syntax: require(predicate, reason_code, message_key).
Output: VALUE_TYPE_ASSERTION.
Operands: predicate of type VALUE_TYPE_BOOLEAN.
Parameters: reason_code (required), message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), FORMAT_ASSERTIONS_V1 (20).
Succeeds when the predicate is true. On failure the format step is invalid with reason_code and, when declared, message_key. reason_code is restricted to codes that prove invalidity.
Surface syntax: luhn(expr).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30), CHECKSUM_LUHN_V1 (31).
Applies the Luhn algorithm to expr, whose rightmost digit is the check digit. valid when the weighted sum is a multiple of ten, invalid/invalid_checksum otherwise. Indeterminate, hence unsupported/unsupported_checksum, when expr is absent, shorter than two code points, or contains a non ASCII digit.
Surface syntax: iso7064_mod97_10(expr).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: expr of type VALUE_TYPE_STRING.
Parameters: message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30), CHECKSUM_MOD97_V1 (32).
Expands every ASCII letter of expr to its base 36 decimal value, every ASCII digit to itself, then requires the resulting decimal string to be congruent to one modulo 97. Indeterminate when expr is absent, shorter than three code points, or contains a code point outside 0..9 and A..Z.
Surface syntax: compare_digit(int_expr, string_expr, index).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: int_expr of type VALUE_TYPE_INTEGER, string_expr of type VALUE_TYPE_STRING.
Parameters: index (required), message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Compares int_expr to the ASCII digit of string_expr at code point position index. Indeterminate when the integer is indeterminate, the string is absent, index is out of range or the code point is not an ASCII digit.
Surface syntax: compare_slice(int_expr, string_expr, start, end).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: int_expr of type VALUE_TYPE_INTEGER, string_expr of type VALUE_TYPE_STRING.
Parameters: start (required), end (required), message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Compares int_expr to the decimal value of string_expr[start:end). The compiler requires end - start between one and 18. Indeterminate when the integer is indeterminate, the string is absent, the slice is out of range or holds a non ASCII digit.
Surface syntax: choose(branches...).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: then at least 1 and unbounded repeated branch of type VALUE_TYPE_CHECKSUM_OUTCOME.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Evaluates its branches in order and returns the outcome of the first applicable one. A WHEN branch whose predicate is false is not applicable; any other branch is always applicable. When no branch applies the result is unsupported/unsupported_checksum.
Surface syntax: when_checksum(predicate, checksum_rule).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: predicate of type VALUE_TYPE_BOOLEAN, checksum_rule of type VALUE_TYPE_CHECKSUM_OUTCOME.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Applicable only when the predicate is true, in which case it yields the outcome of its checksum operand. It is accepted only as a direct operand of CHOOSE; its non applicable state is never observable elsewhere.
Surface syntax: all_checks(checksum_rule...).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: then at least 1 and unbounded repeated checksum_rule of type VALUE_TYPE_CHECKSUM_OUTCOME.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Evaluates every operand in order. Returns the first invalid outcome, otherwise the first unsupported outcome, otherwise valid. WHEN is not accepted as an operand.
Surface syntax: any_check(checksum_rule...).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: then at least 1 and unbounded repeated checksum_rule of type VALUE_TYPE_CHECKSUM_OUTCOME.
Parameters: none; any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Evaluates every operand in order. Returns valid as soon as one operand is valid, otherwise the first unsupported outcome, otherwise the first invalid outcome. WHEN is not accepted as an operand.
Surface syntax: unsupported_checksum(reason_code).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: none.
Parameters: reason_code (required), message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30).
Always yields unsupported with reason_code, restricted to unsupported_checksum and checksum_not_published.
Surface syntax: compare_constant(int_expr, constant).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: int_expr of type VALUE_TYPE_INTEGER.
Parameters: constant (required), message_key (optional); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CHECKSUM_TRISTATE_V1 (30), CHECKSUM_COMPARE_CONSTANT_V1 (34).
Compares int_expr to the literal constant. COMPARE_DIGIT and COMPARE_SLICE can only compare against part of the value being checked, so a rule stating that a remainder must equal a fixed number had nothing to compare with. Indeterminate when the integer is indeterminate, which never proves an identifier wrong.
Surface syntax: use_format.
Output: VALUE_TYPE_ASSERTION.
Operands: input of type VALUE_TYPE_STRING.
Parameters: program_id (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CAPTURES_AND_CALLS_V1 (11), FORMAT_ASSERTIONS_V1 (20).
Runs the format program program_id with input as its subject and propagates its reason code and message key unchanged.
Surface syntax: apply_checksum(checksum_reference, string_expr).
Output: VALUE_TYPE_CHECKSUM_OUTCOME.
Operands: input of type VALUE_TYPE_STRING.
Parameters: program_id (required); any other field present in the message is refused.
Capabilities: CORE_GRAPH_V1 (1), CAPTURES_AND_CALLS_V1 (11), CHECKSUM_TRISTATE_V1 (30).
Runs the checksum program program_id with input as its subject and propagates its outcome unchanged.
| Reason code | Status it may carry |
|---|---|
checksum_not_published |
unsupported |
country_mismatch |
invalid |
empty |
invalid |
incompatible_ruleset |
unsupported |
input_too_long |
unsupported |
invalid_characters |
invalid |
invalid_checksum |
invalid |
invalid_encoding |
unsupported |
invalid_format |
invalid |
invalid_length |
invalid |
invalid_ruleset |
unsupported |
missing_country_code |
unsupported |
not_requested |
not_run |
not_run_format_invalid |
not_run |
not_run_format_unsupported |
not_run |
ok |
valid |
registry_not_configured |
unsupported |
unsupported_checksum |
unsupported |
unsupported_country |
unsupported |
unsupported_format |
unsupported |
unsupported_kind |
unsupported |
REQUIRE only accepts a reason code that proves an invalidity: empty,
invalid_length, invalid_characters, invalid_format and country_mismatch.
CHECKSUM_OP_KIND_UNSUPPORTED and IdentifierDefinition.absent_checksum_reason
only accept unsupported_checksum and checksum_not_published.
trim ASCII removes only U+0009..U+000D and U+0020 at both ends. A kind
token matches [a-z][a-z0-9_-]{0,63} after trim and lower casing, a country
token matches [A-Z]{2} after trim and upper casing, and a declared prefix is 1
to 8 ASCII alphanumeric characters compared case sensitively.
-
Refuse an input that is not valid UTF-8: the step is
unsupportedwithinvalid_encoding, and the value is reported verbatim. An identifier is a sequence of code points, and bytes that do not form one have none to evaluate. This check is also what keeps canonicalization non growing: a step that filters by code point would otherwise substitute U+FFFD for each malformed byte, tripling the value and making two engines disagree on the canonical value they report.This reason is reachable only through an API whose string type admits ill formed text, and no conformance case can carry it: a proto3
stringis valid UTF-8 by definition, on the wire and in the corpus. Nor is there one portable malformed value to carry - an invalid byte in a language whose strings are bytes, an unpaired surrogate in a language whose strings are UTF-16 code units, and nothing at all in a language whose strings are always well formed. An engine MUST therefore pin this step with a native test naming the malformed form its own string type admits, and an engine whose string type admits none MUST document that, and why, rather than widening its public API with a byte oriented entry point that exists only to reach this branch. -
Normalize the kind by trim ASCII, lower case ASCII and the
kind_aliasestable. -
If no dispatcher matches, return
unsupported_kindwithout running any program. -
Run the
pre_canonicalization_programexactly once on the raw value. It runs as soon as the dispatcher is resolved, before any country decision, so a result that stops at step 5 still carries the pre-canonical value. -
Normalize an explicit country by trim ASCII, upper case ASCII and the
country_aliasestable. A syntactically invalid token returnsunsupported_country; in a country specific dispatcher a country without target also returnsunsupported_country. An empty token behaves like an absent context. -
Select the target owning the longest exactly matching
accepted_prefix. -
If an explicit country and a recognized prefix designate two different targets, return
country_mismatch. -
Select, in this order, the country target, the prefix target, the GLOBAL target, then the single
allow_unprefixed_without_countrytarget. -
If nothing is selectable, return
missing_country_code. -
Run the canonicalization program of the selected definition exactly once on the pre-canonical value.
A pre-canonicalization program is restricted to SEQUENCE, TRIM_WHITESPACE,
REMOVE_WHITESPACE, UPPERCASE_ASCII and REMOVE_CHARS. It can never add,
replace or interpret a prefix.
| Situation | kind |
canonicalValue |
countryCode |
|---|---|---|---|
| Input above the byte limit | requested token after trim and lower casing | raw input | raw context |
| Dispatcher not resolved | requested token after trim and lower casing | raw input | raw context |
| Dispatcher resolved, definition not selected | canonical kind | pre-canonical value | normalized country when one exists, otherwise the raw context |
| Definition selected | canonical kind | canonical value | country code of the target, or the normalized country context for a GLOBAL target |
A GLOBAL target keeps a well formed country context in the result without using
it for routing. The country of a country target is its ISO 3166-1 alpha-2 code,
even when its business prefix differs, for example country GR with the
canonical VAT prefix EL.
Two distinct notions share the name here, and confusing them changes results.
The countryCode of the report is the normalized country context, which a
GLOBAL target preserves. The COUNTRY_CODE string operation yields the country
of the selected target, which is absent for a GLOBAL target because such a
target has none. A GLOBAL definition can therefore report a country while
COUNTRY_CODE yields the absent string within the same evaluation.
The profile is resolved in two moments, because a definition cannot supply its own default before it has been selected.
Dispatch uses the profile of the caller, or compatible when the caller gives
none. This is the profile PROFILE_IS observes in a pre-canonicalization
program.
Once a definition is selected, its default_profile applies when, and only
when, the caller supplied no profile. A caller who passes a profile explicitly
always overrides the default.
The rule that follows is worth stating as ownership: everything owned by the dispatcher runs under the dispatch profile, everything owned by the definition runs under the resolved one. So the pre-canonicalization program uses the dispatch profile, while the canonicalization, format and checksum programs of the selected definition use the resolved profile, which is also the one reported in the result.
All current definitions declare compatible, so no conformance case separates
these rules today. They are normative regardless.
-
Reject a raw input longer than 1024 UTF-8 bytes with format
unsupported/input_too_longand checksumnot_run/not_run_format_unsupported.The bound counts the bytes of the input's UTF-8 encoding. Text that is not well formed has no such encoding, and this step runs before the one that refuses it, so the count would have to be invented: a Java or Kotlin string holding one unpaired surrogate encodes to a single replacement byte through the platform encoder, and to three through the encoding that surrogate would have had. An engine whose string type admits ill formed text therefore chooses, and MUST state which: count what its own encoder produces, or refuse the input as
invalid_encodingbefore measuring it. Both answers areunsupported, no conformance case can carry such an input, and the choice only shows on text that is both ill formed and near the bound. -
Run the dispatch algorithm of section 5, each canonicalization phase at most once.
-
country_mismatchproduces formatinvalid/country_mismatchand checksumnot_run/not_run_format_invalid. -
Any other dispatch failure produces format
unsupportedwith that reason and checksumnot_run/not_run_format_unsupported. -
Run the format program on the canonical value.
-
An invalid format gives checksum
not_run/not_run_format_invalid. -
An unsupported format gives checksum
not_run/not_run_format_unsupported. -
A valid format without checksum program gives checksum
unsupportedwithabsent_checksum_reason. -
A valid format with a checksum program runs it.
validateFormat stops after step 5 and reports checksum
not_run/not_requested when the format is valid. validateChecksum returns
exactly the report of validate. canonicalize stops after step 2 and returns
valid/ok, invalid/country_mismatch, or unsupported with
unsupported_kind, unsupported_country, missing_country_code,
input_too_long or invalid_encoding.
Results produced before a rule assertion never carry a message key. An assertion or a checksum declared by the bundle keeps its message key exactly, including when it is absent.
A declared message_key is never empty. An empty one is refused at load time,
because a present but empty key cannot be told apart from an absent one in an
idiomatic API: two engines could then report differently on the same bundle,
which 2.4 forbids. Absence is expressed by omitting the field.
The V1 classes are ASCII only: digits U+0030..U+0039, upper letters
U+0041..U+005A, lower letters U+0061..U+007A. UPPERCASE_ASCII maps only
a..z to A..Z and never consults a locale. Positions and lengths are counted
in Unicode code points.
The frozen whitespace_v1 table is:
U+0009..U+000D, U+0020, U+0085, U+00A0, U+1680,
U+2000..U+200A, U+2028, U+2029, U+202F, U+205F,
U+3000, U+FEFF
A runtime must never delegate this definition to its own Unicode tables.
The character mappings of WEIGHTED_SUM and ISO7064_MOD97_10 are:
| Mapping | Domain | Value |
|---|---|---|
CHAR_MAPPING_DIGIT_VALUE |
0..9 |
the decimal value |
CHAR_MAPPING_ALNUM_BASE36 |
0..9 and A..Z |
the decimal value, then 10 to 35 |
Every limit is normative. An engine may raise an internal limit, never lower it.
maximum binary bundle 16777216 bytes
maximum identifiers 10000
maximum total nodes 500000
maximum nodes per program 4096
maximum call depth 32
maximum constant string 4096 UTF-8 bytes
maximum user input 1024 UTF-8 bytes
maximum steps per validation 100000
code points billed as one step 64
maximum captures per format 128
The bundle shaped limits are enforced when the bundle is accepted, whether by a loader or by a generator, and no longer apply once code has been emitted. The user input limit is an obligation of the engine itself: a longer input is refused without being processed. The step budget applies only to an engine that interprets.
integer constant signed int64
modulus / complement 2..1000000000
absolute value of a weight 0..1000000
weights per operation 1..256
elements of a remainder map 1..1000000
index / slice bound 0..4096
comparison constant -1000000000..1000000000
operands of concat 1..256
provable digits of digits_to_integer 1..18
code points of a custom alphabet 1..256, each one distinct
The compiler emits every repeated field in one normative order, and an engine refuses a bundle that does not respect it: a stable order is what makes the published SHA-256 meaningful and removes any dependence on the iteration order of a hash map.
| Field | Order |
|---|---|
RuleBundle.required_feature_ids |
strictly ascending |
RuleBundle.programs |
ascending id |
RuleBundle.identifiers |
kind, then GLOBAL first, then country_code |
RuleBundle.dispatchers |
UTF-8 bytes of kind |
IdentifierDefinition.sources |
UTF-8 bytes of id |
IdentifierDispatcher.kind_aliases |
UTF-8 bytes |
IdentifierDispatcher.country_aliases |
UTF-8 bytes of alias |
IdentifierDispatcher.targets |
GLOBAL first, then country_code |
DispatchTarget.accepted_prefixes |
UTF-8 bytes |
PredicateOperation.lengths and values |
ascending, deduplicated |
Two equal sort keys are a rejected duplicate, never a tie broken by the input order. Assertions, checksum branches, canonicalization steps, weights, remainder tables and the topological node list keep their semantic order and are never sorted.
Identifier and program ids are assigned from 1 after sorting the fully qualified symbol names of the sources by UTF-8 bytes. Nodes are emitted by a deterministic post-order walk of the operands in syntactic order, with an optional deduplication whose key is the deterministic Protobuf encoding of the node itself, so two nodes are merged only when they are observationally identical, operands, reason code and message key included.
The bundle carries no timestamp. generatedAt exists only in the
external manifest and is the UTC RFC 3339 representation of
SOURCE_DATE_EPOCH. An engine verifies the SHA-256 of the file it
received and never re-serializes a decoded message to recompute it,
because Protobuf is not a canonical serialization.
An engine performs these checks, in this order, before executing anything:
- binary size at most 16 MiB
- complete Protobuf decoding
- supported
format_version - every
required_feature_idsentry known, strictly ascending - absence of any unknown field at any depth
rules_versionnon empty, at most 64 bytes, and made only of ASCII letters, digits, dot, dash and underscoresource_digestof exactly 32 bytes- program ids unique and non zero, program kinds specified
- node count within the per program and total limits
- every operation known, with its declared output type
- operand count, operand types and strictly lower operand indices
- only the parameters the operation declares, and every required parameter
- arithmetic bounds: moduli, weights, remainder tables, indices, provable integer widths and the alphabet of a custom mapping, and the shape of a parameter list: the order section 9 states, then the single element length of a
prefix_in, in that order - expansion within the evaluation budget once repeated operands are inlined
- root, subject and capture nodes inside the program, correctly typed, and a subject node that does not read the subject it defines
- program shape: the accepted root and the accepted operation categories of the kind, both as section 2 states them,
WHENonly insideCHOOSE, and a pre-canonicalization program restricted to its five permitted operations - identifier ids unique, kinds and countries well formed, serialization order respected
- exactly one checksum program or one absence reason per definition
- dispatcher kinds and aliases globally unique, sorted, and never ambiguous
- country aliases sorted, unique, never self mapping and never shadowing a target
- targets sorted, unique per country, prefixes claimed by at most one target
- GLOBAL targets alone, without prefix and without country alias
- every definition referenced by exactly one dispatch target
- call graph acyclic, typed and of static depth at most 32
- no capability used without being declared
A size, structural, arithmetic or graph violation is invalid_ruleset. An
unsupported format_version and an unknown capability id are
incompatible_ruleset. An engine never executes a partially validated graph and
never re-serializes a decoded message to verify the published SHA-256.
An unknown operation is therefore invalid_ruleset, not incompatible_ruleset,
and the distinction is deliberate. A bundle that legitimately uses a newer
operation declares the capability that introduced it, so an engine too old to
understand it stops at check 4 with incompatible_ruleset. Reaching check 10
with an unknown operation means the bundle used one without declaring it, which
is a forged bundle rather than a version gap.
That distinction only survives if check 2 stays at the wire level. An engine that resolves an opcode, an enum value or a capability id while decoding reports a newer bundle at check 2, before the capability check can excuse it, and the version gap is again reported as a malformed bundle. Decoding proves the bytes parse and nothing more: an unresolved opcode is carried to check 10, an unknown field to check 5, and an unrecognised enum value to the check that owns its field. This is not a hint. Two engines have already been caught by it, once on a new field and once on a new opcode, and the second was found only because the first had already happened.
The version checks precede the unknown field scan, and the order carries a meaning. A bundle built against a later version holds fields this runtime has never heard of; reporting those as unknown fields would call a legitimate version gap a forged bundle. Asking first whether the bundle announces something unsupported yields the accurate answer, and tells an operator to upgrade rather than to suspect the file.
A bundle whose encoding repeats a singular field, or carries two branches of the
same oneof, is not a valid bundle. A generator SHOULD refuse it. One that does
not inspect the wire encoding decodes it with the Protobuf semantics, and the
determinism of specification 2.4 is stated over valid bundles.