Skip to content

Bug(expr): string-literal lexer decodes octal escapes RFC 0006 does not define, bypassing re_sub's group-reference rejection #311

Description

@uberware

Summary

Two connected behaviors, one root cause:

  • A. A non-raw string literal '\1' evaluates to the single control character U+0001.
    §1.1.5's escape table lists no octal escapes (\0, \1, … are absent), so this is
    Python's lexer behavior leaking through rather than a documented EXPR feature.
  • B. Consequence: re_sub('a1', '(\d)', '\1') (replacement written NON-raw) is not
    rejected — it returns 'a' followed by an invisible U+0001. The group-reference check
    itself is fine: \1 written raw (r'\1'), \g<1> in either form, $1, and ${1} are
    all correctly rejected with "Group references in replacement strings are not supported".
    The non-raw \1 escapes it only because the lexer has already decoded it to U+0001 before
    the check runs.

Versions

  • openjd-model 0.11.1 and 0.11.2 from PyPI
  • openjd-rs main at 1a89f3a — both still reproduce

Reproduction

from openjd.expr import parse_expression
r = parse_expression(r"'\1'").evaluate_with_metrics()
print(hex(ord(str(r.value))))          # -> 0x1

r = parse_expression(r"re_sub('a1', '(\d)', '\1')").evaluate_with_metrics()
print(repr(str(r.value)))              # -> 'a\x01'  (not rejected)

# control: the same replacement written raw IS rejected
parse_expression(r"re_sub('a1', r'(\d)', r'\1')").evaluate_with_metrics()
# -> ExpressionError: Group references in replacement strings are not supported

Why this is a bug

§1.1.5's table defines exactly which escapes exist, and octal escapes are not among them
(nor are \a, \b, \f, \v — the table is deliberately smaller than Python's). A
template author who writes '\1' gets an invisible control character instead of either the
two literal characters or an error, and re_sub silently produces output containing U+0001
where every other spelling of the same intent is rejected loudly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions