Heredoc bodies lose trailing blank lines and spaces to a greedy rstrip

[kkozik-amplify]

Describe the bug

Heredoc bodies are right-stripped of all trailing newlines, tabs and spaces, so distinct sources collapse to the same value. A heredoc with a trailing blank line and one with trailing spaces both yield "x", identical to a plain one-line body.

The extraction regex gets the body right; the loss happens in the rstrip applied afterwards.

Software:

• python-hcl2: main (1432aea)
• Python 3.9, lark 1.3.1
• Long-standing, not a v8 regression — see below

Snippet of HCL2 code causing the unexpected behaviour:

a = <<MARKER
x

MARKER

(the blank line before the closing marker is the point)

Expected behavior

The blank line is part of the body, so it should survive. Per the HCL specification the heredoc content is the lines between the markers.

Actual behavior

With preserve_heredocs=False (the flattening path), showing the body the regex extracts alongside the final value:

Source body	Extracted by regex	Result
x\n\n (trailing blank line)	x\n\n	"x"
x\n\n\n (two blank lines)	x\n\n\n	"x"
x \n (trailing spaces)	x \n	"x"
x\n\ny\n (interior blank)	x\n\ny\n	"x\ny" — interior correctly preserved

Three different sources produce the same "x".

Round-trip consequence

The trimming makes the heredoc unrecoverable, because strings_to_heredocs only restores values that still contain a newline:

src  = 'a = <<MARKER\nx\n\nMARKER\n'
flat = {'a': '"x"'}          # preserve_heredocs=False
# flatten -> restore with strings_to_heredocs=True:
'a = "x"\n'

The blank line is gone and so is the heredoc — the restored output is a plain quoted string.

Root cause

hcl2/rules/strings.py:106:

_trim_chars = "\n\t "

used at lines 126, 132 and 163 as body.rstrip(self._trim_chars). rstrip with a character set removes every trailing character in that set, not just one newline.

Not a regression. 7.x did exactly the same thing — DictTransformer.heredoc_template in 21b7cf8^:hcl2/transformer.py used the same trim_chars = "\n\t " and the same .rstrip(trim_chars). So this predates the v8 rewrite.

Scope note, and why this is filed narrowly

Dropping the single final newline is a separate and much larger question. Terraform evaluates a one-line heredoc to "x\n", whereas this library yields "x" — but that behaviour is encoded in every existing heredoc golden (e.g. "trimmed": "indented1\nindented2"), so changing it would be breaking.

This report is deliberately limited to the greedy part: removing more than one trailing newline, and removing trailing spaces/tabs. Replacing the blanket rstrip with "remove exactly one trailing newline" would fix the collapsing cases above without touching the established final-newline behaviour. Whether to also revisit that final newline is worth deciding separately.

Impact

Low frequency, silent when it happens. Matters for heredocs whose trailing blank lines are meaningful — generated scripts, policy documents, files whose content is hashed or compared against what Terraform produces.

Workaround

Use the default preserve_heredocs=True and extract the body yourself from the preserved source form; the raw text still contains the blank lines.

---

Found while reviewing #309/#312 (empty heredocs). It is the same code path but a distinct, pre-existing defect, so it is filed separately rather than folded into that fix.

[kkozik-amplify]

Released in v8.1.3, now on PyPI — pip install python-hcl2==8.1.3. Fixed by #318.