fix: a heredoc ends its own line wherever it is written (#338)

CHANGELOG.md

@@ -7,7 +7,15 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## \[Unreleased\]
 
-- Nothing yet.
+### Fixed
+
+- Flattened heredoc bodies match the values Terraform and OpenTofu evaluate the same source to. Three things differed, all checked against OpenTofu v1.12.5 rather than read off the spec: the newline terminating the last content line was dropped (`<<EOT\nline\nEOT` returned `'line'`, not `'line\n'`); `<<-` measured its indent in spaces alone, so a tab-indented body was not dedented at all; and a whitespace-only line was excluded from the measurement but trimmed anyway. This is not a regression — 7.2.1 returned the same values — so it changes long-standing behaviour rather than restoring anything.
+- A carriage return in a flattened heredoc body is written as `\r` rather than left raw. `preserve_heredocs=False` returns quoted-string *source*, and a quoted string cannot hold a literal carriage return: OpenTofu rejects one with "No closing marker was found for the string". A heredoc read out of a CRLF file therefore flattened to source that would not parse again. The value form (`strip_string_quotes=True`) is unchanged and still hands back real carriage returns. `strings_to_heredocs` resolves `\r` when it writes a body, so the two halves stay each other's inverse: a heredoc interprets no escape, so a body carrying a backslash and an `r` would be those two characters rather than the carriage return the value held.
+- A `<<-` heredoc whose closing marker is indented with something other than spaces or tabs no longer appends that indentation to the value. The dedent already measured whitespace rather than spaces, matching OpenTofu, but the marker's own indent was stripped as `[ \t]*`, so a body indented with a non-breaking space, a vertical tab, a form feed or an ideographic space came back with one of those characters on the end. Four such cases are now in the table that `bin/heredoc_ground_truth` re-derives from OpenTofu.
+- A heredoc written inside a list or an object ends its own line. It ends at its closing marker, on a line of its own, so the separator that follows has to start the next one -- `EOF,` closes nothing, and the file this library had just written did not parse, here or in Terraform. A top-level attribute survived only because the newline after it comes from the document. Only a heredoc built by the deserializer needs this: `HEREDOC_TEMPLATE` matches through the newline after the marker, so one that came from the parser already ends the line, and reconstructing a parsed document is byte for byte what it was. ([#338](https://github.com/amplify-education/python-hcl2/issues/338))
+- `strings_to_heredocs` leaves a value carrying a lone carriage return quoted. A heredoc body is read literally, so it can hold a `\r` only where one ends a line: OpenTofu rejects `<<EOF\nx\ry\nEOF` with "No closing marker was found for the string", while the quoted `"x\ry\n"` it came from is valid. Such a value stays quoted, for the same reason one that does not end in a newline does.
+- `strings_to_heredocs` picks a delimiter the body cannot close. It wrote `<<EOF` over every value, so a string holding a line reading `EOF` -- a log excerpt, a shell script, an embedded config, the payloads heredocs are for -- ended its own heredoc early and produced a file that no longer parsed. A numbered variant is used when the body occupies `EOF`, and ordinary values are written exactly as before. The lines that count as markers are Terraform's, which are looser than this grammar's: OpenTofu ends a heredoc on `EOF  ` while `HEREDOC_TEMPLATE` here requires the newline to follow the word. A CRLF body counts too -- it is split on `\n`, so its lines carry their own `\r`, and OpenTofu ends a heredoc on `EOF\r` as readily as on `EOF `. ([#330](https://github.com/amplify-education/python-hcl2/issues/330))
+- `strings_to_heredocs` no longer adds a line to the body it writes. The value's own trailing newline is the one that precedes the closing marker, so a heredoc was being emitted one line longer than the string it came from. A value that does not end in a newline is now left as a quoted string, since no heredoc can express it. Flattening a document and restoring it now yields HCL that OpenTofu evaluates identically to the original; five of the eleven values in the round-trip fixture did not survive it before.
 
 ## \[8.1.3\] - 2026-08-26
 

bin/heredoc_ground_truth

@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+"""Check the heredoc expectations in the test suite against Terraform itself.
+
+`test/unit/test_heredoc_matches_terraform.py` asserts what a heredoc body
+evaluates to. Those values did not come from this library or from reading the
+spec -- each one was produced by handing the same source to OpenTofu. That
+provenance is a docstring, which a reader has to take on trust and which
+nothing re-checks if the reference implementation ever moves.
+
+This script re-derives them. It reads the `CASES` table out of that test module,
+evaluates every source with `tofu console` (or `terraform console`), and
+compares. It is not part of the test run: the suite must not depend on a
+Terraform binary, and these values change about as often as the HCL spec does.
+
+Usage:
+    bin/heredoc_ground_truth            # verify; non-zero exit on any mismatch
+    bin/heredoc_ground_truth --print    # print the table as Python, to paste
+
+Requires `tofu` or `terraform` on PATH.
+"""
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+
+from test.unit.test_heredoc_matches_terraform import CASES  # noqa: E402
+
+BINARIES = ("tofu", "terraform")
+
+
+def find_binary():
+    """Return the first Terraform-compatible binary on PATH, or None."""
+    for name in BINARIES:
+        path = shutil.which(name)
+        if path:
+            return path
+    return None
+
+
+def evaluate(binary, source):
+    """Return the value `binary` evaluates the given heredoc expression to.
+
+    The source is written as a local rather than an output so that nothing has
+    to be applied, and `jsonencode` is what carries the exact string back --
+    the console's own rendering escapes newlines for display.
+    """
+    with tempfile.TemporaryDirectory() as directory:
+        # newline="" so a case testing CRLF is written with the bytes it names.
+        with open(os.path.join(directory, "main.tf"), "w", encoding="utf-8", newline="") as handle:
+            handle.write("locals {\n  x = %s\n}\n" % source)
+        result = subprocess.run(
+            [binary, "console"],
+            cwd=directory,
+            input="jsonencode(local.x)\n",
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+        if result.returncode != 0:
+            raise RuntimeError(result.stderr.strip())
+        return json.loads(json.loads(result.stdout.strip().splitlines()[-1]))
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--print",
+        dest="print_table",
+        action="store_true",
+        help="print the evaluated table as Python instead of verifying",
+    )
+    args = parser.parse_args()
+
+    binary = find_binary()
+    if binary is None:
+        print("neither `tofu` nor `terraform` is on PATH", file=sys.stderr)
+        return 2
+
+    print("using %s\n" % binary, file=sys.stderr)
+    mismatches = 0
+    for source, expected in CASES:
+        actual = evaluate(binary, source)
+        if args.print_table:
+            print("    (%r, %r)," % (source, actual))
+            continue
+        if actual == expected:
+            print("ok   %r" % source)
+        else:
+            mismatches += 1
+            print("BAD  %r\n       expected %r\n       %s says %r" % (source, expected, binary, actual))
+
+    if args.print_table:
+        return 0
+
+    # stdout, so it lands after the per-case lines rather than ahead of them
+    # when the output is piped.
+    print("\n%d of %d cases disagree" % (mismatches, len(CASES)))
+    return 1 if mismatches else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

cli/json_to_hcl.py

@@ -180,7 +180,7 @@ def main():  # pylint: disable=too-many-branches,too-many-statements,too-many-lo
     parser.add_argument(
         "--strings-to-heredocs",
         action="store_true",
-        help="Convert strings containing escaped newlines to heredocs",
+        help="Convert newline-terminated escaped strings to heredocs",
     )
 
     # FormatterOptions flags

docs/01_getting_started.md

@@ -74,7 +74,7 @@ data = loads(text, serialization_options=SerializationOptions(
 | `wrap_objects` | `bool` | `False` | Wrap object values as inline HCL2 strings                                                                                                       |
 | `wrap_tuples` | `bool` | `False` | Wrap tuple values as inline HCL2 strings                                                                                                        |
 | `explicit_blocks` | `bool` | `True` | Add `__is_block__: True` markers to blocks. **Mandatory for JSON->HCL2 deserialization and reconstruction.**                                    |
-| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. |
+| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb\n"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. Either way the body keeps the newline that terminates its last line, as Terraform's does. |
 | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations                                                                                                         |
 | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is                                                                                                                  |
 | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** |
@@ -127,7 +127,7 @@ text = dumps(data, deserializer_options=DeserializerOptions(
 | Field | Type | Default | Description |
 |---|---|---|---|
 | `heredocs_to_strings` | `bool` | `False` | Convert heredocs to plain strings |
-| `strings_to_heredocs` | `bool` | `False` | Convert strings with `\n` to heredocs |
+| `strings_to_heredocs` | `bool` | `False` | Convert newline-terminated strings to heredocs. A value that does not end in a newline is left as a quoted string, because a heredoc body always ends in one and writing it as a heredoc would change the value. |
 | `object_elements_colon` | `bool` | `False` | Use `:` instead of `=` in object elements |
 | `object_elements_trailing_comma` | `bool` | `True` | Add trailing commas in object elements |
 

docs/06_migrating_to_v8.md

@@ -213,17 +213,18 @@ This restores the v7 dict shape but disables round-trip support and comment pres
 
 ```python
 hcl2.loads('x = <<-EOT\n  line1\n  line2\n  EOT\n', serialization_options=V7_COMPAT)
-# {'x': 'line1\nline2'}
+# {'x': 'line1\nline2\n'}
 ```
 
-Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable.
+Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2\\n"'`), because that output is meant to be reconstructable.
 
-Two details of heredoc values are easy to trip over, and both match how HCL itself behaves:
+Three details of heredoc values are easy to trip over, and all three match how HCL itself behaves:
 
+- **The body ends with a newline.** Every content line is terminated by its own newline, the last one included, so `<<EOT\nline\nEOT` is `'line\n'` — the same value Terraform evaluates it to. v7 returned `'line'`, and so did 8.1.x; both were wrong. Only an empty body has no trailing newline, because it has no content line.
 - **Backslash escapes are not interpreted in heredocs.** `strip_string_quotes` resolves `\n` inside a *quoted* string, but a heredoc body containing the two characters `\n` keeps them verbatim. HCL only processes escape sequences in quoted templates.
 - **Line endings come through as written.** A heredoc in a CRLF file yields a body with `\r\n`, because a carriage return inside the body is content rather than structure. Normalize on your side if you need `\n`.
 
 ```python
 hcl2.loads('x = <<EOT\na\\nb\nEOT\n', serialization_options=V7_COMPAT)
-# {'x': 'a\\nb'}  — the backslash and the "n" are two literal characters
+# {'x': 'a\\nb\n'}  — the backslash and the "n" are two literal characters
 ```

hcl2/deserializer.py

@@ -63,6 +63,75 @@
 from hcl2.transformer import RuleTransformer
 from hcl2.utils import HEREDOC_PATTERN, HEREDOC_TRIM_PATTERN
 
+_HEREDOC_BODY_ESCAPES = {"n": "\n", "r": "\r"}
+
+# A line that could end a heredoc: the delimiter word alone, give or take
+# surrounding spaces and tabs -- and a carriage return, because the body is
+# split on "\n" and a CRLF line hands back its own `\r`. OpenTofu ends a
+# heredoc on `EOF\r` exactly as it does on `EOF `, so a CRLF body carrying
+# the delimiter has to count. This grammar is stricter than Terraform, whose
+# scanner ends the heredoc on `EOF  ` while `HEREDOC_TEMPLATE` here requires
+# the newline to follow the word itself. The looser reading is the safe one to
+# pick a delimiter against: emitting a body that only Terraform would treat as
+# closed writes a file this library can read and Terraform cannot.
+_CLOSING_MARKER_LINE = re.compile(r"[ \t]*([a-zA-Z][a-zA-Z0-9._-]*)[ \t\r]*")
+
+
+def _heredoc_delimiter(content: str) -> str:
+    """Return a delimiter the body does not close on its own.
+
+    `EOF` unless the body holds a line that would end the heredoc there, in
+    which case a numbered variant is used. The word matters: a log excerpt, a
+    shell script or an embedded config is exactly the sort of value people put
+    in a heredoc, and `EOF` is exactly the word such a payload tends to
+    contain. Writing one blindly produced a file that no longer parsed.
+    """
+    occupied = set()
+    for line in content.split("\n"):
+        match = _CLOSING_MARKER_LINE.fullmatch(line)
+        if match is not None:
+            occupied.add(match.group(1))
+
+    if "EOF" not in occupied:
+        return "EOF"
+
+    suffix = 1
+    while f"EOF_{suffix}" in occupied:
+        suffix += 1
+    return f"EOF_{suffix}"
+
+
+def _expressible_as_heredoc(content: str) -> bool:
+    """Whether *content* can be a heredoc body without changing.
+
+    A heredoc body is read literally, so it can hold a carriage return only
+    where one ends a line. A lone `\r` makes the file unreadable rather than
+    merely different: OpenTofu rejects `<<EOF\nx\ry\nEOF` with "No closing
+    marker was found for the string", while the quoted `"x\ry\n"` it came
+    from is valid and evaluates to that carriage return. Such a value stays
+    quoted, for the same reason one that does not end in a newline does.
+    """
+    return "\r" not in content.replace("\r\n", "")
+
+
+def _unescape_heredoc_body(inner: str) -> str:
+    r"""Resolve the escapes a heredoc body carries literally: \n, \r, \" and \\.
+
+    A heredoc interprets no backslash sequence -- its body is the characters
+    themselves -- so anything the quoted form spelled as an escape has to be
+    resolved before it is written into one. `\r` is here because the flattened
+    form escapes carriage returns: without it, a heredoc read out of a CRLF
+    file and written back came out holding a literal backslash and an `r`.
+
+    Single-pass, so an escaped backslash cannot combine with the character
+    after it.
+    """
+    return re.sub(
+        r'\\(n|r|"|\\)',
+        lambda m: _HEREDOC_BODY_ESCAPES.get(m.group(1), m.group(1)),
+        inner,
+    )
+
 
 @dataclass
 class DeserializerOptions:
@@ -71,8 +140,10 @@ class DeserializerOptions:
     # Convert heredoc values (<<EOF...EOF) to regular escaped strings during
     # deserialization. When False, heredoc syntax is preserved as-is.
     heredocs_to_strings: bool = False
-    # Convert multi-line escaped strings (containing \n) back into heredoc
-    # syntax (<<EOF...EOF) during deserialization.
+    # Convert newline-terminated escaped strings back into heredoc syntax
+    # (<<EOF...EOF) during deserialization. A value that does not end in a
+    # newline is left quoted: a non-empty heredoc body always does, so writing
+    # one as a heredoc would hand back a different value on the next read.
     strings_to_heredocs: bool = False
     # Use colon (:) instead of equals (=) as the separator in object elements.
     object_elements_colon: bool = False
@@ -179,9 +250,18 @@ def _deserialize_text(self, value: Any) -> LarkRule:
                         return self._deserialize_heredoc(value[1:-1], False)
 
                 if self.options.strings_to_heredocs:
-                    inner = value[1:-1]
-                    if "\\n" in inner:
-                        return self._deserialize_string_as_heredoc(inner)
+                    content = _unescape_heredoc_body(value[1:-1])
+                    # A heredoc's closing marker sits on a line of its own, so
+                    # any body with content in it ends with a newline. A value
+                    # that does not cannot be written as one without gaining
+                    # that character, so it stays a quoted string.
+                    #
+                    # The empty string is the one value this excludes that a
+                    # heredoc could in fact express -- `<<EOF\nEOF` evaluates
+                    # to "" in Terraform and here. It stays quoted anyway,
+                    # because `x = ""` says the same thing in one line.
+                    if content.endswith("\n") and _expressible_as_heredoc(content):
+                        return self._deserialize_string_as_heredoc(content)
 
                 return self._deserialize_string(value)
 
@@ -259,15 +339,10 @@ def _deserialize_heredoc(
             return HeredocTrimTemplateRule([HEREDOC_TRIM_TEMPLATE(value)])
         return HeredocTemplateRule([HEREDOC_TEMPLATE(value)])
 
-    def _deserialize_string_as_heredoc(self, inner: str) -> HeredocTemplateRule:
-        """Convert a quoted string with escaped newlines back into a heredoc."""
-        # Single-pass unescape: \\n → \n, \\" → ", \\\\ → \
-        content = re.sub(
-            r'\\(n|"|\\)',
-            lambda m: "\n" if m.group(1) == "n" else m.group(1),
-            inner,
-        )
-        heredoc = f"<<EOF\n{content}\nEOF"
+    def _deserialize_string_as_heredoc(self, content: str) -> HeredocTemplateRule:
+        """Wrap an unescaped body, already newline-terminated, in heredoc syntax."""
+        delimiter = _heredoc_delimiter(content)
+        heredoc = f"<<{delimiter}\n{content}{delimiter}"
         return HeredocTemplateRule([HEREDOC_TEMPLATE(heredoc)])
 
     def _deserialize_expression(self, value: str) -> ExprTermRule: