write_pdx_file() emits a corrupt PDX from the second call onwards (regression in 11.4.2)

[ken-otf]

Summary

write_pdx_file() produces a corrupt PDX from the second call onwards within the same process. The first call is fine; every subsequent call emits a file whose COMPARAM-SUBSET-REF has lost its DOCREF/DOCTYPE attributes, so loading it back fails with an unresolvable ODXLINK reference.

This is a regression: last good = 11.4.1, first bad = 11.4.2.

It is independent of whether the database was modified — writing the same unmodified database twice is enough to trigger it.

Reproducer

import sys, tempfile, os, odxtools

src = sys.argv[1]          # any PDX; we used the somersault example
d = tempfile.mkdtemp()
print("odxtools", odxtools.__version__)
for i in (1, 2, 3):
    db = odxtools.load_pdx_file(src)
    out = os.path.join(d, f"out{i}.pdx")
    odxtools.write_pdx_file(out, db)
    try:
        odxtools.load_pdx_file(out)
        print(f"  write #{i}: reload OK")
    except Exception as e:
        print(f"  write #{i}: reload FAILED -> {type(e).__name__}: {str(e)[:90]}")

Output on 11.5.3:

odxtools 11.5.3
  write #1: reload OK
  write #2: reload FAILED -> KeyError: "ODXLINK reference OdxLinkRef('SAE_J2411_SWCAN') could not be resolved for any of the docu
  write #3: reload FAILED -> KeyError: "ODXLINK reference OdxLinkRef('SAE_J2411_SWCAN') could not be resolved for any of the docu

Note that each iteration loads a fresh database from the original file, so no state is carried over on the caller's side.

What differs in the output

Diffing ISO_15765_3_on_ISO_15765_2.odx-c between write #1 and write #2:

@@ -14 +14 @@
-      <TEAM-MEMBER-REF ID-REF="CPS_ISO_15765_3_on_ISO_15765_2.Softing.HC" />
+      <TEAM-MEMBER-REF ID-REF="CPS_ISO_15765_3_on_ISO_15765_2.Softing.HC" DOCREF="ISO_15765_3_on_ISO_15765_2" DOCTYPE="COMPARAM-SPEC" />
@@ -138 +138 @@
-       <COMPARAM-SUBSET-REF ID-REF="SAE_J2411_SWCAN" DOCREF="SAE_J2411_SWCAN" DOCTYPE="COMPARAM-SUBSET"/>
+       <COMPARAM-SUBSET-REF ID-REF="SAE_J2411_SWCAN"/>

The second one is the fatal one: without DOCREF/DOCTYPE the subset reference can no longer be resolved on load.

Root cause

_get_jinja_env() is memoised with @cache (added in 11.4.2), so the same jinja2.Environment — including its compiled-template cache — is reused across write_pdx_file() calls.

write_pdx_file() creates a fresh jinja_vars dict per call and binds it into the environment's globals:

jinja_env = _get_jinja_env(templates_dir)
jinja_vars: dict[str, Any] = {}
...
jinja_env.globals["set_layer_docfrag"] = lambda lname: set_layer_docfrag(jinja_vars, lname)
jinja_env.globals["make_ref_attribs"] = lambda ref: make_ref_attribs(jinja_vars, ref)

Because the templates (and the macro modules they import) were already compiled and cached during the first call, the rebound globals do not reach them. The macros keep calling the first call's closures.

Verified by instrumenting make_ref_attribs and holding strong references to every jinja_vars dict it is called with (strong refs so that id() reuse cannot produce a false positive):

write #1: make_ref_attribs saw 1 distinct jinja_vars
write #2: make_ref_attribs saw 1 distinct jinja_vars
write #3: make_ref_attribs saw 1 distinct jinja_vars
leftover cur_docfrags: ['SAE_J2411_SWCAN', 'somersault_assiduous']

So from the second call on, cur_docfrags is whatever was left at the end of the previous call. make_ref_attribs() then sees SAE_J2411_SWCAN as "already in the current document fragments" and omits DOCREF/DOCTYPE, producing the unresolvable reference.

Version bisect

version	@cache on _get_jinja_env	repeated writes
10.6.1	no	OK
11.0.0	no	OK
11.1.0 – 11.4.1	no	OK
11.4.2	yes	broken from write #2
11.5.3	yes	broken from write #2

Suggested fix

The per-call state should not live in Environment.globals. Passing the three helpers through the render variables instead of the environment globals would remove the coupling to the cached templates entirely, e.g.:

jinja_vars["set_category_docfrag"] = lambda cname, ctype: set_category_docfrag(jinja_vars, cname, ctype)
jinja_vars["set_layer_docfrag"] = lambda lname: set_layer_docfrag(jinja_vars, lname)
jinja_vars["make_ref_attribs"] = lambda ref: make_ref_attribs(jinja_vars, ref)

(since every template is rendered with **jinja_vars, these would be resolved per render).

A smaller fix is to drop the @cache, or to clear the environment's template cache at the start of write_pdx_file() — I confirmed that either of those makes repeated writes work.

Workaround for users

import odxtools.writepdxfile as w
w._get_jinja_env.cache_clear()      # before each write_pdx_file() call

Why this matters

Any application that edits and saves in a loop — an ODX editor, a batch converter, a test that round-trips more than once — hits this on the second save. If the application validates by reading its own output back (as ours does) it looks like "saving suddenly stopped working"; if it does not validate, it silently writes PDX files that other tools cannot read.

Environment

• odxtools 11.5.3 (also reproduced on 11.4.2)
• Python 3.14.7, Jinja2 3.1.6
• Windows 11

[andlaus]

I reverted #489, can you check if the issue persists?

[ken-otf]

Confirmed fixed on main (11.5.5.dev6+g00c938289). Thanks for the quick turnaround.

Repeated writes now round-trip cleanly:

odxtools 11.5.5.dev6+g00c938289
  write #1: reload OK
  write #2: reload OK
  write #3: reload OK

Two further checks:

• Reusing the same Database object across five writes, mutating it in between, round-trips every time.
• The written ISO_15765_3_on_ISO_15765_2.odx-c is now byte-identical across writes, and the attribute that used to disappear is present in all of them:

write #1: <COMPARAM-SUBSET-REF ID-REF="SAE_J2411_SWCAN" DOCREF="SAE_J2411_SWCAN" DOCTYPE="COMPARAM-SUBSET"/>
write #2: (identical)
write #3: (identical)

Feel free to close this — I'll drop our workaround once a release including the revert is out.

[ken-otf]

Also confirmed on the released 11.5.4 — I had only checked main before:

odxtools 11.5.4
  write #1: reload OK
  write #2: reload OK
  write #3: reload OK