Enhanced yaml provenance for autosubmit application by MuhammadShafeeque · Pull Request #3 · esm-tools/yaml-provenance

MuhammadShafeeque · 2026-03-02T11:02:56Z

⚠️ Merge dependency: This PR targets feat/yaml_dumper and can only be merged into main after feat/yaml_dumper has been merged.

Summary

Extends yaml-provenance with serialization support, copy.deepcopy preservation, and helper functions needed for real-world integration (e.g. Autosubmit). These additions ensure that provenance-wrapped objects survive pickling, JSON encoding, ruamel.yaml re-dumping, and deep copying without silently losing their provenance metadata.

What's new

Helper functions (_helpers.py)

wrap_computed(value, source) — Attach provenance to programmatically injected values (env vars, computed parameters) using a human-readable source string instead of file/line/col.
transfer_provenance(original, result) — Re-attach provenance after string operations (str.upper(), str.strip(), etc.) that return plain str.
annotate_dict(d, source_prefix) — Recursively wrap every scalar leaf of a plain dict with per-key provenance, in-place.

Serialization (_serialization.py — new module)

register_pickle_reducers() — Patch __reduce__ on all WithProvenance classes so pickle.dumps/pickle.loads work.
register_yaml_representers() — Register WithProvenance types on ruamel.yaml's SafeRepresenter and RoundTripRepresenter.
ProvenanceJSONEncoder — json.JSONEncoder subclass that handles BoolWithProvenance and NoneWithProvenance.

`copy.deepcopy` support

Added __deepcopy__ to all wrapper types (_wrapper.py), DictWithProvenance (_dict.py), and ListWithProvenance (_list.py) so that copy.deepcopy preserves provenance instead of silently reducing to plain builtins via __reduce__.

YAML dumper improvements (`_yaml_dumper.py`)

Omit line: and col: from provenance comments when both are 0 (computed values via wrap_computed).
Add provenance comments to empty dict/list entries (e.g. AQUA_SETUP: {}) by falling back to key provenance.

YAML loader improvements (`yaml_loader.py`)

Accept file-like objects (with .name attribute) in addition to paths.
Wrap string dict keys with provenance so even entries whose values are empty dicts/lists carry a provenance trace.

Files changed

File	Change
init.py	Export new helpers + serialization API
_helpers.py	Add `wrap_computed`, `transfer_provenance`, `annotate_dict`
_serialization.py	New — pickle, JSON, YAML serialization
_wrapper.py	`__deepcopy__` for dynamic + unsubclassable wrappers
_dict.py	`__deepcopy__` for `DictWithProvenance`
_list.py	`__deepcopy__` for `ListWithProvenance`
_yaml_dumper.py	Computed-value comments + empty-container key provenance
yaml_loader.py	File-like support + key provenance wrapping
test_new_helpers.py	New — 127 lines covering new helpers
test_serialization.py	New — 274 lines covering pickle/JSON/YAML serialization
test_yaml_dumper.py	Extended with empty-container and computed-value tests
dependencies.yaml	New — test fixture for DEPENDENCIES-style entries
api.rst	Added new helpers + Serialization section
CONTEXT.md	Updated package structure + test listing
README.md	Added new feature bullets

Testing

All 123 tests pass (pytest tests/ -q). No new dependencies added.

MuhammadShafeeque · 2026-03-02T11:12:02Z

The latest experiment was run with autosubmit branch esm-provenance-to-as in t0ht with workflow main branch. Here is the /appl/AS/AUTOSUBMIT_DATA/t0ht/conf/metadata/experiment_data.yml

mandresm · 2026-03-02T12:59:35Z

Hi @MuhammadShafeeque,

Thanks for this PR. Before I review it, could you clarify the following questions?

These additions ensure that provenance-wrapped objects survive pickling, JSON encoding, ruamel.yaml re-dumping, and deep copying without silently losing their provenance metadata.

Can you explain why do we need that the provenance survives pickling in Autosubmit? What is Autosumit doing that requires to load a pickle? I have assumed that AS parses the yamls, merges the info inside and then all it does is to use that config.

Having pickable provenance is good, but might come with some overhead in the pickling or reading the pickled files, have you profiled how long does it take for a realistic experiment?

Please, let me know when it is ready for review and turn off draft mode

MuhammadShafeeque · 2026-03-03T14:00:28Z

Hi @MuhammadShafeeque,

Thanks for this PR. Before I review it, could you clarify the following questions?

These additions ensure that provenance-wrapped objects survive pickling, JSON encoding, ruamel.yaml re-dumping, and deep copying without silently losing their provenance metadata.

Can you explain why do we need that the provenance survives pickling in Autosubmit? What is Autosumit doing that requires to load a pickle? I have assumed that AS parses the yamls, merges the info inside and then all it does is to use that config.

Having pickable provenance is good, but might come with some overhead in the pickling or reading the pickled files, have you profiled how long does it take for a realistic experiment?

Please, let me know when it is ready for review and turn off draft mode

What this PR is really about: Extending yaml-provenance to track provenance for list items and dictionary keys (especially DEPENDENCIES). In the previous version, list items like NOTIFY_ON: [FAILED] and dependency keys like FDB_INFO_FILE in DEPENDENCIES: {FDB_INFO_FILE: {}} had no provenance tracking at all. With the new changes, they now carry full file:line:col provenance.

Why pickling/deepcopy/JSON/YAML representers? Once list items and dict keys are wrapped in WithProvenance types, they flow through Autosubmit's Job persistence pipeline (pickle to .pkl and SQLite), config merging (copy.deepcopy for FOR-loop expansion), JSON export to the historical DB, and YAML re-dumping. Without the serialization support, Autosubmit would crash at any of these points. The pickle reducers intentionally strip provenance (reducing to plain builtins) — zero size/speed overhead. The deepcopy support preserves provenance during config processing where it's still needed.

Performance: The pickle output is byte-identical to without yaml-provenance. A real expid t0lm with our yaml-provenance+autosubmit (provenance tracking) and with ONLY autosubmit (no provenance tracking) time autosubmit create t0lm produces:

with provenance:

real	0m8.822s
user	0m8.437s
sys	0m0.446s

without provenance:

real	0m6.175s
user	0m5.814s
sys	0m0.603s

MuhammadShafeeque · 2026-03-03T14:04:37Z

@mandresm the PR is ready for review.

mandresm · 2026-05-22T20:14:28Z

I'd like to reduce the 2-3 seconds difference. But again, I need to review the whole paclage and then I should be able to improve its performance.

mandresm

Thanks for this PR @MuhammadShafeeque. Looks good to me in it's new functionalities and the architectural decisions made. I have a few suggestions for improvement though.

…upport - Added `wrap_computed`, `transfer_provenance`, and `annotate_dict` functions to `_helpers.py`. - Introduced `_serialization.py` for pickle reducers, YAML representers, and a JSON encoder. - Updated `load_yaml` to accept file-like objects. - Added tests for new helper functions and serialization features.

…puted values

…on support

… full path

…r pickling support

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

…r helper functions and load_yaml file-object support

…test for full history transfer

…e; update deepcopy test description

…ontext in CONTEXT.md and rename test file for consistency

…tion and update related imports; register YAML representers at class creation in wrapper and enhance tests for serialization

… of NoneWithProvenance; update test for None handling (debugging after review)

…ce in _dict.py and _helpers.py

…tion

mandresm

Hi @MuhammadShafeeque, here another round of questions and suggestions

mandresm · 2026-07-01T16:03:21Z

+    if value is None:
+        return NoneWithProvenance(value, provenance)


This was removed from wrapper_with_provenance_factory. If None is never subclassable, then keep that through out the code. Or am I understanding something wrong?

Both are intentional. YAML-loaded None stays plain None (the factory) because its provenance lives on the key, the loader wraps keys precisely so empty values keep a trace, so is None still works and nothing is lost. wrap_computed/set_provenance inject values whose key isn't provenance-wrapped, so there the value itself must carry provenance via NoneWithProvenance. Added a one-line comment at each site to make the split explicit

I think this behavior is very confusing. I understand you want is None to be evaluated correctly, but having some Nones be Nones and others be NoneWithProvenance is really inconsistent and dangerous.

I'll have to think about it.

Please, refer to the comment above for further discussion on the NoneWithProvenance

Agreed, and fixed. None is now wrapped in exactly one place. The factory (_wrapper.py, elif value is None: return NoneWithProvenance(value, provenance)). Every path (YAML loader, put_provenance, set_provenance, wrap_computed, transfer_provenance) routes through it, so there's no more half-and-half.

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

…e classes

…unctions

…acking in nested structures

…y to clarify provenance handling

…nce to ensure proper memoization

mandresm

This PR at the moment changes turns some Nones to NoneWithProvenance and not others. I'm not okay with this inconsitency and I don't fully understand why it is needed. @MuhammadShafeeque, please see my comments below. IMO this PR is getting very close to a mergable state :)

mandresm · 2026-07-07T08:05:07Z

                    val, provenance.get(key, []), config=self._config
                )
            elif hasattr(val, "provenance"):
                self[key].provenance.extend(provenance.get(key, {}))


Suggested change

self[key].provenance.extend(provenance.get(key, {}))

elif val is None:

# No provenance-wrapped key here (unlike the YAML loader), so carry it on the value.

self[key] = NoneWithProvenance(val, provenance)

I went a slightly different way than this literal suggestion, and I think it lands where you wanted: instead of special-casing None here, I restored None-wrapping in the factory (your _dict.py:115 suggestion), so put_provenance's existing else: wrapper_with_provenance_factory(val, provenance.get(key, None)) already produces a NoneWithProvenance. Adding the elif too would re-introduce the per-method special-case that this review is (rightly) getting rid of. Same outcome — the null keeps its provenance — with one code path. Guarded by test_none_value_keeps_provenance. Happy to add the explicit branch back if you'd rather have it spelled out.

mandresm · 2026-07-07T08:08:56Z

+    prov = getattr(original, "provenance", None)
+    if not prov:
+        return result
+    return wrapper_with_provenance_factory(result, prov)


No support for NoneWithProvenance, but again, I think the factory should be doing that for you

mandresm · 2026-07-07T08:46:15Z

+            elif val is None:
+                # No provenance-wrapped key here (unlike the YAML loader), so carry it on the value.
+                self[key] = NoneWithProvenance(val, provenance)


I understand the decision of not having NoneWithProvenance in the factories to conform with PEP 8 and have is None be evaluated correctly. But then why should we keep it here? I would suggest to remove completly NoneWithProvenance or keep it in the factories.

The whole design of the tool is based on keeping provenance in the values, consequently, I'd keep the NoneWithProvenance. I don't have a problem with NoneWithProvenance is None evaluating to False. Maybe this is a problem for AS but then one can simply change in AS my_val_that_it_a_NoneWithProvenance is None or isinstance(my_val_that_it_a_NoneWithProvenance, NoneWithProvenance. One can even make a function out of it (yaml_provenance.is_none_like).

In summary, removing NoneWithProvenance from the wrappers is a decission that goes against the ground design of this package and could have very important implications and lead to missfunction. Please, consider bringing it back.

Brought it back and centralized it in the factory. That's now the single place NoneWithProvenance is constructed. I also added the exact helper you proposed: yaml_provenance.is_none_like(x) → value is None or isinstance(value, NoneWithProvenance), exported at top level. The special-case that was here in set_provenance is gone (folded into the factory). Value-level provenance for nulls is preserved end-to-end — e.g. empty_value: null now dumps as empty_value: # file,line:2,col:14 instead of "no provenance".

mandresm · 2026-07-07T08:47:32Z

+    if value is None:
+        return NoneWithProvenance(value, provenance)


Please, refer to the comment above for further discussion on the NoneWithProvenance

mandresm · 2026-07-07T08:49:53Z

-        return NoneWithProvenance(value, provenance)
+        # Return plain None so that `x is None` identity checks work as expected.
+        # NoneWithProvenance breaks Python's idiomatic `is None` test (PEP 8).
+        # Provenance on a None value is inaccessible anyway (None has no attributes).
+        return None


As suggested above, I would revert this back.

Please, let me know what are the consequences of reverting this back. There might be alternative solutions to just partially removing NoneWithProvenance from the package.

I cannot fully confirm the consequences of reverting this yet. It will likely require corresponding changes on the Autosubmit side and validation with a real experiment.

The VM is currently unavailable, so I cannot test it immediately. I will check this later once the VM is available and report back.

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

…updating wrapper behavior for None values

…d remove redundant imports

MuhammadShafeeque requested a review from mandresm March 2, 2026 11:02

MuhammadShafeeque self-assigned this Mar 2, 2026

MuhammadShafeeque added enhancement New feature or request autosubmit features specifically added/modified to work with autosubmit labels Mar 2, 2026

MuhammadShafeeque linked an issue Mar 2, 2026 that may be closed by this pull request

Integrating yaml-provenance into autosubmit #1

Open

MuhammadShafeeque marked this pull request as ready for review March 3, 2026 14:03

MuhammadShafeeque force-pushed the enhanced-yaml-provenance branch from 483688a to 82b918c Compare March 23, 2026 15:37

mandresm requested changes Jun 7, 2026

View reviewed changes

MuhammadShafeeque and others added 17 commits June 9, 2026 17:57

Improve provenance comment formatting to omit line and column for com…

64f34ad

…puted values

Fix: handling DEPENDENCIES

360dfb9

fix 2: Dependencies

3a2a5b3

Enhance documentation to include new helper functions and serializati…

e5308b3

…on support

Enhance ProvenanceLoader to accept file-like objects while preserving…

cc2ae71

… full path

Remove unnecessary directories from .gitignore

5ce45d7

Add __reduce__ method to DictWithProvenance and ListWithProvenance fo…

d7022b7

…r pickling support

Apply suggestions from code review 1

263eae4

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

Refactor imports in _helpers and _serialization modules; add tests fo…

b206157

…r helper functions and load_yaml file-object support

Enhance transfer_provenance to preserve full provenance history; add …

420c839

…test for full history transfer

Update annotate_dict to handle keys with dots; add test for key quoting

5696006

Remove register_pickle_reducers references from documentation and cod…

ed3357a

…e; update deepcopy test description

Refactor import statements in _serialization.py for clarity; update c…

dad43f0

…ontext in CONTEXT.md and rename test file for consistency

Refactor serialization module: remove register_yaml_representers func…

1b422bc

…tion and update related imports; register YAML representers at class creation in wrapper and enhance tests for serialization

Refactor wrapper_with_provenance_factory to return plain None instead…

8882142

… of NoneWithProvenance; update test for None handling (debugging after review)

Enhance provenance handling for None values by using NoneWithProvenan…

35a9d3d

…ce in _dict.py and _helpers.py

MuhammadShafeeque force-pushed the enhanced-yaml-provenance branch from e52b7c4 to 35a9d3d Compare June 9, 2026 16:05

Enhance provenance handling for None values in _add_eol_comments func…

58fdd8e

…tion

mandresm requested changes Jul 1, 2026

View reviewed changes

MuhammadShafeeque and others added 10 commits July 6, 2026 13:20

Update src/yaml_provenance/_dict.py

abc353c

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

Update src/yaml_provenance/_dict.py

b33919e

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

Update src/yaml_provenance/yaml_loader.py

116d08d

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

Update src/yaml_provenance/yaml_loader.py

08f4c9b

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

Refactor deepcopy methods in DictWithProvenance and ListWithProvenanc…

7af5eda

…e classes

Clarify provenance handling in DictWithProvenance and wrap_computed f…

2e3b5b0

…unctions

Enhance annotate_dict and add _annotate_list for better provenance tr…

97ca35f

…acking in nested structures

Improve __deepcopy__ documentation in wrapper_with_provenance_deepcop…

0914cb9

…y to clarify provenance handling

Refactor __deepcopy__ method in ListWithProvenance

0345322

Fix deepcopy implementation in DictWithProvenance and ListWithProvena…

47b50f5

…nce to ensure proper memoization

mandresm requested changes Jul 7, 2026

View reviewed changes

MuhammadShafeeque and others added 5 commits July 7, 2026 12:09

Apply suggestions from code review

96e436e

Co-authored-by: Miguel <63242832+mandresm@users.noreply.github.com>

Enhance provenance handling by introducing is_none_like function and …

dc04cb0

…updating wrapper behavior for None values

Refactor _register_yaml_representer to clarify dependency handling an…

6097be1

…d remove redundant imports

Fix provenance tracking for None values in ProvenanceLoader

20bac53

Fix clean_provenance to handle ruamel ScalarFloat and prevent crashes

b423cc5

		if value is None:
		return NoneWithProvenance(value, provenance)

Uh oh!

Conversation

MuhammadShafeeque commented Mar 2, 2026

Summary

What's new

Helper functions (_helpers.py)

Serialization (_serialization.py — new module)

copy.deepcopy support

YAML dumper improvements (_yaml_dumper.py)

YAML loader improvements (yaml_loader.py)

Files changed

Testing

Uh oh!

MuhammadShafeeque commented Mar 2, 2026

Uh oh!

mandresm commented Mar 2, 2026

Uh oh!

MuhammadShafeeque commented Mar 3, 2026

Uh oh!

MuhammadShafeeque commented Mar 3, 2026

Uh oh!

mandresm commented May 22, 2026

Uh oh!

mandresm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mandresm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mandresm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

`copy.deepcopy` support

YAML dumper improvements (`_yaml_dumper.py`)

YAML loader improvements (`yaml_loader.py`)