Add Toolkit template variable substitution for toolkit YAML reads - #1615
Add Toolkit template variable substitution for toolkit YAML reads#1615poocog wants to merge 7 commits into
Conversation
Resolve {{ variable }} placeholders from Toolkit config during format=toolkit imports, with a compact once-per-directory config load.
Co-authored-by: Cursor <cursoragent@cursor.com>
Thread a single options object through importers and session internals, share Args docs, and dedupe YAML load/quote paths. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1615 +/- ##
==========================================
+ Coverage 91.85% 91.92% +0.06%
==========================================
Files 126 131 +5
Lines 7468 7983 +515
==========================================
+ Hits 6860 7338 +478
- Misses 608 645 +37
🚀 New features to boost your workflow:
|
…reads. Treat module-local default.config.yaml as a variable source, not the project root, and ignore build_info/config files during directory import. Co-authored-by: Cursor <cursoragent@cursor.com>
doctrino
left a comment
doctrino
left a comment
There was a problem hiding this comment.
See my longer comment on the ToolkitReadOptions.
I think this is a question of should we do this? And if so, how should we do this? Not the implementation itself. I am happy to take a discussion on it when I am back, or take it without me before then.
| reuse the flattened map for every YAML file under that directory. | ||
| """ | ||
|
|
||
| from __future__ import annotations |
There was a problem hiding this comment.
This is for Python 3.9 and earlier which we have never supported for neat.
| data_model_file: Path | None = None, | ||
| data_model_file: Path | str | None = None, | ||
| fix: bool = False, | ||
| options: ToolkitReadOptions | None = None, |
There was a problem hiding this comment.
Ideally, we want to make the public API, which this is part of only basic types (string, integers, boolean). Here you are introducing a new class that the user must import in their notebook. This class is not publicly exposed to the user, so they cannot import it.
Looking at ToolkitReadOptions, I see that it might be necessary, but I am not really sure. I think this needs some pondering.
Current implementation looks like this
from cognite.neat._data_model.importers._toolkit_variables import ToolkitReadOptions
neat.physical_data_model.read.yaml(io="path-to-toolkit-directory", format="toolkit", options=ToolkitReadOptions(...))This is starting to feel clunky and unintuitive.
An alternative is to split this into two methods, one for reading neat_yaml and one for toolkit_yaml
neat.physical_data_model.read.yaml_toolkit(io="path-to-toolkit-directory", format="toolkit", env="dev")and
neat.physical_data_model.read.yaml_neat(io="path-to-neat-yaml", format="toolkit", fix=True)Then the toolkit version is for running a CI, while the neat version is for developing. However, that raises more questions like should we support neat format in CI through an interface that is intended to be used in a notebook?
Yamale validation schemas such as data_model_container.yaml are no longer treated as DMS container resources during directory reads. Co-authored-by: Cursor <cursoragent@cursor.com>
…ree. Load variables from each YAML file's module path so nested variables.modules keys apply correctly. Skip parsing sibling DataModels and views when data_model_file selects one model. Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
Description
Resolve Toolkit
{{ variable }}placeholders when reading module YAML withformat="toolkit", using Toolkit config (default.config.yaml, env overlays,cdf.toml, and module overrides). Options are collated inToolkitReadOptionsand config is loaded once per directory; strategy is documented in_toolkit_variables.py.Bump
Changelog
Added
format="toolkit"YAML reads viatoolkit_env,toolkit_config, andtoolkit_version.