Interface to extract only the *.yaml metadata

[scarlehoff]

So that the whole eko doesn't need to be extracted.

[felixhekhorn]

There is

eko/src/ekobox/cli/inspect.py

Line 36 in 895f37f

def sub_cards(operator: EKO):

but I'm not sure someone ever used it ... one would need to add the metadata.yaml though

[scarlehoff]

The problem is here

eko/src/eko/io/struct.py

Line 376 in 895f37f

def read(

in order to have the theory card so that it is available to read (with .raw or otherwise), you have to extract it.

[felixhekhorn]

@lol782 if you wish to have another small project - this one might be better suited. It should be a simple file operation involving tarfile.TarFile.extractfile or similar in one way or another. The crucial goal here is to avoid the extraction of the actual (very) big operators, but just obtain the metadata + theory card + operator card (i.e. the (small) yaml files).

[felixhekhorn]

Note that you need to have the metadata already available before you can load the theory or ops card

eko/src/eko/io/struct.py

Lines 133 to 137 in bf40621

	def theory_card(self):
	"""Provide theory card, retrieving from the dump."""
	raw_th = yaml.safe_load(self.paths.theory_card.read_text(encoding="utf-8"))
	if self.metadata.data_version in [1]:
	raw_th = v1.update_theory(raw_th)

In order to have a clear interface it might make sense to strip the loading away from the main class (eko.io.struct.EKO) since that mostly refers to the main (big) operators and then just reuse the new code there

[lol782]

🫡

[lol782]

Did some resarch before starting. The expensive bit is in EKO.read: with extract=True it unpacks the whole archive (big operators included) just to reach the small yamls (just like u mentioned). And theory_card / operator_card read from the extracted folder, so there's currently no way to get cards without full extraction.
One thing I noticed: the cards already have from_dict, but Metadata.load couples reading-the-file with building-the-object — so metadata is the missing seam.

Plan:

• Add Metadata.from_dict (and have load use it), so metadata can be built from a parsed dict.
• Add standalone functions that read only the small yamls straight from the tar via tarfile.extractfile — metadata first (for data_version), then theory, then operator.
• Factor the version-upgrade logic out of the existing properties so both paths reuse it.

Questions:

• New functions in struct.py, or a separate module?
• Is Metadata.from_dict the right seam, or did you have another in mind?

[felixhekhorn]

Mmm, it is slightly more complicated ...

Add Metadata.from_dict (and have load use it), so metadata can be built from a parsed dict.

eko/src/eko/io/metadata.py

Line 77 in ab85bf7

content = cls.from_dict(raw)

so Metadata.from_dict exists already (from inheritance) and Metadata.load is using it.

New functions in struct.py, or a separate module?

I've no strong opinion, but since that file is long (and complicated) already, I would rather attribute it to runcards.py (since we return 2 objects from there)

Is Metadata.from_dict the right seam, or did you have another in mind?

without having it though fully through (all names are mere suggestions), I would say we need

• one function load_meta_and_cards_from_tar(path) (or similar name),
• which will call Metadata.from_raw(yaml), which takes raw yaml output and does the necessary adjustments before from_dict
• which calls TheoryCard.from_raw(yaml, data_version), which takes raw yaml + data version and does the necessary adjustments before from_dict.
• which calls OperatorCard.from_raw(yaml, data_version, th_yaml), which takes raw yaml + data version + raw theory yaml and does the necessary adjustments before from_dict.
• Those can then called directly by EKO in turn