Re-evaluate usage of `recipe-id` to store output artifacts to PVC

[rushirajnenuji]

Re-evaluate usage of recipe-id to store output artifacts to PVC

Currently, we expect the outputs of a recipe to be stored on qgnet-ogdc-workflow-pvc, under a directory called {recipe_id}.

If that directory already exists, the runner assumes the workflow has already run and published outputs. The --overwrite flag can be used to completely remove the recipe_id directory.

recipe_id is currently something that we let recipe authors write. We expect it to be unique, but there's not a way to enforce that at the moment. Two recipes could use the same ID, which could result in one recipe overwriting another!

Instead of having authors provide a recipe_id, we could construct one from a name + a unique identifier (hash/checksum of the recipe itself?).

[trey-stafford]

We might want to consider some other mechanism for indicating that a recipe has been published successfully (other than a directory's existence). Perhaps a lookup of workflows matching the recipe ID and checking for success status? Or some other database that tracks OGDC recipe executions?

[trey-stafford]

I added a separate issue to address generating a recipe_id rather than requiring that the recipe author provide it: #93

[trey-stafford]

#95 implements the update to generate a recipe_id from the recipe name.

[trey-stafford]

Some thoughts copied from #93:

One idea I had would be to compute a checksum for the meta.yml file and use it as part of the recipe ID. Maybe something like:

    @computed_field
    @cached_property
    def id(self) -> str:
        k8s_name = self.name.lower().replace(" ", "-")
        model_json_str = self.model_dump_json(exclude={"id"})
        checksum = hashlib.sha256(model_json_str.encode("utf-8")).hexdigest()

        recipe_id = f"{k8s_name}_{checksum}"

        return recipe_id

Which would produce an id that looks like: test-ogdc-workflow_d621a8c2fc8c94a8f9b26e9538a1fcc21968bdffff03d797b18a36c372e41a9f

(I think we could shorten that checksum).

Using a checksum like this makes the contents of the yaml file the driver for the resulting ID, which has some consequences:

• Any change to the yaml file result in a new ID.
  • Our "publish-to-pvc" approach names the workflow/output directory after the recipe_id, so changes to config would be clearly reflected in different directories.
  • This could be a "good" thing if someone changes a parameter and expects a new output
  • This could be a "bad" thing if someone tweaks a step that is intended to fix an existing recipe that didn't complete properly.
• Any change to sidecar recipe files (like recipe.sh) would not generate a new checksum, which means the good/bad things mentioned above don't happen in this scenario.

If we go this route, we'd likely want to create a checksum for the whole recipe dir, sidecar files included.

---

Thinking about use cases, with respect to determining if a workflow needs to run or not, maybe we should look at this from the perspective of the final output - if the thing at the final output location exists, then the recipe has been published and nothing needs to be done. I'm imagining e.g., a DataONE publication configuration in the meta.yaml that we can load and use to query for if the requested output has been published. If so, we can tell the user that they need to increment the version and update associated configuration to upload to a new destination if they want to make an update.

In the rare case of a config collision, the new recipe author knows to rename their recipe or output configuration - someone else has already published using this config.