Automatically re-generate stubs on cfa-dataops update

[damonbayer]

CDCgov/cfa-dataops#119 Added new stubs, but this catalog doesn't have them yet. We can update them manually, but it would be great to do it automatically somehow.

[uhc7]

@ryanraaschCDC -- Thanks for assigning this to me.

My understanding is that new stubs are generated in cfa-dataops, but they currently need to be updated manually in cfa-catalog-pub. The goal of this issue is to automate that synchronization so the catalog stays up to date whenever new stubs are added or regenerated.

• Do we have documented the current manual workflow between cfa-dataops and cfa-catalog-pub.? if not i'll start from documenting these first.
• Identify where and how the stubs are generated.
• Evaluate the best automation approach (GitHub Actions/workflow trigger, reusable workflow, or other mechanism).
• Implement and test an automated solution that keeps the catalog in sync while minimizing manual intervention.

I'll update this issue with the proposed design before implementation so everyone is aligned.

[ryanraaschCDC]

sounds good, @uhc7. There's a section on generating stubs in the repo. https://raw.githubusercontent.com/CDCgov/cfa-dataops/refs/heads/main/README.md . There might be a way to run the stubs when the catalog gets installed. i think that would be ideal. Please do some research on what's possible but hold off on any implementation. I'll talk through it with Damon.

[uhc7]

@ryanraaschCDC Update:

I reviewed the current stub-generation flow in cfa-dataops and reproduced the issue locally with cfa-catalog-pub.

• cfa-catalog-pub currently tracks the generated stubs under typings/cfa/dataops/.
• Installing cfa-catalog-pub successfully installs cfa-dataops and the public catalog is correctly discovered by cfa-dataops, but installation itself does not regenerate the tracked stubs.
• Running dataops_catalog_stubs manually after installation successfully regenerates them.
• The regenerated output differs from what is currently committed in cfa-catalog-pub (23 insertions / 9 deletions across the two stub files).
• The differences include newer cfa-dataops API/type information such as VersionMetadata, resolve_version(), get_ref(), blob access methods, and updated method signatures.

I also looked into the idea of generating the stubs when the catalog is installed. It appears technically possible to trigger generation after the catalog is available, but the existing generator is intentionally project-local/environment-aware (./typings) and depends on the catalogs installed in the active environment. Because of that, tying it directly to Python package installation may have some packaging/location implications that we should agree on first.

I’m holding off on implementation as discussed. I documented the reproduction and options so we can review whether install-time generation, CI validation/regeneration, or another trigger is the best fit before making changes.

[ryanraaschCDC]

@damonbayer I'm going to share a document Jayaraj put together on different approaches to solve this PR. Specifically, there is a question section that I've responded to in red font, but we would love your input. I think you have a better understanding than I do of how this stub generation should work.