Leverage dataplugin types - #440
Draft
ctrl-schaff wants to merge 10 commits into
Draft
Conversation
This change only updates the dumper templates at the moment. It deletes the dumper templates and replaces it with a metatype and a factory for generating the dumper class instances. The docker dumper template is replace with a transport instance that determines what protocol we use with our dumper instance. If more work is needed for the docker dumper template, we can always make a metatype for it as well Once this proof of concept has been tested in the job manager loop, we can then do the same with the uploader, which while more complex should go faster as I've already thought about how to do it through the dumper transistion
Was causing a warning when running pytest, so this eliminates that warning
Adds the metatypes module with the shared infrastructure between the dumper and uploader type that's defined along with our definition to build our data type
This still uses the older version of how I thought the class would look. We need to apply the changes learned from the dumper to the uploader, mainly in separating out the function calls to be external to the type defintion
The loader no longer is responsible for creating the type definitions from the templates
ctrl-schaff
requested review from
DylanWelzel,
everaldorodrigo,
jal347,
newgene and
shuchenliu
March 26, 2026 20:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces our dataplugin templates with type based class generation in order to help us eventually migrate to updating our job manager strategy to use forkserver