Skip to content

Improve lazy asset loading: use tarred files and checksums - #303

Merged
sibocw merged 1 commit into
dev-v2.1.1from
tarred-remote-assets
Jul 3, 2026
Merged

Improve lazy asset loading: use tarred files and checksums#303
sibocw merged 1 commit into
dev-v2.1.1from
tarred-remote-assets

Conversation

@sibocw

@sibocw sibocw commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Clean up asset lazy loading module which is currently a code smell:

  1. remote asset files on s3 are always tarred and paired with an sha256 checksum (.tar and .checksum)
  2. flygym downloads the tar file into a temp dir and verifies data integrity via checksum
  3. flygym decompresses the tarball into a folder
  4. flygym calls pathlib.Path(temp_folder_path),replace(target_cache_dir), which is atomic and raises an error if some other thread has already created the target dir in the meantime (in which case it discards the temp dir holding the downloaded data)
  5. added a script to make the tar and checksum files.

… loading. Add script to generate tarballs and checksum files
@sibocw
sibocw merged commit ba9ed22 into dev-v2.1.1 Jul 3, 2026
5 checks passed
@sibocw
sibocw deleted the tarred-remote-assets branch July 3, 2026 13:24
@sibocw sibocw changed the title Implement tarred assets for S3 and add tarball generation script Improve lazy asset loading: use tarred files and checksums Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant