Skip to content

Preserve Hugging Face metadata when checkpointing#647

Open
dylantirandaz wants to merge 1 commit into
PsycheFoundation:mainfrom
dylantirandaz:fix/631-checkpoint-metadata
Open

Preserve Hugging Face metadata when checkpointing#647
dylantirandaz wants to merge 1 commit into
PsycheFoundation:mainfrom
dylantirandaz:fix/631-checkpoint-metadata

Conversation

@dylantirandaz

Copy link
Copy Markdown

Summary

  • Download text model files so tokenizer merges are available during checkpoint initialization
  • Preserve tokenizer, config, and custom Python files when writing checkpoints
  • Add focused coverage for model download selection and checkpoint metadata filtering

Fixes #631

Testing

  • cargo check -p psyche-client -p psyche-data-provider --tests --features tch/doc-only
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

a checkpoint hf repo != the init repo does not upload all the original repo fiules, *only* the safetensors

1 participant