Skip to content

Graphnetcore dataload compat - #22

Merged
JoKircher merged 8 commits into
mainfrom
graphnetcore-dataload-compat
Aug 5, 2026
Merged

Graphnetcore dataload compat#22
JoKircher merged 8 commits into
mainfrom
graphnetcore-dataload-compat

Conversation

@JoKircher

Copy link
Copy Markdown
Member

Summary

Brings the package up to GraphNetCore v0.4 and adds first-class data-import
converters
(VTK and CSV) with a new user guide and a regression-tested suite.
The Documenter build is green and Aqua checks pass.

GraphNetCore v0.4 migration

  • Depends on the registered GraphNetCore v0.4.1 (no git [sources]).
  • Model/params/state now live in a Lux TrainState; parameter updates go through
    Setfield.@set! instead of field reassignment.
  • Online-normaliser state is reconstructed from checkpoints; df_step schema widened
    and legacy 2-col checkpoints auto-migrated.
  • Pins GPUCompiler to ~1.9 for Reactant/CUDA compatibility.
  • Updated dataset.jl / graph.jl / solve.jl / strategies.jl, the example scripts,
    and test_normalizer.jl accordingly. GPU-verified on both fixtures.

Data import (new)

  • vtk_to_hdf5 — converts ParaView / DualSPHysics VTK particle trajectories into the
    training HDF5 layout, one trajectory group per source:
    • XML .pvd / .pvtu / .vtu: the wrapper formats are parsed internally (ReadVTK can
      open neither), and only leaf .vtu pieces are handed to ReadVTK; empty (0-point)
      frames are detected and skipped before ReadVTK is called.
    • Legacy BINARY .vtk (POLYDATA, e.g. DualSPHysics PartAll_*.vtk) via a built-in
      reader — ReadVTK doesn't handle legacy .vtk.
    • Case-insensitive field mapping with alias fallback, id-based row alignment across time,
      recompute_acc (also automatic when no acceleration field is present), and either a
      user-provided or auto-generated (write_meta) meta.json.
    • New deps: ReadVTK, LightXML.
  • csv_to_hdf5 refactored to share the on-disk writer (_write_trajectory!) with the
    VTK path so both stay byte-compatible.
  • Fixed an always-throwing Dataset(datafile, metafile, args) constructor
    (format check used || where && was intended).

Docs

  • New "Loading Data" guide covering every import path (directory/split loader, explicit
    Dataset, HDF5/JLD2, the meta.json schema, CSV, and VTK).
  • Documented previously-undocumented exports (ModelConfig, save/load_model_config,
    update_meta!, visualize_eval) and fixed two README links that broke when the README is
    reused as the docs home — Documenter now builds with no warnings/errors.
  • Project logo + favicon.

Tests

  • test/test_converters.jl — CV1–CV11 (43 assertions), wired into runtests.jl. Fixtures are
    generated on the fly (WriteVTK for leaf .vtu; hand-written .pvd/.pvtu/empty/legacy .vtk)
    so the tests exercise our own parsing, empty-frame guard, and legacy reader. Also covers the
    csv_to_hdf5 round-trip.
  • End-to-end verified against real data: WashTec .pvtu collection and a legacy .vtk
    trajectory (517 particles × 151 frames), both loading through Dataset/getobs.
  • WriteVTK added as a test-only dependency.

Notes / follow-ups

  • Variable particle count across frames is currently rejected (trim to a constant-count window).
  • The skeleton meta.json only estimates default_connectivity_radius — set it explicitly
    for real training.
  • Not yet covered: a .vtkhdf reader and guided meta.json creation.

🤖 Generated with Claude Code

@JoKircher
JoKircher merged commit cfbe629 into main Aug 5, 2026
2 checks passed
@JoKircher
JoKircher deleted the graphnetcore-dataload-compat branch August 5, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant