Python: beamtime read/list, ingest progress, notebooks - #36
Merged
Conversation
…atures - Introduced a three-panel FITS preview in the TUI, displaying raw, Gaussian-filtered images, and a cropped view with Gaussian fit contours. - Added a new function to convert 2D f32 arrays to RGBA using a rainbow colormap, improving visualization capabilities. - Updated the `Cargo.toml` to include additional features for the `plotters` dependency, enhancing plotting options. - Refactored the TUI preview logic to support the new three-panel layout and improved error handling for image processing. - Enhanced AGENTS.md documentation to reflect the new FITS preview features and usage instructions.
- Added new dependencies for NEXAFS functionality in `pyproject.toml`, including `scienceplots`, `datasets`, and `huggingface-hub`. - Updated the development dependencies to include `nbconvert`. - Removed obsolete NEXAFS files and components, including the `nexafs_widget.ipynb`, `directory.py`, `normalization.py`, and `widget.py`, streamlining the codebase. - Refactored the NEXAFS module to utilize the new accessor for improved data handling.
…ts handling - Updated the optical constants loading mechanism in the UniTensorSLD class to use interp1d instead of PchipInterpolator, improving performance and simplifying the code. - Added validation for required columns in the optical constants dataframe, ensuring robustness in data handling. - Refactored the calculation of the complex index of refraction to utilize a new property for better clarity and maintainability.
- Updated the n property in the UniTensorSLD class to explicitly convert the energy value to a float, ensuring consistent data type handling and preventing potential type-related issues in calculations.
- Introduced a new catalog layout that consolidates multiple beamtime databases into a single catalog per parent directory, improving data organization and access. - Updated the TUI to support the new catalog structure, allowing for better navigation and management of beamtime entries. - Implemented parallel ingestion features to optimize data processing and reduce wall time during catalog updates. - Enhanced the FITS preview functionality with improved error handling and visualization options. - Refactored various components for clarity and maintainability, including updates to the AGENTS.md documentation to reflect the new features and usage instructions
- Added details on thread safety and concurrency for `CatalogHandle` and `BeamtimeState`, clarifying the handling of background ingest threads and their interaction with the main event-loop. - Revised the conditions for marking experimentalists as `ExptPolicy::Auto`, ensuring a minimum folder count for auto-resolution. - Enhanced the description of the ingest process, including cancellation behavior and partial write visibility during ingestion. - Updated the parameter documentation for `ingest_beamtime` to reflect new options while maintaining backward compatibility.
Move Launcher-specific fields from App struct to LauncherScreenState variant on the navigator stack. App now has exactly 8 fields as specified: navigator, preview_tx, beamspot_rx, preview_cmd_rx, layout, keymap, theme, needs_redraw. Changes: - Add LauncherScreenState wrapper struct with launcher and open_dir fields - Add ScreenState::Launcher variant on navigator - Remove: screen, launcher_state, open_dir_active/path/entries/list_state/focus - Add current_screen() derived from navigator.stack - Add accessor methods for launcher state and open_dir fields - Update run.rs: use current_screen() and accessor methods - Update ui.rs: use current_screen() and open_dir_active() - Update launcher methods to access state from navigator.stack - Update browser.rs: use current_screen() and accessor methods cargo check --features tui,catalog,parallel_ingest: PASS (0 errors)
…olution modal Implement the NAS directory navigator (Explorer screen) for data root browsing: Core Components: - ExplorerState: maintains current directory, entries list, sorting, and layout policy - HeuristicClassifier: detects beamtime patterns and auto-classifies directory structures - ConfigModal: interactive resolution UI for unresolved experimentalist folders Features: - Auto-detect experimentalist folders and beamtime subdirectories - Heuristic pattern matching for date-based folder names (YYYYMMM, YYYY-MM-DD, etc.) - Interactive modal for manual policy configuration (custom pattern, ignore, accept default) - List navigation with sorting support (Name, Modified, Size, FileCount) - CLI mode auto-detection: Explorer for data roots, Beamtime for individual beamtimes UI: - Explorer table view: Name | Kind | Modified | FITS | Status columns - Marked entries: ⚑ flag for experimentalists needing resolution - Bottom status bar with keybind hints - ConfigModal with pattern validation and preview of classified entries Architecture: - Navigator stack extends to support Explorer and ConfigModal screens - Screen enum adds Explorer variant - Browser.rs detects mode based on directory content (all dirs → Explorer, has .fits → Beamtime) Note: Phase 3 catalog integration (fits_count, catalog_status) deferred - shows "--" placeholders cargo check --features tui,catalog,parallel_ingest passes
Add missing cases for Explorer and ConfigModal ScreenState variants in the current_screen() method so they correctly return Screen::Explorer instead of falling through to Screen::Beamtime.
…d CatalogHandle - Add schema migration for experimentalist and data_root columns to bt_beamtimes table - Create explorer_query module with list_experimentalists, list_beamtimes_for_expt, and catalog_status_for_path functions - Define DbCatalogStatus enum (Indexed, Stale, NotIndexed) for catalog status tracking - Implement CatalogHandle methods for querying experimentalists, beamtimes, and catalog status - Adapt ingest_beamtime signature with data_root, experimentalist, and cancel parameters - Add resolve_ingest_target to determine correct catalog.db location based on data_root - Add upsert_beamtime_record to update bt_beamtimes with context metadata and timestamps - Add ingest_beamtime_pipelined_with_context for parallel ingest with new parameters - Maintain backward compatibility - old ingest_beamtime callers unchanged - cargo check --features tui,catalog,parallel_ingest passes
Implement auto-ingest workflow when entering Beamtime entries from Explorer: - Add cancel_flag to BeamtimeState for ingest cancellation on Esc - Implement auto-ingest with background thread when user enters Beamtime entry with NotIndexed or Stale catalog_status - Enhance try_recv_ingest to handle ingest completion and reload from DATA-level catalog using CatalogHandle - Update Explorer entry status to Indexed when ingest completes - Handle cancel ingest: Esc during IngestingDirectory pops the screen - BeamtimeState can be popped directly from Beamtime screen if data_root is set (Phase 4 vs legacy launcher) - Export ingest_beamtime_with_context from catalog module This completes the auto-ingest -> DATA catalog query -> status update cycle for Phase 4, removing dependency on per-beamtime catalogs when data_root context is available.
- Added new dependencies: `tiled`, `ipyaggrid`, `altair`, and `panel` to `pyproject.toml`. - Introduced a new `nsls` section in `pyproject.toml` for additional package management. - Updated `uv.lock` with new package entries including `accessible-pygments`, `alabaster`, and `altair`. - Enhanced the `continual-learning` state management with updated timestamps and paths. - Improved the `tui` application structure by refining the handling of catalog-related features and explorer state management. This commit lays the groundwork for improved data visualization and management within the TUI, aligning with ongoing enhancements in the catalog structure.
- Added .DS_Store to .gitignore to prevent macOS metadata files from being tracked. - Deleted CLAUDE.md and CONTRIBUTING.md as they are no longer needed. - Updated AGENTS.md to reflect changes in project documentation and structure.
Migrate ingest/query to normalized fits_files and catalog_tags; add layout detection; strict catalog.db resolution in Rust and Python.
Replace rusqlite catalog with Diesel + embedded migrations; single-pass ingest writes SQLite rows and local zarr. Add paths (directories, env overrides), IngestParallelism (Rayon pool, PyO3/TUI), and schema updates. Remove materialize/profile_persist split; update AGENTS and Python IO.
…lite-diesel Diesel catalog: migrations, unified ingest + zarr, parallelism
Add read_beamtime/list_beamtimes Python APIs; ingest layout and progress callbacks (tqdm/rich); per-scan parallel FITS reads; scan_from_catalog_for_beamtime. Include exploration notebooks and lockfile updates.
HarlanHeilman
marked this pull request as ready for review
April 13, 2026 20:34
Remove vendored .github/skills, replace lint-format workflow, bump publish Python to 3.12, pin 3.12 with .python-version. Update catalog/TUI, IO helpers, ty config, and drop nexafs test.
Ignore Cargo target trees, loose rlib/o/obj/a and MSVC coverage noise; add pyd/dll, ty cache, pip-wheel-metadata, and clarify tracked .python-version.
Member
Author
|
failing one un-important test. LGTM 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
read_beamtime/list_beamtimes, catalog helpers (scan_from_catalog_for_beamtime,beamtime_ingest_layout), ingest progress callbacks for tqdm/Rich, per-scan parallel ingest in Rust, exploration notebooks, and continual-learning sections inAGENTS.md.Also ignores
.cursor/hooks/state/in git.Base
main(draft for review).