Datacatalog#123
Open
karlafalcao wants to merge 190 commits into
Open
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
karlafalcao
marked this pull request as draft
June 3, 2026 22:49
…es; the list should update smoothly without flicker, layout shift, or perceived reload.
…output is a tuple or list.
…reconnecting an edge correctly restores data propagation. The connected data should propagate immediately upon reconnection, when executing the upstream node(s), and when the workflow is executed with the “Play” button.
Restructure utk_curio/backend/app/datasets/ from a flat module set with a half-migrated services/ package into clean architecture layers, with no behavior change. - Layout: domain/ (manifest, catalog_item, dedup, provenance, computed, constants, errors), application/ (catalog_service, listing, mutations, paths, preview, export, auto_install), repositories/ (local, registry, installed), install/ (installer, bundle), infrastructure/ (storage, file_meta, output_paths, catalog_utils), schemas/ (CatalogItem TypedDict, request parsers). Package root keeps only __init__.py, routes.py, service.py. - Collapse the double facade: service.py imports directly from the layers; remove services/service.py. - Replace the CatalogListing/Mutations/Path mixins with composition — DatasetCatalogService composes a shared PathResolver plus CatalogListing and CatalogMutations collaborators and delegates its public methods; collaborators route internal cross-reads through the facade to preserve overridability. - Split export/lineage helpers into application/export.py; move live-output request parsers into schemas/requests.py; thin routes.py with a shared _map_catalog_errors decorator. - Update all external and test imports to the new paths. Full datasets + projects test suites and the dataset-palette browser test pass.
added 13 commits
July 8, 2026 10:40
Importing a dataset only refetched the drawer's own catalog instance and never dispatched the shared DATASET_CATALOG_REFRESH_EVENT, so the palette provider and palette dropdown (which hold separate cache keys) kept serving their stale pre-import list until a full page reload. Every other mutation (install/uninstall/publish/computed auto-install) already fans out this event; import was the sole exception. Fire notifyDatasetCatalogRefresh() from onPickImport after a successful import, matching onInstall/onUninstall, so the imported dataset shows up immediately across all catalog surfaces.
OSM PBF files were selectable in the Data Catalog import picker (accept was null, so the OS dialog offered every file), but the importer has no PBF support — the backend rejected them with a generic "Unsupported dataset format: .pbf" and no guidance toward the path that does work. OSM PBF is consumed by an Autark node (UrbanSpec type: 'osm' -> db.loadOsm), not the catalog importer. - Front end: scope the import picker's accept list to the formats the importer actually ingests (plus .pbf/.osm.pbf, kept selectable on purpose), and short-circuit onPickImport for OSM PBF with an explicit warning toast naming the Autark node instead of a silent request that returns a generic error. - Back end: special-case .pbf in import_dataset with the same explicit message so direct API callers get the redirect too. - Single-source the extension list and message as shared constants on each side.
The user store (.curio/users/<user>/datasets/) is account-level storage, but list_catalog only ever surfaced user-store datasets *through* project refs, so a dataset registered without a project ref was invisible. This is additive groundwork for decoupling import from install: it adds a UserDatasetRepository that lists *imported* user-store datasets as standalone catalog items (origin 'imported', installed False) and merges them into list_catalog under include_hub. dedupe_items collapses the account row with a project-ref row by id, so an installed dataset still reads installed=True. Computed node-output copies (computed.* dirs) are deliberately excluded — their per-project path and auto-install behavior are unchanged.
…flow) Importing a file auto-attached it to the open dataflow via 'if dataflow_id: install_dataset(...)', conflating account-level registration with workflow-specific install. Remove that block: import now only registers the dataset in the account-level catalog (user store), surfaced by UserDatasetRepository. A node/dataflow linkage is created only when the user explicitly installs/selects the dataset (the existing install path already handles origin 'imported'). dataflow_id is still accepted for API compatibility but no longer triggers an attach. Computed node-output auto-install on execution is unchanged.
With register-only imports, importing no longer attaches to the open dataflow, so the drawer must reflect that: - onPickImport no longer upserts dataflowDatasets and reports a register (not install) success toast; it keeps the catalog-refresh fan-out. - importDataset hook no longer sends dataflowId (intent is explicit; backend no longer auto-installs). - Card install-state is now driven by the installed flag, not origin. The old 'origin is not hub/computed => installed' heuristic treated every imported dataset as installed (valid only when imports were auto-installed); a register-only import would otherwise show as installed with no Install button. source_node datasets stay installed-by-nature. The Installed tab / installed count already gate on installed === true, so a register-only import correctly stays in Browse with an Install action.
Replaces the Issue 3 redirect with real ingestion. A .pbf/.osm.pbf import is now converted server-side to a single GeoParquet: all non-empty GDAL OSM layers (points/lines/multilinestrings/multipolygons/other_relations) are read via geopandas+pyogrio and concatenated into one EPSG:4326 GeoDataFrame with an 'osm_layer' discriminator column, then installed through the normal imported- file pipeline (fmt='parquet'), so the existing loader, preview, export and the account-level listing all work unchanged. - new install/osm_pbf.py: convert_osm_pbf_to_geoparquet() with lazy geo imports and graceful OsmPbfError when the geo extras / GDAL OSM driver are missing or the extract has no features (mirrors the backend's existing optional-geo handling; framework deps stay non-geo). - import_dataset special-cases OSM_PBF_SUFFIXES to run the conversion and seeds the feature count (count_file doesn't parse parquet). - drop the obsolete OSM_PBF_IMPORT_MESSAGE reject constant. Tests: real sample PBF imports to a parquet dataset that previews with the osm_layer column; converter raises OsmPbfError (not ImportError) when geo is absent. Test skips cleanly if pyogrio/OSM driver is unavailable.
Now that the backend imports .pbf/.osm.pbf as a standalone GeoParquet dataset, treat it as a supported import format on the client: - add .pbf/.osm.pbf to IMPORTABLE_DATASET_EXTENSIONS (drives the picker accept list); remove the OSM-PBF redirect helpers (isOsmPbfFilename, OSM_PBF_IMPORT_MESSAGE, OSM_PBF_EXTENSIONS). - onPickImport no longer short-circuits OSM PBF; it flows through the normal register-only import path. - test: OSM PBF now goes through the importer and reports the register toast.
Instead of merging all OSM geometry into one mixed-geometry dataset, register one standalone GeoParquet catalog dataset per non-empty layer (points / lines / multilinestrings / multipolygons / other_relations) — each homogeneous-geometry and titled "<file> (<layer>)". This matches how OSM data is conventionally consumed and keeps each dataset directly usable. - osm_pbf.py: convert_osm_pbf_layers() returns a list of OsmLayer (name, geoparquet bytes, feature count); the merged converter + osm_layer column are gone. - mutations.py: extract _install_imported_bytes() (shared register-only install + item build) and _import_osm_pbf_layers() which installs one dataset per layer. The import route returns the first; the rest surface via the account-level catalog listing on refresh. The primary carries importedDatasetCount. - frontend: DatasetCatalogItem.importedDatasetCount; onPickImport reports "Registered N datasets from <file>" when a PBF yields multiple layers. Verified end-to-end on a real sample PBF (4 homogeneous datasets, EPSG:4326).
Foundation for presenting an OSM PBF import as one grouped, tabbed catalog entry. Each per-layer dataset now records a shared group_id (deterministic osm.x<pbf-hash>) and its own layer_name in the manifest, surfaced on the catalog item as groupId/layerName. - manifest: new optional group_id/layer_name fields (parse groupId/layerName, serialize back). - installer.install_imported_file: accept + persist group_id/layer_name. - mutations._import_osm_pbf_layers: mint one shared group id per PBF and thread it + layer name through _install_imported_bytes. - catalog_item base_item/item_from_manifest + CatalogItem TypedDict: surface groupId/layerName. Additive: items just gain two fields; no grouping behavior yet.
The per-layer OSM datasets now fold into a single group entry whose id is the shared group id and whose format is 'bundle', so the existing bundle card + tabbed preview render it and one action installs all layers: - domain/osm_group.py: build_osm_group_item() (synthetic bundle item: title = base name, parts = layers, installed = all-layers-installed) + collapse_osm_groups(). - listing.list_catalog(group_osm=True): fold members into the group entry (drawer); default off so the palette keeps individual layers. get_dataset and preview intercept a group id — get returns the synthetic item, preview returns one bundle 'part' per layer (part=<i> paginates layer i). - mutations.install_dataset/uninstall_dataset: a group id expands to install/ uninstall every member layer. - routes: /datasets/catalog accepts groupOsm. - domain/constants: OSM_GROUP_ID_PREFIX + is_osm_group_id + canonical layer order. Route test covers grouped vs flat listing, tabbed group preview, and install-all.
Turns on the bundle-shaped OSM group in the catalog drawer and installs it as 'all layers': - backend: the group item exposes groupLayerIds (its real per-layer dataset ids) + CatalogItem TypedDict; test asserts it. - frontend: catalog query carries groupOsm; the drawer sets groupOsm:true (the palette does not, so it keeps individual draggable layers). The backend then returns a format:'bundle' group entry, so the existing bundle card + tabbed preview render it with no new UI. - drawer install/uninstall expand an OSM group over its groupLayerIds and install/uninstall each REAL layer dataset — keeping the dataflow refs (which feed the saved spec via generateTrill) accurate, instead of writing a phantom group ref. Toast reports 'Installed N layers'. groupOsm is part of the catalog cache key so the drawer's grouped fetch and the palette's flat fetch stay separate.
The synthetic OSM group entry now displays as its own 'osm' dataset type (label 'OSM PBF', abbr 'OSM') instead of the generic 'bundle'. The tabbed detail preview is unchanged — it's driven by the preview response's 'bundle' flag, not the item's display format — so grouping/tabs still work. - new DatasetFormat value 'osm' with label/abbr, format-keyed CSS across every catalog surface (browse strip/card/dot/badge/link, list badge, detail chip, card avatar/accent) so the format-CSS completeness guard stays satisfied. - osm_group.build_osm_group_item emits format 'osm' (tags osm/pbf); catalog facet seed includes 'osm'; export stays disabled for it (no single file). Backend + frontend tests updated (group card format == 'osm').
Imported datasets previously left their account-level store folder behind on uninstall and reused it on re-import (id derived from file content), and the detail panel rendered both "Created" and "Last updated" from the same Curio-record timestamp with no field for the source file's own date. - Mint a unique imported.x<uuid> per import (drop content-hash keying and the identical-file fast-path); OSM group ids likewise become per-import uuids so a repeat import forms a separate group. - Uninstall now removes the imported store folder (manifest, data file, counts sidecar), guarded by dataset_usage so a dataset used by another dataflow is kept. - Add a source_updated_at manifest field fed by the browser File.lastModified; expose createdAt + sourceUpdatedAt on catalog items and show a distinct "Source updated" row, with "Created" now reading createdAt. - Cover the lifecycle with backend + frontend tests.
Change the "Created" by "Imported" label
added 8 commits
July 14, 2026 16:06
An OSM PBF import expands into one dataset per layer, all sharing a groupId. The palette rendered each layer as a separate top-level row; group them into one collapsible parent (the multilayer import) instead, matching the chicago_loop reference. - Add groupDatasetsForPalette: a pure, first-seen-order fold of same-groupId datasets into group entries (singles pass through), deriving the base title by stripping the "(layer)" suffix. - Render a non-draggable DatasetGroupRow header (OSM PBF badge, title, IMPORTED chip, timestamp, expand caret); expanding lists the layers as ordinary, individually draggable DatasetRows inside an accent-barred container. - Wire the dropdown to render grouped entries; cover grouping + the row with unit and component tests.
Catalog items exposed import time (createdAt) but not when a dataset was installed into a dataflow, so the palette couldn't sort by install time. - Surface the project ref's installedAt onto the catalog item (both folder and legacy-ref paths); default it in base_item; preserve it through catalog dedup merge so the account-level row winning the merge can't drop it. - Add installedAt to the frontend item type; extend the palette grouping helper with representative importedAt/installedAt and sortDatasetPaletteEntries (groups sort as a unit; unknown timestamps last; pure, persisted-metadata only). - Add a palette sort toggle (Import date / Install date) via a new PaletteAccordion trailing slot; sorting works for single datasets and OSM groups alike. - Tests for surfacing, grouping timestamps, and sort ordering.
Dragging the grouped parent now creates one Data Loading node representing the whole OSM PBF import — its loader reads every extracted layer into a single `layers` dict. Dragging an individual layer already creates a node for just that layer (unchanged). The group node references the real per-layer dataset ids (via a new `groupLayers` drag payload field), never the synthetic group id, so the saved `dataflow.datasets` never gains a phantom, unresolvable ref. The group id is kept only as the datasetSource linkage marker. - Add osmGroupLoaderSnippet + createOsmGroupDragPayload; expand a group payload to its member refs in buildDatasetLoaderNodeOptions / applyDatasetToNodeData. - Split the group header: icon = drag handle (full dataset), meta/caret = expand toggle. - Tests for the group payload, node options (real refs, all-layer loader), and the draggable handle.
Interacting with a dataset palette row highlights the canvas nodes linked to it (focusLinkedNodes). This missed grouped OSM imports two ways: the group parent's click only toggled expand (no highlight at all), and the layer/parent linkage check looked only at datasetSource — so a node created by dragging the whole group (datasetSource = synthetic group id, datasetRefs = real layer ids) matched neither a layer row nor the parent. - Add nodeLinkedDatasetIds / isNodeLinkedToAnyDataset: resolve a node's linked dataset ids from datasetSource + datasetRefs + appliedDatasets. Use it in the DatasetRow and node DATASET/OUTPUT chip highlight predicates so any node referencing a dataset highlights consistently. - Give the group parent its own highlight: clicking the meta area selects every node linked to the group id, any member layer, or a member's producer. - Restructure the group header UI: icon = drag handle, meta = highlight (like a single row), a dedicated caret button = expand/collapse; scope the tooltip to the header. Preserves the collapsible grouped behavior. - Tests for the linkage predicate and the group highlight/caret split.
Focusing a node with a palette panel open (e.g. clicking a dataset/OSM-group row to reveal its canvas node) framed the node off the right edge instead of fitting it on screen. fitViewWithMenuOffset computed the zoom against the full pane width and then only shifted the result right by half the occluded strip — so content sized for the whole pane overflowed the visible area to the right of the wide open dock. Compute the fit against the visible width (pane minus the dock-occluded strip) and shift it right by the full occluded width, so the node is both correctly sized and centered in the area right of the palette. This makes reveal/focus frame single datasets, OSM group parents, and individual layers consistently.
Introduce a single source of truth for the high stacking band in curioTokens.css (node/dataset drawers, modal, dialog, and a toast tier above the whole band). A guard test locks in that toasts sit strictly above every overlay so a future edit can't re-bury action feedback.
Portal the toast container to <body> so it escapes any ancestor stacking context, raise it to the top of the layering scale (--curio-z-toast, was a bare 10000 below the drawer's 10045), and expose an aria-live region so feedback is announced while an aria-modal drawer/dialog is focus-trapped. Errors escalate to role=alert. Adds portal, z-index, a11y, and auto-dismiss regression tests.
Point the dataset/node catalog drawers, ModalShell overlays, and the install-permissions dialog at the --curio-z-* tokens instead of bare literals. Effective stacking values are unchanged; this only coordinates the band through one source so the ordering stays consistent.
Computed node outputs are now account-level Data Catalog assets. Generating an output saves it to the user's account store; it is no longer auto-installed into the open project's palette and is never auto-published to the global catalog. Attaching to a project, publishing to the hub, and permanent deletion are all explicit user actions. Each computed dataset keeps lineage linking it to its producing workflow, its source node, and the upstream inputs used to produce it, so the connection survives independently of any project reference. Identity & lineage - Computed ids are dataflow-namespaced (computed.<dataflowId>.<nodeId>) so the same node id reused in two dataflows yields two distinct account-level datasets instead of colliding on one store folder. Helpers computed_dataset_id / node_segment_from_computed_id / dataflow_segment_from_computed_id tolerate both the namespaced and the legacy (computed.<nodeId>) forms; producer resolution gates on the dataflow segment so a reused node id is never cross-attributed. - The dataset manifest gains producer_node_id / _node_type / _dataflow_id / _dataflow_name / upstream_inputs, written by the computed installers and surfaced on catalog items. upstream_inputs is derived on the backend from the saved spec (edges + node provenance); only the producer label/type is threaded from the client on OutputRef. - An idempotent migration renames legacy computed store dirs to the namespaced form, rewriting the manifest and any matching project refs; unattributable dirs are left untouched. It runs once per process from the account listing. Persistence & listing - Saving/executing writes only the account store (no dataflow.datasets ref). The manifest-durability gate resolves the deterministic account-store dir directly, so an output stays durable and reloadable with no project ref. - The account-level user store now lists computed datasets (was imports-only); the dataflow-scoped catalog also surfaces this dataflow's own account-store outputs, so a generated dataset appears in the open project's drawer as available-not-installed immediately after execution. - The Data Catalog drawer no longer drops not-installed computed rows: only genuinely ephemeral live outputs (no store folder) are hidden; persisted account assets are shown. - The obsolete _preserve_persisted_computed_refs was removed (it re-added explicitly-uninstalled refs once the account dir was retained). Explicit lifecycle - Uninstall from a project removes only the ref and keeps the account asset, so it can be reinstalled later. - New delete_dataset (DELETE /api/datasets/<id>) permanently removes an account-level dataset, cascading an unpublish and ref removal across every referencing dataflow before deleting the store folder. Exposed in the drawer as a distinct Delete action on computed cards. Display - The dataflow-namespaced store id (whose dataflow segment is an opaque project UUID) never surfaces in the UI: titles/subtitles and the publish-title fallback show the node-scoped folder (computed.<nodeId>@n), preserving the readable name behavior. A resolved node label is still shown as the title. Tests updated across the dataset/project suites to the new model, with new coverage for namespacing, lineage, migration, account-level listing, uninstall-keeps-asset, delete cascade, drawer visibility, and title display.
| """ | ||
| from __future__ import annotations | ||
|
|
||
| import json |
| """ | ||
| from __future__ import annotations | ||
|
|
||
| import json |
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import os |
|
|
||
| import json | ||
| import os | ||
| from pathlib import Path |
Comment on lines
+29
to
+35
| from utk_curio.backend.app.datasets.install.installer import ( | ||
| computed_dataset_id, | ||
| dataflow_segment_from_computed_id, | ||
| install_computed_file_for_node, | ||
| node_segment_from_computed_id, | ||
| sanitize_node_id_segment, | ||
| ) |
Comment on lines
+37
to
+40
| from utk_curio.backend.tests.test_datasets.computed_test_helpers import ( | ||
| auth_headers, | ||
| create_project, | ||
| ) |
| from pathlib import Path | ||
| from unittest.mock import MagicMock | ||
|
|
||
| from utk_curio.backend.app.datasets.install.installer import computed_dataset_id, sanitize_node_id_segment |
| import os | ||
| from pathlib import Path | ||
|
|
||
| from utk_curio.backend.app.datasets.install.installer import computed_dataset_id, sanitize_node_id_segment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Datacatalog
Adds a Dataset Catalog: a decoupled dataset layer separate from node packs so users can browse hub data, inspect metadata and previews, install datasets into a dataflow, and use them from the canvas.
Backend
DatasetCatalogServicewith routes under/api/datasets/*and/api/dataflows/<id>/datasets/*: catalog listing (search/facets/sort), detail, preview, import, publish/unpublish, install, and uninstall.datasets/(manifest +data/layout, mirroring node packs).dataflow.datasets).liveOutputsfrom the frontend).Frontend
/data-hub) with filters, sorting, and dataset cards.datasetApplication/datasetLoaderSnippets.DatasetCatalogDrawerProviderand React Query hooks for catalog state.Data & infra
Tests
tests/test_datasets/: catalog routes (hub list, install, preview) and computed-dataset indexing/install/path tests.Follow-up commits on this branch fix install/publish/unpublish, dataset drop on the canvas, Data Hub styling, drawer pin, and CI fixtures.
Issue resolved by this PR (if any)
Type of change (Check all that apply)
Parts of Curio impacted by this PR:
Testing
./scripts/test.shand it passed all tests.pytest utk_curio/backend/tests/test_datasets/— 15 passedSuggested manual checks
.curio/users/.../datasets/.Screenshots
Screen.Recording.2026-06-22.at.5.26.04.PM.mov
Drawer
Full Browse
Checklist (Check all that apply)
tests/folder into Curio, ran all the nodes one by one, and checked that they run without errors and give the expected results