Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8f98d13
add all dataclasses and tests
linglp Jul 14, 2026
78ec656
fix docstring
linglp Jul 15, 2026
879a2c8
added create replica endpoint, dataclasses, updated documentation
linglp Jul 15, 2026
b7d80dd
add test for create replica data classes and add add a unit test of c…
linglp Jul 15, 2026
df9d8bd
if nothing returned, then it should raise error
linglp Jul 15, 2026
bdf168c
raise an error if query_request is none when querying the grid
linglp Jul 15, 2026
9ff1bad
update documentation; export filter and column selection data classes…
linglp Jul 16, 2026
3f6f584
add the connect method
linglp Jul 17, 2026
146ac3c
add connect to documentation
linglp Jul 17, 2026
050a448
restore thing
linglp Jul 17, 2026
16ac436
when no validation result returned, it should not raise an error
linglp Jul 17, 2026
224ddd4
added otel trace method
linglp Jul 17, 2026
a5ea43d
update docstring
linglp Jul 17, 2026
8fb46ce
add test
linglp Jul 17, 2026
6449b72
make sure that GridQuery does not allow empty column selection
linglp Jul 17, 2026
d07551b
make create replica a private method
linglp Jul 20, 2026
028b57a
update docstring
linglp Jul 20, 2026
cf7b2d4
remove fill_from_dict in selectItem classes and filter classes; edit …
linglp Jul 20, 2026
67a045f
update import; docstring example; and documentation
linglp Jul 20, 2026
efacbfe
Merge branch 'develop' into SYNPY-1880-clean
linglp Jul 20, 2026
17cb35e
update to reference the grid previously created
linglp Jul 20, 2026
3b81365
query result should be returned instead of none when there is no row
linglp Jul 20, 2026
b853a52
add to grid class docstring
linglp Jul 20, 2026
2c81990
update docstring example to avoid recreating another grid instance
linglp Jul 21, 2026
990fac7
added integration tests
linglp Jul 22, 2026
740c76d
update docstring
linglp Jul 22, 2026
056e25c
add import back
linglp Jul 22, 2026
d69cfd4
Add documentation for using in-session validation
thomasyu888 Jul 22, 2026
10d2312
Remove redundant imports
thomasyu888 Jul 22, 2026
e5ffab1
Add clarifying point
thomasyu888 Jul 22, 2026
22f6f4a
Potential fix for pull request finding
thomasyu888 Jul 22, 2026
533ad4a
Potential fix for pull request finding
thomasyu888 Jul 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 92 additions & 10 deletions docs/guides/extensions/curator/metadata_contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ print(df)

# Example only: fills 4 rows with random integers regardless of column type.
# Replace this with real edits that match your task's schema before importing —
# schema validation runs in Step 6 and will reject values that don't fit.
# schema validation runs in Step 7 and will reject values that don't fit.
df = pd.DataFrame(
np.random.randint(0, 100, size=(4, len(df.columns))),
columns=df.columns,
Expand All @@ -147,7 +147,74 @@ latest_grid = latest_grid.import_csv(path=edited_path)
print(f"Upserted edits into grid session: https://www.synapse.org/Grid:default?sessionId={latest_grid.session_id}")
```

### Step 6: Export the grid back to the RecordSet
### Step 6: Validate your edits in-session (before exporting)

Before you export, you can check your edits against the JSON schema bound to the RecordSet directly on the Grid session — without creating a new RecordSet version. This lets you catch and fix problems while iterating, then export once (Step 7) with clean data.

Open the session with `connect()` (or `connect_async()`), which binds a replica for the duration of the `with` block, then call `validate_rows()` with a `QueryRequest`. Each returned row carries its own `validation_results`. `SelectAll()` returns every column, so each row's full data comes back alongside its validation results.

Reuse the `latest_grid` session you've been editing:

```python
from synapseclient.models.curation import GridQuery, QueryRequest, SelectAll

with latest_grid.connect() as grid:
query_request = QueryRequest(query=GridQuery(column_selection=[SelectAll()]))
query_result = grid.validate_rows(query_request=query_request)

if query_result.rows is None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if query_result.rows is None:
if not query_result.rows:

This will also check for a 0-length number of rows.

print("No rows matched the query.")
else:
for row in query_result.rows:
print(f"Row ID: {row.row_id}, Validation Result: {row.validation_results}")
Comment on lines +163 to +169
```

If you're landing here with only a `record_set_id` (no session yet), `connect()` will create — or, with `.connect(attach_to_previous_session=True)`, reattach to — a session for you:

```python
from synapseclient.models import Grid
from synapseclient.models.curation import GridQuery, QueryRequest, SelectAll

with Grid(record_set_id="syn123456789").connect() as grid:
query_request = QueryRequest(query=GridQuery(column_selection=[SelectAll()]))
query_result = grid.validate_rows(query_request=query_request)

for row in query_result.rows:
print(f"Row ID: {row.row_id}, Validation Result: {row.validation_results}")
Comment thread
thomasyu888 marked this conversation as resolved.
```

To focus on just the rows that currently fail, add a `RowIsValidFilter` and set `include_validation_messages=True` so each row’s `validation_results` includes the full `all_validation_messages` list:

```python
from synapseclient.models.curation import (
GridQuery,
QueryRequest,
RowIsValidFilter,
SelectAll,
)

with latest_grid.connect() as grid:
query_request = QueryRequest(
query=GridQuery(
column_selection=[SelectAll()],
filters=[RowIsValidFilter(value=False)],
include_validation_messages=True,
)
)
query_result = grid.validate_rows(query_request=query_request)

for row in query_result.rows:
print(f"Invalid row {row.row_id}: {row.validation_results}")
```
Comment thread
thomasyu888 marked this conversation as resolved.

!!! note "Requires a bound schema"
In-session validation only produces results when the administrator has bound a
JSON schema to the RecordSet. Without one, rows still return but their
`validation_results` is `None` for each row.

In-session `validate_rows()` checks the current Grid rows without creating a new RecordSet version — use it to iterate quickly. The export report reviewed in Step 8 (`get_detailed_validation_results()`) reflects the last export from Step 7. Each method has a matching `_async` counterpart (`connect_async`, `validate_rows_async`) for async contexts.

### Step 7: Export the grid back to the RecordSet

> **Important:** Until you call `export_to_record_set()`, your edits live only inside the Grid session — they aren't visible on the RecordSet and won't be validated. Apply changes whenever you reach a logical checkpoint.

Expand All @@ -158,9 +225,9 @@ latest_grid.export_to_record_set()
print(f"Exported to RecordSet version: {latest_grid.record_set_version_number}")
```

### Step 7: Review your validation results
### Step 8: Review your validation results

When you exported the grid in Step 6, Synapse validated each row against the JSON schema bound to the RecordSet and generated a row-level report. Reviewing this report before handing the task back to the administrator lets you catch and fix problems in your own data first — saving a round trip.
When you exported the grid in Step 7, Synapse validated each row against the JSON schema bound to the RecordSet and generated a row-level report. Reviewing this report before handing the task back to the administrator lets you catch and fix problems in your own data first — saving a round trip. (For a quick check before you commit an export, use the in-session validation in Step 6 instead.)

#### Prerequisites for validation results

Expand All @@ -170,7 +237,7 @@ A validation report is only generated when **all** of the following are true:
2. You have entered data through a Grid session
3. The Grid session has been exported back to the RecordSet — this is the step that triggers validation and populates the RecordSet's validation_file_handle_id

If the Grid was never exported (Step 6), there is nothing to review yet.
If the Grid was never exported (Step 7), there is nothing to review yet.

#### Retrieve and inspect the results

Expand All @@ -185,7 +252,7 @@ if isinstance(curation_task.task_properties, RecordBasedMetadataTaskProperties):
validation_df = record_set.get_detailed_validation_results()

if validation_df is None:
print("No validation results yet — make sure the Grid was exported in Step 6.")
print("No validation results yet — make sure the Grid was exported in Step 7.")
else:
total = len(validation_df)
valid = validation_df["is_valid"].sum()
Expand Down Expand Up @@ -230,11 +297,11 @@ Row 2:

#### Fix and re-export

If any rows are invalid, recreate a Grid session against the RecordSet (see Step 3), correct the offending rows, and re-run Steps 4–6 to re-export. The validation report is regenerated on each export, so iterate until the report is clean before letting the administrator know your task is ready.
If any rows are invalid, recreate a Grid session against the RecordSet (see Step 3), correct the offending rows, and re-run Steps 4–7 to re-export. The validation report is regenerated on each export, so iterate until the report is clean before letting the administrator know your task is ready.

> **If get_detailed_validation_results returns None after exporting:** check that record_set.validation_file_handle_id is set after the re-fetch. If it isn't, the export did not complete — re-run export_to_record_set() on an active Grid session against the same RecordSet.

### Step 8: Mark the curation task as COMPLETED
### Step 9: Mark the curation task as COMPLETED

Once your validation report is clean and you've cleaned up the Grid session, transition the curation task to COMPLETED. This signals the administrator that the task is ready for their review — they can list tasks in the project and pick up the ones whose status is COMPLETED.

Expand All @@ -244,7 +311,7 @@ curation_task.set_task_state(state="COMPLETED")

## File-Based Curation Tasks

File-based tasks follow the same overall flow as record-based tasks (Steps 1–8 above), with three key differences:
File-based tasks follow the same overall flow as record-based tasks (Steps 1–9 above), with three key differences:

**No CSV import.** `import_csv` is not currently supported for file-based grids. Instead, you can either:

Expand All @@ -259,7 +326,20 @@ latest_grid.synchronize()

This writes the Grid annotation values back to each file as Synapse annotations. There is no versioned RecordSet — the files themselves are updated in place.

**No per-row validation report.** Validation is enforced by the JSON schema bound to the folder containing the files, not by a row-level export report. After you call `synchronize()`, the administrator verifies schema compliance on their end — there is nothing to retrieve from the contributor side. If the administrator reports violations, correct the flagged annotations in the Grid UI and re-synchronize.
**No per-row export report — but in-session validation still works.** There is no versioned RecordSet, so the export report reviewed in Step 8 (`export_to_record_set()` → `get_detailed_validation_results()`) does not apply. However, the in-session validation from Step 6 works identically for file-based grids: a file-based session created from an `initial_query` still carries a bound JSON schema (`grid_json_schema_id`), so `connect()` + `validate_rows()` returns the same per-row `validation_results`. This is your primary contributor-side check for file-based tasks — run it before `synchronize()`.

```python
from synapseclient.models.curation import GridQuery, QueryRequest, SelectAll

with latest_grid.connect() as grid:
query_request = QueryRequest(query=GridQuery(column_selection=[SelectAll()]))
query_result = grid.validate_rows(query_request=query_request)

for row in query_result.rows:
print(f"Row ID: {row.row_id}, Validation Result: {row.validation_results}")
Comment thread
thomasyu888 marked this conversation as resolved.
```

After you call `synchronize()`, the administrator also verifies schema compliance on their end. If they report violations, correct the flagged annotations in the Grid UI and re-synchronize.

## Appendix

Expand All @@ -280,6 +360,8 @@ Deleting is permanent — you can no longer re-export from this session. If you
- [CurationTask.get][synapseclient.models.CurationTask.get] - Fetch a CurationTask by id
- [CurationTask.create_grid_session][synapseclient.models.CurationTask.create_grid_session] - Create a Grid session for a CurationTask and link it to the task status
- [CurationTask.set_task_state][synapseclient.models.CurationTask.set_task_state] - Set the state on a CurationTask's status
- [Grid.connect][synapseclient.models.Grid.connect] - Connect to a Grid session and bind a replica for in-session validation
- [Grid.validate_rows][synapseclient.models.Grid.validate_rows] - Validate a Grid session's rows against the bound JSON schema
- [Grid.download_csv][synapseclient.models.Grid.download_csv] - Download Grid contents as a local CSV
- [Grid.import_csv][synapseclient.models.Grid.import_csv] - Upsert CSV edits back into a Grid session (record-based grids only)
- [Grid.export_to_record_set][synapseclient.models.Grid.export_to_record_set] - Export Grid data back to RecordSet and generate validation results
Expand Down
116 changes: 116 additions & 0 deletions docs/reference/experimental/async/curator.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,126 @@
- import_csv_async
- delete_async
- list_async
- connect_async
- validate_rows_async
---
[](){ #query-reference-async }
::: synapseclient.models.Query
options:
inherited_members: true
members:
---
[](){ #GridQuery-reference-async }
::: synapseclient.models.curation.GridQuery
options:
inherited_members: true
members:
---
[](){ #QueryRequest-reference-async }
::: synapseclient.models.curation.QueryRequest
options:
inherited_members: true
members:
---
[](){ #SelectItem-reference-async }
::: synapseclient.models.curation.SelectItem
options:
inherited_members: true
members:
---
[](){ #SelectByName-reference-async }
::: synapseclient.models.curation.SelectByName
options:
inherited_members: true
members:
---
[](){ #SelectAll-reference-async }
::: synapseclient.models.curation.SelectAll
options:
inherited_members: true
members:
---
[](){ #CountStar-reference-async }
::: synapseclient.models.curation.CountStar
options:
inherited_members: true
members:
---
[](){ #SelectSelection-reference-async }
::: synapseclient.models.curation.SelectSelection
options:
inherited_members: true
members:
---
[](){ #Filter-reference-async }
::: synapseclient.models.curation.Filter
options:
inherited_members: true
members:
---
[](){ #RowValidationResultFilter-reference-async }
::: synapseclient.models.curation.RowValidationResultFilter
options:
inherited_members: true
members:
---
[](){ #CellValueFilter-reference-async }
::: synapseclient.models.curation.CellValueFilter
options:
inherited_members: true
members:
---
[](){ #RowSelectionFilter-reference-async }
::: synapseclient.models.curation.RowSelectionFilter
options:
inherited_members: true
members:
---
[](){ #RowIsValidFilter-reference-async }
::: synapseclient.models.curation.RowIsValidFilter
options:
inherited_members: true
members:
---
[](){ #RowIdFilter-reference-async }
::: synapseclient.models.curation.RowIdFilter
options:
inherited_members: true
members:
---
[](){ #ValidationOperator-reference-async }
::: synapseclient.models.curation.ValidationOperator
options:
inherited_members: true
members:
---
[](){ #CellValueOperator-reference-async }
::: synapseclient.models.curation.CellValueOperator
options:
inherited_members: true
members:
---
[](){ #GridQueryResult-reference-async }
::: synapseclient.models.curation.GridQueryResult
options:
inherited_members: true
members:
---
[](){ #GridRow-reference-async }
::: synapseclient.models.curation.GridRow
options:
inherited_members: true
members:
---
[](){ #SelectColumn-reference-async }
::: synapseclient.models.curation.SelectColumn
options:
inherited_members: true
members:
---
[](){ #GridQueryValidationResult-reference-async }
::: synapseclient.models.curation.GridQueryValidationResult
options:
inherited_members: true
members:
---
Loading
Loading