[SOP Request] Dataset Withdrawal from Node Beacon

[M-casado]

SOP topics

Technical infrastructure & software development

SOP type

Node-specific

SOP Title

Dataset Withdrawal from Node Beacon

Detailed Description

Derived from #55.
This SOP would be triggered at Step 7 (see PR #62).

The SOP will define how a node's Beacon service (likely Beacon-v2) checks for the presence of a dataset requested for withdrawal and removes it if found.

Foreseeable steps (not limited to!) based on my limited knowledge about the Beacon:

1. Locate dataset in Beacon registry

   • Query the Beacon dataset list for the information available from the withdrawal request (e.g., PID, accession, or internal dataset UUID).
   • Identify all variant collections or allele count tables linked to that dataset.

2. Assess withdrawal scope

   • Soft or hard deletion:
     • If soft-deletion, take into account the accessioning of new dataset IDs (versioned).
   • Full or partial withdrawal:
     • Full withdrawal: remove every variant entry associated with the dataset.
     • Partial withdrawal: remove only the variant lines contributed by specified sample IDs, then re-calculate allele counts.

3. Execute removal

   • Trigger Beacon back-end process (e.g., SQL delete, Elasticsearch re-index, or FASTA/VCF re-import) to purge variants.
   • Update dataset status in the Beacon registry (e.g., WITHDRAWN or DEPRECATED).
   • If soft deletion is requested, flag the dataset as withdrawn but retain data files internally for the audit window.

4. Synchronise allele counts

   • Re-compute population allele frequencies if partial removal changed counts.
   • Verify that queries for withdrawn variants now return zero hits.

5. Audit and logging

   • Record user, timestamp, dataset ID, variant count removed, and reason in Beacon logs/documentation (if applicable).
   • Store log excerpt or job report to send it to the GDI Virtual Helpdesk (VHD).

6. Report completion

   • Post confirmation in the VHD ticket with evidence of successful re-index and the new dataset status.

Motivation

Beacon services expose variant-level summaries that remain searchable long after raw files are removed. Without a standard withdrawal procedure, sensitive allele data could persist in public query results, violating GDPR "right to erasure" (Art. 17) and undermining data-controller trust. This SOP ensures all node Beacons respond uniformly and audibly to dataset-withdrawal triggers.

Existing Procedures or References

• GDI SOP0009 “Withdraw a dataset” (European-level). It has the overarching workflow. See issue [SOP Request] Withdraw a dataset #55 and PR SOP0009: Withdraw a dataset #62.
• ELIXIR Beacon v2 API specification (GA4GH-Beacon-v2).
• Node-specific SQL scripts for allele-count recalculation (where available).

Impact

This SOP will guarantee that variant data belonging to withdrawn datasets is no longer discoverable via any node Beacon, maintaining compliance and preventing misleading search results. Beacon administrators gain a tested checklist; data controllers gain assurance; users receive an accurate catalogue of active datasets.

Stakeholders

• Node Beacon administrators / dev-ops teams
• Node data-management teams
• WP4 infrastructure leads
• GDI Virtual Helpdesk (VHD)
• OC and SDPC
• Dataset controllers and submitters

Additional Information

We should make sure that Beacon v2 is the only version in use by the whole GDI. Otherwise, we would need to include different examples for both Beacon v1 and Beacon v2 dataset removals.

Requester GDI role

Yes

Requester GDI Node

EMBL-EBI

Confirmation

• I have searched the existing SOPs and this request does not duplicate an existing SOP.
• I understand that submitting this request does not guarantee the creation of the SOP.

[M-casado]

Additionally, it is imperative to include a retained metadata stub for findability (where appropriate), and require documenting relations between old and new dataset IDs. Essentially letting data consumers know that there was something (i.e., the record) there, but it's been withdrawn, and if there's a new version of the dataset, a link to it.

This helps to trace different versions of the datasets after withdrawal and maintain PIDs.