Skip to content

Add source mapping configuration to sources page - #454

Draft
jal347 wants to merge 4 commits into
1.1.xfrom
source_metadata_feature
Draft

Add source mapping configuration to sources page#454
jal347 wants to merge 4 commits into
1.1.xfrom
source_metadata_feature

Conversation

@jal347

@jal347 jal347 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

This pull request enhances how source field mappings are handled and exposed in metadata, especially for multi-uploader sources. It ensures that per-source field mappings are merged, properly formatted, and only exposed in development mode, while also adding field-to-source annotations in the /metadata/fields endpoint.

Source mapping handling and exposure:

  • Merges all sub-source field mappings into a single mapping at the main source level (_meta.src.<source>.mapping) during metadata collection, ensuring multi-uploader sources have a unified mapping. [1] [2]
  • Exposes per-source field mappings only in development mode (/metadata?dev), flattening them to match the /metadata/fields representation, and strips them from public responses.
  • Adds a utility method _transform_source_mappings to safely apply transformations (e.g., flattening or removing mappings) to each source's mapping without mutating cached metadata.

Field annotation improvements:

  • Annotates each field in the /metadata/fields response with the main source(s) that define it, based on the merged and flattened per-source mappings, making data provenance clearer for consumers.

Imports and dependencies:

  • Adds necessary imports for merge_struct and copy to support the new merging and deep-copy logic. [1] [2]

@jal347 jal347 linked an issue Jul 21, 2026 that may be closed by this pull request
@jal347
jal347 marked this pull request as draft July 21, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Assign metadata fields to each data source

1 participant