Proposition to extend /search/calls & /search/allelematrix functionalities

[GuilhemSempere]

Add studyDbIds, sampleDbIds, and germplasm filters to POST /search/calls; add studyDbIds to POST /search/allelematrix; formally document AND-intersection semantics

Summary

• Add studyDbIds to both POST /search/allelematrix and POST /search/calls
• Add sampleDbIds, germplasmDbIds, germplasmNames, and germplasmPUIs to POST /search/calls (these already exist on AlleleMatrixSearchRequest)
• Formally document AND-intersection semantics for all high-level filters on both endpoints
• Add dimensionColumnAggregation parameter to override default column aggregation granularity

Background

An allele matrix is a 2D structure where genotype calls are attached to CallSets:

• Row dimension (variants): Study → VariantSet → Variant → Call
• Column dimension (materials): Study → Germplasm → Sample → CallSet → Call

The column dimension can be entered at any level: callers may filter by germplasmDbIds, sampleDbIds, or callSetDbIds directly. Note that Germplasm objects do not carry a studyDbId but can however be filtered by that field.

Currently, /search/calls only supports column selection via callSetDbIds, forcing clients to resolve the full Germplasm → Sample → CallSet chain themselves before querying. /search/allelematrix supports germplasm and sample filters but lacks studyDbIds. Intersection semantics for simultaneous use of multiple filters are undefined on both endpoints.

Proposed Changes

New fields on CallSearchRequest

studyDbIds:
  type: array
  items:
    type: string
  description: >
    Filter results to calls associated with the specified studies.
    The server resolves studyDbIds across both dimensions: to VariantSets on the row
    dimension, and to CallSets on the column dimension.
    Acts as an AND constraint alongside all other filters.

sampleDbIds:
  type: array
  items:
    type: string
  description: >
    Filter results to calls belonging to CallSets derived from the specified samples.
    Acts as an AND constraint alongside all other filters.

germplasmDbIds:
  type: array
  items:
    type: string
  description: >
    Filter results to calls belonging to CallSets derived from samples associated with
    the specified germplasm.
    Acts as an AND constraint alongside all other filters.

germplasmNames:
  type: array
  items:
    type: string
  description: As germplasmDbIds but matched against germplasm names.

germplasmPUIs:
  type: array
  items:
    type: string
  description: As germplasmDbIds but matched against germplasm PUIs.

dimensionColumnAggregation:
  type: string
  enum: [callSet, sample, germplasm]
  description: >
    Override the default column aggregation granularity (see Genotype aggregation level below).
    When provided, the server groups calls at the specified level regardless of which filter
    parameters were used to select the material. For example, filtering by sampleDbIds but
    setting dimensionColumnAggregation to "germplasm" will return one aggregated column per
    Germplasm rather than one per Sample.

New fields on AlleleMatrixSearchRequest

studyDbIds:
  type: array
  items:
    type: string
  description: >
    Filter the matrix to calls associated with the specified studies.
    The server resolves studyDbIds across both dimensions: to VariantSets on the row
    dimension, and to CallSets on the column dimension.
    Acts as an AND constraint alongside all other filters.

dimensionColumnAggregation:
  type: string
  enum: [callSet, sample, germplasm]
  description: >
    Override the default column aggregation granularity (see Genotype aggregation level below).
    When provided, the server groups calls at the specified level regardless of which filter
    parameters were used to select the material. For example, filtering by sampleDbIds but
    setting dimensionColumnAggregation to "germplasm" will return one aggregated column per
    Germplasm rather than one per Sample.

Genotype aggregation level

The biological entity type used to filter the column dimension determines the default granularity at which genotype calls are aggregated and returned:

• callSetDbIds: one column per CallSet — finest granularity, no merging.
• sampleDbIds: calls are grouped per Sample (aggregating all CallSets of that Sample).
• germplasmDbIds / germplasmNames / germplasmPUIs: calls are grouped per Germplasm (aggregating all Samples and their CallSets for that Germplasm).
• studyDbIds (column dimension): calls are grouped per Germplasm within the study.

When multiple filters from different tiers are provided simultaneously, the finest-grained tier governs the default aggregation. This default can be overridden using the dimensionColumnAggregation parameter.

Note: this proposal does not define the merging strategy for conflicting allele calls within an aggregation group (e.g. two CallSets of the same Sample carrying different genotypes). This is left to a follow-up discussion.

Intersection semantics (AND logic)

All filters stack as AND constraints across and within dimensions:

• Providing only studyDbIds is sufficient to retrieve all calls for those studies (both dimensions implicitly resolved).
• Any additional filter further narrows the result within the study scope.
• Filters from different tiers of the same dimension are intersected: e.g. studyDbIds: ["S1"] + sampleDbIds: ["Samp1"] returns only calls from CallSets of Samp1 if and only if Samp1 belongs to S1.
• Non-overlapping filter combinations return HTTP 200 with an empty data array (standard BrAPI empty result).
• All pre-existing fields (variantDbIds, variantSetDbIds, callSetDbIds, etc.) retain their current semantics.

Examples

Minimal study query — all calls for one study, grouped by germplasm (default):

{ "studyDbIds": ["study1"] }

Study query with explicit aggregation override — same query, but return one column per Sample instead of per Germplasm:

{ "studyDbIds": ["study1"], "dimensionColumnAggregation": "sample" }

Study + germplasm — calls restricted to the specified germplasm within the study:

{ "studyDbIds": ["study1"], "germplasmDbIds": ["germ1", "germ2"] }

Sample filter with germplasm-level aggregation — select by sample but aggregate up to germplasm:

{ "sampleDbIds": ["samp1", "samp2"], "dimensionColumnAggregation": "germplasm" }

Study + variantSet — row dimension restricted to variants in both the study and the specified VariantSet:

{ "studyDbIds": ["study1"], "variantSetDbIds": ["vs1"] }

Non-overlapping filters — returns 200 + empty data:

{ "studyDbIds": ["study1"], "sampleDbIds": ["samp_from_study2"] }

Affected endpoints

• POST /search/calls — add studyDbIds, sampleDbIds, germplasmDbIds, germplasmNames, germplasmPUIs, dimensionColumnAggregation
• POST /search/allelematrix — add studyDbIds, dimensionColumnAggregation

Notes

• Server implementations MAY return 202 Accepted with a search result URL for large result sets, consistent with existing BrAPI async search behaviour.

[a-caballero-r]

Hi @GuilhemSempere

Thank you for the elaboration on the filters and groupings that can be done!

A note on this paragraph.

Currently, /search/calls only supports column selection via callSetDbIds, forcing clients to resolve the full Germplasm → Sample → CallSet chain themselves before querying. /search/allelematrix supports germplasm and sample filters but lacks studyDbIds. Intersection semantics for simultaneous use of multiple filters are undefined on both endpoints.

While the filters look good, just to be on the same page, the response should also contain the fields used as filters.

e.g.

When we use /search/calls with filters

{ "sampleDbIds": ["samp1", "samp2"], "dimensionColumnAggregation": "sample" }

The response should contain the sample field

e.g.

{
"callSetDbId": "somecallsetdbid",
"genotypeMetadata": [],
"genotypeValue": "A/T",
"variantDbId": "somevariantdbid",
"variantName": "somevariantname",
"variantSetDbId": "somevariantset",
"sampleDbId": "somesampledbid",
// Additional supported filter fields so response can be easily mapped
"germplasmName":"somegermplasmname",
"germplasmPUI":"somegermplasmpui",
"germplasmDbId":"somegermplasmdbid",
"studydbid":"somestudydbid",
}

This makes the results mapping with the filter used easier, otherwise we would fall to a Germplasm → Sample → CallSet like chain. The response then should include all available filter fields, germplasmName germplasmPUI, germplasmDbId, studyDbId.

This also applies to the /search/allelematrix API.

I await your comments on this

[aliceboizet]

To simplify pagination, I would suggest using a generic pagination dimension called COLUMNS (or another more appropriate name). This dimension would represent callset, sample, or germplasm, depending on the value of dimensionColumnAggregation.
For backward compatibility, we should continue accepting "callset".

This approach would avoid the need to introduce separate pagination parameters such as dimensionCallSetPageSize, dimensionSamplePageSize, and dimensionGermplasmPageSize in the GET /alleleMatrix. Instead, we would add only these two parameters : dimensionColumnPageSize, and dimensionColumnPageSize

Depending on the value of dimensionColumnAggregation, the /search/allelematrix response will contain callsetDbIds, sampleDbIds or germplasmDbIds , and the /search/calls will contain callsetDbId, sampleDbId or germplasmDbId.

@a-caballero-r in your example, if you filter /search/calls with

{ "sampleDbIds": ["samp1", "samp2"], "dimensionColumnAggregation": "sample" }

You would get

{
  "genotypeMetadata": [],
  "genotypeValue": "A/T",
  "variantDbId": "somevariantdbid",
  "variantName": "somevariantname",
  "variantSetDbId": "somevariantset",
  "sampleDbId": "somesampledbid"
}

if you filter on "germplasmDbIds" with :

{ "germplasmDbIds": ["germ1", "germ2"], "dimensionColumnAggregation": "germplasm" }

you would get :

{
  "genotypeMetadata": [],
  "genotypeValue": "A/T",
  "variantDbId": "somevariantdbid",
  "variantName": "somevariantname",
  "variantSetDbId": "somevariantset",
  "germplasmDbId": "somegermplasmdbid"
}

Would that be acceptable to you? If I'm not mistaken, you would already have the mapping between sampleDbId, germplasmDbId, germplasmName or germplasmPUI in EBS database, am I right ? So you won't need to get these from Gigwa ?

[a-caballero-r]

Hi @aliceboizet . It looks good, as long as we receive in the response the field we used as filter (germplasmName, sampleDbId, etc.) we can map each object to an EBS counterpart, one more thing to mention:

The values for the different materials are formed with this character §, it's still confusing how some of them are structured
Example, sampleDbId seems to have a structure like this

AATTOL_rnaseq§BA1-1-r1

In our case, we would only want to filter the sampleCode (EBS) which would be AATTOL_rnaseq

so filter would be {"sampleDbId": "AATTOL_rnaseq"}

Another example for germplasmName

We have Germplasm001 in EBS, so we would want to filter like {"germplasmName":"Germplasm001"}

All of that without concatenating the § character, or bulding the filter itself with other data (germplasmName + study/run,etc.), goal is to only use the raw EBS value for the filter.

[aliceboizet]

Hi @a-caballero-r,

The format of sampleDbId depends on how genotypes are imported. In Gigwa, you can import genotyping data at the "individual" or "sample" level. The individual corresponds to the germplasm. By default, the import is done at individual level.

• If genotypes are provided at the individual/germplasm level (default case), then sampleDbId is built as:
  [database]§[individualName]-[projectId]-[runId]
  where individualName comes from the genotype file.

• If genotypes are provided at the sample level (i.e. the “Genotypes provided for samples, not individuals” option is enabled in the UI, or the providingSamples query parameter is used in the API), then sampleDbId follows this format:
  [database]§[sampleName]
  where sampleName comes directly from the genotype file.
  If you don't provide a mapping file between sampleNames and individualNames, then individualName would be the same as sampleName.

In all cases, the germplasmDbId is defined as: [database]§[individualName]

In your example: AATTOL_rnaseq§BA1-1-r1, AATTOL_rnaseq is the database and the sampleName is BA1-1-r1. The data must have been imported on individual level in the AATTOL_rnaseq database, project_id 1 and run r1and the individual/germplasm name of this sample is BA1.

Currently, for the allelematrix and calls endpoints:

• The database must be specified in the request. (it is inferred from the provided germplasmDbIds / sampleDbIds or variantSetDbIds, and is defined as the substring before the first § in the ID)
• It is not possible to query across multiple databases within a single request.

Because of this constraint, what you are requesting would require a significant change in the API design.

Could you clarify why you need to avoid specifying the database? Could you precise the different queries that you would like to do to the search/calls or search/allelematrix ?

To answer to your exact need, it seems to me that you should use this workflow:

If you want to get the genotypes at sample level, filtering on "sampleNames":

1. POST search/samples with programDbId or studyDbId, and sampleNames as filter parameters to get the mapping between sampleNames and sampleDbIds.
2. POST search/calls or search/alleles with sampleDbIds and "dimensionColumnAggregation": "sample" as filter parameters. You will get "sampleDbIds" in the response that you can easily map with sampleNames according to the search/samples response.

If you want to get the genotypes at germplasm level, filtering on "germplasmNames":

1. POST search/germplasm with programDbId or studyDbId, and germplasmNames as filter parameters to get the mapping between germplasmNames and germplasmDbIds.
2. POST search/calls or search/alleles with germplasmDbIds and "dimensionColumnAggregation": "germplasm" as filter parameters. You will get "germplasmDbIds" in the response that you can easily map with germplasmNames according to the search/germplasm response.

Would that be acceptable to you ?

[a-caballero-r]

Hi @aliceboizet

One of the requirements is to have the filter supported at the genotype endpoints search/calls and search/allelematrix, as specified in the first comment by @GuilhemSempere.

Now that we know that the germplasmDbId is the combination of [databaseName][separator][IndividualName], we can use that as filter in the genotype endpoints without too much of an elaboration. Only concat the database + § + individualName, be it sample or germplasm, am I correct?

To answer to your questions:

1. Could you clarify why you need to avoid specifying the database?
   We were thinking about passing the EBS name as is, mainly for performance, so we don't have to mutate every single filter (sampleName, germplasmName, etc)

2. Could you precise the different queries that you would like to do to the search/calls or search/allelematrix?
   These are the scenarios:
   Retrieve the genotype values (A,T,C,G) for individuals belonging to a particular study
   Retrieve the genotype values (A,T,C,G) for a given set of individuals (independent of their study/variantSet/run, etc.)

We were evaluating the use of the allelematrix API instead of the calls, but since we need to use the raw value of the genotype (A,T,C,G) it's so much easier for us to use the calls API, since that one gives us exactly the value we need.

[GuilhemSempere]

Hi!
I would not recommend computing IDs on your side based on your understanding of how Gigwa handles them internally, because there is no guarantee that things will remain like this in the long term, and your client code would break if you tried to let it talk to another datasource.
In terms of pure performance:

• even for a few thousands of germplasm getting the IDs from the names by specifying a studyDbId will take something like a second
• for large data volumes allelematrix is going to be significantly more performant, because of its compactness; up to you really to decide which endpoint to use
  What really matters is that Gigwa won't be able to efficiently query on names if it doesn't know which DB you are referring to. Knowing that the next Gigwa version will return a studyDbId after each import, it is important that you manage to submit it along with the query that asks for the genotypes.

[a-caballero-r]

Hi @GuilhemSempere

You are correct about the unviability of computing each element in the filter, that's why we ask for the implementation of them with direct sampleName/germplasmName support, so when we have to query for thousands of elements we don't have to build each one.

As for the studies, our use cases also require querying for individuals regardless of their project/study nor variants/variantsets/runs associations, we only want to know the genotype values (A,T,C,G) for individuals (germplasmOrSampleName1, germplasmOrSampleName2...germplasmOrSampleNameN).

[GuilhemSempere]

But I guess you only want to query germplasm from a same species at a time, and since I understood you will be using one database per species, you should be able to simply:

• get all studyDbIds for a given database (that is, program or trial, in BrAPI terms)
• pass that list to /search/calls or /search/allelematrix along with the germplasm or sample names
  This would work

[aliceboizet]

@a-caballero-r in both scenario, do you confirm that you want to get a synthetic genotype for each individual and not for each individual and each run (which corresponds to callset level) ?

[a-caballero-r]

@GuilhemSempere @aliceboizet

We are assigning a database per species, so the suggestion of retrieving all studies, perhaps could be simplified by just passing the database name, along with the individual names.

What we need is something like this:

{
	"germplasmName": "GermplasmName01", // Just the germplasm name as is so we don't have to compute/match with the EBS name by removing §, databaseName, run, etc.
	"sampleName": "SampleName01", // Just the sample name as is so we don't have to compute/match with the EBS name by removing §, databaseName, run, etc.
	"genotypeValue":"A/T",
	"variantName":"Marker01" // Same marker name as EBS, having to map so we don't have to compute/match with the EBS name by removing §, databaseName, run, etc.
	"variantSetDbId":"run01" // Run might be of some relevance, but for now we are only using 1 run per study
}

[GuilhemSempere]

We've got the feeling that this is possibly not going to be accepted as BrAPI compliant because it is quite far from what is currently supported. But we can of course implement something on purpose. In whichever way, we will always need your client to specify which DB needs to be queried in some way. Can you confirm this is not a problem?

[a-caballero-r]

@GuilhemSempere
If the request would look like this:

{
	"database":"databaseName",
	"germplasmNames":[] // List of EBS germplasm names
	// OR
	"sampleNames":[] // List of EBS sample names
}

On both endpoints, then that should be good.