Skip to content

STRY0020932: metadatatransformation: Migrate nf-validation to nf-schema #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emarinier wants to merge 16 commits into
base: dev
Choose a base branch
from
Open

STRY0020932: metadatatransformation: Migrate nf-validation to nf-schema #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3e41e53
First pass of converting to nf-schema.
emarinier Jul 3, 2026
aa23e64
JSON input.
emarinier Jul 6, 2026
e8e02f4
Converting over age tests.
emarinier Jul 6, 2026
0f16ddf
age_pnc tests to json
emarinier Jul 6, 2026
4b8373e
Updating more tests to be JSON format.
emarinier Jul 6, 2026
fca9dec
Updating remaining tests.
emarinier Jul 6, 2026
25de106
Removing support for CSV input.
emarinier Jul 6, 2026
ae1fa30
Removing CSV version of the example sample sheet.
emarinier Jul 6, 2026
9988d20
Fixing extension
emarinier Jul 6, 2026
4fa6b37
Formatting JSON files with prettier.
emarinier Jul 6, 2026
3e46df3
Removing nf-validation parameters, etc.
emarinier Jul 6, 2026
c161e33
Removing check max function.
emarinier Jul 7, 2026
94e04a6
Removing old timestamp definition.
emarinier Jul 8, 2026
f132082
CSV to JSON in tests.
emarinier Jul 8, 2026
f58a0a1
Updating docs CSV->JSON.
emarinier Jul 8, 2026
bf4607c
prettier
emarinier Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,24 @@ This pipeline transforms metadata from IRIDA Next.

# Input

The input to the pipeline is a sample sheet (passed as `--input samplesheet.csv`) that looks like:
The input to the pipeline is a sample sheet (passed as `--input samplesheet.json`) that looks like:

| sample | sample_name | metadata_1 | metadata_2 | metadata_3 | metadata_4 | metadata_5 | metadata_6 | metadata_7 | metadata_8 | metadata_9 | metadata_10 | metadata_11 | metadata_12 | metadata_13 | metadata_14 | metadata_15 | metadata_16 |
| ------- | ----------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Sample1 | SampleA | meta_1 | meta_2 | meta_3 | meta_4 | meta_5 | meta_6 | meta_7 | meta_8 | meta_9 | meta_10 | meta_11 | meta_12 | meta_13 | meta_14 | meta_15 | meta_16 |
```
[
{
"sample": "sample1",
"sample_name": "ABC",
"metadata_1": "1.1",
"metadata_2": "1.2",
"metadata_3": "1.3",
"metadata_4": "1.4",
"metadata_5": "1.5",
"metadata_6": "1.6",
"metadata_7": "1.7",
"metadata_8": "1.8"
}
]
```

The amount and meaning of the metadata columns may be different for each metadata transformation.

Expand Down Expand Up @@ -68,7 +81,7 @@ The following parameters can be used to rename CSV-generated output columns and
The metadata headers are automatically converted to lowercase. For example, the following code:

```
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/age/success_failure_mix.csv --outdir results --transformation age --metadata_1_header "date_of_birth" --metadata_2_header "collection_date" --age_header "age_at_collection"
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/age/success_failure_mix.json --outdir results --transformation age --metadata_1_header "date_of_birth" --metadata_2_header "collection_date" --age_header "age_at_collection"
```

would generate the following `results.csv` file:
Expand Down Expand Up @@ -100,7 +113,7 @@ The age metadata column in the output can be renamed as follows, but this is not
The following code:

```
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/age/basic.csv --outdir results --transformation age --age_header calc_host_age --metadata_1_header host_date_of_birth_dob --metadata_2_header calc_earliest_date --metadata_3_header host_age --metadata_4_header host_age_unit
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/age/basic.json --outdir results --transformation age --age_header calc_host_age --metadata_1_header host_date_of_birth_dob --metadata_2_header calc_earliest_date --metadata_3_header host_age --metadata_4_header host_age_unit
```

would generate the following `results.csv` file:
Expand Down Expand Up @@ -184,7 +197,7 @@ In order to ensure these columns are recognized, the metadata header parameters
For example, the following code:

```bash
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/categorize/basic.csv --outdir results --transformation categorize --metadata_1_header host_scientific_name --metadata_2_header host_common_name --metadata_3_header food_product --metadata_4_header environmental_site --metadata_5_header environmental_material
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/categorize/basic.json --outdir results --transformation categorize --metadata_1_header host_scientific_name --metadata_2_header host_common_name --metadata_3_header food_product --metadata_4_header environmental_site --metadata_5_header environmental_material
```

would generate the following `results.csv` file:
Expand Down Expand Up @@ -231,7 +244,7 @@ For example, if the 2nd metadata column of the sample sheet corresponds to the i
The following code:

```
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/pnc/basic.csv --outdir results --transformation pnc -c pnc.config
nextflow run phac-nml/metadatatransformation -profile singularity --input tests/data/samplesheets/pnc/basic.json --outdir results --transformation pnc -c pnc.config
```

would generate the following `results.csv` file:
Expand Down Expand Up @@ -294,10 +307,10 @@ Other parameters (defaults from nf-core) are defined in [nextflow_schema.json](n
To run the pipeline, please do:

```bash
nextflow run phac-nml/metadatatransformation -profile singularity -r main -latest --input assets/samplesheet.csv --outdir results --transformation lock
nextflow run phac-nml/metadatatransformation -profile singularity -r main -latest --input assets/samplesheet.json --outdir results --transformation lock
```

Where the `samplesheet.csv` is structured as specified in the [Input](#input) section.
Where the `samplesheet.json` is structured as specified in the [Input](#input) section.

For more information see [usage doc](docs/usage.md)

Expand Down
4 changes: 0 additions & 4 deletions assets/samplesheet.csv
View file
Open in desktop

This file was deleted.

38 changes: 38 additions & 0 deletions assets/samplesheet.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
[
{
"sample": "sample1",
"sample_name": "ABC",
"metadata_1": "1.1",
"metadata_2": "1.2",
"metadata_3": "1.3",
"metadata_4": "1.4",
"metadata_5": "1.5",
"metadata_6": "1.6",
"metadata_7": "1.7",
"metadata_8": "1.8"
},
{
"sample": "sample2",
"sample_name": "DEF",
"metadata_1": "2.1",
"metadata_2": "2.2",
"metadata_3": "2.3",
"metadata_4": "2.4",
"metadata_5": "2.5",
"metadata_6": "2.6",
"metadata_7": "2.7",
"metadata_8": "2.8"
},
{
"sample": "sample3",
"sample_name": "GHI",
"metadata_1": "3.1",
"metadata_2": "3.2",
"metadata_3": "3.3",
"metadata_4": "3.4",
"metadata_5": "3.5",
"metadata_6": "3.6",
"metadata_7": "3.7",
"metadata_8": "3.8"
}
]
70 changes: 35 additions & 35 deletions assets/schema_input.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"$schema": "https://json-schema.org/draft-07/schema",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://raw.githubusercontent.com/phac-nml/metadatatransformation/main/assets/schema_input.json",
"title": "phac-nml/metadatatransformation pipeline - params.input schema",
"description": "Schema for the file provided with params.input",
Expand All @@ -11,7 +11,6 @@
"type": "string",
"pattern": "^\\S+$",
"meta": ["irida_id"],
"unique": true,
"errorMessage": "Sample name must be provided and cannot contain spaces."
},
"sample_name": {
Expand All @@ -20,118 +19,119 @@
"errorMessage": "Sample name is optional, if provided will replace sample for filenames and outputs"
},
"metadata_1": {
"type": "string",
"type": ["string"],
"meta": ["metadata_1"],
"errorMessage": "Metadata associated with the sample (metadata_1).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"

@emarinier emarinier Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the JSON sample sheet contained entries like "", then this failed, because the + says there needs to be one or more things, but when loaded from the JSON, there can be zero.

},
"metadata_2": {
"type": "string",
"type": ["string"],
"meta": ["metadata_2"],
"errorMessage": "Metadata associated with the sample (metadata_2).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_3": {
"type": "string",
"type": ["string"],
"meta": ["metadata_3"],
"errorMessage": "Metadata associated with the sample (metadata_3).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_4": {
"type": "string",
"type": ["string"],
"meta": ["metadata_4"],
"errorMessage": "Metadata associated with the sample (metadata_4).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_5": {
"type": "string",
"type": ["string"],
"meta": ["metadata_5"],
"errorMessage": "Metadata associated with the sample (metadata_5).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_6": {
"type": "string",
"type": ["string"],
"meta": ["metadata_6"],
"errorMessage": "Metadata associated with the sample (metadata_6).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_7": {
"type": "string",
"type": ["string"],
"meta": ["metadata_7"],
"errorMessage": "Metadata associated with the sample (metadata_7).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_8": {
"type": "string",
"type": ["string"],
"meta": ["metadata_8"],
"errorMessage": "Metadata associated with the sample (metadata_8).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_9": {
"type": "string",
"type": ["string"],
"meta": ["metadata_9"],
"errorMessage": "Metadata associated with the sample (metadata_9).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_10": {
"type": "string",
"type": ["string"],
"meta": ["metadata_10"],
"errorMessage": "Metadata associated with the sample (metadata_10).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_11": {
"type": "string",
"type": ["string"],
"meta": ["metadata_11"],
"errorMessage": "Metadata associated with the sample (metadata_11).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_12": {
"type": "string",
"type": ["string"],
"meta": ["metadata_12"],
"errorMessage": "Metadata associated with the sample (metadata_12).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_13": {
"type": "string",
"type": ["string"],
"meta": ["metadata_13"],
"errorMessage": "Metadata associated with the sample (metadata_13).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_14": {
"type": "string",
"type": ["string"],
"meta": ["metadata_14"],
"errorMessage": "Metadata associated with the sample (metadata_14).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_15": {
"type": "string",
"type": ["string"],
"meta": ["metadata_15"],
"errorMessage": "Metadata associated with the sample (metadata_15).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
},
"metadata_16": {
"type": "string",
"type": ["string"],
"meta": ["metadata_16"],
"errorMessage": "Metadata associated with the sample (metadata_16).",
"default": "",
"pattern": "^[^\\n\\t\"]+$"
"pattern": "^[^\\n\\t\"]*$"
}
},
"required": ["sample"]
}
},
"uniqueEntries": ["sample"]
}
34 changes: 17 additions & 17 deletions conf/base.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

process {

cpus = { check_max( 1 * task.attempt, 'cpus' ) }
memory = { check_max( 6.GB * task.attempt, 'memory' ) }
time = { check_max( 4.h * task.attempt, 'time' ) }
cpus = { 1 * task.attempt }
memory = { 6.GB * task.attempt }
time = { 4.h * task.attempt }

errorStrategy = { task.exitStatus in ((130..145) + 104) ? 'retry' : 'finish' }
maxRetries = 1
Expand All @@ -24,30 +24,30 @@ process {
// If possible, it would be nice to keep the same label naming convention when
// adding in your local modules too.
withLabel:process_single {
cpus = { check_max( 1 , 'cpus' ) }
memory = { check_max( 6.GB * task.attempt, 'memory' ) }
time = { check_max( 4.h * task.attempt, 'time' ) }
cpus = { 1 }
memory = { 6.GB * task.attempt }
time = { 4.h * task.attempt }
}
withLabel:process_low {
cpus = { check_max( 2 * task.attempt, 'cpus' ) }
memory = { check_max( 12.GB * task.attempt, 'memory' ) }
time = { check_max( 4.h * task.attempt, 'time' ) }
cpus = { 2 * task.attempt }
memory = { 12.GB * task.attempt }
time = { 4.h * task.attempt }
}
withLabel:process_medium {
cpus = { check_max( 6 * task.attempt, 'cpus' ) }
memory = { check_max( 36.GB * task.attempt, 'memory' ) }
time = { check_max( 8.h * task.attempt, 'time' ) }
cpus = { 6 * task.attempt }
memory = { 36.GB * task.attempt }
time = { 8.h * task.attempt }
}
withLabel:process_high {
cpus = { check_max( 12 * task.attempt, 'cpus' ) }
memory = { check_max( 72.GB * task.attempt, 'memory' ) }
time = { check_max( 16.h * task.attempt, 'time' ) }
cpus = { 12 * task.attempt }
memory = { 72.GB * task.attempt }
time = { 16.h * task.attempt }
}
withLabel:process_long {
time = { check_max( 20.h * task.attempt, 'time' ) }
time = { 20.h * task.attempt }
}
withLabel:process_high_memory {
memory = { check_max( 200.GB * task.attempt, 'memory' ) }
memory = { 200.GB * task.attempt }
}
withLabel:error_ignore {
errorStrategy = 'ignore'
Expand Down
2 changes: 1 addition & 1 deletion conf/test.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,5 @@ params {
max_time = '1.h'

// Input data
input = 'https://raw.githubusercontent.com/phac-nml/metadatatransformation/main/assets/samplesheet.csv'
input = 'https://raw.githubusercontent.com/phac-nml/metadatatransformation/nf-schema/assets/samplesheet.json'

@emarinier emarinier Jul 8, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will need to change this to main later.

}
2 changes: 1 addition & 1 deletion conf/test_full.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,5 @@ params {
config_profile_description = 'Full test dataset to check pipeline function'

// Input data for full size test
input = 'https://raw.githubusercontent.com/phac-nml/metadatatransformation/main/assets/samplesheet.csv'
input = 'https://raw.githubusercontent.com/phac-nml/metadatatransformation/main/assets/samplesheet.json'
}
Loading
Loading