diff --git a/docs/metadata.md b/docs/metadata.md index 26fdd7c..bf0a6c7 100644 --- a/docs/metadata.md +++ b/docs/metadata.md @@ -4,11 +4,7 @@ In order to upload data with metadata to Alexandria, your metadata must conform Users are encouraged to include as much metadata as possible, including metadata attributes which are the same for all samples in the study (ex. sequencing technology) as they will be important to users comparing data between studies. -The [Alexandria Metadata Convention](https://docs.google.com/spreadsheets/d/1r2r4XM7trosTQQFDjz4UcnLLXKZAoC9X80V5yTquFck/edit?usp=sharing) is accessible here as a Google Spreadsheet to those with a Broad Institute email address (@broadinstitute.org). To those without a Broad email account, we will grant access to outsiders eventually but for now you can view a [semi-complete version on our Github](https://github.com/ShalekLab/alexandria/blob/master/Docker/metadata_type_map.tsv). - -The [Alexandria Metadata Convention](https://docs.google.com/spreadsheets/d/1r2r4XM7trosTQQFDjz4UcnLLXKZAoC9X80V5yTquFck/edit?usp=sharing) when first opened appears as such: ![](/imgs/metadata/amc.png) - -Each attribute name in the metadata can be a column name in your metadata file. To enforce some structure to the database, several fields are required and others require specific formatting guidelines. +The [Alexandria Metadata Convention](https://github.com/broadinstitute/scp-ingest-pipeline/blob/master/tests/data/AMC_v1.1.3.tsv) enforces structure upon metadata across studies in Alexandria, several fields are required and others require specific formatting guidelines. ## Metadata categories @@ -18,12 +14,12 @@ The Alexandria Metadata Convention contains 4 categories of structured metadata Examples: disease, species, organ -To allow query at varying levels of specificities, the Alexandria Metadata Convention uses ontology-valued metadata whenever possible. These ontologies are tree-structured heirarchies curated by experts and maintained by EBI. For the purposes of the Alexandria Metadata Convention, please select the most specific value in the heirarchy as possible. For use of this metadata during query of the Alexandria database, a query result will include all child values of the query term. Single Cell Portal treats these values as group, or categorical variables. +To allow query at varying levels of specificities, the Alexandria Metadata Convention uses ontology-valued metadata whenever possible. These ontologies are tree-structured hierarchies curated by experts and easily accessible through a [Lookup Service](https://www.ebi.ac.uk/ols/index) maintained by EBI. For the purposes of the Alexandria Metadata Convention, please select the most specific value in the hierarchy as possible. Single Cell Portal treats these values as group, or categorical variables. -**Navigating ontologies to manually annotate your metadata:** The EBI OLS (ontology lookup service) provides an interface as well as an API to explore these ontologies which may be useful in preparing your metadata. The API is documented [here](https://www.ebi.ac.uk/ols/docs/developer), and the API endpoints for a given ontology are found in the `ontology` column in the AMC spreadsheet. The graphical interface can be accessed through the same URL with `api` substituted with `ols` (ex. `https://www.ebi.ac.uk/ols/ontologies/UBERON`). Because some ontologies include more areas than are covered by Alexandria, we have specified a _root_ for some ontologies, specified in the `ontology_root` column of the AMC spreadsheet. Entries in Alexandria metadata should come from below that root in the ontology tree. To visit the page described by the root use the URL: `https://www.ebi.ac.uk/ols/ontologies//terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2F`. For example, for the species attribute, from the `NCBITaxon` ontology with root `NCBITaxon_2759` can be accessed using the URL `https://www.ebi.ac.uk/ols/ontologies/ncbitaxon/terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FNCBITaxon_2759`. +**Navigating ontologies to manually annotate your metadata:** The EBI OLS (ontology lookup service) provides an interface as well as an API to explore these ontologies which may be useful in preparing your metadata. The API is documented [here](https://www.ebi.ac.uk/ols/docs/developer), and the API endpoints for a given ontology are found in the `ontology` column in the AMC spreadsheet. The graphical interface can be accessed through the same URL with `api` substituted with `ols` (ex. `https://www.ebi.ac.uk/ols/ontologies/UBERON`). Because some ontologies include more areas than are covered by Alexandria, we have a recommended _root_ for some ontologies, specified in the `ontology_root` column of the AMC spreadsheet. Entries in Alexandria metadata should come from below that root in the ontology tree. To visit the page described by the root use the URL: `https://www.ebi.ac.uk/ols/ontologies//terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2F`. For example, for the species attribute, from the `NCBITaxon` ontology with root `NCBITaxon_2759` can be accessed using the URL `https://www.ebi.ac.uk/ols/ontologies/ncbitaxon/terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FNCBITaxon_2759`. -**Formatting ontology-valued metadata in the metadata file:** In the Alexandria Metadata Convention, two columns should be included for each ontology-valued metadata entry. The first, named `` contains the ontology ID in the structure `_` or `:`. The second column, `__ontology_label`, should contain the human-readable label for the ontology ID and should exactly match the name in the EBI database. This name is the title of the page on OLS. Ex. for attribute `species`, value: `NCBITaxon_9606` and `species__ontology_label`, value: `homo sapiens`. The label attribute is required for required metadata but may be left blank for optional metadata. It is recommended that users fill in this value to allow for more through validation of metadata files. +**Formatting ontology-valued metadata in the metadata file:** In the Alexandria Metadata Convention, two columns should be included for each ontology-valued metadata entry. The first, named `` contains the ontology ID with the format `_` or `:`. The second column, `__ontology_label`, should contain the human-readable label for the ontology ID and should exactly match the label (or listed synonym) in the EBI database. The label is the title of the page on OLS. Ex. for attribute `species`, value: `NCBITaxon_9606` has `species__ontology_label`, with value: `homo sapiens`. The label attribute is required for required metadata but may be left blank for optional metadata. It is recommended that users fill in this value to allow for more thorough validation of metadata files. ### Numeric-valued metadata @@ -76,29 +72,52 @@ ex. disease, vaccinations For several attributes, it does not make semantic sense to only allow one value for each metadata row. While we expect that in most controlled studies, these values will contain a single entry, we chose to allow multiple values for studies investigating combinitorial effects. Even if one of the values is not of specific interest to the study, please include it if possible. Ex. a study about flu vaccines would include the flu vaccine as a value for the vaccination attribute, but if the clinical data about other vaccines is also available, these vaccinations should be included as well. -**Syntax in metadata file:** Array-valued metadata should be formatted as follows: `["value1""value2""value3"]` or `"value1""value2""value3"`. If the array-valued metadata only contains a single value it should still follow the same format: ex `"value1"` or `["value1"]`. +**Syntax in metadata file:** Array-valued metadata should be delimited using the pipe symbol (|): `[value1|value2|value3"]`. If the array-valued metadata only contains a single value, no special formatting is necessary. -If the array-valued metadata attribute is a dependant metadata attribute, the order in the array should correspond with the order of the array the attribute depends on. For example, the following metadata entries for `vaccination__ontology_label`: `"Influenza Virus Vaccine""BCG Vaccine"`, for `vaccination__time_since` : `"1","2"`, and for `vaccination__time_since__unit_label`: `"year","year"` would indicate that the Influenza vaccine was recieved 1 year ago and the BCG Vaccine was recieved 2 years ago. If information is not available for one of the metadata entries in a dependant array, include an empty string(`""`) for that value. +If the array-valued metadata attribute is a dependant metadata attribute, the order in the array should correspond with the order of the array the attribute depends on. For example, the following metadata entries for `vaccination__ontology_label`: `Influenza Virus Vaccine|BCG Vaccine`, for `vaccination__time_since` : `1|2`, and for `vaccination__time_since__unit_label`: `year|year` would indicate that the Influenza vaccine was recieved 1 year ago and the BCG Vaccine was recieved 2 years ago. If information is not available for one of the metadata entries in a dependant array, include a pipe symbol (`|`) at the appropriate location to indicate the lack of data. Example: `vaccination__time_since` : `|2`. The AMC spreadsheet can be used as a guide in writing an Alexandria metadata file but includes more information than is necessary for this process. -See the below table for descriptions of columns in this spreadsheet that are useful in building this file: w - - -**Column**|**Description** -:---------|:-------------- -attribute | Serve as valid metadata column headers in the [input_csv_file](https://alexandria-scrna-data-library.readthedocs.io/en/latest/dropseq_scCloud/#formatting-your-input_csv_file) for dropseq_scCloud tool or the cell-level metadata file. Spaces between words are denoted by a single '\_' while subattributes are denoted with a double '\_' between the attribute parent and the attribute child. -required | Whether the attribute **MUST** be included as a column in the metadata file of the data you are uploading to Alexandria. -default | The default value which you should include in the metadata file for a required value if it does not apply to your data. _There is currently no system to automatically fill this data._ -type | The datatype of the attribute that Alexandria expects. `strings` can be text without quotation marks `booleans` can be either `True` or `False`. `numbers` are any numeric character, e.g. 0, 1, 2, ... -array | TRUE if the metadata is an array-valued metadata type. -class | The class of a string-type metadata attribute.
  • **[blank]**: no class, ontology: this attribute should be an ontology ID.
  • **ontology_label**: This attribute should be the human-readable label corresponding to it's ontology ID.
  • **enum**: a value from a controlled list.
  • **unit_label**: the name of the unit, either a free-text string or the ontology label from the unit ontology
-ontology | A URL to the ontology entry on the European Bioinformatics Insitute domain. -ontology_root | The highest value in the ontology tree applicable to this metadata attribute -controlled_list_entries | For attributes of class enum, the value is expected to be one value from the list of entries displayed here. -dependency | The parent ontology that the attribute is dependent upon. -dependency_condition | The condition on the dependency under which the attribute is dependant -dependent | The attribute that is dependent upon this attribute. -attribute_description | A description of what the attribute is. +See the below table for descriptions of columns in this spreadsheet that are useful in building this file: + + +```eval_rst ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| **Column** | **Description** | ++=========================+==========================================================================================================================+ +| attribute | Serve as valid metadata column headers in the `Alexandria Sheet `_ for | +| | dropseq_cumulus workflow or the cell-level metadata file. Spaces between words are denoted by a single '_' while | +| | subattributes are denoted with a double '\_' between the attribute parent and the attribute child. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| required | Whether the attribute **MUST** be included as a column in the metadata file of the data you are uploading to Alexandria. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| type | The datatype of the attribute that Alexandria expects. ``strings`` can be text without quotation marks `booleans` can | +| | be either ``True`` or ``False``. ``numbers`` are any numeric character, e.g. ``0``, ``1``, ``2``, ... | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| array | ``TRUE`` if the metadata is an array-valued metadata type. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| class | The classes of a string-type metadata attribute: | +| | | +| | - **[blank]**: no class +| | - **ontology**: this attribute should be an ontology ID. | +| | - **ontology_label**: This attribute should be the human-readable label (or synonym) corresponding to it's ontology ID. | +| | - **enum**: a value from a controlled list. | +| | - **unit_label**: the name of the unit, either a free-text string or the ontology label from the unit ontology | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| ontology | A URL to the ontology entry on the European Bioinformatics Insitute domain. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| ontology_root | The highest value in the ontology tree applicable to this metadata attribute. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| controlled_list_entries | For attributes of class enum, the value is expected to be one value from the list of entries displayed here. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| dependency | The parent ontology that the attribute is dependent upon. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| dependency_condition | The condition on the dependency under which the attribute is dependent. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| dependent | The attribute that is dependent upon this attribute. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +| attribute_description | A description of what the attribute is. | ++-------------------------+--------------------------------------------------------------------------------------------------------------------------+ +``` diff --git a/docs/metadata_validation.md b/docs/metadata_validation.md index fd6c1df..e1e7dde 100644 --- a/docs/metadata_validation.md +++ b/docs/metadata_validation.md @@ -5,6 +5,7 @@ git clone https://github.com/broadinstitute/single_cell_portal.git cd single_cell_portal python3 -m venv env --copies source env/bin/activate +pip install --upgrade pip pip install -r requirements.txt ```