Skip to content

Update README.md #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
proccaserra wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update README.md #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 64 additions & 35 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
# ISA Plug & Play Zendro Data Warehouse

The ISA Model provides a flexible standard for scientific research data. As it is based on the scientific method it is very versatile and can be applied to almost any research project. The ongoing [reprodicibility crisis](https://en.wikipedia.org/wiki/Replication_crisis) has long been recognized and one of the mayor causes is missing meta-data for published scientific results. This is where the ISA Model shines and helps scientists to fill the gap.
The [ISA Model](https://isa-tools.org/format/specification.html) provides a flexible standard for scientific research data. As it is based on the scientific method, it is very versatile and can be applied to almost any research project. The ongoing [reproducibility crisis](https://en.wikipedia.org/wiki/Replication_crisis) has long been recognized and one of the major causes is missing meta-data for published scientific results. This is where the ISA Model can help scientists to fill the gap.

While research projects are typically well equipped to carry out the research modern studies often integrate and produce highly diverse and voluminous data, both in terms of type and format as well as size. To administer this data in an adecuate way and ensure scientific results are reproducible, a fully featured data-warehouse is required. Ideally such a warehouse has both a graphical, point and click, user interface (GUI) as well as a application programming interface (API). The former allows scientists to administer the data and also explore it, while the latter enables data analysts to perform exhaustive data searches and directly include data in anaylis scripts without the need to download large data files. Additionally, data-warehouses need to offer access functions that make the stored data findable, accessible, interoperable, and reusable (FAIR). These requirements imply that a data-warehouse no longer is a simple file-server in the web with short readme files explainng the data and how it was produced. Rather, programming a data-warehouse, especially when adhering to standards like the ISA Model becomes a suizable software development job itself. Resources, in terms of work time, experts, and funding, for such a task a often hard to come by. To fill this gap we present a ready to use, no programming required plug&play ISA Model data warehouse.
While research projects are typically well equipped to carry out the research, modern studies often integrate and produce highly diverse and voluminous data, both in terms of type and format as well as size. To manage such data in an adequate manner and ensure availability of scientific results, a fully featured data-warehouse is required. Ideally, such a resource should have both a graphical, point and click, user interface (GUI) as well as an application programming interface (API). The former allows scientists to manage and explore data, while the latter enables data analysts to perform exhaustive data searches and directly include data in analysis scripts without the need to download large data files.
Furthermore, modern data-warehouses should offer the means to make hosted data Findable, Accessible, Interoperable, and Reusable (FAIR). These requirements imply that data-warehouses should no longer be a simple file-server on the web, with short readme files explaining how data was produced. Rather, a modern data-warehouse would benefit from embedding a suitable domain metadata model to ensure that both data and metadata are tracked at the same time.
Hence, the idea to develop a Zendro Database which would take advantage of the ISA Model and offer the service to the community.
This is all the more important as resources, in terms of work time, experts, and funding, for developing such a software tools are often hard to come by. By working with with ISA, Zendro filled a capability gap in the ISA ecosystem and our teams now present a *ready to use, no programming required plug&play ISA Model data warehouse*.

This repository holds a Plug & Play [Zendro](https://zendro-dev.github.io/) instance created for the latest [ISA](https://isa-specs.readthedocs.io/en/latest/isamodel.html) data model definitions. It is also customized to enable file attachments via a [minio](https://min.io/) fileserver as well as special handling of images through an [imgproxy](https://github.com/imgproxy/imgproxy) server. This particulary enables the user not only to administer scientific meta-data, but also host the research result data itself in the same warehouse.
This repository holds a `Plug & Play` [Zendro](https://zendro-dev.github.io/) instance created for the latest [ISA data model definitions](https://github.com/ISA-tools/isa-api/tree/develop/isatools/resources/schemas/isa_model_version_1_0_schemas/core). It is also customized to enable:
* file attachments via a [MinIO](https://min.io/) fileserver
* as well as special handling of images through an [imgproxy](https://github.com/imgproxy/imgproxy) server.

>This therefore enables the user to both *administer scientific meta-data* and *host the research result data itself* **in the same warehouse**.

## Prerequisites

The plug&play ISA Model warehouse based on Zendro uses Docker. Please make sure, the following requirements are met:
The plug&play ISA Model warehouse based on Zendro uses Docker. Please make sure the following requirements are met:

- [docker](https://docs.docker.com/get-docker/)
- [docker-compose](https://docs.docker.com/compose/install/)
Expand All @@ -17,58 +24,79 @@ The plug&play ISA Model warehouse based on Zendro uses Docker. Please make sure,

Download or clone the repository.

```
```bash=
git clone https://github.com/Zendro-dev/ISA-plug-and-play-warehouse
```

## Configuration

Configuration can be done via the `.env` files in the separate sub-repositories (`graphql-server`, `single-page-app` and `graphiql-auth`).
Configuration can be done via the `.env` files, which can be found in each of the separate sub-repositories (`graphql-server`, `single-page-app` and `graphiql-auth`).

There are `.env.example` files included that can be used by simply removing the `.example` extension. They should work out of the box as is.

### Database configuration (Advanced)
```bash=
cd graphql-server
mv .env.example .env
```

```bash=
cd single-page-app # perform similarly from graphiql-auth directory
mv .env.production.example .env.production
mv .env.development.example .env.development
```


In case you want Zendro to talk to other databases that can be configured in `config/data_models_storage_config.json`. Currently a default PostgreSQL database is used. Support exists for a great variety of storage technologies. See [the Zendro manual](https://zendro-dev.github.io/) for more details.

### Database configuration (advanced)

Currently a default PostgreSQL database is used.
Support exists for a great variety of storage technologies.
In case you want Zendro to talk to databases other than the default (PostgreSQL), that can be configured in `config/data_models_storage_config.json`.
See [the Zendro manual](https://zendro-dev.github.io/) for more details.

## Start up

Build and start the docker containers via:

```
```bash
docker-compose -f docker-compose-prod.yml up -d --force-recreate --remove-orphans

```
This will fetch all the necessary docker images and install them as well as starting up the containers. The containers will be avaialable at:
This will fetch all the necessary docker images, install them, and then start-up the containers.
The containers will be available at:

| service|location |
|:---|:---|
|keycloak|http://10.5.0.11:8081|
|graphql-server|http://localhost:3000/graphql|
|graphiql|http://localhost:7000|
|single-page-app|http://localhost:8080|
|minio|http://localhost:9000|
|imgproxy |http://localhost:8082|

```
graphql-server: http://localhost:3000/graphql
graphiql: http://localhost:7000
single-page-app: http://localhost:8080
keycloak: http://10.5.0.11:8081
minio: http://localhost:9000
imgproxy: http://localhost:8082
```

## Shut down
Stop the docker containers via
```

Stop the docker containers via this docker command
```bash
docker-compose -f docker-compose-prod.yml down
```
In case you want to also delete the local docker volumes with your data run
```

In case you want to also delete the local docker volumes with your data, run the following command
```bash
docker-compose -f docker-compose-prod.yml down -v
```

## Creating the Minio Buckets

By default this setup uses 2 separate minio buckets (something like directories in which the respective files are stored) to hold image and other data. The names of the buckets can be controlled via environment variables. The defaults are:
By default, this setup uses 2 separate MinIO `buckets` (which are something like directories in which the respective files are stored) to hold images in one bucket, and data files in the other one.
The names of the buckets can be controlled via `environment variables`.
The currently defaults are:
```
IMG_BUCKET_NAME = "images";
FILES_BUCKET_NAME = "files";
```

Those need to be created manually in the minio admin surface at
Those need to be created manually in the `MinIO admin interface` at

```
http://localhost:9000
Expand All @@ -83,9 +111,9 @@ password: miniominio

_**Note:**_ This step can probably be automated in the future.

## Keycloak admin console (Advanced)
## Keycloak admin console (advanced)

In case there is some manual configuration needed (like adding different identity providers) for keycloak, the admin console is reachable at:
In case keycloak needs manual configuration tweaks (like adding different identity providers) , the `keycloak admin console` is reachable at:
```
http://10.5.0.11:8081
```
Expand All @@ -101,21 +129,22 @@ To access the graphql-server from the console, e.g. via `curl` or `wget` command
```
curl http://localhost:3000/help
```
to get information on how to get an access_token and how to use it to access the server with simple http requests via `curl`. This is of course directly applicable to any other script like e.g `python`, `ruby`, `perl`, `R`, etc.
To get information on how to get an access_token and how to use it to access the server with simple http requests via `curl`. This is of course directly applicable to any other script like e.g `python`, `ruby`, `perl`, `R`, etc.

## Plots
In the Sandbox an example of how to supply a simple plot to the SPA. To make the plot work a table is include in the repository in `data`, that can be added as a `data` record for example through the single-page-app interface.
- click on data at the left hand menu
In the Sandbox, an example of how to supply a simple plot to the single-page-app (SPA). To make the plot work, a table is included in the repository in `data`, that can be added as a `data` record for example through the single-page-app interface.
- Click on data at the left hand menu
- Add new record by clicking the `+` icon
- Add the record and attach the file `data/20220127_Analysis_Tomato_both_cpm_merged.csv`

After adding the record clicking on the `Plots` Menu in the Topbar should render a bar plot of Gene-expression counts for a specific Tomato genes under different conditions.
After adding the records, clicking on the `Plots` Menu in the Topbar should render a bar plot of Gene-expression counts for a specific Tomato genes under different conditions.

The data used is published and accessible at
```
Reimer JJ, Thiele B, Biermann RT, Junker-Frohn LV, Wiese-Klinkenberg A, Usadel B, et al. Tomato leaves under stress: a comparison of stress response to mild abiotic stress between a cultivated and a wild tomato species. Plant Mol Biol. 2021;107: 177–206.
```

>Reimer JJ, Thiele B, Biermann RT, Junker-Frohn LV, Wiese-Klinkenberg A, Usadel B, et al. Tomato leaves under stress: a comparison of stress response to mild abiotic stress between a cultivated and a wild tomato species. Plant Mol Biol. 2021;107: 177–206.
>[doi:10.1007/s11103-021-01194-0](https://doi.org/10.1007/s11103-021-01194-0)

## Manual customization
Every sub-repoistory is fully customizable by simply changing the code and restarting the docker-compose.
Each sub-repository is fully customizable by simply changing the code and restarting the services with docker-compose.

See also documentation on how to customize the single-page-app [here](https://github.com/Zendro-dev/single-page-app#customization) and for more detailed documentation on nextjs [here](https://nextjs.org/docs/getting-started).