About htsget

The htsget protocol was selected for the Data Reception part of the GDI starter kit. Htsget enables for retrieval of files from the storage-and-interfaces archive. The specification of the protocol can be found here. The implementation can be found here.

This repository contains the implementation for the GDI starter kit, packaged in the docker-compose.yml file. For a demo setup, you also need to setup the GDI starter-kit storage-and-interfaces. More details regarding that and the way to run the services can be found in the sections below.

HTSGET Configuration

The configuration for the htsget server exists in the folder called config-htsget-rs. Detailed description of the configuration options can be found in the reference implementation repository.

Running the services

The htsget product of the starter kit depends on the storage-and-interfaces product. Specifically, the data served has to be ingested and stored in the archive included in the storage-and-interfaces repository. This is achieved by using the docker-compose-demo.yml, as below.

Preparations

The htsget-rs requires a crypt4gh key pair which will be used in communication with the sda-download service Tool that is to generate the key is neicnordic/crypt4gh.

crypt4gh generate -n config-htsget-rs/c4gh

NOTE: This key should be generated without a password, as the htsget-rs does not support passworded keys

Starting the services

To start the services, start the individual docker compose environments from their respective root directories:

docker compose -f docker-compose-demo.yml up -d  # in the folder starter-kit-storage-and-interfaces
docker compose up -d # in the folder starter-kit-htsget

The logs for the two docker compose files can be accessed using the following commands for storage-and-interfaces and htsget respectively

docker compose -f docker-compose-demo.yml logs -f # in the folder starter-kit-storage-and-interfaces
docker compose -f docker-compose.yml logs -f # in the folder starter-kit-htsget

Accessing data with htsget

In order to test the htsget implementation, there needs to be some data ingested into the archive. The demo setup of storage-and-interfaces provides one dataset, DATASET0001, containing the file htsnexus_test_NA12878.bam.

Through the sda-cli tool

The sda-cli tool has functionality to wrap the calls to htsget and then to the sda-download service and will prepare the final file with the relevant parts corresponding the requested refence name

Preparations

Download and install the sda-cli tool at version v0.3.2

Prepare a config.yaml file to use with the sda-cli

echo ' 
access_token='$(curl -s -k https://localhost:8080/tokens | jq -r '.[0]')'
access_key=dummy_key
host_base=dummy_host_base 
' > config.yaml

You also need a crypt4gh key pair for the client. This will be used for (re-)encrypting the file before it's sent to you. Tool that is to generate the key is neicnordic/crypt4gh.

crypt4gh generate -n client.c4gh
pubkey=$(base64 -w0 client.c4gh.pub.pem)

Retrieving data

Running the following sda-cli command will now call htsget-rs and get the relevant content ranges for the specified refence and download each range and concatenate the ranges to a file which can be decrypted with the generated client.c4gh.sec.pem

./sda-cli -c config.yaml htsget --dataset DATASET0001 --filename htsnexus_test_NA12878 --reference 11 --host http://localhost:8088 --pubkey client.c4gh.pub.pem

Decrypting the file

Running the following sda-cli command will decrypt the downloaded file

./sda-cli -c config.yaml decrypt --key client.c4gh.sec.pem htsnexus_test_NA12878.c4gh     

Through the APIs

Follow the steps below to access the information exposed by the APIs in htsget-rs.

Preparations

Get an authentication token from the auth service of storage-and-interefaces using

token=$(curl -s -k https://localhost:8080/tokens | jq -r '.[0]')

You also need a crypt4gh key pair for the client. This will be used for (re-)encrypting the file before it's sent to you. Tool that is to generate the key is neicnordic/crypt4gh.

crypt4gh generate -n client.c4gh
pubkey=$(base64 -w0 client.c4gh.pub.pem)

Retrieving data

Now you should be able to make the requests to the htsget server. To request the byte range of chromosome 11 of the file htsnexus_test_NA12878.bam run:

curl -v -H "Htsget-Context-Public-Key: $pubkey" -H "Authorization: Bearer $token" -H -k http://localhost:8088/reads/DATASET0001/htsnexus_test_NA12878?encryptionScheme=C4GH&referenceName=11

The request will return a ticket of how to download the requested partial file:

{
 "htsget": {
   "format": "BAM",
   "urls": [
     {
       "url": "data:;base64,Y3J5cHQ0Z2gBAAAAAgAAAA=="
     },
     {
       "url": "http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam.c4gh",
       "headers": {
         "Range": "bytes=16-123",
         ...
       }
     },
     {
       "url": "data:;base64,ZAAAAAAAAACxHxjMhagEVY+4bVEZYuqYGK5Ph3jrffrMhXpc3wYWenp2ofohEUwSBOuZF3kH6TEiQsjSPGaE1bvdMQ2uUuuHLWicplUneE77G079sTW8rJIJJ1VgZecPi9cTfQ=="
     },
     {
       "url": "http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam.c4gh",
       "headers": {
         "Range": "bytes=124-1049147",
         ...
       }
     },
     {
       "url": "http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam.c4gh",
       "headers": {
         "Range": "bytes=2557120-2598042",
         "accept": "*/*",
         ...
     }
   ]
 }
}

This repsonse contains byte ranges (eg. "Range": "bytes=124-1049147") as parts of url requests. This should guide you to make requests to http://localhost:8443/s3-encrypted (which is sda-download from storage-and-interfaces) to retrieve data for chromosome 11 from the file:

curl 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam' -H "Authorization: Bearer $token"  -H "Client-Public-Key: $pubkey" -H "Range: bytes=16-123" -o p11-00.bam.c4gh
curl 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam' -H "Authorization: Bearer $token"  -H "Client-Public-Key: $pubkey" -H "Range: bytes=124-1049147" -o p11-01.bam.c4gh
curl 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam' -H "Authorization: Bearer $token"  -H "Client-Public-Key: $pubkey" -H "Range: bytes=2557120-2598042" -o p11-02.bam.c4gh

The response from hstget also lists two data sections:

"url": "data:;base64,Y3J5cHQ0Z2gBAAAAAgAAAA=="

and

"url": "data:;base64,ZAAAAAAAAACxHxjMhagEVY+4bVEZYuqYGK5Ph3jrffrMhXpc3wYWenp2ofohEUwSBOuZF3kH6TEiQsjSPGaE1bvdMQ2uUuuHLWicplUneE77G079sTW8rJIJJ1VgZecPi9cTfQ==

These segments are part of the requested data. Save the data (eg. Y3J5cHQ0Z2gBAAAAAgAAAA==) to files

echo Y3J5cHQ0Z2gBAAAAAgAAAA== | base64 -d > start.b64
echo ... | base64 -d > mid.b64

Then concatenate all segments:

cat start.b64 p11-00.bam.c4gh mid.b64 p11-01.bam.c4gh p11-02.bam.c4gh > htsnexus_11.bam.c4gh 

Make sure that the file can be decrypted with your private key:

crypt4gh decrypt -s client.c4gh.sec.pem -f htsnexus_11.bam.c4gh

Finally, check that samtools can open the new file:

samtools view htsnexus_11.bam

or, if you don't have samtools installed

docker run -it --rm -v $(pwd):/tmp staphb/samtools /bin/bash