Add s3-service.md to document the S3 service inside Safe Haven Services#238
Add s3-service.md to document the S3 service inside Safe Haven Services#238howff wants to merge 12 commits into
Conversation
| There is no general-purpose S3 service within Safe Haven Services, unlike the [EIDF S3 service](../services/s3/). | ||
|
|
||
| However there is a S3 service in SHS with the following caveats: | ||
| * it is only available to the Scottish National Safe Haven, other tenants by arrangement | ||
| * it is a read-only service, as a way of providing access to large collections of files | ||
| * it is not a storage solution for users wanting to create their own files | ||
|
|
||
| ## Access arrangements | ||
|
|
||
| Access to buckets is via keys (Access Key and Secret Access Key) provided to the user by the Research Coordinator. |
There was a problem hiding this comment.
Suggested rewording: Some Safe Havens may provide you with access to data via S3. If this applies to your project, your Research Coordinator will provide you with an access key. This documentation will guide you through how to get access to your data from a terminal as well as programmatically via R and Python.
!!! important Files in S3 buckets are read-only. If you need to transform or make changes to any files, you will need to download them to your project space. If you download files, please be mindful of disk space by only downloading what is necessary and deleting them as soon as no longer needed.
|
|
||
| Access to buckets is via keys (Access Key and Secret Access Key) provided to the user by the Research Coordinator. | ||
|
|
||
| ## How to use the service |
| To access files you need the following information: | ||
|
|
||
| * Region is "us-east-1" | ||
| * Endpoint URL is "http://nsh-fs02:7070" | ||
| * Access key | ||
| * Secret access key | ||
| * The web proxy variables must be empty |
There was a problem hiding this comment.
Your RC will provide you with a bucket name, access key ID (bucket name), and secret access key.
| export http_proxy= | ||
| ``` | ||
|
|
||
| ## Use from the command line |
| If this command fails it might be due to a proxy configuration in your environment. To temporarily turn off the proxy in the current window use this first: | ||
| ``` | ||
| export http_proxy= | ||
| ``` |
There was a problem hiding this comment.
Why is this the case? This appears to be a workaround that could become confusing if the user then attempts to download other packages in the same terminal. Users will likely run this without understanding what it does or that they would have to run other installation commands in a separate terminal.
There was a problem hiding this comment.
I don't know if it's technically possible for the web proxy to be configured to pass traffic onto nsh-fs02. Maybe a question for Barry or similar. If so it would reduce confusion, but on the other hand I don't think it's a good idea to put the web proxy between the client and the S3 server because all it does is cause additional unnecessary load on the proxy and slow everything down; there's no benefit from authentication either.
There was a problem hiding this comment.
Had a chat with Barry about this, he recommends the use of the NO_PROXY variable set in bashrc instead, so this would look like export NO_PROXY="$NO_PROXY,nsh-fs02:7070" (I tested and can confirm works). I also asked Susan, and users are never told to go and edit their bashrcs, which we may want to avoid. The other option is for systems to add this to their bashrcs (either to all or on-demand).
There was a problem hiding this comment.
Last time I tested this the NO_PROXY variable was ignored!
There was a problem hiding this comment.
NO_PROXY will work here, if properly configured:
Routes to the proxy
rmacleod@nsh-rc-desktop01:~$ NO_PROXY=''; curl -LI nsh-fs02:7070
HTTP/1.1 503 Service Unavailable
Server: squid
...Routes directly to nsh-fs02
rmacleod@nsh-rc-desktop01:~$ NO_PROXY=nsh-fs02; curl -LI nsh-fs02:7070
HTTP/1.1 400 Bad Request
Server: VERSITYGW
...A more generalised solution would be to set NO_PROXY=localhost,127.0.0.1,.nsh.loc and then to fully-qualify the server as nsh-fs02.nsh.loc.
There was a problem hiding this comment.
NO_PROXY doesn't work; R ignores it and Python ignores it. awscli obeys it. Python does obey no_proxy, R ignores it.
| At this point it will probably complain that it can't locate your credentials. In fact it requires a bit more information in order to find the bucket: the region, endpoint, access key and secret key: | ||
| ``` | ||
| export AWS_DEFAULT_REGION=us-east-1 | ||
| export AWS_ENDPOINT_URL=http://nsh-fs02:7070 | ||
| export AWS_ACCESS_KEY_ID=put_your_key_here | ||
| export AWS_SECRET_ACCESS_KEY=put_your_secret_here | ||
| export http_proxy= | ||
| aws s3 cp s3://extraction_5_CT/123/456/789.dcm copy_of_789.dcm | ||
| ``` | ||
|
|
||
| It should go without saying that the access key details are confidential and must never be shared or allowed to be seen by others. Note that all file accesses are logged. |
There was a problem hiding this comment.
Avoid repeating this by placing it in the environment setup section. Maybe show this as an example env file and source command.
|
|
||
| It should go without saying that the access key details are confidential and must never be shared or allowed to be seen by others. Note that all file accesses are logged. | ||
|
|
||
| ## Performance tips |
There was a problem hiding this comment.
These apply to programmatic methods as well, so why not put them after access methods?
|
|
||
| ## Performance tips | ||
|
|
||
| * consume the file directly in memory if possible, don't save it to disk. Saving to disk will waste disk space and it will take 3 times longer to do your processing. See the example code below. |
There was a problem hiding this comment.
Process S3 files directly in memory wherever possible. Saving files to disk is not recommended as this will harm performance (expected to be up to 3 times slower). If this cannot be avoided, please delete files when no longer needed to recover disk space.
| pip install boto3 | ||
| ``` | ||
|
|
||
| ### Download an object |
| import boto3 | ||
| resource = boto3.resource("s3") | ||
| bucket = resource.Bucket("epcc-test") | ||
| bucket.download_file("test.txt", "copy_of_test.txt") |
There was a problem hiding this comment.
Show how the env vars would be used to configure the connection, like you have in R.
| bucket.download_file("test.txt", "copy_of_test.txt") | ||
| ``` | ||
|
|
||
| ### Load object into pydicom dataset |
There was a problem hiding this comment.
Generalise this to "load a file for further processing" or similar, and give as an example passing it to pydicom.
| my_endpoint_host <- "nsh-fs02:7070" | ||
| my_object_path <- "studyid/seriesid/instanceid-an.dcm" | ||
| save_object( my_object_path, file = "output.dcm", bucket = my_bucket, base_url=my_endpoint_host, region="", use_https=FALSE, key = my_access_key, secret = my_secret_key ) | ||
| ``` |
There was a problem hiding this comment.
No examples of loading files given as with Python. Examples must be equivalent.
| * A "bucket" name (similar to a folder name) | ||
| * An "access key" (similar to a username) | ||
| * A "private key" (similar to a password) |
There was a problem hiding this comment.
The user only needs the access key id (or bucket name), and secret access key. I think this is confusing, implies there's two access keys, one of which is "similar to a username", despite this not being used in the configuration?
There was a problem hiding this comment.
Actually it is as described, the bucket name is not the access key
There was a problem hiding this comment.
When defining the variables below, and as defined in the file received from the RC:
AWS_ACCESS_KEY_ID=bucketname
AWS_SECRET_ACCESS_KEY=password
There was a problem hiding this comment.
The bucket name is like the directory name, the access key is like the username and the secret key is like the password, I'm not sure where you're getting the information that the access key is the same as the bucket name.
There was a problem hiding this comment.
As specified in my previous comment, I'm getting it from the documentation and the file received from the RC. Maybe an example would help clarify where I'm coming from.
I have a user bprodan in study 2024-0000 and was told by the RC that I will be given access to a bucket called dummydata. According to the user documentation, which follows the standard minio convention, the configuration should be:
| variable | value |
|---|---|
| BUCKET_NAME | dummydata |
| ACCESS_KEY_ID | bprodan |
| SECRET_ACCESS_KEY | 1234 |
According to the RC documentation, keys are created at study level, so this should be more like:
| variable | value |
|---|---|
| BUCKET_NAME | dummydata |
| ACCESS_KEY_ID | 2024-0000 |
| SECRET_ACCESS_KEY | 1234 |
But instead, in the file received from the RC, I have:
| variable | value |
|---|---|
| BUCKET_NAME | dummydata |
| ACCESS_KEY_ID | dummydata |
| SECRET_ACCESS_KEY | 1234 |
Currently, the user documentation assumes knowledge of how minio is configured and calling the access key ID a username just to then receive the bucket name would make a user question which value maps to that variable and if that is their username or not.
There was a problem hiding this comment.
The bucket name is like the directory name, the access key is like the username and the secret key is like the password, I'm not sure where you're getting the information that the access key is the same as the bucket name.
I agree with this, so perhaps the RC documentation needs to be updated?
As an aside, we are not using MinIO.
There was a problem hiding this comment.
The technology is not important in this case, this is about user guidance on how to configure access. I am not disagreeing with Andrew's statement, I'm saying the way the documentation presents this assumes knowledge about what the access key ID variable represents and is unclear about what value is associated with it.
| save_object( my_object_path, file = "downloaded.dcm", bucket = my_bucket, base_url=my_endpoint_host, region="", use_https=FALSE, key = my_access_key, secret = my_secret_key ) | ||
| ``` | ||
|
|
||
| Note! You need to have the region set in the environment variable and pass `region=""` to the functions, otherwise you get a `cannot resolve host` error. The `base_url` is just the host and port, no `http://` prefix, and `use_https` is false. |
There was a problem hiding this comment.
Note formatting is inconsistent within this page as well as compared with the rest of the docs. For important pieces of text, this is probably the most visible.
| os.environ["AWS_ENDPOINT_URL"] = "http://nsh-fs02:7070" | ||
| os.environ["AWS_DEFAULT_REGION"] = "us-east-1" | ||
| os.environ["AWS_ACCESS_KEY_ID"] = "dummydata" | ||
| os.environ["AWS_SECRET_ACCESS_KEY"] = "put_your_secret_here" | ||
| os.environ["http_proxy"] = "" |
There was a problem hiding this comment.
The user was already told to set the environment variables. Does boto automatically pick these up or is something like this needed?
kavousan
left a comment
kavousan
left a comment
There was a problem hiding this comment.
If the answer to the pip install comment is that it's not a problem, this is an approval.
There was a problem hiding this comment.
Can anyone do pip install?
There was a problem hiding this comment.
Minor points you may not care about just now:
-
Standardise "Important" signposting; they are all the same (unless one wants to use an md construct), so:
- Under "Access arrangements", add - after it, like the first instance
- Under "How to use" replace "Note!" with "Important -"
- Under R usage replace "Note!" with "Important -" (needs done three times)
-
Remove "n" from "recover then memory"
There was a problem hiding this comment.
Yes, anyone can run pip install as part of access to PyPi via the web proxy.
Added instructions for using a GUI to view DICOM files from the S3 service, including container setup and script creation.
|
@howff, we are approaching this year's pentest for the NSH and would benefit from having access to this documentation, even as a hidden page. Can I please ask you to bring this up-to-date with main and remove the addition of this page to |
| All of the above will be supplied to you by your Research Coordinator. | ||
| It should go without saying that the access key details are confidential and must never be shared or allowed to be seen by others. | ||
| Note that all file accesses are logged. | ||
|
|
There was a problem hiding this comment.
I would replace this with a drier note: " S3 access key details are confidential and must never be shared or allowed to be seen by others. All file accesses are logged."
| ## Performance tips | ||
|
|
||
| * consume the file directly in memory if possible. Saving to disk is not recommended; it wastes disk space and will take 3 times longer to do your processing. See the example code below. | ||
| * If you need to save into a file temporarily (e.g. whilst converting to NIFTI) then save into a RAM disk in `/run/user/$(id -u)/` but delete it straight after use to recover then memory. |
There was a problem hiding this comment.
...to recover memory. (or recover the memory, not sure what applies)
|
|
||
| * consume the file directly in memory if possible. Saving to disk is not recommended; it wastes disk space and will take 3 times longer to do your processing. See the example code below. | ||
| * If you need to save into a file temporarily (e.g. whilst converting to NIFTI) then save into a RAM disk in `/run/user/$(id -u)/` but delete it straight after use to recover then memory. | ||
| * If it's too large for RAM then save into a file on the system disk, not in your home directory, in `/tmp/$(id)/` but check the disk has space first (using `df -h /tmp/`) and delete it straight after use to recover the disk space. |
There was a problem hiding this comment.
Is /tmp/$(id)/ only readable by the user (and maybe their group)?
| ## Example GUI to view DICOM files | ||
|
|
||
| This example uses a customised program that can view and download DICOM files from the S3 service. | ||
| It is not installed in the National Safe Haven but you can import it as a container image. |
There was a problem hiding this comment.
As per first sentence, this appears to be drafted as a Tenant-agnostic page, so replace with "If it is not installed in your Safe Haven, you can import is as a container image."
| REPOSITORY TAG IMAGE ID CREATED SIZE | ||
| ghcr.io/howff/dcmaudit cpu 4f994194efa8 11 days ago 2.67 GB | ||
| ``` | ||
| Now you will need a 's3' directory in your home directory so create that, or run: |
There was a problem hiding this comment.
so to create that, run the following command:
| ``` | ||
| ces-pm-run --opt-file <(echo -v $HOME/.dcmaudit:/root/.dcmaudit -v $HOME/s3:/root/s3 --http-proxy=false) ghcr.io/howff/dcmaudit:cpu | ||
| ``` | ||
| That is difficult to type every time you need it, so we will create a script: |
There was a problem hiding this comment.
so you can create a script as follows:
kavousan
left a comment
kavousan
left a comment
There was a problem hiding this comment.
I made some simple suggestions and let more minor points go. If there is no time, and if the answer to the /tmp comment is that only user (and maybe their group, if IG happy with that) can read this dir, I can approve without these changes/.
| * If you need to save into a file temporarily (e.g. whilst converting to NIFTI) then save into a RAM disk in `/run/user/$(id -u)/` but delete it straight after use to recover then memory. | ||
| * If it's too large for RAM then save into a file on the system disk, not in your home directory, in `/tmp/$(id)/` but check the disk has space first (using `df -h /tmp/`) and delete it straight after use to recover the disk space. | ||
| * If you need to save into a file temporarily (e.g. whilst converting to NIFTI) then save into a RAM disk in `/run/user/$(id -u)/` but delete it straight after use to recover the memory. | ||
| * If it's too large for RAM then save into a file on the system disk, not in your home directory, for example in `/tmp/$(id -un)/` but check the disk has space first (using `df -h /tmp/`) and delete it straight after use to recover the disk space. |
There was a problem hiding this comment.
I spoke to Donald. The Safe Setting does not enforce permissions on /tmp; it would be at least a near-miss to put Safe Data there without correct permissions. There is also no real performance benefit between /tmp and the user account. This line should be deleted.
Donald's advice is to delete both lines; the advice should be to just work in memory.
Don't suggest a private directory underneath /tmp
kavousan
left a comment
kavousan
left a comment
There was a problem hiding this comment.
Many thanks to all who contributed text and reviews.
4 participants
EIDF Documentation Pull Request
Description
Add s3-service.md to document the S3 service inside Safe Haven Services
Fixes #237 237
Type of change
Please delete options that are not relevant.
What has to be reviewed
s3-service.md
Checklist