Suggestion: Single download of all files/databases

[jakobnissen]

MMASeq does a lot of downloading. This means lots of URLs are stored in the repo to various different internal databases and files. This is fragile, as these URLs may change or go offline in the future.

I suggest collecting all downloads in one or a few tarred and zipped folders, and then have the commands download and untar+unzip these in the beginning. Some advantages:

• It makes it clearer which files the workflow depends on
• You're in control of when databases and files are updated, making results more reproducible
• You're in control of the links, making it less likely that they go offline in the future
• It'll be more efficient and faster
• Download errors surface earlier, since downloads are moved up front in the pipeline

[KasperThystrup]

We are allready handling many databases from this repo:
https://github.com/ssi-dk/ssi_analysis_utility_db/tree/main

Which rules outside this are you thinking about? :-)

[jakobnissen]

Besides the one in your repo, currently, this repo downloads:

• 16 FASTQ files listed in src/mmaseq/data/reads/reads.urls
• From enterobase.warwick.ac.uk:
  • MLST_Achtman_ref.fasta
  • profiles.list.gz
• serovar-list.txt from raw.githubusercontent.com/phac-nml/sistr_cmd
• fimH.fsa and fumC.fsa from CHTyper DB
• elmDB.tar.gz from LRE-Finder
• Various git repos, such as plasmidfinder_db, resfinder_db, pointfinder_db etc.

While the current system works currently, I would probably gather all this data (including all the git repos) and create a single large gzipped tarball to download.

[KasperThystrup]

I think it makes sense to maintain a backup master zip that could be created and hosted on a annual schedule, in case the online locations becomes unavailable. I do still wish to maintain main download functionality as well though, so we can ensure that the latest DB versions can be fetched with minimal interaction.

I don't think we can dump it into a rule structure as the rest, so it would properly be a seperate script evoked by deploy.py.

Ideas are welcome :-)