Run Semi Automatically

Setup

From inside Cnidaria folder, add "scripts" folder to your PATH by:

• type, from this folder, every time you open a new terminal

• add script folder to your PATH by:
  • type, from this folder, every time you open a new terminal

source enable.sh

- add (only once) cnidaria/scripts to your $HOME/.bashrc and restart (only once) your terminal

echo "export PATH=$PWD/scripts:$PATH"                 >> $HOME/.bashrc
echo "export PATH=$PWD/src/libs/Jellyfish/bin/:$PATH" >> $HOME/.bashrc

verify if samtools is installed (only if working with BAM/CRAM)

Make sure samtools is in your PATH

samtools

how to generate jf database files

• jf_from_fasta.sh converts FASTA to JF.
• jf_from_fastq.sh converts FASTA to JF.
• jf_from_cram.sh converts CRAM to JF (requires samtools >= 1.1 in PATH).
• jf_from_bam.sh converts BAM to JF (requires samtools >= 1.1 in PATH).

Create a filelist.csv file

Create a filelist.csv file, a tab delimited file containing the name of your JF files and their "pretty" name:

/home/user/cnidaria/data/input/spp1.fas.21.jf	Species 01
/home/user/cnidaria/data/input/spp2.fas.21.jf	Species 02
/home/user/cnidaria/data/input/spp3.fas.21.jf	Species 03
/home/user/cnidaria/data/input/spp4.fas.21.jf	Species 04
/home/user/cnidaria/data/input/spp5.fas.21.jf	Species 05

Create a test_def.csv file

Create a test_def.csv file, a tab delimited file containing the run names

test01	/home/user/cnidaria/data/input/spp1.fas.21.jf
test01	/home/user/cnidaria/data/input/spp2.fas.21.jf
test01  /home/user/cnidaria/data/input/spp3.fas.21.jf
test01  /home/user/cnidaria/data/input/spp4.fas.21.jf
test01  /home/user/cnidaria/data/input/spp5.fas.21.jf

Run

Automatically

Create a Makefile

Create a Makefile for your runs by using scripts/gen_mkfile.py.

It will create a Makefile for your analysis calling all programs in the correct order for: split, calculate, merge, generate statistics and plot graphs in one go

$ scripts/gen_mkfile.py -h
usage: gen_mkfile.py [-h] [-thr [NUM_THREADS]] [-min [MINVAL]]
                     [-se [SAVE_EVERY]] [-me] [-nm] [-ec] [-nem] [-img]
                     file_list def_file out_dir kmer_size num_pieces

Cnidaria Merger Makefile Creator

positional arguments:
  file_list             file name mapping list
  def_file              definition file
  out_dir               output dir
  kmer_size             kmer size
  num_pieces            number of pieces

optional arguments:
  -h, --help            show this help message and exit
  -thr [NUM_THREADS], --threads [NUM_THREADS]
                        Number of threads. Not compatible with COMPLETE
                        (default: 1)
  -min [MINVAL], --minval [MINVAL]
                        Minimum number of shared species to start counting
                        (default: 2)
  -se [SAVE_EVERY], --save-every [SAVE_EVERY]
                        Count every N k-mers. Speeds analysis while skipping
                        data (default: 1)
  -me, -merge, --merge-only
                        Merge only (default: False)
  -nm, -nomerge, --do-not-merge
                        Do not merge (default: True)
  -ec, -complete, --export-complete
                        Export COMPLETE database (default: False)
  -nem, -nomatrix, --no-export-matrix
                        DO NOT Export MATRIX database (default: True)
  -img, -image, --gen-image
                        Generage PNG images. Requires X11 (default: False)

Example: ./gen_mkfile.py <file_list> <def_file> <out_folder> <kmer_size> <number_of_pieces>

$ ./gen_mkfile.py /home/user/cnidaria/data/filelist.csv \
  /home/user/cnidaria/data/test_def.csv \
  /home/user/cnidaria/data/output 21 20

Run Makefile

Go to the output folder ( /home/user/cnidaria/data/output ) and run:

$ make all

Or do a specific run:

$ make test01

the makefile contains:

# Run Cnidaria for each of the pieces
ulimit -c unlimited && time /home/cnidaria/cnidaria/scripts/cnidaria.py --export-complete \
--num-pieces 20 --piece-num 1 --outfile test01 /home/cnidaria/cnidaria/data/input/spp{1,2,3,4,5}.fas.21.jf

# Merge all the pieces
ulimit -c unlimited && time /home/cnidaria/cnidaria/scripts/cnidaria.py --export-complete \
--num-pieces 20 --merge-only --outfile test01 /home/cnidaria/cnidaria/data/input/spp{1,2,3,4,5}.fas.21.jf

# Create statistics of the results
/home/cnidaria/cnidaria/scripts/cnidaria_stats.py test01_0001_0020.json /home/cnidaria/cnidaria/data/filelist.csv
...
/home/cnidaria/cnidaria/scripts/cnidaria_stats.py test01_0020_0020.json /home/cnidaria/cnidaria/data/filelist.csv

# Verify if the merging occured well
/home/cnidaria/cnidaria/scripts/verify_csvs.py test01_00{01,02,...,19,20}_0020.json.csv test01.json.csv

# Merge statistics in single files
cat test*.json.csv               > test01.all.csv
cat test01_*_0020.json.count.csv > test01.all.count.csv

# Convert trees into PNG
/home/cnidaria/cnidaria/scripts/newick_to_png.py test01_00{01,02,...,19,20}_0020.json.no_scale.jaccard_dissimilarity.nj

The output files are:

# Results of individual pieces
test01_0001_0020.cne
test01_0001_0020.cnm
test01_0001_0020.json
test01_0001_0020.json.count.csv
test01_0001_0020.json.csv
test01_0001_0020.json.no_scale.jaccard_dissimilarity.matrix
test01_0001_0020.json.no_scale.jaccard_dissimilarity.nj
test01_0001_0020.json.no_scale.jaccard_dissimilarity.nj.png
test01_0001_0020.json.no_scale.jaccard_dissimilarity.nj.tree

# Global results
test01.all.count.csv
test01.all.csv
test01.cne
test01.cnm
test01.json
test01.json.count.csv
test01.json.csv
test01.json.no_scale.jaccard_dissimilarity.matrix
test01.json.no_scale.jaccard_dissimilarity.nj
test01.json.no_scale.jaccard_dissimilarity.nj.png
test01.json.no_scale.jaccard_dissimilarity.nj.tree

CNE - a Cnidaria Complete Database file containing both k-mer and presence/absence list

JSON - a Cnidaria Summary Database file containing only the matrix of shared/total k-mers counts

MATRIX - a jaccard distance matrix calculated from the JSON file

NJ - a NEWICK neighbour-joining tree based in MATRIX

PNG - a tree image beased in NJ

TREE - an ASCII art tree based in NJ

To create HTML report

$ /home/cnidaria/cnidaria/scripts/report/stats_report.py test01.json \
  /home/cnidaria/cnidaria/data/filelist.csv \
  test01.json.no_scale.jaccard_dissimilarity.nj

which will generate: test01_kmer_stats_report.html