GTseqTools

Program for filtering GTseq genotype data, conducting data quality assessment, and converting to various file formats.

Citation

A manuscript is in preparation that will describe the program. The citation here will be updated once published:

Mussmann, Steven M. 2026. GTseqTools: reproducible and transparent methods for filtering GTseq matrices in prep

For now, please cite this github repository if you use this software.

Python Version Compatibility and Dependencies

This program has been tested with Python = 3.14 but it should be compatible with Python >= 3.8 (Syntax is used that was first introduced to Python in v3.8). Testing and development was conducted with the following versions of Python libraries:

• holoviews = 1.23.1
• matplotlib = 3.10.7
• numpy = 2.3.4
• openpyxl = 3.1.5
• pandas = 2.3.3
• scipy = 1.16.3

Installation

1. Set up a conda environment. This is accomplished by first installing Miniconda. See the basic account configuration instructions in another of my repositories to see how I install custom software. Do steps 1 - 8 under the heading "Basic account configuration, conda installation, etc." These instructions are written for Windows Subsystem for Linux (WSL) but should be translatable to any Unix-based operating system. You do not need to repeat this procedure if you have already done it for another one of my packages.

2. Next, create a conda environment in which this program can be run. Use the following command, which should install a sufficiently recent version of python:

conda create -n GTseqTools -c conda-forge python=3 holoviews matplotlib numpy openpyxl pandas scipy

3. The environment can be activated when needed with the following command. Make sure the GTseqTools environment is active before running the software.

conda activate GTseqTools

4. Next, download this package to your ~/local/src directory with the following commands:

cd ~/local/src
git clone https://github.com/stevemussmann/GTseqTools.git

5. Make sure the software is executable:

cd ~/local/src/GTseqTools
chmod u+x gtSeqConvert.py

6. Finally, link the software in your $PATH.

cd ~/local/bin
ln -s $HOME/local/src/GTseqTools/gtSeqConvert.py

7. You can test if the software works by printing the help menu. If successfull, the full list of command line options should print to the screen:

gtSeqConvert.py --help

Development Notes

• Additional file format conversion options will be considered upon request.
• Please report any bugs encountered via the github 'issues' menu above. I must be able to replicate errors to fix them, so please attach copies of all input files and include the exact command that caused the error.

Order of operations

The program conducts all filtering procedures prior to file format conversion. Filtering is conducted in the following order:

1. Remove user-specified individuals (-r / --removeinds).
2. Remove all individuals not belonging to retained populations (-P / --keeppops).
3. Remove individuals that do not pass IFI score threshold (-I / --ifi).
4. Remove unwanted locus list (-R / --removeloci).
5. Remove species-identification loci (-s / --species).
6. Remove sex-identifying loci (-d / --sexid).
7. Remove loci (-l / --pmissloc) and individuals (-i / --pmissind) that do not meet the minimum thresholds. Default behavior is to apply the locus filter first, but the user can opt to apply the individual filter first if desired (-o / --order individuals).
8. Remove monomorphic loci (-m / --monomorphic).
9. Remove individuals with duplicate genotypes (-D / --dups).

IMPORTANT: Note that since the -m / --monomorphic and -D / --dups filters (steps 8. and 9.) are applied after the missing data filters (step 7.), the final calculations of missing data may show that a few loci or individuals have missing data values that slightly exceed the user-defined thresholds of -l / --pmissloc and -i / --pmissind.

Input Requirements

Required

• The minimal input is either a .csv file or a Microsoft Excel formatted file (.xlsx).
• If using a .xlsx file, all data should be in a worksheet titled 'Final Genotypes'. Any other worksheets in an .xlsx file will be ignored.
• Regardless of input file type the first row will be a header line containing the following:
  • Cell A1 will specify the individual sample column, and must contain the exact text Sample.
  • Cell B1 must contain the exact text Population ID. This column will contain collection group information for all individuals
  • The remaining columns of the file can appear in any order. Columns containing genotype data should be contain locus names.
  • Special columns need to be included for certain file formats (e.g., Colony, SNPPIT, Sequoia, etc.) These columns need to contain specific names in the header line (see explanations below in File Conversion Input Details) and can appear as the columns either before or after the genotype data.
  • You are encouraged to include the IFI score column from the GTseq-Pipeline output, but this is not required.
• Genotype data should be encoded using the output format native to the GTseq Pipeline. In other words:
  • Only characters 'A', 'C', 'G', 'T', and '-' will be recognized as valid alleles.
  • The alleles for a genotype should be concatenated per locus (e.g., genotype = AA, AT, etc.).
  • Indel alleles should be coded as '-' (e.g., genotype = A-, --, etc.).
  • A missing genotype for a locus should be recorded as a single '0' (e.g., genotype = 0).
  • Usage of other characters in the genotype data will likely result in the program throwing error messages.
  • Exceptions to this rule exist for the sex ID loci, which can be coded as sex genotypes (e.g., XX, XY, etc.), but only if the -d / --sexid option is used since this option will strip these loci from the data file before any file conversions take place.
• Your input file should not contain multiple individuals with the same sample name. The default program behavior is to screen for duplicated sample names and exit if any are detected. Alternatively, you can try to force duplicated names to be unique with the -Q / --identquit option.

If you are using the GTscore pipeline for genotyping, I have forked a copy of this repository and included my transposeDataGTscore.pl script which will mostly transform the GTscore genotype outputs to a format compatible with GTseqTools. Just add and populate the 'Population ID' column, do the same for any desired / necessary optional columns, and then save the file in .csv or .xlsx format. Make sure the worksheet is titled 'Final Genotypes' if using .xlsx format.

Optional Inputs

You can provide plain text files with individuals or loci to be stripped from the input file (see -r, and -R options in the Filtering Arguments below). Each of these files should contain a single column of data listing a single individual or locus per line.

GTseq panels often contain species or sex ID loci that you may not want to include in certain analyses. These loci can be easily removed during the filtering process using the -s or -d options, respectively. For either option specify a plain text file containing these locus names. List one locus per line.

You can also add a 'Sex' column to your input .csv/.xlsx file. The column heading must be exactly 'Sex' (no quotes) to be processed properly. This column is intended to hold phenotypic sex data, and will be transferred to the .sexID.xlsx output if you use the -d option.

Program Options

Required Inputs:

• -x / --infile: Specify an input Excel file containing GTseq data.

Required for SNPPIT conversion only:

• -Z / --snppitmap: Specify a tab-delimited map in which the first column lists each population, the second column lists its status as POP or OFFSPRING, and the third column lists the potential parental POP(s) for each OFFSPRING. See example snppitmap in 'example_files' folder.

Filtering Arguments:

• -D / --dups: Turn on filter to screen for individuals with duplicate genotypes.
• -i / --pmissind: Enter the maximum allowable proportion of missing data for an individual sample. Default = 0.2.
• -I / --ifi: Set maximum allowable IFI score to retain an individual (default = 2.5).
• -k / --keepdups: Method for retaining duplicates. 'all' = keep all duplicates; 'first' = keep first encountered; 'second' = keep second; 'none' = keep none (default)
• -l / --pmissloc: Enter the maximum allowable proportion of missing data for a locus. Default = 0.1.
• -m / --monomorphic: Turn on filter to remove monomorphic loci.
• -r / --removeinds: Provide a list of individuals that should be removed from the input xlsx file. This should be a plain text file with each individual being specified on its own line. These individuals will be removed before missing data proportions are calculated.
• -R / --removeloci: Provide a list of loci that should be removed from the input xlsx file. This should be a plain text file with each locus being specified on its own line. These loci will be removed before any other locus-filtering operations are performed.
• -T / --dupthresh: Maximum number of mismatching loci for identifying individuals with duplicate genotypes (default = 3).

Optional Arguments:

• -d / --sexid: Provide a list of loci that are sex-identifying SNPs. This should be a plain text file with one locus per line. These loci will be removed from the dataset before any data filtering steps are executed.
• -o / --order: Specify whether missing data filters will first be applied to loci or individuals (options: 'loci' = loci filtered first (default), 'individuals' = individuals filtered first).
• -P / --keeppops: Provide a list of populations that will be retained in final outputs. All individuals belonging to populations not specified in this file will be filtered. This input should be a plain text file with each population being specified on its own line. Population names must match those in the 'Population ID' column exactly.
• -s / --species: Provide a list of loci that are species identification SNPs. This should be a plain text file with one locus per line. These loci will be removed from the dataset before any other data filtering steps are executed.
• -Q / --identquit: Attempt to force duplicated sample names in your input file to be unique. This will be accomplished by appending suffixes (_1, _2, etc.) to all instances of a name past the first detection (Boolean; default = off).

Colony Format Arguments:

• -e / --droperr: Enter the assumed allelic dropout rate (default = 0.0005).
• -E / --genoerr: Enter the assumed genotyping error rate (default = 0.0005). If the -f / --genoerrfile option is used, the value of -E / --genoerr will override error values < -E / --genoerr.
• -f / --genoerrfile: Specify a list of marker-specific genotyping error rates (optional). This should be in a tab-delmited format, one locus per line. Column 1 = locus name, Column 2 = error rate.
• -F / --pfemale: Enter the assumed probability of mother being among candidate parents (default = 0.5). Value is ignored if no candidate mothers provided in the dataset.
• -L / --runlength: 1/2/3/4 = Short/Medium/Long/VeryLong run (default = 2).
• -M / --pmale: Enter the assumed probability of father being among candidate parents (default = 0.5). Value is ignored if no candidate fathers provided in the dataset.
• -N / --runname: Enter the name for the colony file format (default = 'gtSeqConvert').
• -y / --mpoly: 0/1 = Polygamy/Monogamy for males [default = 0 (polygamy)].
• -Y / --fpoly: 0/1 = Polygamy/Monogamy for females [default = 0 (polygamy)].

Structure Format Arguments:

• -H / --header: Turn off printing of header line with locus names for Structure output
• -t / --twoline: Use this option to write structure files in two-line format. Default = single-line Structure format.

Current supported file conversions (You are required to specify at least one conversion format):

• -a / --allelematch: Prints a file formatted for the allelematch R package
• -b / --binary: Prints a file in binary format (0 = major allele, 1 = minor allele, 2 = missing data).
• -c / --coancestry: Prints a file formatted for coancestry (or 'related' R package)
• -C / --colony: Prints a file formatted for colony.
• -g / --genepop: Prints a file in genepop format.
• -G / --grandma: Prints a file in gRandma format.
• -n / --newhybrids: Prints a file in newhybrids format.
• -p / --plink: Prints a file in plink format. Result is similar to using the --recode12 option in plink. Output should be valid for the program Admixture
• -q / --sequoia: Prints a sequoia formatted genotype file.
• -S / --structure: Prints a file in structure format (default = single line per individual. See '-t' option above).
• -u / --rubias: Prints a mixture file in rubias format.
• -X / --xlsx: Writes an xlsx-formatted file after user-specified individuals are removed (-r option) but before any other filtering steps are applied.
• -z / --snppit: (under development) Prints a file in snppit format (-Z option is also required for snppit conversion as specified above).

Outputs

• Most outputs retain the input file (-x / --infile) base name, but change the output file extension depending upon format. The Colony format is an exception, because the Colony output will always be named colony2.dat.
• Most file conversions result in a single file. Examples of exceptions include Plink and Structure format. The Structure conversion creates a .distructLabels.txt file which contains a list of population numbers and their associated population names. This file can be input into distruct, or used in the CLUMPAK pipeline for visualizing outputs of the program Structure.
• Population maps are also provided for the Coancestry, Genepop, and NewHybrids formats. These provide you with the order of the samples as they appear in the converted genotype files, as well as the population for each individual (pulled from the 'Population ID' column in your input file). The Sequoia option will output the life history file that is required for this program (i.e., sequoia.lh.txt).
• File formats are output with the file extensions in the table below:

Format	File Name(s) / Extension(s)	Program Option
AlleleMatch	.allelematch	-a
Binary	.bin	-b
Coancestry	.coancestry; coancestry.popmap.txt	-c
Colony	colony2.dat	-C
Excel	.xlsx	-X
Genepop	.gen; genepop.popmap.txt	-g
gRandma	.grandma	-G
NewHybrids	.newhyb; newhybrids.popmap.txt	-n
Plink	.ped and .map	-p
rubias	.rubias.csv	-u
Sequoia	.sequoia; sequoia.lh.txt	-q
SNPPIT	.snppit	-z
Structure	.str; .distructLabels.txt	-S

Loci and individuals discarded via filtering options will be written to Excel files. All outputs retain the input file (-x / --infile) base name, but change slightly according to filtering step:

Filtering Step	Name	Program Option
Individuals with duplicate genotypes	.duplicateGenos.xlsx	-D
Missing data proportion for individuals	.filteredIndividuals.xlsx	-i
Missing data proportion for loci	.filteredLoci.xlsx	-l
Monomorphic loci	.monomorphic.xlsx	-m
Discard unwanted populations	.removed.pops.xlsx	-P
IFI score filtering	.removed.ifi.xlsx	-I
List of individuals for removal	.removed.xlsx	-r
List of loci for removal	.removed.loci.xlsx	-R
Sex-identifying loci	.sexID.xlsx	-d
Species-identifying loci	.speciesID.xlsx	-s

Log file

A log file (plain text format) is also created that documents the following:

• The command used to execute gtSeqConvert.py
• Missing data proportions per individual and locus
• The number of individuals/loci removed at each step
• The number of individuals retained from each sample group (observed and expected)
• A chi-square test that evaluates whether missing individuals are evenly distributed among sample groups (only performed if >1 sample group is analyzed) The log file is named using the input file (-x / --infile) base name with the file suffix .log.

Plots

The program produces several plots to help the user assess quality of data both before and after filtering, as well as plots that transparently show the number of individuals and loci removed by each filtering step. All plots can be found in the plots subdirectory that is created when the program is run.

• Sankey plots are produced to show the number of loci and individuals removed by each filter. Labels correspond to the long-form command line option for each filter (e.g., pmissind, monomorphic, etc.)
• Histograms are produced that show the distributions of missing data per locus and individual sample both before and after filtering.
• If the IFI score column is included, histograms will be produced to show the distribution of IFI scores both before and after filtering.
• Two plots are produced if the program is used to identify duplicate genotypes:
  • A histogram showing the distribution of pairwise genotype mismatches among all pairs of individuals.
  • A quantile-quantile (QQ) plot that shows whether mismatch distribution is normally distributed. Duplicate pairs of individuals, if any, should be represented by the isolated cluster of points in the lower left corner of the plot. Duplicated individuals will usually have approximately 0-5 mismatching loci. This plot can be used to help identify an appropriate -T / --dupthresh threshold (default = 3 mismatching loci). See the two examples below:

The y-axis values are useful for identifying a proper threshold. In the left example, all pairs of duplicate individuals had =< 3 mismatching loci. In the right example, all duplicate pairs had =< 5 mismatching loci. There is usually a large gap between the pairs of duplicates and pairs of non-duplicated individuals.

Example Commands

You can print the program help menu using the -h option:

gtSeqConvert.py -h

The command below would convert the data in the Excel format to a pandas dataframe, remove Species-identifying SNPs listed in the 'speciesIdSNPs.txt' file, remove individuals with the proportion of missing data 0.1, then perform a conversion to a Structure format file with data arranged in two lines per individual.

gtSeqConvert.py -x GTseqData.xlsx -i 0.1 -s speciesIdSNPs.txt -S -t

All file conversions occur independently of one another, so the following command would also be valid to write genepop, newhybrids, and structure files in a single command:

gtSeqConvert.py -x GTseqData.xlsx -i 0.1 -s speciesIdSNPs.txt -S -g -n

SNPPIT format requires some extra information to complete the conversion (see explanation below in File Conversion Input Details:

gtSeqConvert.py -x GTseqData.xlsx -z -Z snppitmap.txt

File Conversion Input Details

AlleleMatch

The allelematch file can be read into allelematch with the following R code, substituting "filename.allelematch" for your actual file name:

library("allelematch")

data <- read.table("filename.allelematch", header=TRUE, sep=",")
amData <- amDataset(data, missingCode="-99", indexColumn=1, metaDataColumn=2)

Colony

Add a column to the file titled exactly colony2. In this column, identify all potential offspring as offspring and all candidate parents by their sex (male or female). The terms offspring, male, and female are all case-insensitive.

You can also specify locus-specific genotyping error rates with the -f / --genoerrfile command line option. The input file for this option should be a tab-delimited plain text file. One locus per line. Column 1 = locus name, column 2 = error rate. See the example file markerErrorRates.txt in the example_files directory of this repository. All error rates in this file that are < -E / genoerr will be overwritten with the value of -E / genoerr. The purpose of this is primarily to remove any zero values, since it is unlikely that any locus actually has a genotyping error rate = 0. All error rates will also be rounded to four decimal places (e.g., 0.0001).

gRandma

A special filter is applied to the gRandma-formatted output to retain only biallelic SNPs for which each of the following conditions is met by at least one individual in the data file:

1. Homozygous for allele 1
2. Homozygous for allele 2
3. Heterozygous for alleles 1 and 2

The output file can be read into gRandma with the following command:

library("gRandma")

genotypes <- read.csv("output.grandma.txt", sep="\t", header=TRUE, na.strings="")

NewHybrids

The NewHybrids conversion allows for optional use of its 'z' option to specify known genotypes. To use this option, add an extra column to your input .xlsx file titled exactly ZOPT. The naming of the column is important so that it will be ignored in conversions for other file formats.

Fill the column with data to designate individuals belonging to the different classes (i.e., z0 for Pure_0, z1 for Pure_1). If you do not want to provide a 'z' designation for a sample then leave that cell empty and it will be ignored. Any data in the ZOPT column will be transferred to your converted file exactly as it appears in your input .xlsx file, so it is important to only enter information that will be valid in a NewHybrids input file.

rubias

The rubias-formatted file can be read into rubias with the following commands:

library("rubias")
library("tidyverse")

data <- read_csv("exampleData.rubias.csv", col_types = cols(.default="c"), na = c("NA"))

Sequoia

The Sequoia conversion relies upon some of the optional SNPPIT columns that are also used for the SNPPIT file conversion (see below). Use the POPCOLUMN_SEX column to specify sex data for all individuals. Only case insensitive versions of f, female, m, and male will be recognized. All other values and blank cells will be converted to unknown sex data value in sequoia (3).

The OFFSPRINGCOLUMN_BORN_YEAR is used to specify the birth year for all individuals. You can enter birth year data in this column even for the 'parental' populations. This will not cause any problems for the SNPPIT file conversion as listed below.

Files can be read into sequoia with the following commands:

library("sequoia")

# genotypes file
geno <- as.matrix(read.csv("filename.sequoia", sep="\t", header=FALSE, row.names=1))

# life history file
lh <- read.csv("sequoia.LH.txt", sep="\t", header=TRUE)

SNPPIT

The SNPPIT conversion has a few special requirements that are not needed for other file formats. Firstly, a special tab-delimited snppit map file is required as supplemental input. An example of the snppit map file is included in the 'example_files' folder. Essentially, each line of this file is intended to contain all of the information of lines starting with the POP and OFFSPRING keywords, as seen on pages 22-23 of the SNPPIT program documentation. However, note that the columns of the snppitmap are in a different order than they appear in the final snppit-formatted file (i.e., 'popname'\tab'POP' rather than 'POP'\tab'popname').

Secondly, the user can utilize SNPPIT's 'optional' columns as seen on pages 24-26 of the SNPPIT program documentation by including the relevant data in their input .xlsx file. To do this, add columns to your input .xlsx file with headings that exactly match the optional columns used by SNPPIT. For example, if you want to use the POPCOLUMN_SEX option in SNPPIT, then include a column named exactly POPCOLUMN_SEX in the 'Final Genotypes' worksheet of your input .xlsx file, and fill this column with the appropriate values for each individual, as necessary.

Generally, the values you input in these optional columns should exactly match the values as they would appear in the final SNPPIT file. Additional details are as follows:

• Values in the POPCOLUMN_SEX column are somewhat flexible. Case-insensitive versions of 'f' and 'female' or 'm' and 'male' will be converted to 'F' and 'M' respectively. Blank cells and any other values entered in this column will be converted to missing data ('?').
• Values in columns containing year data (POPCOLUMN_REPRO_YEARS, OFFSPRINGCOLUMN_BORN_YEAR, OFFSPRINGCOLUMN_SAMPLE_YEAR) must be valid four-digit integers.
• Currently there are no data validation measures implemented for the other two optional columns (POPCOLUMN_SPAWN_GROUP and OFFSPRINGCOLUMN_AGE_AT_SAMPLING) so please make sure anything you enter in these columns is exactly as you want it to appear in the final file.
• Unnecessary data can be left as blank cells. An example of this is that you would not need to enter sex data for OFFSPRING groups in the POPCOLUMN_SEX column.