Skip to content

Fix subprocess pipe handling and testing environment #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nextgenusfs merged 4 commits into from
May 23, 2026
Merged

Fix subprocess pipe handling and testing environment #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1e1a302
fix: run memory monitoring in separate thread to capture stdout/stderr
May 23, 2026
928fbc1
fix: commit missing test updates and pixi configuration
May 23, 2026
dc48568
build: bump gapmm2 and gfftk to 26.5.22
May 23, 2026
bb4c790
build: bump version to 26.5.22 and python min version to 3.10
May 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest] # Focus on Linux for release testing
python-version: ['3.9', '3.11'] # Test on min and stable versions
python-version: ['3.10', '3.11'] # Test on min and stable versions

steps:
- name: Set up Python ${{ matrix.python-version }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest] # Temporarily disable macOS due to mappy build issues
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
python-version: ['3.10', '3.11', '3.12', '3.13']
include:
# Add one macOS test with conda-based installation
- os: macos-latest
Expand Down Expand Up @@ -54,7 +54,7 @@ jobs:
env:
# Force pytantan to build from sdist so all Python versions get the
# same AVX2-off baseline. The published cp3x wheels have caused
# import-time failures on 3.9/3.10/3.11 in this matrix.
# import-time failures on 3.10/3.11 in this matrix.
PIP_NO_BINARY: pytantan
run: |
if [[ "${{ matrix.use_conda }}" == "true" ]]; then
Expand Down
6 changes: 3 additions & 3 deletions CITATION.cff
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
cff-version: version = "26.4.22"
cff-version: version = "26.5.22"
title: 'funannotate2: eukaryotic genome annotation'
message: >-
If you use this software, please cite it using the
Expand All @@ -17,5 +17,5 @@ keywords:
- functional annotation
- consensus gene models
license: BSD-2-Clause
version: version = "26.4.22"
date-released: '2026-04-22'
version: version = "26.5.22"
date-released: '2026-05-23'
46 changes: 36 additions & 10 deletions funannotate2/utilities.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -774,13 +774,24 @@ def logoutput(logname, process_result):
from .memory import MemoryMonitor

monitor = MemoryMonitor()
memory_stats = monitor.monitor_process(
process, process_name or " ".join(cmd[:2])
)
process.wait()
import threading
def monitor_target():
nonlocal memory_stats
memory_stats = monitor.monitor_process(
process, process_name or " ".join(cmd[:2])
)

t = threading.Thread(target=monitor_target)
t.start()
out, err = process.communicate()
process.stdout = out
process.stderr = err
t.join()
except ImportError:
# Fallback if memory module not available
process.communicate()
out, err = process.communicate()
process.stdout = out
process.stderr = err
else:
process = subprocess.run(
cmd,
Expand All @@ -806,13 +817,28 @@ def logoutput(logname, process_result):
from .memory import MemoryMonitor

monitor = MemoryMonitor()
memory_stats = monitor.monitor_process(
process, process_name or " ".join(cmd[:2])
)
import threading
def monitor_target():
nonlocal memory_stats
memory_stats = monitor.monitor_process(
process, process_name or " ".join(cmd[:2])
)

t = threading.Thread(target=monitor_target)
t.start()
out, err = process.communicate()
process.stdout = out
process.stderr = err
t.join()
except ImportError:
# Fallback if memory module not available
pass
process.communicate()
out, err = process.communicate()
process.stdout = out
process.stderr = err
else:
out, err = process.communicate()
process.stdout = out
process.stderr = err
try:
# Handle return code checking for both subprocess.run and subprocess.Popen
if hasattr(process, "check_returncode"):
Expand Down
3,590 changes: 3,174 additions & 416 deletions pixi.lock
View file
Open in desktop

Large diffs are not rendered by default.

16 changes: 11 additions & 5 deletions pixi.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.1.0"
description = "funannotate2 runtime (Docker/HPC) — assembled with pixi"
authors = ["Jon Palmer <nextgenusfs@gmail.com>"]
channels = ["conda-forge", "bioconda"]
platforms = ["linux-64"]
platforms = ["linux-64", "osx-arm64"]

[dependencies]
python = "3.11.*"
Expand All @@ -18,11 +18,9 @@ snap = "*"
glimmerhmm = "*"
diamond = "*"
trnascan-se = "*"
table2asn = "*"

# funannotate2 python deps available via bioconda / conda-forge
gfftk = ">=26.2.12"
gapmm2 = ">=25.8.12"
gapmm2 = ">=26.5.22"
buscolite = ">=26.4.22"
pyhmmer = ">=0.12.0"
pyfastx = ">=2.0.0"
Expand All @@ -39,10 +37,15 @@ h5py = "*"
tensorflow = ">=2.10,<2.16"

# runtime utilities
tini = "*"
ca-certificates = "*"
pytest = "*"

[target.linux-64.dependencies]
tini = "*"
table2asn = "*"

[pypi-dependencies]
gfftk = ">=26.5.22"
# local funannotate2 source (built from repo)
funannotate2 = { path = ".", editable = false }
# pure-pypi packages
Expand All @@ -52,3 +55,6 @@ annorefine = ">=2026.2.9"
# pytantan: bioconda lags at 0.1.3; pull 0.1.4+ from PyPI
pytantan = ">=0.1.4"


[tasks]
install-mac-table2asn = "curl -O https://ftp.ncbi.nlm.nih.gov/asn1-converters/by_program/table2asn/mac.table2asn.gz && gunzip -f mac.table2asn.gz && mv mac.table2asn $CONDA_PREFIX/bin/table2asn && chmod 755 $CONDA_PREFIX/bin/table2asn"
8 changes: 4 additions & 4 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,20 @@ build-backend = "hatchling.build"

[project]
name = "funannotate2"
version = "26.4.22"
version = "26.5.22"
description = "Funannotate2: eukarytoic genome annotation pipeline"
readme = {file = "README.md", content-type = "text/markdown"}
authors = [
{name = "Jon Palmer", email = "nextgenusfs@gmail.com"}
]
requires-python = ">=3.9.0,<3.14"
requires-python = ">=3.10.0,<3.14"
dependencies = [
"natsort",
"numpy",
"mappy",
"gfftk>=26.2.12",
"gfftk>=26.5.22",
"buscolite>=26.4.22",
"gapmm2>=25.8.12",
"gapmm2>=26.5.22",
"pyhmmer>=0.12.0",
"pyfastx>=2.0.0",
"requests",
Expand Down
Binary file modified tests/data/test_invalid.fasta.fxi
View file
Open in desktop
Binary file not shown.
Binary file modified tests/data/test_valid.fasta.fxi
View file
Open in desktop
Binary file not shown.
82 changes: 78 additions & 4 deletions tests/integration/data/mock_lineage/hmms/busco1.hmm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,81 @@
HMMER3/f [3.1b2 | February 2015]
HMMER3/f [3.4 | Aug 2023]
NAME busco1
ACC PF00001.1
DESC Test BUSCO HMM
LENG 300
LENG 20
ALPH amino
RF no
MM no
CONS yes
CS no
MAP no
DATE Sat May 23 08:09:23 2026
COM [1] generate_hmms.py
NSEQ 1
STATS LOCAL MSV -7.5203 0.80217
STATS LOCAL VITERBI -8.2197 0.80217
STATS LOCAL FORWARD -2.4528 0.80217
HMM A C D E F G H I K L M N P Q R S T V W Y
m->m m->i m->d i->m i->i d->m d->d
COMPO 2.63972 3.73943 2.95908 2.71288 3.15345 2.80504 3.47799 2.92902 2.84823 2.44993 3.70203 3.13380 3.04237 3.19612 2.97300 2.71429 2.95006 2.77821 3.60501 3.24619
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
1 1.27959 4.19958 3.57405 3.03404 3.87264 2.67681 4.42214 3.15850 3.15119 2.66797 4.06769 3.82993 3.35896 3.55827 3.24540 2.37585 2.92798 2.70846 5.43730 4.13926 - a - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
2 2.34737 1.13399 3.68797 3.78387 3.66861 3.42664 4.53606 2.95447 3.58306 2.46394 3.86366 3.94385 3.79083 3.99015 3.67727 2.80772 3.04191 2.82238 4.91532 3.93523 - c - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
3 3.20437 5.17050 1.04751 2.09726 4.20766 3.01183 4.12126 3.81147 3.16826 3.63889 4.72066 2.89315 3.37603 3.25739 3.58042 2.71087 3.26301 3.67938 5.77232 4.47428 - d - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
4 2.94385 5.54588 2.37674 1.20083 4.26509 3.38721 3.86074 3.86890 2.58978 3.37837 4.46014 3.26853 3.43346 2.67892 3.00195 2.76830 3.32043 3.41886 5.51180 4.21376 - e - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
5 3.05258 4.70076 3.75727 3.53522 1.19430 3.49594 3.96947 2.70582 3.65237 2.21529 3.61502 4.01316 4.17809 4.05945 3.74658 3.19498 3.42916 2.89169 4.03077 2.41478 - f - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
6 2.53706 5.13910 3.24176 3.33766 4.17625 0.75477 4.40781 4.09802 3.45481 3.60749 4.68926 3.17969 3.66258 3.86189 3.54902 2.67946 3.54955 3.64798 5.10501 4.44288 - g - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
7 3.14672 5.11285 3.21551 2.67551 3.51410 3.27213 1.20205 3.75382 3.11061 3.26329 4.34506 2.83549 3.63633 3.19974 2.88687 2.97117 3.52331 3.62173 5.07876 2.82688 - h - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
8 2.83697 4.48516 3.85962 3.63757 3.20436 3.91624 4.70772 1.53637 3.75471 1.68174 3.39941 4.11551 3.96249 4.16180 3.84893 3.29732 3.21356 1.72223 5.40492 3.78894 - i - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
9 2.90017 5.18426 3.28692 2.42896 4.22141 3.34354 4.13501 3.82522 1.27430 3.01674 4.09851 3.22485 3.38978 2.95319 2.32237 2.72462 3.27676 3.37519 5.46812 4.17009 - k - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
10 2.84795 4.49614 4.18856 3.64855 3.21534 3.92722 4.71870 2.18325 3.44775 1.05682 3.09244 4.12649 3.97347 3.85483 3.54196 3.30831 3.22454 2.36912 5.09795 3.79992 - l - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
11 2.82260 4.47078 3.84525 3.30524 3.18998 3.58392 4.37539 2.47585 3.10444 1.66736 2.11323 3.78318 3.63016 3.19357 3.19865 2.96500 3.19919 2.34376 4.75465 3.77456 - m - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
12 3.20423 5.17036 2.63712 2.73302 4.20752 2.69374 3.48522 3.81133 2.85017 3.32080 4.40257 1.30325 3.69384 3.25725 2.94438 2.39278 2.94491 3.67924 5.77218 4.15619 - n - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
13 2.85151 5.13559 3.23825 3.01620 4.49070 3.29487 4.40430 3.77656 3.13335 3.28603 4.36780 3.81209 0.79751 3.54043 3.54551 2.99391 3.22810 3.32652 5.73741 4.43937 - p - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
14 2.87233 5.15641 2.94112 2.08317 4.19356 3.31569 3.78922 3.79738 2.51826 2.98890 3.75272 3.19701 3.36194 1.65354 2.61247 2.69678 3.24891 3.34734 5.12232 3.82429 - q - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
15 2.81678 5.10087 3.52148 2.66353 4.13802 3.26015 3.73367 3.74184 2.14477 2.93335 4.01513 3.14146 3.62434 2.86981 1.28512 2.95918 3.19337 3.60975 5.38473 4.08670 - r - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
16 2.27283 4.55692 2.97753 2.75548 3.91202 2.71620 4.14358 3.51584 2.87262 3.02531 4.10708 2.91546 3.39835 3.27971 3.28479 1.46138 2.64942 3.38375 5.47669 4.17865 - s - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
17 2.55499 4.52112 3.25968 3.03763 3.87623 3.31630 4.42573 3.16209 3.15478 2.67156 4.07128 3.19762 3.36255 3.56186 3.24899 2.37944 1.34182 2.71205 5.12294 4.14285 - t - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
18 2.54132 4.50746 3.88192 3.34192 3.54461 3.62059 4.73002 1.87662 3.45906 2.02199 3.42171 4.13781 3.66684 3.86615 3.87123 3.31963 2.91791 1.42658 5.42722 3.81124 - v - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
19 3.36566 4.69589 4.07036 3.53035 2.77919 3.17312 4.28255 3.65481 3.64750 2.84633 3.92810 4.32624 4.17322 3.73664 3.74171 3.50806 3.42430 3.52272 0.84639 2.72787 - w - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.04082 3.91202 3.91202 0.51083 0.91629 0.51083 0.91629
20 3.13147 4.77965 3.83617 3.29616 2.22705 3.57483 3.09451 3.10267 3.41331 2.61214 4.01186 3.77410 3.93903 3.50244 3.50752 3.27387 3.50806 2.97058 3.79171 1.22187 - y - - -
2.54091 4.18910 2.92766 2.70561 3.22625 2.66633 3.77575 2.83006 2.82275 2.33953 3.73926 3.18354 3.03052 3.22984 2.91696 2.68331 2.91750 2.69798 4.47296 3.49288
0.02020 3.91202 * 0.51083 0.91629 0.00000 *
//
Loading
Loading