SQLite "disk I/O error" when running under SLURM (works on login node)

[isidoramorel]

Hi,

I'm encountering a reproducible issue when running SingleM under SLURM on an HPC cluster.

Summary

• SingleM runs successfully on a login node
• The same command fails under SLURM (including interactive sessions)
• The error occurs during the query-based taxonomy assignment step
• The failure is SQLite-related (OperationalError)

Environment and command

• SingleM version: 0.20.3
• Installed via: conda
• Python version: tested with 3.10 and 3.12
• Running on: SLURM cluster (Linux)

I've been just trying with the test dataset:
singlem pipe -1 GCA_000309865.1_genomic.fna --metapackage S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb -p /dev/stdout
and

export SINGLEM_METAPACKAGE_PATH='/scartch/imorel/S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb'
singlem pipe -1 GCA_000309865.1_genomic.fna -p /dev/stdout

Error

2026/03/31 11:20:51 AM INFO: SingleM v0.20.3
2026/03/31 11:20:51 AM INFO: Acquiring SingleM metapackage from Zenodo backpack directory specified ..
2026/03/31 11:20:51 AM INFO: Retrieval successful. Location of backpack is: S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb
2026/03/31 11:20:51 AM INFO: Loaded 59 SingleM packages
2026/03/31 11:20:51 AM INFO: Using as input 1 different sequence files e.g. GCA_000309865.1_genomic.fna
2026/03/31 11:20:51 AM INFO: Filtering sequence files through DIAMOND blastx
2026/03/31 11:20:53 AM INFO: Finished DIAMOND prefilter phase
2026/03/31 11:20:53 AM INFO: Assigning sequences to SingleM packages with DIAMOND ..
2026/03/31 11:21:20 AM INFO: Running taxonomic assignment ..
2026/03/31 11:21:20 AM INFO: Assigning taxonomy by singlem query ..
2026/03/31 11:21:20 AM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2
2026/03/31 11:21:20 AM INFO: Searching with SMAFA NAIVE by nucleotide sequence ..
2026/03/31 11:21:20 AM INFO: Querying index for S3.1.ribosomal_protein_L2_rplB
2026/03/31 11:21:20 AM INFO: Loading taxonomy cache
2026/03/31 11:21:25 AM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2
2026/03/31 11:21:25 AM INFO: Searching with SMAFA NAIVE by nucleotide sequence ..
2026/03/31 11:21:25 AM INFO: Querying index for S3.2.ribosomal_protein_L3_rplC
Traceback (most recent call last):
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/imorel/miniconda3/envs/singlem/bin/singlem", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/main.py", line 776, in main
    singlem.pipe.SearchPipe().run(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 73, in run
    otu_table_object = self.run_to_otu_table(**kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 473, in run_to_otu_table
    otu_table_object = self.assign_taxonomy_and_process(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 506, in assign_taxonomy_and_process
    assignment_result = self._assign_taxonomy(
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 1238, in _assign_taxonomy
    query_based_assignment_result = PipeTaxonomyAssignerByQuery().assign_taxonomy(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe_taxonomy_assigner_by_query.py", line 122, in assign_taxonomy
    query_single_set(queries, 0)
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe_taxonomy_assigner_by_query.py", line 102, in query_single_set
    for hit in querier.query_with_queries(queries, sdb, max_species_divergence, method, SequenceDatabase.NUCLEOTIDE_TYPE, 1, None, False, None):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/querier.py", line 355, in query_by_sequence_similarity_with_smafa_naive
    m = sdb.sqlalchemy_connection.execute(query).first()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
           ^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[SQL: SELECT markers.id 
FROM markers 
WHERE markers.marker = ?]
[parameters: ('S3.2.ribosomal_protein_L3_rplC',)]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

What I tested

• Running on login node → works
• Running under SLURM (batch and interactive sessions, in /scratch and in /home directories) → fails
• Copying the metapackage to scratch → same error
• Fresh conda environment → same error
• Using --working-directory and --working-directory-dev-shm → same error

Question

Is there a known issue with SQLite usage in SingleM under SLURM or HPC environments?
Or any recommended configuration/workaround for this type of setup?
Happy to provide more details or test suggestions if needed.

Thanks!

[wwood]

Claude's response. It could be onto something?

SingleM SQLite Errors Under SLURM: Troubleshooting Guide

This is a classic HPC issue with SQLite on network/parallel filesystems. The core problem is that SQLite relies on file locking (POSIX fcntl locks), and many HPC parallel filesystems (Lustre, GPFS, BeeGFS) don't implement these locks correctly — or the SLURM job runs on a node where the filesystem is mounted with different options than the login node.

Here are the things to try, roughly in order of likelihood to fix it:

1. Check Your Filesystem Type

Run this from within a SLURM job to see what filesystem the metapackage is on:

stat -f -c %T /path/to/metapackage.smpkg.zb

If it's Lustre or a similar parallel filesystem, that's almost certainly the cause. Login nodes sometimes have different mount options or use a local cache layer that masks the issue.

2. Copy the Metapackage to Node-Local Storage (/tmp or Node-Local SSD)

You mentioned copying to /scratch, but on many HPC systems /scratch is still a shared parallel filesystem. What you want is the node-local temporary storage:

#!/bin/bash
#SBATCH ...

# Copy to node-local storage
cp -r /path/to/metapackage.smpkg.zb /tmp/metapackage.smpkg.zb

singlem pipe -1 GCA_000309865.1_genomic.fna \
  --metapackage /tmp/metapackage.smpkg.zb \
  -p /dev/stdout

rm -rf /tmp/metapackage.smpkg.zb

Your cluster might use $TMPDIR or /local or similar for node-local disk — check with echo $TMPDIR in a SLURM job or ask your sysadmins.

3. Try SQLite's PRAGMA journal_mode Workaround

This isn't something you can set directly with SingleM, but you can test whether SQLite itself is the problem by setting this environment variable before running:

export SQLITE_TMPDIR=/tmp

This ensures SQLite's temporary files go to local disk even if the database itself is on a network filesystem.

4. Check for nolock Mount Options

From a compute node:

mount | grep -i scratch

Look for nolock or flock options — nolock would explain why it works on the login node but not on compute nodes.

5. Verify Basic SQLite Functionality on Compute Nodes

A quick sanity check from within a SLURM job:

srun python3 -c "
import sqlite3, tempfile, os
db = tempfile.NamedTemporaryFile(suffix='.db', dir='/tmp', delete=False)
conn = sqlite3.connect(db.name)
conn.execute('CREATE TABLE test (id INTEGER)')
conn.execute('INSERT INTO test VALUES (1)')
print('Local /tmp:', conn.execute('SELECT * FROM test').fetchone())
conn.close()
os.unlink(db.name)

# Now test on the shared filesystem
conn2 = sqlite3.connect('/scratch/imorel/test.db')
conn2.execute('CREATE TABLE test (id INTEGER)')
conn2.execute('INSERT INTO test VALUES (1)')
print('Scratch:', conn2.execute('SELECT * FROM test').fetchone())
conn2.close()
os.unlink('/scratch/imorel/test.db')
"

If the /tmp test passes but the /scratch test fails, you've confirmed it's a filesystem locking issue.

Summary

The best guess is that option 2 (node-local /tmp or $TMPDIR) will fix it. The metapackage .zb file contains SQLite databases internally, and those need proper file locking support to query. The fact that it gets through the first marker package query and fails on the second is consistent with intermittent locking failures rather than a total inability to read the file.

If the metapackage is too large for /tmp, ask your sysadmins about node-local SSD paths or whether they can enable flock on the scratch filesystem.

[wwood]

@codex attempt fix by opening sqlite3 packages read only -\------------- Ben Woodcroft Group leader, Centre for Microbiome Research, QUT

…

________________________________ From: Isidora Morel Letelier ***@***.***> Sent: Tuesday, March 31, 2026 7:27:29 PM To: wwood/singlem ***@***.***> Cc: Subscribed ***@***.***> Subject: [wwood/singlem] SQLite "disk I/O error" when running under SLURM (works on login node) (Issue #293) [https://avatars.githubusercontent.com/u/57356919?s=20&v=4]isidoramorel created an issue (wwood/singlem#293)<#293> Hi, I'm encountering a reproducible issue when running SingleM under SLURM on an HPC cluster. Summary * SingleM runs successfully on a login node * The same command fails under SLURM (including interactive sessions) * The error occurs during the query-based taxonomy assignment step * The failure is SQLite-related (OperationalError) Environment and command * SingleM version: 0.20.3 * Installed via: conda * Python version: tested with 3.10 and 3.12 * Running on: SLURM cluster (Linux) I've been just trying with the test dataset: singlem pipe -1 GCA_000309865.1_genomic.fna --metapackage S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb -p /dev/stdout and export SINGLEM_METAPACKAGE_PATH='/scartch/imorel/S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb' singlem pipe -1 GCA_000309865.1_genomic.fna -p /dev/stdout Error 2026/03/31 11:20:51 AM INFO: SingleM v0.20.3 2026/03/31 11:20:51 AM INFO: Acquiring SingleM metapackage from Zenodo backpack directory specified .. 2026/03/31 11:20:51 AM INFO: Retrieval successful. Location of backpack is: S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb 2026/03/31 11:20:51 AM INFO: Loaded 59 SingleM packages 2026/03/31 11:20:51 AM INFO: Using as input 1 different sequence files e.g. GCA_000309865.1_genomic.fna 2026/03/31 11:20:51 AM INFO: Filtering sequence files through DIAMOND blastx 2026/03/31 11:20:53 AM INFO: Finished DIAMOND prefilter phase 2026/03/31 11:20:53 AM INFO: Assigning sequences to SingleM packages with DIAMOND .. 2026/03/31 11:21:20 AM INFO: Running taxonomic assignment .. 2026/03/31 11:21:20 AM INFO: Assigning taxonomy by singlem query .. 2026/03/31 11:21:20 AM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2 2026/03/31 11:21:20 AM INFO: Searching with SMAFA NAIVE by nucleotide sequence .. 2026/03/31 11:21:20 AM INFO: Querying index for S3.1.ribosomal_protein_L2_rplB 2026/03/31 11:21:20 AM INFO: Loading taxonomy cache 2026/03/31 11:21:25 AM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2 2026/03/31 11:21:25 AM INFO: Searching with SMAFA NAIVE by nucleotide sequence .. 2026/03/31 11:21:25 AM INFO: Querying index for S3.2.ribosomal_protein_L3_rplC Traceback (most recent call last): File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1967, in _exec_single_context self.dialect.do_execute( File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/default.py", line 952, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: disk I/O error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/imorel/miniconda3/envs/singlem/bin/singlem", line 9, in <module> sys.exit(main()) ^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/main.py", line 776, in main singlem.pipe.SearchPipe().run( File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 73, in run otu_table_object = self.run_to_otu_table(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 473, in run_to_otu_table otu_table_object = self.assign_taxonomy_and_process( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 506, in assign_taxonomy_and_process assignment_result = self._assign_taxonomy( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 1238, in _assign_taxonomy query_based_assignment_result = PipeTaxonomyAssignerByQuery().assign_taxonomy( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe_taxonomy_assigner_by_query.py", line 122, in assign_taxonomy query_single_set(queries, 0) File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe_taxonomy_assigner_by_query.py", line 102, in query_single_set for hit in querier.query_with_queries(queries, sdb, max_species_divergence, method, SequenceDatabase.NUCLEOTIDE_TYPE, 1, None, False, None): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/querier.py", line 355, in query_by_sequence_similarity_with_smafa_naive m = sdb.sqlalchemy_connection.execute(query).first() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1419, in execute return meth( ^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/sql/elements.py", line 527, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1846, in _execute_context return self._exec_single_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1986, in _exec_single_context self._handle_dbapi_exception( File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1967, in _exec_single_context self.dialect.do_execute( File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/default.py", line 952, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error [SQL: SELECT markers.id FROM markers WHERE markers.marker = ?] [parameters: ('S3.2.ribosomal_protein_L3_rplC',)] (Background on this error at: https://sqlalche.me/e/20/e3q8) What I tested * Running on login node → works * Running under SLURM (batch and interactive sessions, in /scratch and in /home directories) → fails * Copying the metapackage to scratch → same error * Fresh conda environment → same error * Using --working-directory and --working-directory-dev-shm → same error Question Is there a known issue with SQLite usage in SingleM under SLURM or HPC environments? Or any recommended configuration/workaround for this type of setup? Happy to provide more details or test suggestions if needed. Thanks! ― Reply to this email directly, view it on GitHub<#293?email_source=notifications&email_token=AAADX5FN4AOHKIS2Q4WR46L4TOFQDA5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBRG43TKNJWHE2DTJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAADX5DX33TOYH6PFTWHNML4TOFQDAVCNFSM6AAAAACXHOEHM2VHI2DSMVQWIX3LMV43ASLTON2WKOZUGE3TONJVGY4TIOI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[chatgpt-codex-connector]

Summary

• Updated SequenceDatabase.acquire() to open otus.sqlite3 using a SQLite URI configured for mode=ro, immutable=1, and uri=true, via a new helper method so packaged DBs are treated as read-only immutable during query-time access (to reduce lock/journal interactions on HPC filesystems). singlem/sequence_database.pyL58-L63 singlem/sequence_database.pyL250-L252
• Updated MetapackageReadNameStore.acquire() to use the same read-only immutable SQLite URI pattern via a dedicated helper, while leaving DB generation/write paths unchanged. singlem/metapackage_read_name_store.pyL18-L25 singlem/metapackage_read_name_store.pyL72-L79
• Added focused tests to assert both acquire paths construct read-only immutable SQLite URIs (mode=ro, immutable=1, uri=true). test/test_sqlite_readonly_mode.pyL5-L29 test/test_sqlite_readonly_mode.pyL31-L51
• Committed changes on the current branch: 038376f with message “Open read-only SQLite databases in immutable mode”.

Testing

• ✅ pixi run -e dev pytest test (result: 225 passed, 24 skipped).

View task →

[wwood]

@isidoramorel does #294 fix it for you? I haven't seen this before so cannot test unfortunately.

[isidoramorel]

@wwood unfortunately no, got a slightly different error though

2026/03/31 03:32:35 PM INFO: SingleM v0.20.3
2026/03/31 03:32:35 PM INFO: Retrieval successful. Location of backpack is: /slurm-databases/SingleM/S5.4.0.GTDB_r226.metapackage_20250331.smpkg.zb
2026/03/31 03:32:35 PM INFO: Loaded 59 SingleM packages
2026/03/31 03:32:35 PM INFO: Using as input 1 different sequence files e.g. GCA_000309865.1_genomic.fna
2026/03/31 03:32:35 PM INFO: Filtering sequence files through DIAMOND blastx
2026/03/31 03:32:38 PM INFO: Finished DIAMOND prefilter phase
2026/03/31 03:32:38 PM INFO: Assigning sequences to SingleM packages with DIAMOND ..
2026/03/31 03:33:08 PM INFO: Running taxonomic assignment ..
2026/03/31 03:33:08 PM INFO: Assigning taxonomy by singlem query ..
2026/03/31 03:33:09 PM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2
2026/03/31 03:33:09 PM INFO: Searching with SMAFA NAIVE by nucleotide sequence ..
2026/03/31 03:33:09 PM INFO: Querying index for S3.1.ribosomal_protein_L2_rplB
2026/03/31 03:33:09 PM INFO: Loading taxonomy cache
2026/03/31 03:33:14 PM INFO: Querying against species database with 1 sequences, using method smafa-naive and max divergence 2
2026/03/31 03:33:14 PM INFO: Searching with SMAFA NAIVE by nucleotide sequence ..
2026/03/31 03:33:14 PM INFO: Querying index for S3.2.ribosomal_protein_L3_rplC
Traceback (most recent call last):
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/imorel/miniconda3/envs/singlem/bin/singlem", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/main.py", line 776, in main
    singlem.pipe.SearchPipe().run(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 73, in run
    otu_table_object = self.run_to_otu_table(**kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 473, in run_to_otu_table
    otu_table_object = self.assign_taxonomy_and_process(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 506, in assign_taxonomy_and_process
    assignment_result = self._assign_taxonomy(
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe.py", line 1238, in _assign_taxonomy
    query_based_assignment_result = PipeTaxonomyAssignerByQuery().assign_taxonomy(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe_taxonomy_assigner_by_query.py", line 122, in assign_taxonomy
    query_single_set(queries, 0)
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/pipe_taxonomy_assigner_by_query.py", line 102, in query_single_set
    for hit in querier.query_with_queries(queries, sdb, max_species_divergence, method, SequenceDatabase.NUCLEOTIDE_TYPE, 1, None, False, None):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/querier.py", line 461, in query_by_sequence_similarity_with_smafa_naive
    for result in future.result():
                  ^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/querier.py", line 449, in process_chunk
    for qres in self.query_result_batch_from_db(sdb, batch_for_db_queries, sequence_type, batch_for_db_hit_indices, 
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/querier.py", line 572, in query_result_batch_from_db
    for row in sdb.sqlalchemy_connection.execute(query2):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
           ^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/imorel/miniconda3/envs/singlem/lib/python3.12/site-packages/singlem/../sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[SQL: SELECT otus.marker_id, otus.sample_name, otus.sequence, otus.num_hits, otus.coverage, otus.taxonomy_id, otus.marker_wise_sequence_id 
FROM otus 
WHERE otus.marker_wise_sequence_id IN (?) AND otus.marker_id = ?]
[parameters: (276, 12)]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

I also tried copying the metapackage to /tmp, but that also didn't work :(

[wwood]

Hi,

Sorry for the slow response here. Is /tmp actually local, or is it bind-mounted to scratch? This is painful to debug.

I have made a new branch nolock where we can test this out. Would you mind cloning that and then runnign this from the base directory. It should take <30s to fast to queue.

singlem$ pixi run singlem pipe -1 test/data/1_pipe/small.fa --otu-table /dev/stdout --debug --metapackage test/data/4.11.22seqs.gpkg.spkg.smpkg/ 2>log

And then report the result on stdout and the log file please?

[l-gallucci]

Hi @wwood, I send an update on this. Me and @isidoramorel are working on the same IT infrastructure, here is the log. Looks, that worked out in that way. But I'm not sure what I need to check.

We all here get the same I/O error.

test.log