Skip to content

migrate irods pyfilesystem2 filesource to fsspec backed filesource#23029

Open
PlushZ wants to merge 4 commits into
galaxyproject:devfrom
PlushZ:migrate-irods-fs-to-fsspec
Open

migrate irods pyfilesystem2 filesource to fsspec backed filesource#23029
PlushZ wants to merge 4 commits into
galaxyproject:devfrom
PlushZ:migrate-irods-fs-to-fsspec

Conversation

@PlushZ

@PlushZ PlushZ commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

This PR migrates iRODs pyFileSystem2 outdated filesource to fsspec backed with https://gitlab.kuleuven.be/setit/rdm/mango-fsspec
Configuration of file source is not changed.
https://gitlab.kuleuven.be/setit/rdm/mango-fsspec library should be published before merging and changed in conditional requirements.txt

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. Run local iRODS demo server from https://github.com/irods/irods_demo. Start demo server with Docker Compose using the irods-catalog-provider and irods-client-icommands services. Expose the iRODS provider port to the host with a docker-compose.override.yml:
services:
  irods-catalog-provider:
    ports:
      - "1247:1247"
  1. Configure local Galaxy instance, use lib/galaxy/files/templates/examples/production_irods.yml as a template in config/file_source_templates.yml
  2. Run Galaxy instance, configure Galaxy file source in User Preferences/Manage Your Repositories/Create/iRODS using the demo credentials:
  type: irods
  host: localhost
  port: 1247
  username: rods
  password: rods
  zone: tempZone
  root: /tempZone/home/rods
  writable: true
  1. Verifiy through the Galaxy UI that the file source can:
  • connect to the iRODS demo server
  • browse the configured root collection
  • create directories
  • upload files into iRODS
  • list uploaded files and directories
  • import files from the iRODS file source into Galaxy
  1. Verifiy the same files from the iRODS side using iCommands inside the demo client container:
docker compose exec irods-client-icommands ils -r /tempZone/home/rods
docker compose exec irods-client-icommands ils -L /tempZone/home/rods/<path>
docker compose exec irods-client-icommands iget -f /tempZone/home/rods/<path>/<file> /tmp/<file>

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@PlushZ

PlushZ commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Ping @hechth

@mvdbeek

mvdbeek commented Jul 7, 2026

Copy link
Copy Markdown
Member

Looks good to me, can you update the PR @PlushZ ?

@PlushZ
PlushZ force-pushed the migrate-irods-fs-to-fsspec branch from 4850481 to 422b402 Compare July 9, 2026 16:36
@mvdbeek

mvdbeek commented Jul 14, 2026

Copy link
Copy Markdown
Member

can you run make format ?

@hechth

hechth commented Jul 20, 2026

Copy link
Copy Markdown

for the time being, we're still testing our implementation of the fsspec here: https://github.com/RECETOX/fs-irods/tree/fsspec

omero-py #type: omero
iiif-fsspec # type: iiif
fs-irods # type: irods, depends on python-irodsclient
mangofs @ git+https://gitlab.kuleuven.be/setit/rdm/mango-fsspec.git@1.1.0 # type: irods, depends on python-irodsclient

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want this resolved before merging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, waiting for the package being tested and published.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Needs Review

Development

Successfully merging this pull request may close these issues.

3 participants