guidelines for setting up Python environments for use in R/Bioconductor

[HelenaLC]

• basilisk requires minimal specifications (e.g., pip="anndata==1.0.0"), which are prone to breaking due to dependency changes/breaks further up the hierarchy (difference in philosophies in terms of versioning and fixing dependencies?).
• We struggled a lot in OSTA with a comparably simple env setup, and resorted to 1. build env locally using pip, 2. extract requirements from the command line and write to file, and 3. build env using reticulate

# hidden chunk
req <- read.delim("requirements.txt"), header=FALSE)[[1]]
req <- paste0("'", req, "'", collapse=", ")

# copy-paste from above
# (100+ fixed dependency versions)
req <- c( 
    'access==1.1.9', 'affine==2.4.0', 'anndata==0.9.2', 'anndata2ri==1.3.1',
    'attrs==23.1.0', 'backports.zoneinfo==0.2.1', 'beautifulsoup4==4.12.2', ...)
install_python(version=ver <- "3.8.18")
virtualenv_create(envname=env <- "OSTA", python=ver, packages=req)
use_virtualenv(env, required=TRUE)

• Such a requirements.txt file could, in principle, be shipped with an R/Bioconductor package. There might be other (better/cleaner) solutions? I'd be interested in hearing/discussing people's thoughts/ideas on this!
• In general, might be worth to compile some guidelines (for programmatically less inclined users) who'd be interested in running Python methods from R. By contrast, basilisk/reticulate are relatively technical and don't focus on specific use-cases (e.g., here, interop with anndataR, call a Python method, continue in R, etc.).
• Such guidelines could live on the public Bioc domain (e.g., package guidelines), as well as OSCA and/or OSTA eventually, depending on the exact examples/recommendations we deem reasonable.

[mjemons]

renv offers the option to ship a requirements.txt with an renv.lock file and creates then a venv for python. The lockfile then specifies the python requirements and the requirements.txt all python dependencies. In practice when @sgunz and I tried it for our work, it was not an easy endevour to ship this and install it on a new machine. See this and this

In our CI/CD pipeline it works then quite nicely, just on local computers we never got it to work well.

Making this experience better would be really helpful!