bug: `ImportError: Must have notebook>=5.3 or jupyter_server installed'` when installing `nb_conda_kernels`

[nekomeowww]

Summary

Today, our integrated system reported this issue that we no longer be able to install the nb_conda_kernels:

channels:
    - defaults
    - conda-forge
default_threads: 4
dependencies:
    - python=3.12.3
    - pip
    - ipykernel
    - nb_conda_kernels
name: pip
prefix: /home/neko/.conda/envs/pip

An ModuleNotFoundError will be thrown with the following tracebacks:

Traceback (most recent call last):
  File "/home/neko/.conda/envs/pip/lib/python3.12/site-packages/nb_conda_kernels/install.py", line 17, in <module>
    from jupyter_server.config_manager import BaseJSONConfigManager
ModuleNotFoundError: No module named 'jupyter_server'

During handling of the above exception, another exception occurred:
ERROR conda.core.link:_execute(950): An error occurred while installing package 'defaults::nb_conda_kernels-2.5.2-py312h06a4308_0'.

Traceback (most recent call last):
  File "/home/neko/.conda/envs/pip/lib/python3.12/site-packages/nb_conda_kernels/install.py", line 20, in <module>
    from notebook.config_manager import BaseJSONConfigManager
ModuleNotFoundError: No module named 'notebook'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/neko/.conda/envs/pip/lib/python3.12/site-packages/nb_conda_kernels/install.py", line 22, in <module>
    raise ImportError("Must have notebook>=5.3 or jupyter_server installed")
ImportError: Must have notebook>=5.3 or jupyter_server installed

done
Rolling back transaction: ...working... done

LinkError: post-link script failed for package defaults::nb_conda_kernels-2.5.2-py312h06a4308_0
location of failed script: /home/neko/.conda/envs/pip/bin/.nb_conda_kernels-post-link.sh
==> script messages <==
Traceback (most recent call last):
  File "/home/neko/.conda/envs/pip/lib/python3.12/site-packages/nb_conda_kernels/install.py", line 17, in <module>
    from jupyter_server.config_manager import BaseJSONConfigManager
ModuleNotFoundError: No module named 'jupyter_server'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/neko/.conda/envs/pip/lib/python3.12/site-packages/nb_conda_kernels/install.py", line 20, in <module>
    from notebook.config_manager import BaseJSONConfigManager
ModuleNotFoundError: No module named 'notebook'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/neko/.conda/envs/pip/lib/python3.12/site-packages/nb_conda_kernels/install.py", line 22, in <module>
    raise ImportError("Must have notebook>=5.3 or jupyter_server installed")
ImportError: Must have notebook>=5.3 or jupyter_server installed

I appears that the problem is caused by nb_conda_kernels:

python                 3.12.3  h996f2a0_1       pkgs/main      37MB
nb_conda_kernels        2.5.2  py312h06a4308_0  pkgs/main      45kB

Such issue won't occur if environment is set to a specific version with 2.5.1:

channels:
  - defaults
  - conda-forge
default_threads: 4
dependencies:
  - python=3.12.3
  - pip
  - ipykernel
  - nb_conda_kernels=2.5.1
name: pip
prefix: /home/neko/.conda/envs/pip

Details

Investigations

I checked the diff between 2.5.1 and 2.5.2, but didn't find anything useful.

2.5.1...2.5.2

But I found that

--- nb_conda_kernels-2.5.1-pyh707e725_2/info/info/index.json	2024-04-27 00:49:24
+++ nb_conda_kernels-2.5.2-py312h06a4308_0/info/info/index.json	2024-11-05 23:59:51
@@ -1,20 +1,22 @@
 {
-  "arch": null,
-  "build": "pyh707e725_2",
-  "build_number": 2,
+  "arch": "x86_64",
+  "build": "py312h06a4308_0",
+  "build_number": 0,
+  "constrains": [
+    "notebook >=5.3.0"
+  ],
   "depends": [
-    "__unix",
     "jupyter_client >=4.2",
-    "notebook",
+    "jupyter_core",
     "psutil",
-    "python >=3.8",
-    "requests"
+    "python >=3.12,<3.13.0a0",
+    "traitlets"
   ],
   "license": "BSD-3-Clause",
+  "license_family": "BSD",
   "name": "nb_conda_kernels",
-  "noarch": "python",
-  "platform": null,
-  "subdir": "noarch",
-  "timestamp": 1714150164539,
-  "version": "2.5.1"
+  "platform": "linux",
+  "subdir": "linux-64",
+  "timestamp": 1730822391556,
+  "version": "2.5.2"
 }

from 2.5.1 to 2.5.2:

1. channel switched from conda-forge to pkgs/main.
2. nb_conda_kernels is no longer a noarch build, rather to have variants of different platforms.
3. depended notebook was removed, instead, constrains added: notebook >=5.3.0, this means the dependency of notebook is still present with >=5.3.0 constraint but no depends, even with notebook@7.2.0-pyhd8ed1ab_0 in our base environment (which met the constraints), the notebook dependency will not be installed when a new environment is created.
4. depended jupyter_core and traitlets was added.

Expected behavior

• There is no created and written Release for 2.5.1 and 2.5.2, we don't know such breaking change.
• Even though we could add an extra dependency of notebook when installing, this is still worth to write in Release or README for users and developers to understand what they should do, or potential workaround.
• If this is indeed a breaking change, according to semantic versioning, at lease 2.6.0 should be used as new version (3.0.0 would be recommended if fully made it compilant with semantic versioning).

Proposed fix

Before fixing, there are questions that needs to be answered:

• Should declare the reason of why notebook was removed?
• Does constrains work with depends to install the needed dependencies?

Possible workaround

• Pin nb_conda_kernels at 2.5.1 with nb_conda_kernels=2.5.1 if known issue or dependency constraint should be enforced.
• Add the extra notebook dependency when install if satisfied with other modules & dependencies.

More context

mamba info

          mamba version : 1.5.8
     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/neko/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 24.5.0
    conda-build version : not installed
         python version : 3.11.9.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=icelake
                          __conda=24.5.0=0
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /home/neko/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/neko/.conda/envs
               platform : linux-64
             user-agent : conda/24.5.0 requests/2.32.3 CPython/3.11.9 Linux/5.15.0-122-generic ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
                UID:GID : 1000:0
             netrc file : None
           offline mode : False