Sync APIs corrupt Jupyter kernels on ipykernel >= 7 (nest_asyncio vs anyio-based kernel)

[alexandraabbas]

Summary

All sync APIs that route through inspect_ai._util._async.run_coroutine (read_eval_log, list_eval_logs, the inspect_ai.analysis dataframe builders, Inspect Scout's scan(), ...) corrupt the kernel event loop when called from a Jupyter notebook running ipykernel >= 7. In a notebook, run_coroutine applies nest_asyncio2 and calls asyncio.run() on top of the kernel's running loop. ipykernel 7 (released 2025-10-13, now the default install) rewrote message handling as anyio tasks with per-message contexts (Kernel.shell_main, _async_in_context), and nest_asyncio's re-entrant loop running is incompatible with that.

Note the failure is noisy rather than always fatal — cells often still return correct results beneath the error spew — but the nest_asyncio2.apply() patch is process-global, so a single sync call degrades the kernel for the rest of the session. (This may be why it's gone unreported.)

Symptoms

ERROR Exception in callback Task.__step()
RuntimeError: cannot enter context: <_contextvars.Context object at 0x...> is already entered
ERROR Task was destroyed but it is pending!
  task: <Task pending name='Task-24' coro=<Kernel.shell_main() running at .../ipykernel/kernelbase.py:597>
RuntimeWarning: coroutine 'Kernel.shell_main' was never awaited

Errors recur on later cells, including cells that don't touch inspect_ai.

Repro

inspect-ai 0.3.246, nest-asyncio2 1.7.2, Python 3.12, macOS
ipykernel 7.3.0  -> corrupted (above errors)
ipykernel 6.31.0 -> clean (same code)

1. pip install inspect-ai "ipykernel>=7" jupyterlab
2. In a notebook cell, call any sync log API, e.g. from inspect_ai.log import list_eval_logs; list_eval_logs("logs/") (any log dir).
3. Errors appear in that cell's output and in subsequent cells.

Scriptable repro (no live Jupyter needed): execute a two-cell notebook via nbclient — cell 1 calls a sync API, cell 2 is import time; time.sleep(2) — and grep the outputs for Task was destroyed / cannot enter context.

Found while running transcript analysis in JupyterLab.

[alexandraabbas]

Repro script: Runs the sync APIs inside a real kernel via nbclient — the same path JupyterLab uses — and greps the outputs for the corruption markers. Point it at any directory containing at least one .eval log:

"""Repro: inspect_ai sync APIs vs ipykernel >= 7.

Usage: python repro.py path/to/logs
Requires: pip install inspect-ai nbclient nbformat ipykernel

With ipykernel >= 7 this prints the corruption markers found; with
ipykernel 6.x it prints CLEAN for the identical code.
"""

import json
import sys

import nbformat
from nbclient import NotebookClient

LOG_DIR = sys.argv[1]

MARKERS = (
    "Task was destroyed but it is pending",
    "cannot enter context",
    "Exception in callback",
    "was never awaited",
)

nb = nbformat.v4.new_notebook()
nb.cells = [
    nbformat.v4.new_code_cell(
        "from inspect_ai.log import list_eval_logs, read_eval_log\n"
        f"infos = list_eval_logs({LOG_DIR!r})\n"
        "header = read_eval_log(infos[0].name, header_only=True)\n"
        "print(header.eval.task)\n"
    ),
    # a second cell catches corruption that only surfaces on the
    # kernel's *next* message after the loop was patched
    nbformat.v4.new_code_cell("import time; time.sleep(2); print('ok')"),
]

NotebookClient(nb, timeout=120, kernel_name="python3").execute()

rendered = json.dumps(nb)
found = [m for m in MARKERS if m in rendered]
if found:
    print(f"CORRUPTED - markers in notebook outputs: {found}")
else:
    print("CLEAN")

Observed:

ipykernel 7.3.0  -> CORRUPTED - markers in notebook outputs: ['Task was destroyed but it is pending', 'cannot enter context', ...]
ipykernel 6.31.0 -> CLEAN

(Both cells still return their results in the 7.x case — the corruption shows up in the output streams, and in an interactive kernel it persists across subsequent cells.)

[jstar0]

I'd like to take this issue.

I'll start with the two-cell nbclient reproduction on ipykernel 7 and a
focused regression around run_coroutine. My intended boundary is to keep
notebook sync execution off the kernel's running event loop, so it no longer
needs to globally patch and re-enter that loop, while preserving the existing
non-notebook asyncio/trio and AsyncFilesystem behavior.

I'll keep the change scoped to the async bridge and its tests, and run the
focused utility suite plus a real ipykernel 7 notebook smoke before opening a
PR.