Repository navigation
Conversation
…contrast test Tokens gains the colours the chrome was hard-coding, each with a light and a dark value: on_accent (text on an accent fill), an eight-colour category palette (theme::Cat), the paper colour inside icons, reference colours for the formula bar and for the sheet, the message box warning disc, the tab bar scroll thumb, and sheet_control_* colours that follow the sheet and so stay light in both themes. What now comes from the tokens: - icons: paper (white pages glared in dark mode) and the accent constants; icons placed on the sheet use icons::paint_on_sheet, which keeps the light colours; - reference colours (editor::REF_COLORS), which were hard to read on the dark formula field; - white text on the accent in OK, Solve, slicer tiles and the AutoSave switch, 2:1 on the dark accent; - the comments pane (a white card with light text in dark mode), gallery hover and swatch outlines, the message box disc, the tab bar thumb, chart selection frames, and the sheet's filter and validation buttons. A unit test (theme::contrast_tests) checks every token pair used by the ribbon, popups, dialogs, panes, formula bar, status bar and sheet in both themes against WCAG 2: 4.5:1 for text and 3:1 for glyphs. It caught the dark danger red (3.5:1 on hover) and the selected header text (4.1:1 light), which are adjusted.
- widgets::big_split_button: the icon runs the default action, the
label and its arrow open the menu. It returns both responses, with
accessible names ("AutoSum" and "AutoSum options"). big_button shares
its label layout.
- Formulas › AutoSum uses it: the icon inserts SUM as before, and the
arrow, which used to do the same, opens the Sum / Average / Count /
Max / Min / More Functions menu that Home › AutoSum already had
(shared as autosum_menu). Its keytip is unchanged.
- When a tab's groups are wider than the window, arrows at the card's
edges scroll them; before, only the mouse wheel did and nothing showed
that the tab went on. ribbon::RibbonOverflow records the groups' and
the card's sizes each frame.
- A test checks every tab fits the card's three rows. It found View ›
Show stacking four checkboxes, with Headings cut off; Show is now two
columns (Ruler, Gridlines / Formula Bar, Headings).
Tests: tests/ribbon_layout.rs (the split button's two parts, AutoSum's
icon and menu, the scroll arrows, the card height).
The formula bar is a top-aligned row: the Name Box and the cancel, enter and fx buttons stay on the first line while the field grows. Its height is the content's laid-out line count (Alt+Enter or wrapped text) times the font's row height, from one line (30 px, as before) to five, then the field scrolls with the caret in view. This holds while editing and while viewing a cell. The height eases over 120 ms when the line count changes, and the chevron pins it at five lines (it used to give a fixed 72 px). The in-cell editor likewise stops at five lines and scrolls. formula_bar::BarLayout records where the parts went, for tests. widgets::code_editor(ui, id, &mut text, Lang, max_lines) is a small code editor for features that edit code: the monospace font, syntax colours from Tokens::syntax (new, with light and dark values in the contrast test) for Python (keywords, strings including triple-quoted ones, numbers, comments and builtins) and LaTeX (commands, scripts and comments), Tab inserting four spaces (or indenting every selected line, Shift+Tab dedenting), Enter keeping the line's indentation (one level more after a Python colon), and a scroll area that follows the caret. The tokenizer and the edits are pure functions with tests, including hostile input that must keep spans on char boundaries. kittest (tests/formula_bar.rs) covers Alt+Enter up to eight lines with the Name Box and buttons fixed and the caret visible, the commit, viewing the committed cell, the chevron, the ease, the in-cell cap and the code editor's Tab and Enter.
Session::grouped(label, f) runs f as one undo step on the active document. On success the group leaves exactly one undo entry holding the workbook from before the group; on error it restores the workbook, selection, undo and redo stacks as they were and returns the error. group_begin, group_end and group_rollback do the same across separate calls, for an action that runs one step per call. While a group is open on a document, undoable commands on it add no undo steps of their own, so a long group neither grows the history nor pushes older entries past the undo limit, and Undo and Redo are disabled there. Groups nest: the outermost one makes the step, and a failing inner group rolls back only its own changes. Closing a document drops its groups, and a command that panics drops the groups it opened. history.list reports whether a group is open. Insert Copied Cells folds its nested steps into one undo step by counting revisions; inside a group the nested steps push no entries, so it now leaves the history alone there instead of removing entries from before the group.
A macro recorder needs more than a journal index: the journal is trimmed past 10,000 entries, it also holds the commands that other commands run (so playback would run them twice), and it keeps no selection, which relative references need. Session now tracks how deeply execute calls are nested. While recording, a buffer (record.rs) keeps each successful, journaled top-level call with the document uid, the active sheet (index and name) and the selection it ran in, then its id and parameters without secrets, as the journal stores them. Queries, failed commands and the command that starts recording are not kept. The buffer holds up to 100,000 steps and counts any it drops. Session::start_recording, stop_recording, is_recording and depth are the interface. Automate › Record Actions keeps recording through the journal as before.
Active content such as macros must not run only because a workbook was opened. DocState gets a trust state: Local for workbooks made in the session (new workbooks and samples) and FromFile for anything opened with file.open. Active content in a FromFile workbook asks before it runs. A second window on a workbook shares its trust. The clipboard carries the trust of its source document. Pasting formulas, formats (conditional formats hold formulas) or validation from a FromFile workbook into another workbook makes that workbook FromFile too; pasting values, column widths or notes carries nothing active and changes nothing. Sheet moves and copies stay inside one workbook, so its trust is unchanged. document.inspect reports the trust.
GridCraft keeps workbook data in hidden defined names so it travels inside the xlsx file. The existing items write one name holding one string literal, which past Excel's 255-character literal and 8,192-character formula limits gives a file Excel does not accept. cmd/stash.rs stores an item `key` as hidden workbook-scope names _gc_<key>_0, _gc_<key>_1, ..., each a formula of string literals of at most 255 characters (UTF-16 units, as Excel counts them) joined with &, at most 8,192 characters per name. Reading concatenates the chunks in order and stops at a gap; it also accepts the older single _sc_<key> name. Names are matched case-insensitively and only at workbook scope. Keys are limited to letters, digits, _ and ., items to 4,194,304 characters, and junk around or inside the literals, duplicate or out-of-order chunks, non-JSON and over-deep JSON all read safely. keys(prefix) lists the stored items that start with a prefix. Custom views, scripts, the watch list and scenarios keep their current storage; nothing reads or writes the new names yet. docs/architecture.md describes the engine building blocks added in this series: undo groups, the recording buffer, document trust and this storage.
…its JSON-lines protocol
Python code for cells runs in Monty 1.1.0 (MIT), a Python interpreter
written in Rust for sandboxed code, inside a separate worker process:
the app binary started again with --python-worker. Code such as
'x' * 10**12 makes an interpreter allocate until the process aborts, and
10**10**8 never checks the interpreter's time limit; catch_unwind can
stop neither, so only a separate process that the app can kill is a
reliable harness. RustPython was considered first, but every recent
release depends on the LGPL-3.0 malachite crates, which would be linked
into the app binary.
The new crate gridcraft-python (layer 1, on core) holds:
- protocol: one JSON object per line. The parent sends Init {limits,
init_code}, Run {id, code}, XlValue {id, reply} and Reset; the worker
answers Ready {init_error}, Xl {id, reference, headers} and Done {id,
output, stdout, stderr, error {line, message}, elapsed_ms}. Worker
frames start with a marker, so other text that reaches its standard
output is never taken for a frame. Values travel as
gridcraft_core::Value.
- worker (feature `python`, not on wasm32): one Monty REPL whose
namespace is shared by every cell, with the initialization code run at
start and on Reset. print output is collected per run (1 MB, then cut
off). xl() asks the parent and waits for the reply; file, environment
and network access are refused with PermissionError. Limits: the time
limit, recursion depth 200, 10,000 xl() calls per run and 1,000,000
output cells. A closed input pipe ends the worker even while code
runs.
- conversions: None to empty, bool, int and float to numbers (NaN,
infinities and integers too large for a double give #NUM!), str to
text, a list to a column, a list of lists to a table padded with #N/A,
a dict to a key/value table, and any other object to its repr. xl()
gives a scalar for one cell, a list of rows for a range, and a list of
dicts keyed by the first row with headers=True.
- client: PyWorker starts the worker and reads its frames on a thread,
so poll() never blocks longer than asked. A run past its time limit
plus one second is killed (#TIMEOUT!), a watchdog reads the worker's
resident set size every 50 ms and kills it above 512 MB, and a worker
that exits fails its runs; the next run starts a fresh worker. An
in-process transport runs the worker on a thread for tests.
apps/gridcraft and apps/gridcraft-cli check for --python-worker first
thing in main, before the logger, panic hook or window.
Tests: protocol round trips, conversions in both directions, error
lines, refused host access and the output cap; in-process runs with a
shared namespace, xl(), initialization code, reset and initialization
errors; and, on a real process, `while True: pass` and 10**10**9 time
out, 'x' * 10**12 is stopped by the memory watchdog, a killed worker is
replaced, and the host keeps running throughout.
…! errors
Cell gains `py: Option<Arc<PyCode>>`, read with a serde default and not
written for ordinary cells, so older native files load and ordinary
cells serialize as before. PyCode holds the code, the output mode
(Python object or cell values), the last result (value, type name,
preview, whether it converted to cells) and a stale flag for manual
calculation. The cell's `value` stays the cached result, so formulas,
display and saving read a Python cell like any other. is_blank,
CellStore::has and input_text (which returns the code) take the code
into account; Cell::has_content and clear_content cover value, formula
and code together. The model also reads and writes the file form,
PY("chunk"&"chunk",0|1) with string literals of at most 255 characters,
and finds the xl("...") literals in code, skipping comments and other
strings. Code is limited to 7,000 characters so that the formula stays
under the 8,192-character formula limit.
Edits: cell.set over a Python cell, Clear Contents, Clear All and the
cells a merge empties remove the code; Clear Formats keeps it. Copy and
paste (All, Formulas), Fill, sort, cut-move, sheet copy and undo carry
it unchanged; Paste Values and Paste Link do not, and paste operations
do not combine numbers with a Python cell. Fill copies Python cells
instead of extending them as a series. cell.set with =PY(...) is
refused with a message that points to python.set. The sheet-protection
check compares the code, so a locked Python cell cannot be changed.
cell.get reports {python: code, output, typeName, preview, stale}.
The xl() literals follow reference-moving edits: rewrite_all_formulas
(row and column insert and delete, cell insert and delete, cut-move),
sheet rename and sheet delete adjust each literal like a formula
reference, and replace it only when it changed.
Errors: CellError gains Python, Busy, Timeout and Blocked, and ALL has
14 entries, so the lexer reads them as literals. ERROR.TYPE gives 11 for
#BLOCKED!, the number Excel returns. Microsoft documents no ERROR.TYPE
number for #PYTHON!, #BUSY! or #TIMEOUT!; they take 19, 20 and 21, after
the numbers Excel is known to use. The xlsx writer caches the four as
#VALUE!, as it already does #SPILL! and #CALC!, because they are not
error codes of the file format.
Tests cover the =PY refusal, removal by entry and clearing, undo, copy,
paste values, fill and sort, xl() rewrites through row insert, sheet
rename, cut-move and column delete, native JSON round trips, and the
new errors.
…rker per workbook After every top-level command that changed a workbook, Session::execute hands the workbook before and after to the Python runtime (engine/src/python.rs), which finds the cells to run: Python cells whose code changed, whose xl() literals read cells that changed (cell store and spill differences, sheet by sheet; references resolve like a formula's, so names, tables and Sheet! references work), or whose literal names something only known at run time; every Python cell after the first of those, in row-major order sheet by sheet, because they share one namespace; all of them when a call is xl(<non-literal>) or a sheet with Python cells disappeared; and the cells after a removed Python cell. Python cell positions are cached per sheet version and patched from the edit's band differences, so a workbook without Python pays only for the bands an edit touched. Nothing runs when a workbook is opened. Each cell runs once per refresh: run, write the value (an array spills through calc::lay_python_spill, or shows #SPILL! when blocked), let calc recalculate its formula dependents, then the next cell. Results go straight into the workbook and are not undoable, as for data tables: one undo takes back the edit and the results it caused. A Python cell fed, through formulas or directly, by a later Python cell in the same refresh gets #CALC!. xl() is answered from the current workbook and refused for non-references, missing sheets and names, deleted (#REF!) cells and ranges over the output cap. Calc keeps a Python cell's spill: cells_changed no longer drops it as it does a replaced formula's, recalc_all lays Python spills out again from the kept results, and after each edit the runtime lays out again the spills of Python cells showing #SPILL! or whose area the edit touched, so clearing a blocker spills again without running the code. Runs happen inside the command for headless callers (CLI, MCP, tests). With set_python_background(true) a host steps them with step_python(); waiting cells show #BUSY!, and an edit, undo or reset during a refresh restarts it from the earlier of its next cell and the new first cell, discarding the result in flight. Undo, redo and the panic restore mark the namespace dirty: the next refresh resets it and runs every cell. A failure that took the worker down (time limit, memory, crash) starts a fresh worker, which runs the earlier cells again once before the queue continues. In manual and automatic-except-tables calculation, affected cells are only marked stale until Calculate Now or Calculate Sheet runs them. Workbooks opened from a file (Trust::FromFile) run nothing, and new or edited code there shows #BLOCKED!. Without a worker launcher (the web build, or a host that set none) cells show #BLOCKED!. Limits: Prefs.python_timeout_seconds (5 s) per cell, 512 MB per worker, recursion depth 200, 1 MB of output kept, 1,000,000 output cells. Every run is recorded in DocState.py_log, a ring buffer of 500 entries with the cell, status, duration, output and error line. gridcraft-python gains run.rs (the launcher, run outcomes, and the default launcher that apps set at start-up) and a web stand-in for the client with the same API, so the engine needs no cfg of its own. Both apps set the default launcher to their own binary. Tests: order and the shared namespace, xl() dependencies with formula dependents, the Python to formula to Python chain, the cycle error, spills blocked, unblocked and kept through a structural edit, errors with their line and output, time limits, ranges, names and other sheets, the trust gate, manual calculation, undo resetting the namespace, background runs with #BUSY! and restart, and a real worker process killed mid-refresh with the earlier cell run again.
…om files
cmd/python.rs registers the Python commands, all reachable from the
control channel, the CLI and MCP:
- python.insert {cell?} (Formulas > Python, Ctrl+Alt+Shift+P) makes the
cell a Python cell with no code yet;
- python.set {cell?, code, output?} sets the code (at most 7,000
characters) and runs it, answering with the cell's value; the same
code again keeps its result without running;
- python.clear {cell?} (Convert to Normal Cell) turns Python off,
clearing the code and value, undoably; the cells after it run again;
- python.output {cell?, mode: object|value} shows the result as a
Python object (its type name, with no spill) or converts it to values;
- python.reset {clearDiagnostics?} starts Python afresh and runs every
Python cell again;
- python.diagnostics {sheet?, errorsOnly?} returns the last 500 runs;
- python.initialization {code?} returns, or sets and then runs again
with, the workbook's initialization code, kept in the hidden workbook
data item py_init;
- python.options {timeoutSeconds?} reads or sets the per-cell time
limit (0.1 to 3600 s, default 5);
- python.enable lets Python run in a workbook opened from a file
(FromFile becomes Trusted) and runs its Python cells.
None of them opens a dialog. With empty parameters they act on the
active cell or report state, or return an error.
Trust gate: a workbook opened from a file runs no Python; code set
there shows #BLOCKED!, and python.reset does not change that;
python.enable does, by making the workbook Trusted, a new Trust state
for a file whose active content the person allowed to run.
document.inspect reports it as `trusted`. A locked cell on a protected sheet refuses
python.set through the existing protection check, which compares
Python code.
python.set, insert, output and clear refresh inside the command so they
can report the result, and tell Session::execute that the refresh is
done (only as the top-level command; inside another command, that
command's refresh covers them).
Tests: set, rerun avoidance, clear and undo, insert, output modes, the
code limit, parameter errors, protection, reset and diagnostics
filters, initialization code (first, on change, its errors), the trust
gate and enabling, a missing worker, the time-limit setting, and a CLI
test that drives python.set through gridcraft-cli with Python in a real
worker process: `while True: pass` gives #TIMEOUT!, 'x' * 10**12 gives
#PYTHON!, and the next cell still runs.
…s for Python content
A Python cell is written to xlsx as an ordinary formula,
PY("chunk"&"chunk",0|1), with string literals of at most 255
characters, 1 for a Python object result and 0 for values, and no
_xlfn. prefix, together with its cached value. A spilling result is
written like a dynamic array (cm="1", t="array" ref=...) with the
spilled values cached, so other spreadsheet applications show the
values (and #NAME? if they recalculate). The reader turns a formula
that is PY( followed by string literals, and optionally ,0 or ,1, back
into a Python cell, reading the formula's own text so that line breaks
in the code survive, and keeps the cached spilled values as the cell's
result, so a reopened workbook shows its arrays without running
anything. Python errors are cached as #VALUE!, as #SPILL! and #CALC!
already are. The initialization code travels in the hidden workbook
data names. Native JSON keeps Cell.py whole: code, output mode, the
last result and the exact error.
Opening any file adopts PY(...) formulas from every format
(io::adopt_python) and then applies limits to Python content
(io::enforce_python_limits): a cell with more than 7,000 characters of
code keeps only its cached value, with a warning, and a kept result
that is not a well-formed array of at most 1,000,000 cells is dropped.
calc::lay_python_spill refuses such an array too (#SPILL!), so a
crafted file cannot make it loop over a claimed size. Opening a file
still runs nothing: the workbook is FromFile, shows its cached values,
and runs Python only after python.enable.
python.set and python.insert refuse a table's header row, which holds
the column's name, as Excel keeps formulas out of headers.
Catalog: Formulas > Python gains Insert Python, Reset, Diagnostics and
Initialization; `cargo xtask parity` gives 278 / 294 (94.6%).
Tests: xlsx round trip of values, a formula over a Python cell, a
spilling table, an object-mode cell, 7,000 characters of code with
quotes and line breaks, an error and initialization code, checking the
written PY() formulas; the native JSON round trip; hostile content (code
over the limit in JSON and xlsx, a result claiming far more cells than
it holds); and a malformed result that never spills.
When the person has Python 3.11 or newer, Python cells run on it, so
pandas and numpy work; otherwise they run on the built-in Monty worker.
GridCraft never installs an interpreter or packages.
Finding the interpreter (gridcraft-python `interp`): the search order
is the path from python.options (a bare name is looked up on PATH),
VIRTUAL_ENV, CONDA_PREFIX, python3 then python on PATH, and on Windows
the `py -3` launcher and the usual install folders, newest first. The
Microsoft Store shortcuts under %LOCALAPPDATA%\Microsoft\WindowsApps
are skipped with a reason, since they open the Store instead of running
Python. Each candidate is probed under a 5 s deadline: first with -I
for its version (anything older than 3.11 is rejected with the version
it reported), then with -P -X utf8 for sys.executable, the pandas and
numpy versions (from package metadata, without importing pandas) and
whether it is externally managed (PEP 668). Cells run with
sys.executable, so a virtual environment and the program behind
`py -3` are used directly. The search takes a Host trait, so tests run
it against fake machines, including a fake Windows one. It runs on its
own thread and never on the web build. The engine caches the result per
session and searches again when the interpreter option changes, on
python.reset and on python.options {detect: true}. Apps opt in with
interp::enable_search(); tests and library users keep the built-in
Python or the launcher they set. Prefs.python_interpreter keeps the
path.
Running on it: the parent's process handling is unchanged (the deadline
kill, the memory watchdog, the restart after a crash and the JSON-lines
protocol, including the xl() round trip). Only the child differs:
`<python> -P -X utf8 -u -c <shim>` instead of the app binary with
--python-worker. -P keeps the current folder off sys.path; -I and -s
are not used because they hide user site-packages, where
`pip install --user` puts pandas.
- crates/python/src/shim.py, embedded with include_str!, is the worker
for CPython. It is data that the user's interpreter runs, not app
code. It keeps frames on a private copy of standard output and points
file descriptor 1 at standard error, so output from C code or child
processes cannot break a frame; reads input on a thread and exits
when GridCraft goes away; collects print output per run up to the
limit; stops a run at the time limit with a timer and interrupt_main
(the parent's kill covers C calls that cannot be interrupted), and
installs the SIGINT handler itself, since a parent started with SIGINT
ignored passes that on; and returns the last expression's value.
SystemExit is a cell error, not an exit.
- Conversions: xl() gives a pandas DataFrame when pandas is installed
(imported on first use; the first row is the header with
headers=True) and lists otherwise, as the built-in Python does.
Results: DataFrame (with the index only when it is not the default
RangeIndex), Series, one- and two-dimensional ndarrays, lists and
dicts spill; numpy scalars become numbers or booleans; datetime, date
and Timestamp become date serials; NaN and infinities become #NUM!;
None, NaT and NA become empty; anything else is an object with its
repr. The limits on output cells and xl() calls apply.
- The recursion limit on CPython is at least 1000, since pandas calls
deeper than the built-in Python's 200 allows.
- Workers start in a process group of their own (a job object on
Windows) through process-wrap (MIT OR Apache-2.0), so a kill also
stops processes the user's code started; CREATE_NO_WINDOW keeps a
console window from appearing on Windows. The worker's standard error
is drained on a thread, keeping its last 8 KB.
- A worker that exits or times out before its Ready message is started
once more and its waiting runs are sent again.
- The engine picks the launcher per run: one set for the session, else
the detected CPython, else the built-in worker. In the background,
cells wait as #BUSY! while the search finishes; headless callers wait
for it. A new search stops the workers so that they start again on
what it finds, from a fresh namespace, and an options change runs the
cells again.
python.options takes `interpreter` and `detect` and reports the
backend; python.diagnostics reports it too: kind, summary line, path,
version, pandas, numpy, source and every rejected candidate with its
reason.
The user's CPython has the access of the person's own account: files,
network and processes. The trust gate is what keeps a workbook opened
from a file from running code on it before python.enable.
Tests: the search against fake machines and this machine; shim protocol
tests that run when this machine has Python 3.11 or newer and are
skipped with a message otherwise (xl() round trip, conversions, print
capture, errors with lines, the time limit, the shim's own and the
parent's kill, a loop stopped with SIGINT ignored, reset, and a child
process killed with the worker's group); pandas tests that run only
when pandas imports; the retry before Ready; and an engine run of cells
on the detected CPython.
…against crash loops The built-in Python (Monty) is the fallback when no CPython is found. It now refuses code it cannot run before any of it runs, instead of running part of it, and no failure can hang the application or make the worker restart in a loop. - Imports: a small tokenizer (crates/python/src/imports.rs) reads the import statements of a cell, with strings and comments blanked, bracketed and backslash-continued lines joined, and statements split on `;` and `:`. A module Monty does not have (pandas, numpy, scipy, statistics and others) refuses the cell with #PYTHON! and "This cell needs CPython: `<module>` isn't available in the built-in Python. Install Python 3.11 or newer and set it in Python Options.", at the import's line. The scan is bounded and never fails. - Constructs: the cell is first compiled on its own; a NotImplementedError from Monty's compiler (match, del, yield, type aliases, wildcard imports and others) refuses it with "This cell needs CPython: the built-in Python doesn't support <what>." Other compile errors still come from the run, as before. - Messages, on both backends: a time limit gives #TIMEOUT! "Stopped after N s (an endless loop?)", whether the interpreter stopped itself or the parent killed it; RecursionError gives #PYTHON! "Too much recursion", as does a worker that died of a stack overflow (read from the end of its standard error); the memory watchdog gives "Used more than N MB". - Crash-loop guard: when the worker dies while running a cell (a kill at the deadline, the memory limit, a crash), the cell is remembered with a fingerprint of its code and of what its xl() calls read. It does not run again (#PYTHON!, "Not run: this cell stopped Python the last time it ran", status `refused` in the diagnostics) until one of those changes or python.reset runs everything again. A call whose reference is only known at run time counts any edit as a change. - On CPython, ModuleNotFoundError names the command for that interpreter, `"<sys.executable>" -m pip install <package>` (with the pip names of sklearn, cv2, PIL, yaml and a few others). When the interpreter is externally managed (PEP 668), it says to make a virtual environment instead and choose its python in python.options. GridCraft never runs pip. Tests: Monty refusals (pandas, statistics, from numpy, match, wildcard imports; nothing of a refused cell prints), the scanner on strings, comments, multi-line statements and hostile input, the exact time limit, recursion and memory messages, a worker dying of a stack overflow, the guard (refused after a kill, run again when its input changes, on reset and on a code change), and the pip hint on this machine's CPython.
…admap) - architecture.md: the python crate in the layer table, and a Python in cells section: worker processes, the protocol, the built-in and CPython backends, the engine runtime, the trust gate (with the Trusted state), files and the web build. - file-format-parity.md: Python cells in xlsx, other imports and native JSON, the initialization code and the limits on open. - function-parity.md: Python in cells, and #PYTHON!, #BUSY!, #TIMEOUT! and #BLOCKED! with their ERROR.TYPE numbers (19, 20 and 21 are GridCraft's own; Microsoft documents none for them). - cli.md: running Python cells headless, which Python runs them, and the trust gate for files opened with --in. - ROADMAP.md, target-app-parity.md and gaps.md: progress log, catalog 278 / 294, and the UI that is still to come. - AGENTS.md and CLAUDE.md: python in the layering list, and the CPython shim as the one exception to the Rust-only rule.
…rkbook
python.stop kills the active workbook's worker and drops its queued run.
Every Python cell that had not finished shows #PYTHON! ("Stopped") and a
"stopped" row in Diagnostics; nothing runs again until the next edit or
python.reset, which start from a fresh namespace. Cells that finished
keep their values. With nothing running it returns {stopped: 0}.
The package installs that follow use the same command to cancel an
install.
…person approves
GridCraft keeps one virtual environment of its own for Python cells,
made from the CPython it found (it never downloads an interpreter), in
the application's data folder: ~/Library/Application Support/GridCraft/
python-env on macOS, $XDG_DATA_HOME/gridcraft/python-env (by default
~/.local/share) on Linux and BSD, and %LOCALAPPDATA%\GridCraft\python-env
on Windows. GRIDCRAFT_PYTHON_ENV or python.options {environment} move it;
tests always do. The search for Python tries it after VIRTUAL_ENV and
CONDA_PREFIX and before PATH, and the probe reports whether a candidate
is a venv or a conda environment and which distributions it has.
- python.install {packages, approve, declare?} installs nothing without
approve: true; otherwise it fails with EngineError::Approve, whose
message names the packages and where they would go. It refuses while
Python is turned off for a workbook opened from a file. Requirements
are checked strictly: a PEP 508 name, plain extras and a version
specifier only (no URLs, paths, options, markers or other
characters), at most 32; import names such as sklearn and PIL map to
their pip names.
- Packages go into the person's own venv or conda environment when that
runs cells, otherwise into GridCraft's environment, made first with
uv venv --seed or python -m venv; never into a system or externally
managed Python.
- Installs run uv pip install when uv is on PATH, else python -m pip
install, always with --only-binary, from an argument list, on a
thread with a 10-minute deadline; python.stop cancels them. Failures
are explained (no network, no wheel for this platform or Python, pip
or venv missing). When an install succeeds, the search runs again and
every workbook's Python cells run again.
- python.installStatus reports the running or last install, with a
one-line summary and the versions installed; every install is a row
in python.diagnostics (kept whatever its sheet filter, and kept when
old runs are trimmed) and a record in the application log.
- python.packages declares the packages a workbook needs, kept in
hidden names so they travel in xlsx and native files, and checked and
capped when read; python.diagnostics reports each as found or missing.
- python.environment reports GridCraft's environment and its packages,
creates it, and deletes it (only a folder holding a pyvenv.cfg, and
only with approve: true).
- The CPython shim no longer appends a pip command to a
ModuleNotFoundError; the run records the missing module's pip name
(missing), for python.install.
- EngineError::Approve is added for commands that need the person's
approval. The macro engine pull request adds the same variant.
- CLI tests point GRIDCRAFT_PYTHON_ENV at a folder that does not exist,
so they never use the real environment. A test that installs from the
package index is ignored by default (it needs the network).
…warm-up after installs
A package's first import after an install can take longer than a cell's
5 s limit (a cold import of scikit-learn took about 22 s on macOS). The
cell then showed #TIMEOUT!, and the crash-loop guard refused to run it
again, so a working install looked like a failed one.
- On CPython, time spent importing modules not loaded yet does not
count against the time limit, up to python.options {importSeconds}
(1 to 600, 60 by default, kept in Prefs). The shim wraps
builtins.__import__ (the outermost import on the main thread) and
replaces its timer with a 50 ms watchdog that checks the run's time
less its import time. While an import runs it sends
FromWorker::Extend {id, ms, module}, and the parent moves its own
deadline by that much, never past the allowance; the parent's kill
stays the backstop. The built-in Python never sends it.
- A worker loads numpy and pandas, when installed, before it reports
ready, under an allowance of its own, and its start-up does not count
as the first cell's time.
- A time limit that interrupts an import replaces the worker, since a
package may be half loaded; the message names the module ("Stopped
after 61 s while importing sklearn (the first import after an install
can be slow)"), and RunOutcome carries it.
- A cell the time limit stopped mid-import, or on its first run on a
new worker, runs once more on a fresh worker after the cells before
it, shown as a "retried" row in Diagnostics; stopped again, it is
refused as before.
- Every install ends with a warm-up that imports what it installed once
under a 120 s deadline (python.installStatus {stage} says so while it
runs); a warm-up failure is a warning on an install that worked. uv
installs pass --compile-bytecode.
# Conflicts: # ROADMAP.md # docs/architecture.md
…round runs with Stop
Python cells get an editing mode of their own in the formula bar, in the
code editor (monospace, Python colours). It starts with =PY then Tab or
Enter, Ctrl+Alt+Shift+P, or editing an existing Python cell. Enter adds an
indented line, Tab and Shift+Tab indent, Ctrl+Enter (Cmd+Enter on macOS)
commits through python.set, and Esc cancels. Clicking or dragging on the
sheet inserts xl("A1:B5"); F2 turns pointing off and on. xl() literals take
reference colours and are outlined on the grid.
The formula bar gains a badge slot beside fx, which shows the PY badge in
Python mode and on a selected Python cell; clicking it, the shortcut, or
Backspace in an empty editor turns Python off for the cell, and committing
that runs python.clear, which undo reverses.
On the grid, Python cells carry a PY tag, stale cells under manual
calculation are greyed and struck through, and the tag or Ctrl+Shift+F5
opens a preview card with the result's type and repr, or the last run's
error. The cell menu gains the output mode, Show Preview Card and Convert
to Normal Cell.
The app runs Python in the background (set_python_background) and steps
it every frame, so a slow cell never blocks the window; waiting cells show
#BUSY!. Esc with no editor open, and a Stop button in the status bar, run
python.stop. ui.inspect reports pythonBusy and editor.python, and
engine.execute takes wait and waitSeconds, holding the reply until Python
is idle.
Theme tokens badge_py, on_badge and stale_text join the contrast test.
The Formulas tab gets a Python group after the Function Library: a big Insert Python split button, then Reset and Diagnostics as labelled small buttons. - Insert Python: the main part starts Python mode on the active cell. Its menu has Python Cell, Insert xl() Reference (Python mode with pointing on), Initialization… and Python in GridCraft, a short help text. - Reset is a small split button: the main part runs python.reset; its menu has Reset Runtime, Reset and Clear Diagnostics, and Stop (Esc), enabled while Python cells run. - Diagnostics opens the Python Diagnostics pane. - Keytips in keytips.json: MY opens the Insert Python menu, MZ the Reset menu, M1 the Diagnostics pane. The Python, Reset and Diagnostics icons are drawn in code. When the Formulas tab is wider than the ribbon card, the Python group folds into a single Python dropdown button with the same commands, so it does not push the groups after it out of view. Only this group folds; the other groups keep their layout, and the ribbon's scroll arrows handle any remaining overflow. The expanded width is remembered, so the decision does not change from frame to frame, and a keytip of the folded group opens its menu. Big and small ribbon buttons, the new small split button and icon buttons carry their label as an accessible name.
…s, Enable Python bar
- The Python Diagnostics task pane begins with a line saying which Python
runs cells, then lists the workbook's Python runs, newest first: the
cell (a link that selects it, switching sheets if needed), the status,
the duration, printed output and errors, and the error with its line.
"This sheet" and "Errors only" filter it, Clear empties it through the
new python.clearDiagnostics command, and Options… opens Python Options.
Formulas › Diagnostics opens it, as do Show Diagnostics on an error
cell's preview card and in its cell menu.
- The Initialization dialog edits the workbook's initialization code in
the code editor; OK runs python.initialization, so every Python cell
runs again with it.
- Python Options shows the backend (for CPython its version, program,
where it was found, and the pandas and numpy versions; the Pythons
passed over under "Not used"), the interpreter path with Browse (a new
Services.pick_program, rfd in the desktop app) and Detect Again, and the
time limit per cell. OK runs python.options. It opens from the Insert
Python menu, the Diagnostics pane and the status bar.
- A message bar under the formula bar, for workbooks opened from a file
that hold Python, says that Python cells are turned off, with the
sandbox wording for the built-in Python or, for CPython, the path and a
note on file and network access; Enable Python runs python.enable, and
the bar can be dismissed.
- The status bar shows "Py: CPython" or "Py: built-in" on workbooks with
Python in them; a click opens Python Options.
- In Python mode, pointing is on from the start, as in a formula's point
mode: a click or drag on the sheet inserts xl("A1:B5") and never commits
the code. F2 turns pointing off and on.
- New UI command ui.pane {name?} shows or closes a task pane, so agents
reach the Diagnostics pane like the dialogs.
The formula bar shows one icon button beside the PY badge on a Python
cell or in Python mode: Run (a play triangle) when the cell has not run,
is out of date, or its code is being edited, committing the code first as
Ctrl+Enter does; Stop (a square) while the cell, or the cells it waits on,
run or are queued; Re-run (a circular arrow) once it has a result or an
error. While Python is turned off for the workbook, Run is shown disabled
and its tooltip points to Enable Python. The icons are drawn in code, and
the button is absent on normal cells.
The engine command python.run {cell?, sheet?} runs a Python cell again
(the active cell by default) and the Python cells after it in run order,
which share its namespace, through the existing scheduling and background
job, so engine.execute's wait works with it. It is an error on a cell that
is not a Python cell, and refused while Python is turned off.
Session::python_cell_pending tells the button whether a cell runs or
waits.
The command-sequence fuzz test's XLSX round-trip check accepts a Python
cell's error coming back as #VALUE!, which is how the file format caches
it; the new command changed the generated sequences and reached that
case.
ROADMAP and ui-parity list the Python user interface.
A save requested while Python cells are running or waiting no longer writes #BUSY! (stored as #VALUE!) into the file. It becomes a pending save that is checked every frame, without blocking the window, and runs once the cells have finished. The wait is bounded: it lasts at most one cell's time limit plus the import allowance plus one second, and starts again whenever a cell finishes. When no cell finishes within that time, the workbook is saved with what the cells hold and a warning is logged. This covers Save, Save As, Cmd+S / Ctrl+S, the Save of the Save Changes? prompt (which closes the workbook once it has been saved), and AutoSave, which waits until no Python cell is pending. A workbook without a file asks where to save before it waits. Session::python_pending_cells counts the cells running or queued, so that progress can be measured. On the control channel, the reply to app.save, and to engine.execute or a direct call with file.save or file.saveAs, is held until the save has happened and then reports its outcome; ui.inspect reports savePending. docs/control-protocol.md describes the behaviour.
Contributor
Author
|
Related: #496 adds a Save / Don't Save / Cancel prompt when the window closes or the application quits. It reuses the per-workbook Save Changes? prompt, whose Save this pull request makes wait for running Python cells, so once both are merged the same waiting applies when the window closes. |
Contributor
Author
|
Closed: this depends on running Python code, which the series described in #2637 does not include. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #482.
Summary
Python cells run in the background, so a workbook saved directly after a Python edit could be written while the cell still showed
#BUSY!, and the file then held#VALUE!in place of the computed result. With this change, a save requested while Python cells are running or waiting becomes a pending save. The pending save is checked every frame, never blocks the window, and runs as soon as the cells have finished.The wait is bounded. A save waits at most one cell's time limit plus the import allowance plus one second, and the wait starts again whenever a cell finishes. If no cell finishes within that time, the workbook is saved with whatever the cells hold, and a warning is logged.
Changes
Session::python_pending_cellscounts the Python cells, across all workbooks, that are running or queued. A package install on its own is not counted. A falling count is treated as progress.SheetApp::rundefersfile.save, andfile.saveAswhen it is given a path, while cells are pending. This covers Save, Save As, Cmd+S / Ctrl+S and the File menu and Backstage entries, which all go throughrun. A workbook with no file asks for a location first; only the save itself waits.app.save, or toengine.execute(or a direct call) withfile.saveorfile.saveAs, is held until the save has happened, and then reports how the save went. Each waiting save has its own number, so several such requests each receive their own outcome.ui.inspectreportssavePending.docs/control-protocol.mddescribes the behaviour.Testing
New
crates/ui-egui/tests/python_save.rscovers these cases:app.save, and toengine.executewithfile.save, is held until the file is written, andsavePendingis reported while the save waits.With the deferral disabled, all of these tests fail. The AutoSave test fails without its guard; the file then holds
#VALUE!.A unit test in
crates/ui-egui/src/lib.rscovers the timeout. The cells never progress, so the save stays pending until its deadline and then writes the file while the cell is still waiting.An engine test checks
python_pending_cellsfor queued cells and for cells that have finished.The following checks all pass:
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings, and thecargo xtasktasks assets, layers, wasm, parity and keytips (no generated-file changes).cargo test --workspace --no-fail-fastpasses except for three tests that already fail on main: the French i18n completeness test and two CSV separator tests, which open PRs CSV/text export: General numbers without exponents (integers below 1e16 exact);&switches to scientific past 21 characters #449 and fix(i18n): correct scattered wrong French translations and add 2 missing keys #454 fix.Generative AI use: The code and descriptions in this pull request were written with the assistance of Claude Opus 5.5 (Claude Code). I directed the work, chose its scope, and reviewed and tested the changes.