Add opt-in workbook macro projects to craft-python

.github/workflows/ci.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches: [main]
   pull_request:
+  workflow_dispatch:
 
 permissions:
   contents: read
@@ -21,7 +22,9 @@ jobs:
         with:
           python-version: ${{ matrix.python }}
       - name: plugin.json is JSON
-        run: python -m json.tool plugin.json >/dev/null
+        run: |
+          python -m json.tool plugin.json >/dev/null
+          python -m json.tool plugin-macros.json >/dev/null
       - name: The versions agree
         run: |
           python - <<'PY'

.github/workflows/release.yml

@@ -31,7 +31,7 @@ jobs:
             exit 1
           fi
           mkdir -p dist/craft-python/craftpython
-          cp plugin.json main.py README.md LICENSE dist/craft-python/
+          cp plugin.json plugin-macros.json main.py README.md LICENSE dist/craft-python/
           cp craftpython/*.py dist/craft-python/craftpython/
           (cd dist && zip -qr "craft-python-$version.zip" craft-python)
       - name: Publish

README.md

@@ -357,3 +357,50 @@ apps' bridges, with the trust flow, time limits and malformed input), and
 ## Licence
 
 MIT
+
+
+## Experimental workbook-project role (GridCraft)
+
+This optional role adds the desktop workbook-project portion of
+[GridCraft #1566](https://github.com/storytold/gridcraft/issues/1566). It builds on
+craft-python 0.3.0 and preserves Run Script, Console, Run File, PY.EVAL, helpers,
+trust handling and WordCraft/DeckCraft support. It requires the companion host
+extension on top of [#1123](https://github.com/storytold/gridcraft/pull/1123).
+The ordinary released host does not support its proposed capabilities yet.
+
+Load `plugin-macros.json` explicitly, alongside the ordinary `plugin.json` when
+PY.* formulas are needed. Set `run.command` to the installed Python executable
+if `python3` is unavailable. Optional arguments are `--uv-executable`,
+`--python-interpreter` and `--macro-cache`. The separate role ID does not register
+duplicate functions or ribbon commands. Both modes use the existing App/Channel;
+only `--workbook-macros` imports the macro runtime.
+
+The shared host owns the editor, document trust, source persistence and one-step
+undo/rollback. This role implements the existing `macros/*` methods for validated
+multi-module projects. Discovery uses AST parsing without executing source. A
+macro is a public top-level synchronous function with no arguments, addressed as
+`analysis.run`. It can import other project modules and use
+`import gridcraft as gc; wb = gc.active_workbook()` for `wb.read`, `wb.cell`,
+`wb.inspect`, literal `wb.write` and explicit `wb.execute` calls.
+
+uv prepares the project's PEP 723 environment offline by default. The host's
+Python Settings can explicitly approve downloads for the current document and
+exact metadata, separately from workbook contents. Source builds and unrelated
+uv configuration remain disabled. Ordinary craft-python never starts this worker.
+A successful worker stays warm for the same document/source, retires after 60
+idle seconds, and restarts after changes or errors. Startup/run have an absolute
+30-second budget, messages a 16 MiB limit, diagnostics an 8 KiB limit. Windows
+kill-on-close jobs own worker descendants. On POSIX, a small helper blocks on an
+owned pipe and kills the worker process group when the provider closes or dies;
+it uses no polling and retires with the worker. Platform regressions cover forced
+shutdown, a normal interpreter exit and abrupt provider death. These are trusted programs
+with the user's OS permissions, not a sandbox.
+
+The host supplies Run/Stop and restores workbook changes on failure; external
+Python side effects cannot be rolled back. Form controls, Pyodide, richer output
+and language tooling remain future phases. This is not a replacement for the
+existing scripting UI, Excel's Python formulas, or the Python-in-cells PR #469.
+
+Run the full provider suite with `python -m unittest discover -s tests -v`.
+Native host integration in the companion GridCraft branch additionally tests
+real uv startup, warm reuse, Stop-after-write rollback and PY.TWICE recalculation.

craftpython/app.py

@@ -112,9 +112,9 @@ def answer(self, msg):
         self.channel.send(reply)
 
 
-def main():
+def main(app_factory=App):
     runner.install()
-    app = App(Channel())
+    app = app_factory(Channel())
     while not app.channel.shutdown:
         try:
             msg = app.channel.read()

craftpython/macro_worker.py

@@ -0,0 +1,91 @@
+"""Private, reusable Python worker. No workbook source executes before a run request."""
+
+import ast as _ast
+import contextlib as _contextlib
+import importlib as _importlib
+import inspect as _inspect
+import json as _json
+import sys as _sys
+import traceback as _traceback
+import types as _types
+
+_wire_in, _wire_out = _sys.stdin, _sys.stdout
+_MAX = 16 * 1024 * 1024
+
+def _send(message):
+    data = _json.dumps(message, allow_nan=False, separators=(',', ':'))
+    if len(data.encode('utf-8')) > _MAX:
+        raise ValueError('Macro message exceeds 16 MiB')
+    _wire_out.write(data + '\n')
+    _wire_out.flush()
+
+def _receive():
+    data = _wire_in.buffer.readline(_MAX + 1)
+    if not data or len(data) > _MAX:
+        raise ValueError('Missing or oversized host response')
+    return _json.loads(data)
+
+def _call(command, params):
+    _send({'kind': 'command', 'command': command, 'params': params})
+    result = _receive()
+    if 'error' in result:
+        raise RuntimeError(result['error'])
+    return result['result']
+
+class _Workbook:
+    def read(self, reference, formulas=False):
+        return _call('sheet.read', {'range': reference, 'formulas': formulas})['values']
+    def cell(self, reference):
+        return _call('cell.get', {'cell': reference})
+    def inspect(self):
+        return _call('document.inspect', {})
+    def write(self, reference, values):
+        return _call('range.setValues', {'range': reference, 'values': values, 'literal': True})
+    def execute(self, command, **params):
+        return _call(command, params)
+
+_bridge = _types.ModuleType('gridcraft')
+_bridge.active_workbook = lambda: _Workbook()
+_sys.modules['gridcraft'] = _bridge
+
+class _Output:
+    def __init__(self, name):
+        self.name, self.remaining = name, 65536
+    def write(self, text):
+        if self.remaining:
+            part = text[:self.remaining]
+            self.remaining -= len(part)
+            _send({'kind': 'output', 'stream': self.name, 'text': part})
+        return len(text)
+    def flush(self):
+        pass
+
+_send({'kind': 'ready', 'interpreter': _sys.executable})
+if _sys.argv[-1] == '--prepare':
+    _sys.exit(0)
+while True:
+    try:
+        _request = _receive()
+    except Exception:
+        break
+    if _request.get('action') == 'shutdown':
+        break
+    try:
+        if _request['modulePath'] not in _sys.path:
+            _sys.path.insert(0, _request['modulePath'])
+        with _contextlib.redirect_stdout(_Output('stdout')), _contextlib.redirect_stderr(_Output('stderr')):
+            _module, _function = _request['macro'].split('.')
+            _loaded = _importlib.import_module(_module)
+            _entry = getattr(_loaded, _function)
+            if not _inspect.isfunction(_entry) or _entry.__module__ != _loaded.__name__ or _function.startswith('_'):
+                raise ValueError('Select a public top-level function defined in the module')
+            if _inspect.signature(_entry).parameters or _inspect.isgeneratorfunction(_entry):
+                raise TypeError('Select a synchronous function with no arguments and no yield')
+            _return = _entry()
+            if _inspect.isawaitable(_return):
+                if _inspect.iscoroutine(_return):
+                    _return.close()
+                raise TypeError('Async macros are not supported')
+        _send({'kind': 'done'})
+    except BaseException:
+        _send({'kind': 'failed', 'error': _traceback.format_exc(limit=12)[-8192:]})

craftpython/process_tree.py

@@ -0,0 +1,121 @@
+"""Own only this macro worker's process group; no process-name scans or global kills."""
+import os
+import signal
+import subprocess
+import sys
+
+# The provider is the sole writer of this pipe. EOF also arrives after SIGKILL/crash, when
+# Python finally blocks cannot run. The helper stays outside the worker's process group and
+# sleeps in the kernel until needed; user modules never inherit the write end.
+_GROUP_OWNER = """
+import os, signal, sys
+try:
+    sys.stdin.buffer.read()
+finally:
+    try:
+        os.killpg(int(sys.argv[1]), signal.SIGKILL)
+    except ProcessLookupError:
+        pass
+"""
+
+
+class ProcessTree:
+    def __init__(self):
+        self.process = None
+        self.owner = None
+        self.job = _WindowsJob() if os.name == 'nt' else None
+
+    def attach(self, process):
+        self.process = process
+        if self.job is not None:
+            self.job.attach(process.pid)
+        else:
+            self.owner = subprocess.Popen([sys.executable, '-c', _GROUP_OWNER, str(process.pid)],
+                                          stdin=subprocess.PIPE, stdout=subprocess.DEVNULL,
+                                          stderr=subprocess.DEVNULL, close_fds=True)
+
+    def close(self):
+        if self.job is not None:
+            self.job.close()
+        elif self.process is not None:
+            try:
+                if self.owner is not None:
+                    self.owner.stdin.close()
+                    try:
+                        if self.owner.wait(timeout=5) != 0:
+                            self._kill_group()
+                    except subprocess.TimeoutExpired:
+                        self._kill_group()
+                        self.owner.kill()
+                        self.owner.wait(timeout=5)
+                else:
+                    self._kill_group()  # Failed ownership setup still cleans its new session.
+            finally:
+                self.owner = None
+                self.process = None
+
+    def _kill_group(self):
+        try:
+            os.killpg(self.process.pid, signal.SIGKILL)
+        except ProcessLookupError:
+            pass
+
+
+if os.name == 'nt':
+    import ctypes
+    from ctypes import wintypes as w
+
+    class _BasicLimits(ctypes.Structure):
+        _fields_ = [('process_time', ctypes.c_longlong), ('job_time', ctypes.c_longlong),
+                    ('flags', w.DWORD), ('minimum_ws', ctypes.c_size_t),
+                    ('maximum_ws', ctypes.c_size_t), ('active_processes', w.DWORD),
+                    ('affinity', ctypes.c_size_t), ('priority', w.DWORD), ('scheduling', w.DWORD)]
+
+    class _IOCounters(ctypes.Structure):
+        _fields_ = [(name, ctypes.c_ulonglong) for name in
+                    ('read_ops', 'write_ops', 'other_ops', 'read_bytes', 'write_bytes', 'other_bytes')]
+
+    class _ExtendedLimits(ctypes.Structure):
+        _fields_ = [('basic', _BasicLimits), ('io', _IOCounters),
+                    ('process_memory', ctypes.c_size_t), ('job_memory', ctypes.c_size_t),
+                    ('peak_process_memory', ctypes.c_size_t), ('peak_job_memory', ctypes.c_size_t)]
+
+    _kernel = ctypes.WinDLL('kernel32', use_last_error=True)
+    _kernel.CreateJobObjectW.argtypes = [ctypes.c_void_p, w.LPCWSTR]
+    _kernel.CreateJobObjectW.restype = w.HANDLE
+    _kernel.SetInformationJobObject.argtypes = [w.HANDLE, ctypes.c_int, ctypes.c_void_p, w.DWORD]
+    _kernel.SetInformationJobObject.restype = w.BOOL
+    _kernel.AssignProcessToJobObject.argtypes = [w.HANDLE, w.HANDLE]
+    _kernel.AssignProcessToJobObject.restype = w.BOOL
+    _kernel.OpenProcess.argtypes = [w.DWORD, w.BOOL, w.DWORD]
+    _kernel.OpenProcess.restype = w.HANDLE
+    _kernel.CloseHandle.argtypes = [w.HANDLE]
+    _kernel.CloseHandle.restype = w.BOOL
+
+    class _WindowsJob:
+        def __init__(self):
+            self.handle = _kernel.CreateJobObjectW(None, None)
+            if not self.handle:
+                raise ctypes.WinError(ctypes.get_last_error())
+            limits = _ExtendedLimits()
+            limits.basic.flags = 0x2000  # JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, no breakaway.
+            if not _kernel.SetInformationJobObject(self.handle, 9, ctypes.byref(limits), ctypes.sizeof(limits)):
+                error = ctypes.WinError(ctypes.get_last_error())
+                self.close()
+                raise error
+
+        def attach(self, pid):
+            process = _kernel.OpenProcess(0x0100 | 0x0001, False, pid)  # SET_QUOTA | TERMINATE
+            if not process:
+                raise ctypes.WinError(ctypes.get_last_error())
+            try:
+                if not _kernel.AssignProcessToJobObject(self.handle, process):
+                    raise ctypes.WinError(ctypes.get_last_error())
+            finally:
+                _kernel.CloseHandle(process)
+
+        def close(self):
+            if self.handle:
+                if not _kernel.CloseHandle(self.handle):
+                    raise ctypes.WinError(ctypes.get_last_error())
+                self.handle = None

craftpython/workbook_macros.py

@@ -0,0 +1,51 @@
+"""Optional workbook-project role, using craft-python's existing App and Channel.
+
+Run Script, Console, Run File, PY.EVAL and the other apps keep their ordinary path.
+Only --workbook-macros loads this adapter or starts the uv-managed worker.
+"""
+import argparse
+
+from .app import App, main
+from .commands import PluginError
+from .wire import HostError
+from . import workbook_projects
+
+MACRO_ID = "io.github.sirmmo.python.macros"
+
+
+class WorkbookMacroApp(App):
+    def __init__(self, channel, options):
+        super().__init__(channel)
+        workbook_projects.configure(channel.request, options.uv_executable,
+                                    options.python_interpreter, options.macro_cache)
+
+    def handle(self, method, params):
+        if method == "initialize":
+            result = super().handle(method, params)
+            if self.host != "gridcraft":
+                raise PluginError("Workbook Python projects require GridCraft")
+            try:
+                workbook_projects.handle(method, params)
+            except RuntimeError as error:
+                raise PluginError(str(error)) from error
+            result["id"] = MACRO_ID
+            return result
+        if method.startswith("macros/"):
+            try:
+                return workbook_projects.handle(method, params)
+            except (RuntimeError, ValueError, SyntaxError, OSError, HostError) as error:
+                raise PluginError(str(error)) from error
+        return super().handle(method, params)
+
+
+def run():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--workbook-macros", action="store_true")
+    parser.add_argument("--uv-executable", default="uv")
+    parser.add_argument("--python-interpreter")
+    parser.add_argument("--macro-cache")
+    options = parser.parse_args()
+    try:
+        main(lambda channel: WorkbookMacroApp(channel, options))
+    finally:
+        workbook_projects.close_worker()