fix: stop bundling awscrt in the installer - #1331
Conversation
The installer's PyInstaller build reads its extras from the `installer` Hatch env, which declared no `features` and so inherited them from `envs.default`. When the `console` extra was added there it began installing botocore[crt], PyInstaller bundled awscrt, and `installer:validate_exe` failed against scripts/pyinstaller/allowlist.py -- breaking the release build on all three platforms after the change had already merged. Pin the env's features explicitly so an extra added to `envs.default` can no longer change what ships in the signed artifact. Console sign-in credential refresh stays available to consumers via `pip install "deadline[console]"`, which is unaffected: the extra is wheel metadata and independent of hatch.toml. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>
| # omitted on purpose -- it pulls in awscrt, a compiled wheel that is not in | ||
| # scripts/pyinstaller/allowlist.py. Consumers who need AWS Console sign-in | ||
| # credential refresh install the extra themselves. | ||
| features = ["gui"] |
There was a problem hiding this comment.
Dropping console from the installer env means AWS Console sign-in is not just absent from the shipped installer — it is unreachable there, and the error message users hit points at a remedy they cannot apply.
_check_console_login_dependency (src/deadline/client/api/_loginout.py:59-63) raises:
Signing in to the AWS Console sign-in profile <p> requires an additional
dependency. Install it with: pip install "deadline[console]"
The installer ships a PyInstaller-frozen bundle (scripts/pyinstaller/make_exe.py → DeadlineClient.zip), which has no pip and no site-packages a user can extend. So for anyone using the installed CLI/GUI rather than a pip install, deadline auth login on a login_session profile fails with instructions that are impossible to follow. The login_session profile itself is one Deadline Cloud monitor can create, so this is a reachable path for installer users, not a corner case.
I agree with the PR description that adding awscrt to the allowlist is a supply-chain decision not worth making as release triage — so the config change here looks like the right call for unblocking the release. The gap worth tracking separately is the messaging: in a frozen build the guard should say something actionable (e.g. "not supported by the Deadline Cloud installer build; use a pip install of deadline[console]") rather than naming a pip command that cannot run. Detecting it is cheap — getattr(sys, "frozen", False), which the codebase already reasons about in _deadline_web_url.py:128.
| # Extras that must never be bundled, mapped to why. An extra listed here brings in | ||
| # a distribution that scripts/pyinstaller/allowlist.py does not allow. | ||
| _EXTRAS_EXCLUDED_FROM_INSTALLER = { | ||
| "console": "pulls in awscrt, a compiled wheel absent from the PyInstaller allowlist", |
There was a problem hiding this comment.
This guard is a denylist, but the bug it is guarding against is an unanticipated addition — so it cannot catch the next instance of the same class of failure.
_EXTRAS_EXCLUDED_FROM_INSTALLER only names console. If someone adds mcp (or a future extra) directly to envs.installer.features, both tests here pass and installer:validate_exe still breaks at release time — exactly the failure mode described in the module docstring. mcp is concretely in that position today: scripts/pyinstaller/deadline_cli.spec:21 strips MCP prefixes out of hiddenimports and mcp is absent from allowlist.py's DEPENDENCIES, so enabling it would produce unallowlisted files.
The invariant that actually matches the allowlist is exact equality, not exclusion:
_INSTALLER_FEATURES = ["gui"] # every extra here must be covered by scripts/pyinstaller/allowlist.py
def test_installer_env_features_are_pinned(hatch_envs: dict) -> None:
assert hatch_envs["installer"].get("features") == _INSTALLER_FEATURES, (
"Changing envs.installer features changes what PyInstaller bundles into the "
"signed installer. Add the new distribution to scripts/pyinstaller/allowlist.py "
"and update this list together."
)That subsumes both current tests (a missing features key fails it too, since .get returns None), makes any change to the bundle a deliberate two-file edit, and removes the need to enumerate reasons per excluded extra.
| try: | ||
| import tomllib | ||
| except ModuleNotFoundError: # Python 3.9/3.10 | ||
| import tomli as tomllib # type: ignore[no-redef] |
There was a problem hiding this comment.
tomli is not a declared test dependency, and if it is absent this fallback raises out of module scope rather than skipping — turning a missing optional dep into a collection error for the whole file on Python 3.9/3.10.
The except ModuleNotFoundError: handler catches the failure of import tomllib, but import tomli inside the handler is unguarded. tomli appears nowhere in requirements-testing.txt; the only path by which it reaches a 3.9/3.10 env is transitively through coverage[toml]'s marker-gated tomli requirement. code_quality.yml:24 runs the unit suite on 3.9 and 3.10, so this file's importability there rests on a transitive dep of a coverage extra that nothing in this repo pins or asserts.
Two ways to make it robust:
tomllib = pytest.importorskip("tomllib" if sys.version_info >= (3, 11) else "tomli")or add tomli; python_version < "3.11" to requirements-testing.txt and keep the current import. The first is preferable if the guard is not considered important enough to warrant a new dependency; the second if it is (a silent skip on 3.9/3.10 still leaves the invariant checked on 3.11+, so either is defensible — an unguarded ImportError is not).
Why
envs.installerinhatch.tomldeclared nofeatures, so Hatch resolved them fromenvs.default. When #1323 added theconsoleextra to the default env, the installer build began installingbotocore[crt], PyInstaller bundledawscrt, andinstaller:validate_exefailed againstscripts/pyinstaller/allowlist.py:This broke
BuildAndStageInstallersin the 0.60.4 release (run 31435049529) — macOS failed and Linux/Windows were cancelled by fail-fast, soRelease,Publish, andPublishToPyPInever ran. 0.60.4 is not on PyPI.What
Pin
features = ["gui"]onenvs.installerso an extra added toenvs.defaultcan no longer change what ships in the signed artifact.consoleis omitted deliberately:awscrtis a compiled wheel that isn't allowlisted.Rejected alternative: adding
awscrtto the allowlist so the installer ships console sign-in support. That puts a compiled binary into three signed artifacts and needs per-platform allowlist entries — a supply-chain decision worth making deliberately, not as release triage. The allowlist is a deliberate control, so the conservative fix is to keep the bundle as it was.No consumer impact. Console sign-in credential refresh stays available via
pip install "deadline[console]". The extra is wheel metadata (Provides-Extra: console→botocore[crt]>=1.42.89), independent ofhatch.toml. Verified in a clean venv: adding the extra to an existing install pullsawscrt,botocore.compat.ECbecomes non-None,_check_console_login_dependencystops raising, and botocore'sloginprovider appears in the credential resolver chain.Testing
New
test/unit/test_installer_env_features.pyasserts the installer env declaresfeaturesexplicitly and omitsconsole. It lives intest/unit/, so it runs in the existing PR CI matrix. Confirmed it fails against the pre-fix config, not just passes against the new one.Locally on arm64 macOS / Python 3.13 (matching the failing CodeBuild host):
awscrtin installer envhatch run installer:make_exehatch run installer:validate_exeValidation passed, 0 allowlist failureshatch run lint(ruff + mypy)hatch run testFollow-ups (not in this PR)
installer:validate_exeruns only in the release pipeline, never in PR CI — which is why feat: support login and logout for AWS Console sign-in profiles #1323 merged green and broke at release. This test closes the specific hole; catching the general case means build+validate on PRs touchinghatch.toml,pyproject.tomlextras,requirements-installer.txt, orscripts/pyinstaller/.hatch.tomltest_build_installerpoints attest/build_installer, which doesn't exist.