Repository navigation
fix(agent): adopt, update and roll back stacks that Portainer runs - #27
Merged
Merged
Conversation
Portainer runs compose inside its own container, so the paths compose writes into the labels (/data/compose/49/docker-compose.yml) are Portainer's, not the host's, and adopting such a stack failed with "No such file or directory". A label path that does not exist on the host is now mapped through the mount of the container that holds it (longest destination wins), and used only if the mapped file exists. The env file compose recorded (Portainer's stack.env) is passed with --env-file, so a re-up from the host interpolates the same values instead of blanks, and adopt reads the env from it.
…instead of failing every task A host installed in monitor mode and switched to managed afterwards keeps the monitor unit: User=vigil-agent under ProtectSystem=strict. Every task it accepted then died on its first write — Update Agent with "[Errno 30] Read-only file system: '/usr/local/bin/...'". The agent now rejects each task up front with the fix (re-run the installer), reports runs_as_root at check-in, and the server counts such a host as refusing every action, so the deploy dialog's "will refuse" chip and skip logic catch it before sending.
The page assumed an operator already knew that "unmanaged" means "seen only in Windows' Apps & features, reported by no package manager" — which is exactly the thing that needs explaining, because it is why such an item can never show as outdated. Each tab now opens with one line saying what it lists, and the unmanaged, outdated and decision columns carry the existing .field-hint markers, so the copy is one hover away instead of a footnote at the bottom of one tab. "By host" jumped to the Monitor page, which read as a broken link: the tab promises a per-host software list, so the row expands in place the way a "By app" row does, fetching /software/hosts/<id>/ once and caching it. The jump to Monitor is still there, now as an explicit link inside the row. MUTATION RED: removing data-app-host-row from the row template turns test_by_host_expands_in_place red; restored and green. End-to-end (rendered template, not source scan): field-hint markers: 8 apps-intro: 3 upd-note: 0 hints: 3x unmanaged (Apps & features / cannot upgrade or uninstall, paragraphs intact), 1 outdated, 3 bulk decisions, 1 decision column (Approved / Declined / Undecided). Ran 2182 tests in 112.371s — OK (skipped=1); manage.py check: no issues.
… escape scope once
… groups The two sections were stacked, so the per-host table pushed remediation off screen and "by fix" rendered an empty div (or nothing at all, since the section was hidden until rows arrived) while the fix-groups request was in flight. Tabs are the pattern the Apps page already uses, so the markup copies that shape and vigil-nav.js wires it with no new JS. The skeleton reuses the modal shimmer's .sk-line rules rather than introducing a second loader; it is painted only when there is nothing on screen, so a search-refetch keeps its rows instead of flashing. With the section in its own tab there is no "hidden until non-empty" state left to manage, and the empty state can say which of the two cases it is.
The host control became a picker button plus a hidden <input>, but _fwPopulateHosts still appended <option>s to it and bound a change listener — an <input> ignores both, so sel.value stayed "" and loadFirewall() never fetched a snapshot. Pick a host at load instead: the one this browser last chose, if it is still eligible, otherwise the first eligible host, and set the button label to match.
The lockout guard refused *every* remove_firewall_rule on a protected port, so the one rule that could undo a self-inflicted lockout was the one nobody could delete — and the guard let `reject` past on add while the agent refused to remove it, so a REJECT rule on 22 was permanent from the UI. Only removing an *allow* can cut access off. The remove branch now returns early when the change claims deny/reject, but only when the stored snapshot confirms that exact rule is a block: the claimed action comes from the browser and Windows removes by rule_id whatever the action says, so a "deny" carrying the RDP allow rule's id must not pass the guard. A block the snapshot doesn't carry falls through and is refused, failing safe. The add branch blocks deny and reject alike (reject was the gap the user hit), and the agent accepts reject so ufw/firewalld can actually carry the removal. The tab mirrors the guard: Remove now shows on every rule except an allow on a protected port, and the surviving protected rows say which service they keep open instead of the old bare "[protected]".
A Vigil session was Django's default: a two-week cookie with no idle check, so a browser tab left open on a shared machine stayed signed in for a fortnight. Sessions now end after 15 idle minutes and after 12 hours at most, both editable in Settings → Account & Security. The rule that makes an idle timeout mean anything is what counts as activity. Vigil's pages poll the API continuously, so treating any request as activity would let an unattended Monitor tab sit out the clock forever. Only a page navigation, a change, or the browser's interaction heartbeat refresh the idle stamp; background GETs are checked but never refresh it. Agents carry a bearer token and no session, so the middleware leaves them alone; a session predating this change gets its clocks started rather than being signed out on upgrade.
manage.py validate_community <dir> [--strict] checks a Vigil-Approved-Scripts checkout with the same parsers the Community page uses, so the repo's CI can stop carrying a second copy of the rules — that copy drifted, and files it passed were refused by the server. On top of parsing it checks what a fork needs: file names match their slugified names, every file has a unique uid, playbook steps and automations resolve to files in the checkout with matching uids, event automations name an event that exists (alert_fired imported fine and never fired), directories are flat, and tasks hardcode no home directory. --strict fails deprecated syntax too. A file with only lenient warnings still reports ok.
… the place after The browser half of the session timeout. Real use (pointer, keyboard, wheel, touch) sends a heartbeat at most once a minute; background polling never does, so an open tab still times out. Two minutes before the end a dialog counts down with Stay signed in (idle only — the hard cap cannot be extended) and Sign out now. When time is up the page goes to the login screen with the reason and the page it was on, and signing in returns there. Tabs share one session, so a tab asks the server before warning or leaving rather than trusting its own timer; an apiJson call that meets the timeout goes to the login page instead of showing an error.
…ject rule A rule whose action the snapshot could not read is refused by the guard like an allow is, so the tab no longer shows a button that is bound to fail.
…one to it Deploying a task definition checked only that the caller was signed in. A viewer could write a definition and deploy it, and a user scoped to one site could target hosts in another, by id or by tag. The server signs what is deployed, and the agent runs it as root within its allowlist. Deploy now requires tasks:run in every target host's site. Hosts outside the caller's scope answer 404 like an unknown id, and tag targeting only ever matches hosts the caller may run on. Creating, editing, forking and archiving definitions needs a role that may run tasks somewhere, and archiving a shared definition needs its owner or an admin. scope_of() had no entry for Host, so it returned None for every host and each per-host capability check (edit, update_agent, firewall, reprovision) fell back to the caller's strongest role anywhere. Per-site operator limits now apply to hosts.
…un tasks Starting or resuming a rollout checked only that the caller was signed in. It now needs permission to run tasks fleet-wide: an admin, or an operator not limited to particular sites. A rollout's waves reach every site. Stack deploy, remove and adopt were admin-only by the caller's strongest role anywhere, so someone who was admin in one site and a viewer in the stack's site passed. Each now needs tasks:run in the stack host's site. The enrolment wizard's pending-host check described any host whose token was supplied to any signed-in user. It is admin-only now, like enrolment itself.
…s sites allow Task history, a task's detail, run history, run detail and a run's hunt results answered for every host in the fleet. A task's output is whatever its host printed, so a user scoped to one site could read another site's results. History and task detail now follow the host's site like the host endpoints do; an out-of-scope task answers 404. A run is shown only when every host it touched is in the caller's scope, since its summary and outputs cover all of them. Unscoped installs see no change.
…he agent runs as SYSTEM execute_script runs scripts from C:\ProgramData\Vigil\scripts as LocalSystem. The installer never created or locked that folder, and C:\ProgramData's default ACL lets any user create folders and files in it. On the Windows test VM a standard user created the folder, owned it, and planted a script that the next signed execute_script would have run as SYSTEM. The agent's writability check was POSIX-only and trusted an installer ACL that was never applied. The installer now locks C:\ProgramData\Vigil (SYSTEM and Administrators full, Users may only list it), creates scripts, and takes back ownership of anything already in it, by SID so it works on any language. The monitor-mode service account keeps write access to its own log only. The agent refuses to run a script unless the file and every folder up to the scripts directory are owned by root (or SYSTEM, Administrators or TrustedInstaller) and writable by nobody else. On Linux that adds the owner and the parent folders to the old mode-bit check. Checked against real ACLs on the VM, and a check that cannot complete refuses.
… before locking it Locking C:\ProgramData\Vigil's ACL did not change who owned what was already inside it, and an owner can always rewrite an ACL. A standard user who created the folder, agent.yml or data before the install would have kept control, and icacls /grant only adds entries, so one they put on a pre-created file would have survived too. The installer now makes Administrators the owner of the whole tree, locks the folder, and resets every child to inherit from it before applying the specific grants for agent.yml, data and the service account. Every grant names its account by SID: SYSTEM and Administrators are spelt differently on non-English Windows, where the old by-name grants would have failed.
…talls stop following planted links The server's public key now arrives in install.sh / install.ps1, over the same TLS download as the agent binary, and is written into agent.yml. The agent accepts only that key: no trust on first use, and a pin file in the data directory cannot override it. Re-running the installer is how an intended key rotation reaches an agent. In monitor mode the installer used to hand agent.yml and the pinned key to the unprivileged vigil-agent account. The mode written in agent.yml decides whether the next install creates a root service, so that account could have asked for root and waited for a reinstall. agent.yml is now root:vigil-agent 0640, and a task-running mode is only believed from a root-owned file or when the installing admin confirms it with VIGIL_MODE. The Windows lock-down from the previous commit discarded icacls failures and recursed with /T as an administrator, so a link a user planted in the tree would have pointed those owner and ACL rewrites somewhere else. Each step now fails the install if it fails. The top folder is locked first, planted links are deleted without being followed, and every recursive step uses /L. A planted junction at C:\ProgramData\Vigil itself is replaced before anything is created in it. Checked on the Windows VM.
…write, and never recurse through links install.sh read single keys back out of agent.yml with sed while the agent reads it as YAML, and the two can disagree about what a file says. It also believed everything but `mode:` in a file the unprivileged service account had owned, so its server_url and allowlist could flow into a root service. A config that is not root-owned and private is now set aside and rebuilt as on a fresh install. Only the agent token is carried over, and only if it has a token's shape: it goes into a sed command run as root, and VIGIL_TOKEN is held to the same rule. On Windows, icacls /T follows directory junctions even with /L (measured on the test VM), so the recursive lock-down could still rewrite owners and ACLs outside the tree through a link planted between the cleanup pass and the recursion. The tree is now walked top-down in PowerShell: each folder is taken over and reset before it is listed, so nothing can be added behind the walk, and links are deleted, never entered. Every later grant uses /L and fails the install if it fails.
… agent The signature covered a task's id, host, action, params, nonce and TTL, but not the time its TTL is counted from, and the agent never checked that the signed host was itself. The agent also checked the TTL before the signature, skipped it altogether when the timestamp would not parse, and forgot nonces after an hour whatever a task's TTL was. An agent now advertises signed_v2, and for it the server signs the dispatch time and a SHA-256 fingerprint of the agent's own token as well. That agent accepts nothing else: it verifies first, measures the TTL from the signed time, refuses a missing or unreadable one, and checks the fingerprint against its own token. Nonces are kept until their task could no longer run. Older agents keep getting v1. An agent can never be newer than its server, so none is cut off. The server's v2 signature is checked with the agent's own verifier in the server tests.
…name opens its run detail Clicking a legend row hid every other host's cards, so an operator could see a group's count but never read the rest in context, and the ring's slices — the bigger target — weren't clickable at all. Now a slice or a legend row slides its group's cards to the top and dims the others, and clicking it again restores the order the cards came in. The host's name in a result was inert text. It opens a second dialog listing every task the run sent that host, in step order: state, action and step label, params as JSON, the step results table, full output, the three timestamps with how long it took, plus TTL, risk, requester, playbook step and branch. The API doesn't expose the nonce or signature, so neither does the dialog.
…s the right role A network vulnerability scan could be launched against any host in the caller's scope by anyone signed in, viewers included. It now needs tasks:run in the host's site, like running a task there. Adding an unmanaged device was open to every signed-in user while editing and deleting one were admin-only; adding is admin-only too.
install.sh only restarted the agent when VIGIL_TOKEN was given. A reinstall over a running agent (upgrading, changing mode, or rebuilding a config that was not trusted) left the old process running with the old binary, privileges and config until something else restarted it, and told the admin to start a service that was already running. It now restarts the agent, and the read-only engine proxy, whenever they were running. Found installing on the Ubuntu test VM.
…owned instead of adopting them Taking ownership of the scripts folder made every file in it Administrators', so a script a standard user had planted before the fix would have looked trusted to the agent afterwards. Files under scripts that SYSTEM, Administrators or TrustedInstaller did not own are now moved to a locked quarantine folder before the takeover. The walk never follows a link, since a recursive Get-ChildItem can descend through a junction and this runs elevated. Run elevated on the Windows test VM: a standard user's planted scripts folder and junction were taken back, the junction removed with its target untouched, the planted script quarantined, and the user could no longer write there.
…ner and by who may change them The quarantine pass judged scripts by owner only, before the scripts folder was locked, and stopped the install when a file could not be moved. An administrator-owned script that users could modify would have been adopted, a user could still swap files while they were being judged, and two same-named files in different subfolders overwrote each other in quarantine. Quarantine now happens inside the top-down lock walk, so a folder is locked before its files are judged and nothing in it can change behind the check. A script is untrusted when a non-administrator owns it or anyone but SYSTEM, Administrators or TrustedInstaller may write it, the agent's own rule. It keeps its relative path in quarantine, and one that cannot be moved is deleted, with the install stopping only if that fails too. Run elevated on the Windows VM: an admin-owned but user-writable script and two same-named user scripts were quarantined, and the trusted one stayed.
…staller, is the guarantee The installer's lock cannot revoke a handle opened before it ran, so a file could still be added through one afterwards. The agent re-checks ownership and write access every time it runs a script, which refuses such a file. Pinned by a test that the check comes before the script runs.
…ack columns A bare window.prompt for a 6-digit code tells the operator nothing about what they are authorizing and matches nothing in the app. totpPrompt(reason) is one styled dialog -- digits-only, Confirm gated on exactly six, Enter confirms, Escape/Cancel/overlay resolve null -- used by all eight writes that ask for a code, each now saying what it confirms. The Containers host <select> sat unlabelled in a hunt-toolbar next to a 100%-wide .form-control; it is now a .fw-controls bar with a Host label, like the Firewall page. Each compose stack rendered its own auto-width table, so Name/Image/State/CPU/Memory landed at a different x in every card: the tables are table-layout: fixed with a colgroup, so they line up down the page.
Registering a host is open by design, so an agent token proves nothing until an admin approves the host. authenticate_agent accepted any token on file, pending and rejected hosts included. Anyone who could reach the server could register a host and at once fetch every registry credential scoped to all hosts, without anyone approving anything. Every agent endpoint now refuses a pending or rejected host with 403. Check-in alone opts out, because a pending host has to check in to reach the enrolment queue, and it already answers rejected hosts itself.
consume_totp blocked reuse of a code but not repeated wrong ones. The TOTP gate on deploys, approvals and firewall changes exists for the case where someone holds a session without the device, and with no limit a six-digit code falls to guessing. After five wrong codes in ten minutes every TOTP-gated action refuses that user for the rest of the window; a right code before the limit clears the count. Kept in the LoginAttempt table, so the limit holds across workers and restarts.
… before checking it The limit added in the previous commit kept its rows in LoginAttempt under "totp:<user id>". A failed sign-in records whatever username was typed, so anyone could type that key a few times and lock the user out of every TOTP-gated action. It also checked the count before recording the failure, so parallel requests could all pass a check none of them had incremented. Attempts now go in a TotpAttempt table keyed by the user, which sign-in never writes to. Each attempt is recorded first and the count is checked after, so racing requests see each other. A right code within the limit clears the user's attempts.
…s every other gate totp_disable checked the code with verify_totp directly, outside the attempt limit and replay check in consume_totp. Disabling is the strongest thing a code unlocks, since afterwards a different device can be enrolled, so a stolen session could guess its way out of every TOTP gate. It now uses consume_totp.
…re it starts a stack
Every round of review of the server's compose check found another way compose
reads the same text differently: variables substituted from the .env after
the check, include and extends merging other files, "true" strings read as
booleans, //run read as /run. Checking the text can't win that race. So
before `up`, the agent now asks compose for its resolved configuration
(`config --format json`, or YAML from podman-compose) and applies the same
rules to exactly what will run: protected binds, including their ancestors,
privileged, host namespaces, dangerous capabilities, raw devices, sockets,
volumes_from other containers, and build, secret and config paths outside the
stack. If the configuration can't be read, nothing runs.
Checked against real compose on the Ubuntu VM: a file hiding /etc in
${HOSTDIR} and privileged in a "true" string came back resolved, and the check
named both.
The server's check stays as early feedback for the editor. Its key walk now
visits each YAML anchor's node once and caps nesting, so an anchor bomb
cannot pin a worker.
… shape compose accepts
The resolved-config check only understood docker's normal form, and `up`
then re-read the original files. podman-compose's `config` hands back the
file much as written, so short volume strings ("/etc:/e"), "true" strings,
list sysctls and a lone cap_add string went unchecked, and anything that
changed between `config` and `up` (a .env, a file a container bound to the
stack folder could rewrite) was never checked at all.
- composecheck: an allowlist of service and top-level keys, like the
server's; every volume, device, sysctl, cap and security_opt shape;
build additional_contexts and dockerfile; env_file; ~ refused; a $ in
any field that decides access refused; security_opt allowlisted
(custom seccomp profiles and label:type are out); bind sources and
devices checked again after following symlinks.
- Binds inside the stack's own folder are allowed again — ./data resolves
under /opt/vigil and was refused by the previous commit.
- _stack_deploy writes the checked configuration to
<data_dir>/stacks/<project>.json (0600, a path no stack may bind) and
runs `up -f` on it. Compose labels the containers with that file, so
stack_update re-runs exactly what was checked. stack_remove deletes it.
Verified on the Ubuntu test VM: up from the resolved JSON keeps $$ literal,
substitutes .env values once and binds ./data where expected.
…s a container could re-link Three gaps in ec0809f's deploy-what-you-checked: - Paths: `up` reads the checked file from <data_dir>/stacks, so a relative path left in it (podman-compose's config does not absolutise) would resolve against the agent's data dir, while the check resolved it against the stack folder — "../" would bind the agent's own state. composecheck.pin makes every host path absolute against the stack folder; the pinned copy is what is checked and what is started. - Symlinks: following links at check time cannot stop a container that can write a folder from swapping one in before the engine mounts. A bind is refused when a component under the stack folder is already a symlink, or when it lies inside a folder that this config or a running Vigil-stack container mounts writable. - Variables: only some fields refused a $. Now every string does, except in fields that only reach inside the container (environment, command, entrypoint, healthcheck, labels, content), where docker's $$ re-reads as a literal $. Checked on the Ubuntu test VM against docker's real resolved output.
…s through planted links, no x-podman - Binds: every folder above a bind source (as written and once links are followed) must be root-owned and not group/other-writable. Whoever can write one can swap a path component for a symlink after the check and have the engine mount, say, /etc into a root container — the "owns the machine but is not admin" case. Stack data belongs in the stack folder or a root-owned path such as /srv; a bind under /home/<user> is refused. - The stack folder: the agent writes compose.yaml and .env there as root. It is refused unless root alone can change the folders above it; the files are replaced (unlink + O_EXCL|O_NOFOLLOW), never written through a link; and no container may mount the stack folder, or a folder holding it, writable — in the new config or already running. - x-podman: an inert extension field to docker, but podman-compose acts on it (podman_args, pod_args, uidmaps). Refused at any depth by the server and by the agent. The agent's variable walk visits shared YAML nodes once. Checked on the Ubuntu test VM: /srv and not-yet-created /opt/vigil paths pass, /home/vboxuser and /tmp are refused.
…ymlink _stack_read runs as root and reads compose and .env from paths in container labels, which may sit in a folder a non-root user owns. A planted .env -> /etc/shadow (or a re-pointed folder) would have been read and posted to the server as the stack's env. Reads now open each folder relative to the last with O_NOFOLLOW, accept regular files only, and stop at 1 MB. _up also refuses a symlinked <data_dir>/stacks.
… itself, and refuses hard links - _read_plain refuses a file with more than one hard link: a link to /etc/shadow is no symlink, and gets past O_NOFOLLOW on a host without fs.protected_hardlinks. - _hash_report ran `compose config --hash` on the labelled paths — a second, link-following read as root after the safe one — and passed compose's error text on, which quotes the line it could not parse. It now runs on private copies of the files already read (same --project-directory, so the hashes match: checked on the Ubuntu test VM) and reports only that compose could not resolve the stack.
The full-screen stack editor (QA-20) validates as the person types, which
needs an answer for every keystroke rather than a status code: bad YAML, a
sandbox refusal and a good file are all 200 with the reason in the body, so
the editor never renders a failed request as a broken file.
describe_compose calls validate_compose and reports its refusal verbatim
instead of re-implementing a lighter copy of the rules — a validation
endpoint that skipped the sandbox would be a way to develop a file that
could reach the host, and the editor would look authoritative about it.
StackError now carries the parser's line (1-based) so the editor can point
the gutter at the place PyYAML actually blamed.
env_refs/missing_env are the side panel's "these need a value" list. A
reference counts as missing only when nothing can settle it: a ${VAR…}
clause of any kind does, and a bare $VAR/${VAR} is settled by the stack's
.env, which is what env_keys is the editor's view of. $$ is an escaped
dollar sign and never a reference.
Phase QA-19.
compose's parse errors quote the line they could not read, of whatever file that was. _check_resolved passed that text back as the task result; it now logs it on the host and returns a generic refusal, as _hash_report does.
Creating or editing a compose stack happened in a 760px modal, which is a cramped place to write a YAML file. The editor is now a page of its own, in the layout the user picked: a sticky bar (breadcrumb, status pill, take down/deploy/save), a highlighted YAML editor with a line-number gutter, and a side column for the masked .env and the revisions. Live validation is the reason for the page: 400ms after the last keystroke the file goes to QA-19's /api/v1/stacks/validate/, and the answer becomes the status pill, the service chips, the gutter's red line, and an "Add VAR" button per variable the file references but the .env does not carry. Responses are sequenced so a stale answer cannot overwrite a newer one. Because the page can now be left — it is navigation, not a dialog — unsaved edits are tracked and Back, Escape and any navigateTo out of the page ask first, and Deploy is disabled until a save: deploy ships a revision, not the bytes on screen. Take down asks twice, the second time about the folder on the host, matching the server's rule that adopted stacks keep their files.
A rollout walked one ladder. Starting Servers, Desktops and Labs side by side took three requests and three TOTP codes, and a machine tagged into two ladders (a shared Canary wave) was patched once per ladder. POST /api/v1/rollouts/ with wave_group_tags (2-12) starts one ordinary rollout per group under a shared PatchRollout.batch id, all at once, inside one transaction. Every machine goes to exactly one of them — the first group listed that reaches it — so parallel ladders never send the same task twice. An unknown group refuses the whole batch before anything starts; a group the split leaves empty is skipped. Each rollout then runs, gates and halts exactly as a single one does. POST /api/v1/rollouts/batch/<batch>/halt/ (admin + one TOTP code) halts every active rollout in the batch: codes are single-use, so halting each group separately would need a fresh code per group. Phase QA-21.
… lanes The start dialog picks wave groups from chips instead of a typed tag: none is every enabled wave, one is that ladder, two or more start a parallel batch. A picked chip shows its position, because pick order decides which group gets a machine that sits on two ladders. Rollouts that share a batch are one card with a lane per group, side by side — 1 2 | 1 2 | 1 2 3 — each lane listing its waves with its own Halt/Resume, and a Halt all that stops the batch with one TOTP code. Lanes are a tinted surface, not a striped edge, and stack under 640px. Phase QA-22.
…e process The parallel runner failed now and then, and hid which test failed behind "cannot pickle 'traceback' object". Two pieces of per-process state leaked between tests in a worker: - SetupRedirectMiddleware sends every page to /setup/ until it has seen an account, then remembers that for the process. Tests that created no user passed only when an earlier test had already set the flag. Found all of them by running the suite with the flag uncached: the Civil SSO disabled and login-start tests and three public status page tests. Each now creates an account, as any installed Vigil has. - The license state is cached in-process; the lapse test now reloads it instead of inheriting a license an earlier test installed. Six consecutive parallel runs pass.
…s exit code The Containers page showed only "exited", so a clean stop and a crash looked the same, stacks had no state of their own, and a stack Vigil manages vanished from the list once it had no containers. - A container reads "stopped (exit 0)" in neutral grey or "crashed (exit 137)" in rose, from Docker's status text, which is also its tooltip. - Each stack header carries one state: running, stopped (exit 0), crashed (exit N), or "1/2 running · 1 crashed (exit 137)". - A managed or adopted stack with no containers left stays listed as down. - The State column is wide enough to hold those on one line.
For teams that build their images rather than publish them. A stack can now take its compose file from https or ssh Git: track a branch (each deploy takes its latest commit, and every revision records the commit it came from) or pin an exact commit or tag. The repository is the source of truth for the compose file; the .env is still edited in Vigil. The server, never a host, talks to the Git server, so a repository credential stays in Vigil (GitCredential: HTTPS token or SSH deploy key, encrypted, write-only, TOTP to add). apps/stacks/gitsource.py: - resolves the branch/pin to one commit, fetches only that commit into a bare repo (no checkout, hooks or filters) and takes `git archive` of the compose file's folder, repacked as a deterministic tar.gz of regular files only — a symlink or special file is refused, not skipped; - allows https and ssh URLs only (no file://, ext::, local paths, embedded passwords or leading '-'), sets GIT_ALLOW_PROTOCOL, ignores user and system git config, follows no HTTP redirects, refuses hosts resolving to loopback / link-local / unspecified addresses (a LAN Git server is fine); - sends a token through GIT_ASKPASS from the environment — never a command line, which every local user can read — and uses an SSH key only with pinned host keys (StrictHostKeyChecking=yes, no trust on first use); - caps time, files and size. Every fetched compose file passes validate_compose like any other. A deploy carries a one-time SourceTicket and the archive's sha256 in the signed task (stack_deploy source_ticket / source_sha256 / source_commit), and is refused for an agent that does not advertise `stack_source`. The server image gains git and openssh-client. Fetched for real from github.com/docker/awesome-compose in testing.
…ollowing links stack_deploy with source_ticket / source_sha256 (advertised as the `stack_source` feature) redeems the ticket for the packed folder, refuses it unless its sha256 is the one in the signed task, and accepts regular files at plain relative paths only. The files are written as root into the stack folder through directory file descriptors: each folder opened relative to the last with O_NOFOLLOW, each file replaced with unlink + O_EXCL|O_NOFOLLOW — a link planted by a container or a user cannot redirect a write. Files an earlier commit placed and this one dropped are removed, known from a manifest in the agent's data dir; data folders and the .env are never touched. The checked compose text is then written over the repo's copy, and the resolved-config check and `up` run as for any stack. Verified on the Ubuntu test VM: docker/awesome-compose's nginx-golang placed, checked, built from its source and started; a wrong hash refused.
…s to the checked address, and fetches are capped on disk Review of the Git stack fetcher: - Exfiltration: any admin session could point a stack at a server it runs and pick an existing token, and git would send it there. A credential now names its Git server (git_host, required) and is refused for any other — checked when it is attached and again at every fetch — and attaching one to a stack asks for TOTP. - DNS rebinding: the host's addresses were checked, then git looked the name up again. git is now held to the checked address (http.curloptResolve for https; HostName + HostKeyAlias for ssh, so the TLS certificate and the pinned host key are still verified for the real name). - Resource caps: git's output was read whole into memory before the size check, and nothing bounded what a fetch wrote to disk. git now runs under `prlimit --fsize` (512 MB per file: its pack and its captured output), output goes to files and is read back only within the limit. The server image installs util-linux for prlimit. Real fetch from github.com still works with the address pinned.
…t a time The per-file size cap (prlimit --fsize) did not bound a fetch that git unpacked into loose objects — one small file per object, each under the cap, as many as the commit holds. fetch.unpackLimit / transfer.unpackLimit = 1 keep it a single pack, which the cap does bound (tested with 120 small objects under a 64 KB cap). A file lock also serialises fetches across server workers, so a burst of requests cannot multiply the disk they use; a second fetch waits up to 30 s, then says another is running.
The one-at-a-time lock from 3926672 let a deliberately slow repository hold it for up to 120 s per git command — several minutes per fetch — blocking every other Git stack's pull and deploy; and its lock file sat in the shared /tmp, where anyone could create it first and hold it for ever. - A whole fetch now has one 180 s deadline; each git command gets only what is left of it, and none starts after it has passed. - Two fetch slots instead of one: a slow repository occupies one, the other keeps working. Still few, so the size caps bound the disk. - The slot lock files live in a 0700 folder owned by the server's user, checked before use.
… from Git QA-23. The server has taken a compose file from a repository since 2f6b00a; the editor had no way to say "this one comes from Git", so the merged half was unreachable from the UI. A new stack chooses its source up front. For a Git stack the compose file is read-only — the repository is the source of truth — and the fields that aim the stack at a repo (url, branch, pin, path, credential) are editable afterwards, a change taking effect by pulling. The .env panel is unchanged: that stays Vigil's, which is why a Git stack's save still PUTs {env, note}. A credential is added from the same select, in a modal, and its secret is only ever sent — never interpolated into markup, cleared on close and after save either way. Adding one asks for TOTP because it is a key to a repository whose contents run as root on a host.
autocomplete=off on the SSH private-key textarea, matching the new-password token field: a password manager offering to fill a pasted key into a form it cannot read back is only a way to leak it.
…, with the .env sent alongside
… use one, and reads right in dark mode The credential form sends git_host (prefilled from the repository URL) and keeps typed fields across a kind switch; saving with a credential asks for TOTP; the commit line no longer overwrites its own commit; only the current revision is marked current; selected chips use on-accent text, which also fixes the rollout group chips; an empty status pill is hidden.
… lane lists its own waves Dispatch and the batch split planned hosts across every ladder's waves, so a host matching two ladders' same-order waves was claimed by whichever sorted first: a Desktops-only rollout could skip a desktop tagged canary because the Servers ladder had a Canary rung too. Each ladder now plans from its own waves, as the wave serializer already counted them; the batch split still gives a host on two ladders to the first listed group.
… bar keeps its layout The Git fields kept the last stack's repository; an empty status pill collapsed and pulled Save to the left; a managed stack chip's revision sat flush against its name.
…ave groups, sessions and the hardening Everything the release added was missing from the wiki. Written up from the code's facts only: container states and the stack editor, stacks sourced from Git (with the guarantee stated where it lives — the host's resolved-config check, not the server's editor check), several wave groups at once, sign-in session limits, the agent/task-signing and installer hardening, the two session settings, validate_community, and the stacks/rollouts/session endpoints.
Words instead: Yes / Partly in the Podman and editions tables; the warnings read as plain sentences.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Portainer runs compose inside its own container, so the paths compose writes
into the labels (/data/compose/49/docker-compose.yml) are Portainer's, not the
host's, and adopting such a stack failed with "No such file or directory".
A label path that does not exist on the host is now mapped through the mount
of the container that holds it (longest destination wins), and used only if
the mapped file exists. The env file compose recorded (Portainer's stack.env)
is passed with --env-file, so a re-up from the host interpolates the same
values instead of blanks, and adopt reads the env from it.