Skip to content

setup.md

Latest commit

 

History

History
2371 lines (1962 loc) · 109 KB

setup.md

File metadata and controls

2371 lines (1962 loc) · 109 KB
Raw

Set up nodkey

This guide sets up nodkey from the start: the broker on a trusted machine, a Telegram bot (or an ntfy server) for the approvals, the secret stores, and the client on the machine where the AI agent works.

DESIGN.md says how nodkey works, and THREAT-MODEL.md says what it protects against. Please read section 5 of the threat model before you rely on nodkey.

The examples use these names. Use your own names instead.

name what it is
broker-host the trusted machine that runs the broker
192.0.2.10 the address of the broker host on your private network
laptop the agent machine, and the name of its client key in the broker config
web-1, web-2 SSH hosts that the agent may log in to
deploy the remote user on these hosts, and the name of the broker's SSH key
github-token a secret that the agent may ask for
123456789 the Telegram user ID of the approver: you

Run the commands for the broker host as root, for example in a shell from sudo -i, on its console or from a trusted computer (section 1). Run the commands for the agent machine as the user who runs the agent. Only the install of nodkey is different: run it as root on each host, also on the agent machines, from a root shell that does not come through the agent's account (section 2).

Contents

  1. Before you start
  2. Get nodkey
  3. The broker host
  4. The Telegram bot
  5. Stores
  6. SSH keys and the broker's known_hosts
  7. The config file
  8. The systemd unit
  9. Account and host safety
  10. Start the broker
  11. A client on the agent machine
  12. Check the audit log
  13. Upgrades and key changes
  14. When something does not work

1. Before you start

You need:

  • A broker host. A Linux machine that is always on, with systemd 250 or newer (systemctl --version; for example Debian 12, or Ubuntu 24.04) and cgroup v2. No AI agent runs on it, and only you have root there. It uses no swap, or only encrypted swap, and no hibernation (section 9). If it has a TPM2 chip, the keys of nodkey can be bound to it.
  • One or more agent machines. The Linux machines where the agent works. For SSH logins, they need OpenSSH 8.9 or newer (ssh -V).
  • A private network between them, such as your LAN or a VPN, and a fixed address of the broker host on it. The clients connect to one TCP port of the broker: 7743 in the examples.
  • Telegram on your phone. This Telegram account approves the requests.
  • A store for the secret values: Bitwarden Secrets Manager (bws), a separate account of the Bitwarden Password Manager (bw), a vault of 1Password (1password), or a project of Infisical (infisical). If you use nodkey only for SSH logins, you need no store.

The broker host is the root of trust. Root on it, or a copy of its memory, gives every key and token (THREAT-MODEL.md section 5, item 5). DESIGN.md section 4 shows how the parts fit together.

Never log in to the approving Telegram account on a machine where an agent runs. Whoever can use that account can approve (section 9).

On a machine where an agent runs, never log in to a Bitwarden account that can read the secrets. Whoever can use such an account can read them without nodkey (section 9).

Where you do the setup

The agent machine is not trusted. The agent may read its clipboard, its terminals, its browser and its files. A token that passes through it may be the agent's from then on. With the bws token alone, the agent could read every secret of its project, without a tap (THREAT-MODEL.md section 5, item 8). So:

  • Do the setup of the broker host on its console, or from a trusted computer where no agent runs. Do not log in to the broker host from an agent machine.
  • Make and copy tokens and passwords only on that trusted computer, or on your phone: the bot token, the bws access token, the master password of each Bitwarden account that can read the secrets, and the API key of a bw account. Never let them pass through an agent machine: not its clipboard, not its terminal, not its browser. Do not save them in a password manager that is open on an agent machine.
  • Keep no way to get root on the broker host on an agent machine: no SSH key for the broker host, no SSH agent or ControlMaster socket that reaches it, no saved password.
  • The user of the agent never downloads, builds or copies the nodkey binary, on any machine: the agent could change the file before it is installed. Install nodkey as root on each host, with the installer. Or copy it straight from a trusted machine where you built or downloaded it, not through an agent machine, and check its SHA-256 on the broker host (section 2).

2. Get nodkey

Install the same version of nodkey on the broker host and on each agent machine, as /usr/local/bin/nodkey, owned by root. When the versions differ, nodkey says so (section 13). Do it as root on each host, never as the user of the agent. The agent can change what its own user downloads, builds or copies (section 1). Get the root shell in a way that does not go through the agent's account:

  • On the broker host: on its console, or from a trusted computer, as for the rest of the setup. Never set it up or update it through an agent machine.
  • On an agent machine: on its console, or with SSH as an admin user from a trusted computer. If you can, do not type sudo in the terminal where the agent works, or in the agent's account. The agent can put its own sudo there (a program of that name earlier in PATH, an alias or a function), or type into that terminal.

This has a limit. On an agent machine that the agent controls, nodkey cannot keep its own client safe. But nodkey's promises come from the broker, which trusts nothing from the client: a changed client binary gets no more than what you approve on the phone (THREAT-MODEL.md T20).

There are two ways to install: the installer, or by hand.

The installer

install.sh in the repository installs the newest release, or the version of --version X.Y.Z. It needs curl, tar, gzip, sha256sum, install, mktemp, stat and uname, and no GitHub login. Before it installs anything, it checks the SHA-256 and the build attestation of the release, and that the new nodkey says it is the version that you want (DESIGN.md section 18). If a check fails, it installs nothing. It restarts nothing. As root, it also refuses a directory or a file that another user than root can change: /usr/local/bin, the old nodkey in it, and the temporary directory (TMPDIR, or /tmp).

The first time, on each host: download the script as root, read it, and run it as root. In a root shell that you got as above:

curl --proto '=https' --tlsv1.2 -fsSL -o /root/install.sh https://raw.githubusercontent.com/code6x/nodkey/main/install.sh
less /root/install.sh
sh /root/install.sh
nodkey version

On the broker host, where no agent runs, the short form works too, in the same root shell. But it runs the script before you can read it:

curl --proto '=https' --tlsv1.2 -fsSL https://raw.githubusercontent.com/code6x/nodkey/main/install.sh | sh

To check the attestation, the installer downloads gh 2.102.0, and checks it against the SHA-256 in the script. To use a gh that is installed already, 2.93.0 or newer, add --gh /usr/bin/gh. Only root may be able to change that gh and its directories: root runs it. The installer needs the Sigstore bundle of the attestation in the release, nodkey_<version>.sigstore.json. 0.1.0 has none: install 0.1.0 by hand.

When that version is installed already, the installer does nothing. It does not check the installed file then. To replace it with a checked copy, add --reinstall.

Later, nodkey update runs the copy of the installer that is built into the installed nodkey (section 13). It takes --version, --gh and --reinstall too.

By hand

Use a trusted machine for this, never an agent machine (section 1). Get nodkey from a release, or from the source.

From a release. Download the tarball for your machine, nodkey_<version>_linux_amd64.tar.gz (nodkey_<version>_linux_arm64.tar.gz on 64-bit ARM), SHA256SUMS, and the bundle nodkey_<version>.sigstore.json if the release has one, from the release page. Check it, as docs/releasing.md says, with all the steps there. From gh 2.96.0 on, gh release download needs no GitHub login. gh attestation verify needs none with --bundle, but needs one without it, such as for 0.1.0. Then take the file nodkey out of the tarball:

# the version you downloaded
VERSION=0.2.0
tar -xzf "nodkey_${VERSION}_linux_amd64.tar.gz" nodkey

From the source. Build nodkey with Go 1.26 or newer. Never build it on an agent machine: the agent could change the build.

git clone https://github.com/code6x/nodkey.git
cd nodkey
CGO_ENABLED=0 go build -trimpath -ldflags "-X main.version=$(git rev-parse --short HEAD)" -o nodkey ./cmd/nodkey

The -ldflags part is optional. It puts the commit into nodkey version and into the broker_started entry of the audit log.

Then, for both ways: copy the file nodkey straight from the trusted machine to each machine, never through an agent machine (section 1). On the broker host, check that it is the same file: sha256sum nodkey must print the same SHA-256 as on the trusted machine. Then install it, and check it:

install -o root -g root -m 0755 nodkey /usr/local/bin/nodkey
nodkey version

On a terminal, nodkey version of a release also asks GitHub whether a newer release is out, and says so in one more line. --offline skips that.

3. The broker host

The user and the directories

The broker runs as a system user of its own, nodkey:

useradd --system --home-dir /var/lib/nodkey --shell /usr/sbin/nologin nodkey
install -d -o root -g root -m 0755 /etc/nodkey
install -d -o root -g root -m 0700 /etc/nodkey/credentials /run/nodkey-setup
  • /etc/nodkey holds the config file and the broker's known_hosts. Only root may change them. The broker only reads them.
  • /etc/nodkey/credentials holds the encrypted credentials. Only root reads them: systemd opens them for the broker at each start.
  • /run/nodkey-setup is a work directory in memory: /run is a tmpfs. Plain keys and tokens stay there only for a moment. They do not reach the disk, unless the machine moves them to swap. So before you make any key, make sure that the machine has no swap, or only encrypted swap (section 9). The directory is gone after a reboot; then make it again with the same command. Use it only for secrets: /run is small (on Debian, 10 % of the RAM by default), so large files that hold no secret, such as the downloads of bws and bw in section 5, go to a directory on disk.
  • /var/lib/nodkey is the state directory: the audit log, and a home directory for each store. systemd makes it at the first start (StateDirectory= in the unit). For some store steps, you make it earlier by hand (section 5).

Encrypted credentials

Every key, token and password of the broker is a systemd credential: a file that systemd-creds encrypted. At each start, systemd decrypts the credentials and gives them to the broker in $CREDENTIALS_DIRECTORY, a directory in memory that only the broker can read. The config names each one as ${CREDENTIALS_DIRECTORY}/NAME (DESIGN.md section 11).

Check whether the machine has a TPM2 chip that systemd can use:

systemd-creds has-tpm2

If the first line of the output is yes, bind the credentials to the TPM2 chip and to this installation:

WITH_KEY=host+tpm2

Otherwise, use the key of this installation only:

WITH_KEY=host

Without the TPM2 chip, anyone with a full copy of the disk can decrypt the credentials: the key is in /var/lib/systemd/credential.secret. So can anyone with a backup that holds both that file and the credentials. With host+tpm2, they also need the TPM2 chip of this machine. On a disk without encryption, systemd-creds warns that this file "is not located on encrypted media"; that is the risk just described. Set WITH_KEY again in each new shell; the commands below use it.

Before systemd 258, host+tpm2 also binds each credential to PCR 7 of the TPM2 chip: the state of Secure Boot. A firmware update or a change of the Secure Boot settings can change PCR 7. Then no credential can be decrypted any more, and systemd does not start the broker (section 14). That fails closed, but you must make the credentials again. If you did not update the firmware or change Secure Boot, find out why before you make new credentials: someone may have changed how the machine starts. systemd 258 and newer bind to no PCR by default. To get that on an older systemd, add --tpm2-pcrs= (with nothing after =) to each systemd-creds encrypt. Then the credentials survive such updates, but the TPM2 chip no longer checks how the machine was started.

Each credential has a name. systemd-creds writes the name into the file, and systemd refuses the file under another name. The example config and the example unit use these names:

name what it holds made in
broker-tls-key the TLS key of the broker this section
telegram-token the token of the bot section 4
bws-token the access token of a Secrets Manager machine account section 5
bws-token-backup the access token of a second machine account section 5
bw-master-password the master password of a separate Bitwarden account section 5
ssh-deploy the SSH key deploy section 6

There are two ways to make a credential:

  • From a file, such as a key. Then remove the plain file:

    systemd-creds encrypt --with-key="$WITH_KEY" --name=NAME FILE /etc/nodkey/credentials/NAME.cred

  • From what you type, for a token or a password. systemd-ask-password shows a * for each character, never the text, and the value never goes to a file, a command line or the shell history. It waits at most 90 seconds. -n leaves out the newline at the end:

    systemd-ask-password -n "NAME:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=NAME - /etc/nodkey/credentials/NAME.cred

To check a credential without showing it, count its bytes:

systemd-creds decrypt --name=NAME /etc/nodkey/credentials/NAME.cred - | wc -c

The TLS key of the broker

The clients and the broker connect over mutual TLS. Each side pins the key of the other side; there is no certificate authority (DESIGN.md section 10). Make the key of the broker:

nodkey keygen --out /run/nodkey-setup/broker-tls-key

nodkey keygen prints the pin of the new key, such as sha256:Q2b9.... Write it down: each client needs it as broker_pin. The broker also logs it at each start. Now make the credential, and remove the plain key:

systemd-creds encrypt --with-key="$WITH_KEY" --name=broker-tls-key \
  /run/nodkey-setup/broker-tls-key /etc/nodkey/credentials/broker-tls-key.cred
rm /run/nodkey-setup/broker-tls-key

4. The Telegram bot

The broker sends each request to you as a Telegram message from a bot of your own. Only your taps count.

  1. Make the bot. Use Telegram on your phone, or on a trusted computer where no agent runs: never use BotFather on an agent machine. Open the chat with @BotFather and send /newbot. Choose a name and a user name for the bot. BotFather answers with the bot token. Use this bot for this one broker only: Telegram lets only one program read the answers of a bot.

  2. Start the chat. Open the new bot in Telegram, and tap Start. A bot cannot write to you before you do this.

  3. Save the token as a credential. On the broker host:

    systemd-ask-password -n "Telegram bot token:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=telegram-token - /etc/nodkey/credentials/telegram-token.cred

    Never paste the token into a chat, an issue, a file in a repository, or a command line.

  4. Find your user ID. The tap on Start sent the message /start to the bot. On the broker host, before the first start of the broker:

    # paste the token, then press Enter; it is not shown
read -rs TOKEN
printf 'url = "https://api.telegram.org/bot%s/getUpdates"\n' "$TOKEN" |
  curl -sS --config - -H 'Content-Type: application/json' \
    -d '{"allowed_updates":["message","my_chat_member"]}'
unset TOKEN

    Your user ID is the number in "from":{"id":123456789. printf is part of the shell, and curl reads the address from its input, so the token is not in the list of processes.

    Never call getUpdates while the broker runs. Telegram lets only one program read the updates of a bot. The broker would see a conflict, send an alert that the token may be stolen, and stop (exit code 76).

  5. A chat for notices (optional). A "never ask" rule (section 7) sends a silent notice for each use. By default, the notices go to your chat with the bot. To keep that chat for approvals only, make a Telegram group with only you in it, and add the bot. Nobody else may join this group: its members see the names of the secrets, the clients and the hosts. Then run the getUpdates command again. The ID of the group is the negative number in "chat":{"id":-.... It starts with -100 for a supergroup. It goes into notice_chat_id.

  6. Keep the bot out of other groups (optional). If the bot is in no group, send /setjoingroups to BotFather and choose Disable. Then nobody can add the bot to a group.

ntfy in place of the Telegram bot

The broker can send its requests through an ntfy server instead of Telegram. A broker has one channel: [telegram] or [ntfy], never both. Skip the steps above if you use ntfy.

ntfy is weaker than Telegram: it does not say who pressed a button, and the server can change a message without the broker seeing it. Read THREAT-MODEL.md T40 first. Two limits follow from it:

  • Run the ntfy server yourself. This guide does not use the public server ntfy.sh: whoever runs the server can approve.
  • No standing grants. A config with [ntfy] and [standing] does not load. Scheduled grants (nodkey preapprove) work.

The examples use the server https://ntfy.example.com, the request topic nodkey-requests and the answer topic nodkey-answers. The broker sends each request to the request topic, and your phone reads it there. A button on the phone sends its answer to the answer topic, which only the broker reads.

  1. The server. Set up ntfy on a machine that you trust (install), with https. In its server.yml:

    base-url: "https://ntfy.example.com"
auth-file: "/var/lib/ntfy/user.db"
auth-default-access: "deny-all"
upstream-base-url: "https://ntfy.sh"

    deny-all means that nobody may read or write a topic without an access rule. upstream-base-url is for iOS: without it, a message can take hours to reach an iPhone. Only the message ID and a hash of the topic address go to ntfy.sh; the phone then fetches the message from your server. Leave the line out if you use only Android.

  2. The users and the access rules. On the ntfy server (access control):

    # asks for a password
ntfy user add nodkey-broker
ntfy user add phone
ntfy access nodkey-broker nodkey-requests write-only
ntfy access nodkey-broker nodkey-answers read-only
ntfy access phone nodkey-requests read-only
ntfy access everyone nodkey-answers write-only

    The buttons send no token, so anyone may write to the answer topic. An answer counts only with the random value of its button, and only the broker can read the answers. Give no other user access to these two topics.

  3. The token of the broker. On the ntfy server:

    ntfy token add --label="nodkey broker" nodkey-broker

    On the broker host, save the token as a credential:

    systemd-ask-password -n "ntfy token:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=ntfy-token - /etc/nodkey/credentials/ntfy-token.cred

    Never paste the token into a chat, an issue, a file in a repository, or a command line. The example unit already has the line LoadCredentialEncrypted=ntfy-token:; remove the line of telegram-token (section 8).

  4. The config. In place of [telegram] (section 7):

    [ntfy]
server = "https://ntfy.example.com"
request_topic = "nodkey-requests"   # the phone subscribes here
answer_topic = "nodkey-answers"     # the buttons answer here; only the broker reads it
token_file = "${CREDENTIALS_DIRECTORY}/ntfy-token"
timeout = "120s"
ssh_timeout = "90s"

    timeout and ssh_timeout are as in [telegram]. There is no notice_chat_id: notices go to the request topic too, with a low priority.

  5. The phone. Install the ntfy app (Android or iOS). In its settings, add the user phone with its password for https://ntfy.example.com. Then subscribe to nodkey-requests on that server. On Android, turn on instant delivery for the subscription. Never subscribe to nodkey-answers.

  6. Keep the request topic off agent machines. Anyone who can read the request topic can approve. Never subscribe to it on a machine where an agent runs: not in the ntfy web app, not in a desktop client, not with curl. Never log in as phone or nodkey-broker there.

A new token. Make a new one with ntfy token add, make the credential again as in step 3, and restart the broker. Then remove the old one with ntfy token remove nodkey-broker <old token>. When the server refuses the token, the broker stops with exit code 77, and systemd does not start it again until you do.

5. Stores

For each value, the broker runs the program of the store: bws or bw (DESIGN.md section 16). The program gets a token or the master password from the broker, so it must be the real program, and nobody else may change it. A 1password store runs no program: see 1password. Nor does an infisical store: see infisical.

Where bws and bw come from

  • Take them only from the official releases: bws from the bws-v... releases of bitwarden/sdk-sm, and the standalone bw from the cli-v... releases of bitwarden/clients. The broker also accepts bw from npm (@bitwarden/cli), but see below. Check each download.
  • Install them owned by root, with mode 0755, in a directory that only root can write to, such as /usr/local/bin. The broker checks this at start: the program and every directory on the way to it must be owned by root (or by nodkey), and group and others must not be able to write to them.
  • A bw from npm is a Node.js script. The broker runs bw with PATH=/usr/local/bin:/usr/bin:/bin, and the script starts the node that it finds there. That node must be owned by root, below directories that only root can write to, also after its links. The broker checks this at start. A node from nvm in a home directory is not found there.
  • There must be no directory named bw-data next to bw, next to the file that bw links to, or next to that node. With it, bw keeps its login and its copy of the vault there, not in the home directory of the store, so the broker refuses to start.

An example with bws 2.1.0 on x86-64 (take the newest release, and aarch64 instead of x86_64 on 64-bit ARM). The checksum file of the release lists the SHA-256 of each file. The downloads hold no secret, so they do not need /run/nodkey-setup, and they may not fit there: the standalone bw zip is about 46 MB, and the unpacked bw about 140 MB. So unpack them in a private directory on disk, and remove it after the install:

install -d -o root -g root -m 0700 /root/bw-setup
cd /root/bw-setup
curl -fLO https://github.com/bitwarden/sdk-sm/releases/download/bws-v2.1.0/bws-x86_64-unknown-linux-gnu-2.1.0.zip
curl -fLO https://github.com/bitwarden/sdk-sm/releases/download/bws-v2.1.0/bws-sha256-checksums-2.1.0.txt
sha256sum -c --ignore-missing bws-sha256-checksums-2.1.0.txt
unzip bws-x86_64-unknown-linux-gnu-2.1.0.zip bws
install -o root -g root -m 0755 bws /usr/local/bin/bws
cd / && rm -r /root/bw-setup

sha256sum must print OK for the zip file. An example with the standalone bw 2026.9.1 (bw-linux-arm64-... on 64-bit ARM). Its release has no checksum file, so compare the SHA-256 with the sha256: value that GitHub shows next to the file on the release page:

install -d -o root -g root -m 0700 /root/bw-setup
cd /root/bw-setup
curl -fLO https://github.com/bitwarden/clients/releases/download/cli-v2026.9.1/bw-linux-2026.9.1.zip
sha256sum bw-linux-2026.9.1.zip
unzip bw-linux-2026.9.1.zip bw
install -o root -g root -m 0755 bw /usr/local/bin/bw
cd / && rm -r /root/bw-setup

With gh, this prints the value from the release page:

gh release view cli-v2026.9.1 --repo bitwarden/clients --json assets \
  --jq '.assets[] | select(.name == "bw-linux-2026.9.1.zip") | .digest'

Prefer the standalone bw. npm install --global runs the install scripts of every package in the tree as root, on the most trusted machine that you have. If you still want the bw from npm, install the node of your distribution (in /usr/bin), and an exact version of the package, as root:

npm install --global @bitwarden/cli@2026.9.1
ls -l /usr/local/bin/bw "$(readlink -f /usr/local/bin/bw)" "$(command -v node)"

All of them must be owned by root. A bad version of @bitwarden/cli was on npm for about 90 minutes on 2026-04-22 (THREAT-MODEL.md T20). So choose the version, and read the release notes before an upgrade.

bws: Bitwarden Secrets Manager

Use a Bitwarden account that is never logged in on an agent machine: no web app in a browser, no browser extension, no desktop app and no CLI there. Every owner and admin of the organization can read every secret in Secrets Manager, and so can each member that you give access to the project (THREAT-MODEL.md T32). If your main Bitwarden account is logged in on an agent machine, make the organization for Secrets Manager with a separate account, and do not add your main account to it.

Bitwarden offers Secrets Manager for Free, Teams and Enterprise organizations. Its Free plan is for up to 2 users, 3 projects and 3 machine accounts. With a Families plan, Bitwarden says to make a new Free organization for Secrets Manager (Secrets Manager plans, seen 2026-10-16).

In the web app of Secrets Manager, with that account, on a trusted computer (section 1):

  1. Make a project for the secrets that nodkey may hand out. Put these secrets into it.
  2. Make a machine account. Give it read access to this project only.
  3. Make an access token for the machine account. It is shown only once.
  4. Copy the ID of each secret, a UUID. It goes into id in the config. nodkey hands out the value of the secret.

On the broker host, make the credential:

systemd-ask-password -n "bws access token:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=bws-token - /etc/nodkey/credentials/bws-token.cred

bws logs in with the token at each run, so there is no login step. Its state is in the home directory of the store, /var/lib/nodkey/stores/NAME, where NAME is the name of the [store.NAME] section: main in the example.

Bitwarden's EU cloud or your own server. Set the server once for each bws store, as the broker runs bws: as the user nodkey, with only HOME in the environment, in the home directory of the store. Do this while the broker is stopped. umask 077 makes the new files private, as UMask=0077 does for the broker. Without it, the files get the umask of your shell, often 022. For the store main:

(
umask 077
install -d -o nodkey -g nodkey -m 0700 /var/lib/nodkey /var/lib/nodkey/stores /var/lib/nodkey/stores/main
sudo -u nodkey env -i -C /var/lib/nodkey/stores/main HOME=/var/lib/nodkey/stores/main \
  /usr/local/bin/bws config server-base https://vault.bitwarden.eu
)

The parentheses keep umask 077 inside this one block, so later files, such as the unit, get their normal mode.

bws answers Profile updated successfully!.

A second machine account. A job with a "never ask" rule (DESIGN.md section 12) should get its secret through a machine account of its own. Put the secret into a project of its own, and give the new machine account access to that project only. Then give it a store section of its own: in the example config, [store.backup-job] with the credential bws-token-backup. Now even a mistake in the config cannot hand out another secret through that store. Its token can read only that one project.

1password: a vault only for nodkey

The 1password store uses the official 1Password SDK inside the broker. There is no program to install. The SDK is version 0.x and carries a WebAssembly core of about 9.5 MB, so the nodkey binary is about 34 MB (DESIGN.md section 4 has the details). The core may reach only the hosts of 1Password. It uses Go's HTTP client, so HTTPS_PROXY applies.

Use a vault only for nodkey. 1Password does not let a service account read the built-in Personal, Private or Employee vault, or the default Shared vault (service accounts, seen 2026-10-29). Everyone who can open the vault in 1Password's own apps reads its items with no tap, so give it to no more people than needed (section 9).

In 1Password, on a trusted computer (section 1):

  1. Make a vault for nodkey. Put into it only the items that nodkey may hand out.
  2. Make a service account. Give it the permission read_items only, on that vault only, and set an expiry. These permissions cannot change later. If you need a change, make a new service account and a new token, then revoke the old one.
  3. Copy the token. It is shown only once.
  4. Copy the ID of the vault and the ID of each item. An ID is 26 small letters and digits. A name does not work. The vault goes into vault, and the item into id, in the config.

On the broker host, make the credential, as for bws:

systemd-ask-password -n "1Password service account token:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=op-token - /etc/nodkey/credentials/op-token.cred

Add LoadCredentialEncrypted=op-token:/etc/nodkey/credentials/op-token.cred to the unit (section 8). The config (section 7) is:

[store.op]
type = "1password"
vault = "VAULT_ID"
token_file = "${CREDENTIALS_DIRECTORY}/op-token"

[secret.github-token]
store = "op"
id = "ITEM_ID"
field = "password"   # optional

A 1password store has no program; that is an error. field is the ID or the title of a field of the item, and the default is password. If two fields have the same title, the broker denies the request: name the field by its ID. A value that is empty is denied too. The broker hands out the value of that one field. An item over 1 MiB is denied.

The broker logs in at start. If the login fails, the broker still starts, and the next request tries again. When 1Password refuses a request, or cannot be reached, the error is always the same text, whatever the cause: a missing item, a wrong or expired token, or the network. The broker never shows the text of the SDK, because it can repeat the token. Check the vault ID, the item ID and the expiry of the token first.

Rate limits. 1Password limits the requests of a service account (seen 2026-10-29, rate limits):

plan limit
Individual and Families 1,000 requests a day for the whole account, and 1,000 reads an hour for each token
Teams 5,000 requests a day
Business 50,000 requests a day

At the limit, the broker denies the request, and the log says why. The broker also sends one alert to your phone for each store in each hour. One fetch can cost more than one request. To see how many, run op service-account ratelimit before and after a fetch (docs/manual-telegram-test.md, check 16).

infisical: a project only for nodkey

The infisical store uses the official Infisical SDK inside the broker. There is no program to install. The SDK adds about 5 MB to the nodkey binary (DESIGN.md section 4 has the details). It works with the Infisical cloud (US or EU) and with your own server. The broker logs in with a machine identity (Universal Auth) and keeps the access token only in memory.

Use a project only for nodkey. Put into it only the secrets that nodkey may hand out. Everyone who can read the project in Infisical's own apps reads them with no tap, and everyone who can write to it can change what a name holds (section 9).

In Infisical, on a trusted computer (section 1):

  1. Make a project for nodkey. Put into it only the secrets that nodkey may hand out.
  2. Make a machine identity with Universal Auth. Give it the built-in Viewer role on the project: read-only, and it covers every environment of the project. A custom role for one environment or one path, and trusted IPs, need a paid plan (seen 2026-10-29). nodkey works with both.
  3. Set a short access token TTL for the identity.
  4. Make a client secret. Set a TTL on it: by default it never ends. Copy it. It is shown only once.
  5. Copy the client ID of the identity, the ID of the project, the slug of the environment, the path of the folder and the name of each secret. The IDs are UUIDs.

On the broker host, make the credential, as for bws:

systemd-ask-password -n "Infisical client secret:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=infisical-secret - /etc/nodkey/credentials/infisical-secret.cred

Add LoadCredentialEncrypted=infisical-secret:/etc/nodkey/credentials/infisical-secret.cred to the unit (section 8). The config (section 7) is:

[store.inf]
type = "infisical"
server = "https://app.infisical.com"   # or the EU cloud, or your own server
client_id = "CLIENT_ID"
client_secret_file = "${CREDENTIALS_DIRECTORY}/infisical-secret"

[secret.ci-token]
store = "inf"
project = "PROJECT_ID"
environment = "prod"
path = "/"
name = "CI_TOKEN"

An infisical store has no program; that is an error. All four keys of the secret are required. path has no default, so that a secret of the same name in another folder is never picked by mistake. A value that is empty, or that has ${ (a reference to another secret), is denied: the tap must show every secret that goes out. An answer over 1 MiB is denied.

The broker logs in at start. If the login fails, the broker still starts, and the next request tries again. It logs in again when the access token ends. The error texts are fixed, and they never show the client secret, the access token or the text of the server. If a request is denied, check the IDs, the environment, the path, the name, the Viewer role and the TTL of the client secret first. When Infisical limits the requests (a 429), the broker denies the request, and sends one alert to your phone for each store in each hour.

Without trusted IPs, the client secret gets an access token from anywhere. So keep it on the broker host only.

bw: a separate Bitwarden account

The Password Manager CLI cannot unlock the vault without the master password, so the broker must keep the password. Anyone who breaks into the broker host gets the whole vault (DESIGN.md section 11, THREAT-MODEL.md T24). So use bw only with a separate Bitwarden account that holds nothing but the items that nodkey may hand out.

Never log in to this account on an agent machine: no web app in a browser, no browser extension, no desktop app and no CLI there. Whoever can use the account can read every item in it, without nodkey (THREAT-MODEL.md T32). Share its items with no other account, and give no account emergency access to it: such an account could read them too.

  1. Make the account, on a trusted computer (section 1). Give it a long random master password, for example from openssl rand -base64 24.

  2. Add the items. Each secret is a login item, and nodkey hands out its password. nodkey refuses an item of another type, and an item in the trash or in the archive. Note the ID of each item, a UUID. One way: on a trusted computer where you use bw with this account, bw get item "NAME OF THE ITEM" | jq -r .id prints only the ID.

  3. Check the KDF settings. The KDF settings are in the security settings of the account in the web vault. An unlock computes the key of the vault from the master password with them, and the broker gives each run of bw 30 seconds. Start with the default settings, and check the unlock after the first start (step 7). If it takes too long, such as on a small ARM board, lower them. With a long random master password, lower settings are still safe.

  4. Make the credential of the master password:

    systemd-ask-password -n "bw master password:" |
  systemd-creds encrypt --with-key="$WITH_KEY" --name=bw-master-password - /etc/nodkey/credentials/bw-master-password.cred

    bw reads only the first line of this file, so the password must be one line.

  5. Log in once, with exactly the environment of the broker. The broker runs bw as the user nodkey, with only HOME and a fixed PATH in the environment, in the home directory of the store. Run the login and the server setup in the same way, or bw keeps the login in another place (DESIGN.md section 16). Do this while the broker is stopped, on the console of the broker host or from a trusted computer. umask 077 keeps the new files private, as for bws. For the store vault:

    # new files private; run `umask 022` before you copy the unit (section 8)
umask 077
install -d -o nodkey -g nodkey -m 0700 /var/lib/nodkey /var/lib/nodkey/stores /var/lib/nodkey/stores/vault
# Only for Bitwarden's EU cloud or your own server:
sudo -u nodkey env -i -C /var/lib/nodkey/stores/vault HOME=/var/lib/nodkey/stores/vault \
  PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/bw config server https://vault.bitwarden.eu
sudo -u nodkey env -i -C /var/lib/nodkey/stores/vault HOME=/var/lib/nodkey/stores/vault \
  PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/bw login --apikey
sudo -u nodkey env -i -C /var/lib/nodkey/stores/vault HOME=/var/lib/nodkey/stores/vault \
  PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/bw status

    login --apikey asks for the personal API key of the account, client_id and client_secret. You find it in the security settings of the account in the web vault. bw shows the client_secret on the screen while you paste it. clear does not remove it from the scrollback of tmux or screen, or from a log of the terminal. So log in on the console of the broker host, or clear the scrollback too, for example with tmux clear-history. Then bw status must show the separate account in userEmail, and "status":"locked". Do not unlock: the broker does it.

  6. Never run bw lock, bw unlock or bw logout by hand in the home directory of a store. Each unlock makes a new session key, and ends the key of the broker. A lock locks the vault for the broker too. Then the next request is denied, and the broker unlocks again. After a logout, every request for that store is denied until you log in again as in step 5.

  7. Check the unlock after the first start (section 10). The journal must show a store started with store=vault. If it says that a store could not start, and the error has bw unlock and took too long, the unlock took more than 30 seconds: lower the KDF settings (step 3).

6. SSH keys and the broker's known_hosts

Skip this section if you use nodkey only for secrets. The broker still needs a known_hosts file then. Make an empty one:

install -o root -g root -m 0644 /dev/null /etc/nodkey/known_hosts

The SSH keys of the broker

Make a new key on the broker host for each purpose. Do not use a key that was ever on an agent machine: the agent may have a copy (THREAT-MODEL.md section 5, item 8). The broker refuses a key with a passphrase; the encrypted credential protects the key instead. It takes Ed25519, ECDSA (P-256, P-384 and P-521) and RSA keys with at least 2048 bits.

ssh-keygen -t ed25519 -N '' -C nodkey-deploy -f /run/nodkey-setup/ssh-deploy
systemd-creds encrypt --with-key="$WITH_KEY" --name=ssh-deploy \
  /run/nodkey-setup/ssh-deploy /etc/nodkey/credentials/ssh-deploy.cred
install -o root -g root -m 0644 /run/nodkey-setup/ssh-deploy.pub /etc/nodkey/ssh-deploy.pub
rm /run/nodkey-setup/ssh-deploy /run/nodkey-setup/ssh-deploy.pub

Add the public key, /etc/nodkey/ssh-deploy.pub, to ~deploy/.ssh/authorized_keys on each target host. Manage authorized_keys with your configuration management, so that a key that the agent adds there is removed again (THREAT-MODEL.md section 5, item 2).

The broker's known_hosts

The broker finds the name of a host in its own known_hosts, never on the agent machine. The name is what the message shows, so the file has stricter rules than the known_hosts of OpenSSH (DESIGN.md section 14):

  • one plain host name on each line: no list of names, no wildcards, no hashed names, no [host]:port, and no @cert-authority or @revoked;
  • plain host keys only, no certificates;
  • at most one key of each key type for a name. A name may be on several lines, one for each key type;
  • each host key only once. Two hosts with the same host key, such as cloned machines, are refused. Give each clone new host keys.

The name is the name in the messages, in the hosts lists of the config, and in nodkey ssh-grant --host. It does not need to be the DNS name.

Get the host keys in a safe way. Never copy them from the known_hosts of an agent machine: the agent can change that file.

  1. On each target host, list the fingerprints of its host keys. Use the console of the host, or an SSH session that you already trust:

    for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

  2. On the broker host, fetch the host keys, and give each line the plain name. -p gives the port, if it is not 22:

    ssh-keyscan -t ed25519,ecdsa,rsa web-1.example.com 2>/dev/null |
  awk '$1 !~ /^#/ {print "web-1", $2, $3}' >> /run/nodkey-setup/known_hosts
ssh-keyscan -t ed25519,ecdsa,rsa -p 2222 web-2.example.com 2>/dev/null |
  awk '$1 !~ /^#/ {print "web-2", $2, $3}' >> /run/nodkey-setup/known_hosts

    ssh-keyscan also prints a comment line for each host, such as # web-1.example.com:22 SSH-2.0-OpenSSH_9.8. Since OpenSSH 9.8 it prints these lines on stdout, not on stderr (commit 1e84d0c), so 2>/dev/null does not remove them. (The message of that commit says the opposite, but its code moves the lines to stdout.) The awk drops every line that starts with #, so the file gets only key lines with each version.

  3. Show the fingerprints of the fetched keys, and compare them with the list from step 1. Each one must be the same. If one differs, stop: something between the broker host and that host may change the keys.

    ssh-keygen -lf /run/nodkey-setup/known_hosts

    ssh-keygen -lf skips a line that is not a key, and says nothing. So also check that the file has one line for each fingerprint. The two numbers must be the same:

    wc -l < /run/nodkey-setup/known_hosts
ssh-keygen -lf /run/nodkey-setup/known_hosts | wc -l

    If the first number is bigger, a line is not a key. This command shows each such line with its number. Remove them, and run the check again:

    awk '$2 !~ /^(ssh-|ecdsa-|sk-)/ {print NR ": " $0}' /run/nodkey-setup/known_hosts

  4. Install the file:

    install -o root -g root -m 0644 /run/nodkey-setup/known_hosts /etc/nodkey/known_hosts
rm /run/nodkey-setup/known_hosts

Keep all host keys of a host in the file: ssh may use any of them. When a host gets a new host key, check its fingerprint in the same way, change the file, and reload the config: nodkey broker reload, as root (section 10).

7. The config file

The config file is /etc/nodkey/broker.toml. A complete example, the same as in DESIGN.md section 14:

listen = "192.0.2.10:7743"          # an address on your private network only
tls_key = "${CREDENTIALS_DIRECTORY}/broker-tls-key"
audit_log = "/var/lib/nodkey/audit.jsonl"
known_hosts = "/etc/nodkey/known_hosts"

[telegram]
token_file = "${CREDENTIALS_DIRECTORY}/telegram-token"
user_id = 123456789                 # the approver
notice_chat_id = 123456789          # optional; default: the approver's chat
timeout = "120s"                    # secrets
ssh_timeout = "90s"                 # SSH signatures

[updates]
check = true                        # optional; false: never ask GitHub for a newer release

# [preapprove], [standing] and preapprove = true are new in 0.3.0; leave them out for 0.2.0
[preapprove]                        # optional; the limits of scheduled grants
max_ahead = "168h"                  # how far ahead a grant may start; default 7 days
max_open = "8h"                     # the longest time from start to end
max_uses = 10                       # the most uses of one grant
max_grants = 5                      # the most per client: approved, not ended, plus waiting

[standing]                          # optional; the limits of standing grants (DESIGN.md section 8)
summary = "mon 09:00"               # the weekly list of standing grants, in broker local time

[client.laptop]
pin = "sha256:Yf3c...base64..."     # printed by `nodkey keygen`

[store.main]                         # any name; `type` picks the program
type = "bws"
program = "/usr/local/bin/bws"
token_file = "${CREDENTIALS_DIRECTORY}/bws-token"

[store.backup-job]                   # a second machine account, one project
type = "bws"
program = "/usr/local/bin/bws"
token_file = "${CREDENTIALS_DIRECTORY}/bws-token-backup"

[store.vault]
type = "bw"
program = "/usr/local/bin/bw"
password_file = "${CREDENTIALS_DIRECTORY}/bw-master-password"

[store.op]                           # 1Password: no `program`
type = "1password"
vault = "abcdefghijklmnopqrstuvwxyz" # the ID of the vault, 26 small letters and digits
token_file = "${CREDENTIALS_DIRECTORY}/op-token"

[store.inf]                          # Infisical: no `program`
type = "infisical"
server = "https://app.infisical.com"
client_id = "00000000-0000-0000-0000-000000000000" # of the machine identity
client_secret_file = "${CREDENTIALS_DIRECTORY}/infisical-secret"

[secret.github-token]
store = "main"
id = "00000000-0000-0000-0000-000000000000"
clients = ["laptop"]
ask = "window"
window = "15m"
preapprove = true                   # new in 0.3.0; optional; default false: allows `nodkey preapprove`

[secret.deploy-token]
store = "op"
id = "abcdefghijklmnopqrstuvwxyz"   # the ID of the item, never its name
field = "password"                  # optional; a field's ID or title; default "password"
clients = ["laptop"]
ask = "always"

[secret.ci-token]
store = "inf"
project = "00000000-0000-0000-0000-000000000000" # the ID of the project
environment = "prod"                # the slug
path = "/"                          # required; no default
name = "CI_TOKEN"
clients = ["laptop"]
ask = "always"

[key.deploy]
file = "${CREDENTIALS_DIRECTORY}/ssh-deploy"
clients = ["laptop"]
ask = "window"
window = "15m"
hosts = ["web-1", "web-2"]          # optional; empty means any known host

Leave out what you do not use, such as a store or a key.

The top. listen is one IP address and port of the broker host on the private network. A host name, 0.0.0.0 and :: are refused: the broker must not listen on every network. tls_key is the credential from section 3. The home directories of the stores are in stores, next to audit_log.

[telegram]. token_file and user_id come from section 4. notice_chat_id is optional. For ntfy, use [ntfy] in its place (section 4). timeout is how long a request for secrets waits for your tap: 15 seconds to 10 minutes, 120 seconds by default. ssh_timeout is the same for an SSH login: 15 to 110 seconds, 90 seconds by default. sshd drops a login that takes longer than its LoginGraceTime (120 seconds by default).

New in 0.3.0. [preapprove], [standing], preapprove, standing and the command nodkey preapprove are new in 0.3.0. nodkey 0.2.0 refuses a config with them (unknown key), so leave them out for 0.2.0.

[preapprove] and preapprove. A secret or key with preapprove = true lets a client ask for a scheduled grant with nodkey preapprove (section 11). It is off by default, and it is an error with ask = "never". The four limits are optional. Their defaults are in the example above. They are safe, so leave them unless you need more.

[standing] and standing. A secret with standing = true lets a client ask for a standing grant: daily or weekly windows, for weeks (section 11). It is a separate flag from preapprove, off by default, and an error with ask = "never". Without a [standing] section, no standing grant is allowed. Its keys and defaults are in section 11.

[updates]. check is optional, true by default. Then the broker asks GitHub 10 minutes after its start, and then once a day, whether a newer release of nodkey is out, and sends you a silent notice when one is (section 13). For this, the broker host needs outbound HTTPS to api.github.com. The check uses the proxy of HTTPS_PROXY, as the calls to Telegram do (section 8). GitHub sees the address of the broker host, or of its proxy, and the version of nodkey (THREAT-MODEL.md T34). If the broker host cannot reach GitHub, or GitHub should not see this, set check = false. Otherwise the journal gets a line about the failed check every hour.

[client.NAME]. One section for each agent machine, with the pin of its TLS key. Make that key on the agent machine first (section 11). The name shows in each message as machine:.

[store.NAME]. type is bws with a token_file, bw with a password_file, 1password with a vault and a token_file, or infisical with a server, a client_id and a client_secret_file. program is the absolute path of the program; a 1password or infisical store has none.

[secret.NAME]. The name is what the client asks for, and what the message shows. store names a store section, and id is the UUID of the item in that store (for 1password, the 26-character ID of the item, and field may name the field; for infisical, no id, but project, environment, path and name). clients lists the clients that may ask for it.

[key.NAME]. file is the credential with the SSH key. clients lists the clients that may use it. hosts lists the names in the broker's known_hosts that it may log in to; without hosts, it may log in to every host there.

The rule of each secret and key, ask (DESIGN.md section 8):

  • "always": ask for each request. The buttons are Approve once and Deny.
  • "window": ask, and also offer Approve N min, where N is the window: 1 minute to 8 hours. A tap on it allows the same client the same secret, or the same key, host and remote user, for N minutes.
  • "never": do not ask. This is an open door for that one secret or key (DESIGN.md section 12). It needs exactly one client, and a key needs hosts. It can have hours, max_per_day and notify.

Two more examples. A secret in the bw store, and a nightly job on a machine of its own (backup-1) that gets its secret without a tap:

[secret.npm-token]
store = "vault"
id = "11111111-1111-1111-1111-111111111111"   # a login item; nodkey hands out its password
clients = ["laptop"]
ask = "always"

[client.backup-1]
pin = "sha256:...base64..."

[secret.backup-token]
store = "backup-job"
id = "22222222-2222-2222-2222-222222222222"
clients = ["backup-1"]
ask = "never"
hours = "03:00-03:20"     # local time of the broker; outside these hours, it asks
max_per_day = 1           # more uses in a day ask again
notify = true             # the default: a silent notice for each use

The broker is strict, so that a typo never turns a rule off. An unknown key, a key in upper case, or a key that is not below the header of its own section stops the start. The error names the line or the key, never a value. Paths are absolute. Only ${CREDENTIALS_DIRECTORY}/ and ~/ at the start of a path are expanded.

Write the file as root, and make it owned by root, so that only root can change it:

chown root:root /etc/nodkey/broker.toml
chmod 0644 /etc/nodkey/broker.toml

The file holds no secret, but it lists your secret names, hosts and item IDs. To hide it from other users, use group nodkey and mode 0640 instead.

8. The systemd unit

Copy the example unit contrib/systemd/nodkey.service, of the same version as your nodkey, to /etc/systemd/system/nodkey.service. Never copy it through an agent machine: whoever can change the unit can run a program as root on the broker host. After a build from the source, the unit is in your copy of the repository. The release tarball does not hold it. Take it from the tag of your version in the repository (contrib/systemd/nodkey.service at the tag vX.Y.Z), on the broker host or on the trusted machine. For a release, on the broker host, this takes the unit of the installed version:

VERSION=$(/usr/local/bin/nodkey version | cut -d' ' -f2)
curl -fL -o /etc/systemd/system/nodkey.service "https://raw.githubusercontent.com/code6x/nodkey/refs/tags/v$VERSION/contrib/systemd/nodkey.service"

Then change it to fit your setup:

  • Keep one LoadCredentialEncrypted= line for each credential that you made, and remove the others. systemd does not start the unit when one of these files is missing.
  • Change the paths if yours are different.
  • If the broker must reach Telegram through a proxy, set it as the comments in the unit say. Never use a proxy that inspects TLS: it sees the bot token. The check for a newer release (section 7, [updates]) uses the same proxy to reach api.github.com.

Check the unit. No output means no problem:

systemd-analyze verify /etc/systemd/system/nodkey.service

systemd older than 253 warns that it does not know MemoryZSwapMax=. You can ignore that: MemorySwapMax=0 alone is enough.

What the settings do:

setting what it does
Wants= and After=network-online.target Start the broker after the network is up.
Type=exec systemd sees a failure when the program cannot start.
User=nodkey, Group=nodkey Run as the user of the broker, not as root.
ExecStart= The program and the config file.
LoadCredentialEncrypted=NAME:FILE Decrypt one credential at each start. The broker reads it as ${CREDENTIALS_DIRECTORY}/NAME.
StateDirectory=nodkey nodkey/stores, StateDirectoryMode=0700 Make /var/lib/nodkey (the audit log) and /var/lib/nodkey/stores (the home directories of the stores), owned by nodkey, with mode 0700. The broker refuses a store directory with another mode or owner.
UMask=0077 New files of the broker and of the store programs are private.
Restart=on-failure, RestartSec=5s Start the broker again 5 seconds after a crash or an error.
RestartPreventExitStatus=64 76 77 Do not start it again after wrong usage (64), a polling conflict at Telegram (76) or a bot token that Telegram refuses (77). Only a person can fix these.
TimeoutStopSec=90s, KillMode=control-group The broker stops in at most 80 seconds. After 90 seconds, systemd kills it and every process of the unit, also a store program that left its process group.
LimitCORE=1 No core dumps. A crashed bws or bw starts no core handler that could copy its token into the journal (section 9). The broker sets this limit itself too.
MemoryMax=1G A memory limit for the broker and its store programs.
MemorySwapMax=0, MemoryZSwapMax=0 The memory of the broker, with keys and tokens, never goes to swap.
LogRateLimitIntervalSec=0 journald never drops a line of the broker. The anchors of the audit log are in these lines (section 12). The broker limits its own lines.
Environment=, EnvironmentFile= (as comments) An optional proxy for Telegram, and for the check for a newer release at GitHub. A proxy with a password goes into a file that only root can read.
NoNewPrivileges=yes and the lines after it The sandbox: no new privileges or capabilities, a read-only system, no access to /home, a private /tmp, no devices, kernel settings, other namespaces or raw sockets, and only the system calls of a normal service.
WantedBy=multi-user.target systemctl enable starts the broker at boot.

What the unit leaves out on purpose:

  • MemoryDenyWriteExecute=yes: bw runs on Node.js, and the 1Password SDK runs its core with wazero. Both need memory that is writable and executable.
  • PrivateUsers=yes: the credentials and the config, owned by root, would look owned by nobody, and the broker would refuse them.
  • PrivateNetwork=, IPAddressDeny= and the like: the broker must reach Telegram, and GitHub for the check for a newer release, and bws and bw must reach Bitwarden, the 1Password SDK must reach 1Password, and the broker must reach your Infisical server. Do not add them.

After systemctl daemon-reload, systemd-analyze security nodkey shows how much the sandbox limits the broker.

9. Account and host safety

The approver's Telegram account

Anyone who can use your Telegram account can approve (THREAT-MODEL.md T7).

  • Never log in to the approving Telegram account on a machine where an agent runs: no Telegram Desktop, and no Telegram Web in a browser there. The agent could tap Approve itself, or copy the session and use it from anywhere. The same holds for any other way to control the phone from that machine, such as adb or KDE Connect.
  • In Telegram, open Settings → Devices, and end every session that you do not need. Better: use an account that is logged in only on the phone.
  • Turn on two-step verification (Settings → Privacy and Security), and a passcode for the app.

The ntfy request topic

With ntfy, anyone who can read the request topic can approve (THREAT-MODEL.md T40). Run the server yourself, with auth-default-access: deny-all. Never subscribe to the request topic on a machine where an agent runs, and never keep an ntfy login there (section 4).

The Bitwarden accounts that can read the secrets

Anyone who can use a Bitwarden account that can read the secrets can read them in Bitwarden's own apps, with no tap and no entry in the audit log (THREAT-MODEL.md T32). For bws, these are at least the owners and admins of the organization, and the members with access to the project. For bw, it is at least the separate account. T32 has the full lists.

  • Never log in to such an account on a machine where an agent runs: no web app in a browser, no browser extension, no desktop app and no CLI there. The agent could read the secrets itself, or record the master password when you type it. The same holds for a phone that the agent machine can control, such as through adb or KDE Connect.
  • Do not save the master password, the API key or the key of the two-step login of these accounts in a password manager that is open on an agent machine, such as your main Bitwarden account. With them, the agent could log in to the account from anywhere.
  • If your main Bitwarden account is logged in on an agent machine, it must not be able to read the secrets. Make the organization for Secrets Manager with a separate account, and do not add your main account to it (section 5).
  • After each change to the organization, check who is an owner or an admin, and who is on the People tab of the project, its secrets and its machine accounts.
  • Turn on two-step login for these accounts, with an authenticator app or a security key (Bitwarden's guide). A code by email does not help if you read that email on an agent machine.

The 1Password vault and who can open it

Anyone who can open the nodkey vault in 1Password's own apps reads its items with no tap and no entry in the audit log (THREAT-MODEL.md T39). So can everyone who holds the token of its service account: a leaked token reads the whole vault until it expires or is revoked.

  • Use a vault only for nodkey, with only the items that nodkey may hand out, and give it to no more people and groups than needed.
  • Never log in to an account that can open the vault on a machine where an agent runs. Do not add your main account to the vault if it is logged in there.
  • The service account has read_items only, on that vault only, and an expiry. Its permissions cannot change later.
  • The token is an encrypted systemd credential on the broker host, never a file on an agent machine.
  • After each change, check who is a member of the vault and which service accounts can read it.
  • Turn on two-step login for the accounts that can open the vault.

The Infisical project and who can read it

Anyone who can read the nodkey project in Infisical's own apps reads its secrets with no tap and no entry in the audit log (THREAT-MODEL.md T41). Anyone who can write to it can change what a name holds. So can everyone who holds the client secret of its machine identity: a leaked client secret gets access tokens until it ends or is revoked.

  • Use a project only for nodkey, with only the secrets that nodkey may hand out. Give it to no more people than needed, and write access to few.
  • Never log in to an account that can open the project on a machine where an agent runs. Do not add your main account to the project if it is logged in there. The admins of the organization may read it too.
  • The machine identity has the Viewer role, a short access token TTL and a TTL on its client secret.
  • The client secret is an encrypted systemd credential on the broker host, never a file on an agent machine.
  • After each change, check who is a member of the project and which machine identities can read it.
  • Turn on two-step login for the accounts that can open the project.

Swap and hibernation

The memory of the broker holds the keys, the tokens and, for a moment, the secret values. Swap and hibernation write memory to the disk, where a copy of the disk can give them.

  • Check for swap with swapon --show. Use no swap, or encrypted swap, such as a swap device in /etc/crypttab with a new random key at each boot. zram keeps its swap in memory, which is fine. The unit also keeps the memory of the broker out of swap (MemorySwapMax=0).

  • Turn off hibernation:

    systemctl mask hibernate.target hybrid-sleep.target suspend-then-hibernate.target

Core dumps

A core dump of a crashed bws or bw would hold its environment, with the token or the session key. The unit sets LimitCORE=1, and the broker sets the same limit for itself and its store programs. With a limit of exactly 1, the kernel writes no core file and starts no core handler, but only when kernel.core_pattern is a pipe (|) or a file name. The newer socket form (@) passes the dump to a program whatever the limit (DESIGN.md section 16).

sysctl kernel.core_pattern
  • kernel.core_pattern = |/usr/lib/systemd/systemd-coredump ... (the default of systemd), |/usr/share/apport/apport ... or core: good.
  • A value that starts with @: change it. Find the file that sets it with grep -rs core_pattern /etc/sysctl.conf /etc/sysctl.d /usr/lib/sysctl.d /run/sysctl.d. Remove that line, or override it with a file such as /etc/sysctl.d/90-nodkey-core-pattern.conf that sets a pipe or the plain kernel.core_pattern = core. Then run sysctl --system, and check again.

A firewall for the broker port

Only the agent machines should reach the broker port. Any other machine can at least make the broker do TLS handshakes (THREAT-MODEL.md T28). An example with nftables, with the agent machines 192.0.2.21 and 192.0.2.22. It only adds rules for port 7743, and leaves your other rules as they are. Save it as /etc/nodkey/firewall.nft:

# The first two lines remove the old version of this table, if there is one.
table inet nodkey
delete table inet nodkey

table inet nodkey {
  set agents {
    type ipv4_addr
    elements = { 192.0.2.21, 192.0.2.22 }
  }
  chain input {
    type filter hook input priority filter; policy accept;
    tcp dport 7743 ip saddr @agents accept
    tcp dport 7743 drop
  }
}

Load it with nft -f /etc/nodkey/firewall.nft. To load it at boot, for example with the nftables service of Debian and Ubuntu, add the line include "/etc/nodkey/firewall.nft" at the end of /etc/nftables.conf. With ufw, allow the port for each agent machine only, for example ufw allow proto tcp from 192.0.2.21 to any port 7743.

The journal

The broker writes a hash of each audit entry to its log, which systemd keeps in the journal. These anchors find changes to the audit log (section 12), but only while the journal keeps them.

  • Keep the journal on the disk: the directory /var/log/journal must exist, or set Storage=persistent in /etc/systemd/journald.conf.
  • Keep it long enough. journald removes old entries when it uses more than SystemMaxUse=: by default 10 % of the file system, at most 4 GiB.
  • Better: send the journal to another machine, for example with systemd-journal-upload. Then not even root on the broker host can change the copies later.

Without a working log, the broker stops. When a write to stderr fails, for example because the program that reads it ended, the broker stops in its normal order and exits with code 1. The example unit then starts it again after 5 seconds. The entry whose anchor did not reach the journal is on disk. At the next start, the broker writes an audit open line at that entry, and nodkey audit verify --anchors then reports that entries were added while the broker was stopped. A failed sync and a power cut can do the same: journald may lose its last minutes of lines. Then look at the log near that entry and at the time of the stop. If you are sure the gap has a cause like this, check the log with only the journal lines after that start: journalctl -u nodkey -o cat --since TIME, where TIME is a moment after that start. This works once the broker has written at least one entry after that start.

10. Start the broker

Write the config file first (section 7), with at least one client (section 11). Then:

systemctl daemon-reload
systemctl enable --now nodkey
journalctl -u nodkey -f

A good start looks like this. Here, the time= at the start of each line, and some lines, are left out:

level=INFO msg="nodkey broker: starting" version=0.2.0 config=/etc/nodkey/broker.toml
level=INFO msg="nodkey broker: the config and the files it names passed their checks"
level=INFO msg="nodkey broker: core dumps are off: the soft and the hard limit of core dumps are 1, and the process is not dumpable"
level=INFO msg="nodkey broker: the TLS key is loaded" pin=sha256:Q2b9...
level=INFO msg="nodkey broker: an SSH key is loaded and can sign" key=deploy fingerprint=SHA256:3SMw...
level=INFO msg="nodkey broker: the audit log is open" path=/var/lib/nodkey/audit.jsonl
audit open 0:0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000
level=INFO msg="nodkey broker: a store started" store=main
level=INFO msg="nodkey broker: the listen address is open; nothing reads from it yet" address=192.0.2.10:7743
audit anchor 1:8318bd865cb1923faea0c3a1837f636ad1f8781898606701949120d612c85da0 8318bd865cb1923faea0c3a1837f636ad1f8781898606701949120d612c85da0
level=INFO msg="nodkey broker: the approver runs: it polls Telegram for the answers"
level=INFO msg="nodkey broker: listening; the broker is ready" address=192.0.2.10:7743

The lines audit anchor SEQ:HASH LOGID and audit open SEQ:HASH LOGID are the anchors of the audit log (section 12). LOGID names the log. On a new log, the open line has a zero head and a zero LOGID, and comes before the first anchor. When something is wrong, the broker logs why (section 14).

A restart of the broker ends all grants, and denies the requests that wait for a tap:

systemctl restart nodkey

Change the config without a restart

After most changes of the config or of known_hosts, a restart is not needed. Run this as root on the broker host:

nodkey broker reload

It asks the broker to load the config again (with SIGHUP), and waits for the result. The broker checks the new config as at a start.

  • When the new config is good, the broker uses it at once, and the command exits with 0. Grants that the new config still allows stay, with the same uses left and the same end. Grants of a client, secret or key that you removed or changed end at once. A removed client, or a client with a new pin, cannot go on with a connection that it opened before.
  • When the new config is not good, the broker goes on with the old one. The command prints why, and exits with 1. The journal says why too. Fix the config, and run the command again.

A reload can change [client.*], [secret.*], [key.*] and known_hosts. A change of listen, tls_key, audit_log, [telegram], [ntfy], [updates], [preapprove], [standing] or a [store.*] section needs a restart: the reload refuses it, and says so. A new key file in ${CREDENTIALS_DIRECTORY} also needs a restart, because systemd loads the credentials only at the start.

The broker writes the result of each reload to reload-status.json, next to the audit log (/var/lib/nodkey/reload-status.json). Each reload that the broker took has a config_reloaded entry in the audit log (DESIGN.md section 14).

Check the setup with sudo nodkey doctor broker (section 14).

11. A client on the agent machine

The key and the config of the client

On the agent machine, as the user who runs the agent, make the TLS key of the client:

install -d -m 0700 ~/.config/nodkey
nodkey keygen --out ~/.config/nodkey/client.key

nodkey keygen prints the pin of the new key. On the broker host, add a section for the client to /etc/nodkey/broker.toml, add its name to the clients lists of the secrets and keys that it may use, and reload the config with nodkey broker reload, as root (section 10):

[client.laptop]
pin = "sha256:Yf3c...base64..."

The agent can copy this key. With it, someone can only ask, like the agent (THREAT-MODEL.md T11). They get what this client gets: what needs a tap still needs your tap, but a grant or a "never ask" rule of this client works for them too. To block a stolen key, remove its pin from the broker config, and run nodkey broker reload as root. Each machine needs its own key: the broker refuses two clients with the same pin.

Write the client config, ~/.config/nodkey/client.toml, with the address of the broker and the pin of the broker's key (section 3):

broker = "192.0.2.10:7743"
broker_pin = "sha256:Q2b9...base64..."
tls_key = "~/.config/nodkey/client.key"

The pin is the only trust of the client in the broker, so other users must not be able to change the file. The client refuses it then:

chmod 0600 ~/.config/nodkey/client.toml

Check the setup with nodkey doctor (section 14).

The first request

nodkey run --secret GH_TOKEN=github-token --reason "test the setup" -- \
  sh -c 'echo "the command got ${#GH_TOKEN} characters"'

The client prints nodkey: waiting for approval (request 7qd2). On the phone, a message comes from your bot with the same short code. Check the lines with ✅: they come from the broker. The lines below "claimed by the agent" come from the agent machine. Tap Approve once. The command prints how many characters the secret has, never the value.

nodkey never prints a value. The value goes only into the environment of the one command. For real work, the agent runs, for example:

nodkey run --secret GH_TOKEN=github-token --reason "publish the release the user asked for" \
  -- gh release create v1.2.0

Tell the agent about it, for example in its instructions for a project: "For a command that needs a secret, use nodkey run --secret VAR=NAME --reason TEXT -- COMMAND." Some variable names are refused, because programs print or run their values (DESIGN.md section 5).

The command itself may print a value, for example in a debug line. So, by default, nodkey replaces each value in the command's stdout and stderr with ***. It masks a value only as it is, not in another form such as base64. The command's stdout and stderr are then pipes, not a terminal.

When stdout and stderr of nodkey run are both a terminal, a person reads the output, so nodkey does not mask it: the command owns the terminal, and editors, ssh and prompts work. --mask masks also on a terminal. --no-mask turns masking off, for a value that is also a common word, for example: then normal output would show ***.

nodkey run exits with the exit code of the command. When nodkey itself stops, its last line on stderr is nodkey: <reason> (exit <code>): 77 when the request was denied or expired, 69 when the broker cannot be reached, 64 for wrong usage (DESIGN.md section 15).

SSH: nodkey ssh-agent

nodkey ssh-agent is an SSH agent without keys. For each login, it asks the broker for a signature. Start it in a terminal of its own. It prints the short code of each request there:

nodkey ssh-agent

Or run it as a user service. Save this as ~/.config/systemd/user/nodkey-ssh-agent.service:

[Unit]
Description=nodkey ssh-agent: SSH logins with the keys of the broker

[Service]
ExecStart=/usr/local/bin/nodkey ssh-agent

[Install]
WantedBy=default.target

Then run systemctl --user enable --now nodkey-ssh-agent. The short codes are in journalctl --user -u nodkey-ssh-agent -f.

In the shell of the agent, for example in ~/.profile:

export SSH_AUTH_SOCK="$XDG_RUNTIME_DIR/nodkey/agent.sock"

Check it:

# the keys that this client may use, with their names
ssh-add -l
# a message comes for the login
ssh deploy@web-1 true

ssh, git and Ansible work through it with no changes. Some things to know:

  • ssh must send session-bind@openssh.com: OpenSSH 8.9 or newer. Paramiko and Go's own SSH client do not, so the broker refuses them. For Ansible, use its default ssh connection.

  • The broker refuses host certificates. If a host offers one, ssh may use it. Then allow only plain host keys for that host in ~/.ssh/config:

    Host web-1 web-2
    HostKeyAlgorithms ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256

  • The message shows NODKEY_REASON from the environment of ssh as the reason.

  • An approved login is one SSH session. That session can stay open, and ssh can share it with later commands (ControlMaster, ControlPersist), without a new tap.

Many hosts at once: nodkey ssh-grant

One Ansible run logs in to each host, so it needs one tap for each host. nodkey ssh-grant asks once for a list of hosts. The key needs ask = "window":

nodkey ssh-grant --key deploy --user deploy --host web-1 --host web-2 \
  --for 15m --reason "apply the nginx fix"
ansible-playbook site.yml

The message lists the key, the remote user, each host and the end of the grants. After Approve, logins as deploy to these hosts need no tap until that time. Other hosts and users still ask. A Deny for the key ends its grants at once.

--for must be at most the key's window. A longer one gets no message: nodkey prints denied by a rule: --for 1h is longer than the key's window (30m); no prompt was sent and exits with 77. Any other check that fails before a message, such as a host that is not in known_hosts, prints denied by a rule of the broker; no prompt was sent. Plain denied comes after the waiting for approval line: the person denied it, or nobody answered in time.

A job that runs later: nodkey preapprove

New in 0.3.0: nodkey 0.2.0 does not have it.

A timer or a job runner cannot tap a button at 03:00. Ask for the grant when the job is planned, while you are awake. Then the job gets what it needs without a tap, inside the time you approved, and the grant ends by itself. First turn it on for the secret or key, in broker.toml. It is off by default:

[secret.backup-token]
# ...
ask = "always"
preapprove = true

Then, when the job is planned (for example in the script that sets up tomorrow's job):

nodkey preapprove --secret backup-token \
  --from 2026-10-28T03:00 --until 2026-10-28T04:00 --uses 1 \
  --reason "job nightly-backup"

--from is optional (default: now): without it, the grant starts when you run the command. The message shows the secret, the uses and the time, in the broker's local time. After Approve, the command prints the ID of the grant and ends. The message keeps one Revoke button. At 03:00 the job runs nodkey run as usual, and gets the secret without a tap. Each use sends a silent notice to the notice chat. When the uses are gone, or at 04:00, the grant is gone, and the message shows how and when it ended. If the job did not run, or ran late, and a use is left at 04:00, a silent notice says not used. To end the grant earlier, tap Revoke on the phone. The client can also cancel it, for example when the job is removed:

nodkey preapprove --cancel ID

Things to know:

  • Anything on that machine can use the grant inside its time, not only the job. Keep the span short and the uses few. If something else uses the grant up, the real job fails, and that shows it.
  • For a key, use --key, --user and --host as for nodkey ssh-grant. Each SSH connection takes one use. A tool that opens many connections, such as Ansible, needs more uses, or ControlPersist in the ssh config so that the connections are shared.
  • A time without an offset is read in the broker's local time. Near the change to summer time, write the offset (2026-10-28T03:00:00+02:00). The broker refuses a local time that the clock skips or shows twice.
  • The window of the rule does not cap a grant. The limits are in [preapprove]: how far ahead it may start (max_ahead, 7 days), how long it is open (max_open, 8 hours), how many uses (max_uses, 10) and how many grants a machine holds (max_grants, 5).
  • A broker restart drops all grants. The job then asks as usual. When the broker stops, one notice lists the grants that it dropped: the ID, the machine, the secret or key, and the uses left. A Deny for the same secret or key ends the grant, also before its start; the message of the grant says so.

A job that runs every night: standing grants

New in 0.3.0: nodkey 0.2.0 does not have it.

A scheduled grant covers one time span. A job that runs every night for weeks would need a new grant, and a tap, each day. A standing grant lets you approve it once: this client may use these secrets in these daily or weekly windows, N times in each window, until a date. It is for secrets only, not for SSH keys.

Turn it on for the secret, in broker.toml. It is off by default, and it is a separate flag from preapprove. The rule must have ask = "always" or ask = "window". The [standing] section must be there, also when it is empty: without it, no standing grant is allowed. These are its keys and their defaults:

[secret.backup-token]
# ...
clients = ["backup"]
ask = "always"
standing = true

[standing]
max_days = 90                       # from now to the end date; at most 366
max_window = "8h"                   # one window; at most 8h
max_uses = 1                        # uses in each window; at most 10
max_grants = 3                      # standing grants of one client; at most 20
warn_days = 7                       # the warning before the end; less than max_days
summary = "mon 09:00"               # the weekly list, in broker local time

Run the job as its own system user. Anything that runs as the same user can read the token file and use the grant. Give the job its own user, with its home folder in /var/lib, and its own client key and config (section 11), with its own [client.backup] section on the broker. Then the agent's user cannot read the token or the key, if it cannot use sudo or become root. On the agent machine, as root:

useradd --system --home-dir /var/lib/nightly-backup --shell /usr/sbin/nologin nightly-backup
install -d -m 0700 -o nightly-backup -g nightly-backup /var/lib/nightly-backup
sudo -H -u nightly-backup sh -c 'install -d -m 0700 ~/.config ~/.config/nodkey &&
  nodkey keygen --out ~/.config/nodkey/client.key'

Write ~nightly-backup/.config/nodkey/client.toml as before, owned by nightly-backup, with mode 0600.

Ask for the grant as that user. --until is a must. --uses is the uses in each window (default 1). --from is optional (default: now). Give --daily HH:MM-HH:MM, or --weekly DAYS HH:MM-HH:MM (such as --weekly mon,thu 03:00-05:00), not both. A window is in broker local time, and does not pass midnight. --grant-token FILE is a must:

sudo -H -u nightly-backup nodkey preapprove --secret backup-token \
  --daily 03:00-05:00 --until 2026-12-31T00:00 \
  --grant-token /var/lib/nightly-backup/token --reason "job nightly-backup"

The message shows the windows, the uses in each window and the end, with Approve and Deny. After Approve, the command writes the token to the file and prints the ID of the grant. It never prints the token. The file must be a new file, or a file that the user owns with mode 0600. The client refuses a token file that is not its own, or that other users can read or write. Keep it in a private folder, not in /tmp.

The job. A systemd service and a timer. The timer starts the job inside the window:

# /etc/systemd/system/nightly-backup.service
[Unit]
Description=Nightly backup

[Service]
Type=oneshot
User=nightly-backup
Group=nightly-backup
StateDirectory=nightly-backup
StateDirectoryMode=0700
Environment=HOME=/var/lib/nightly-backup
ExecStart=/usr/local/bin/nodkey run --grant-token /var/lib/nightly-backup/token \
  --secret BACKUP_TOKEN=backup-token --reason "job nightly-backup" -- /usr/local/bin/backup
# /etc/systemd/system/nightly-backup.timer
[Unit]
Description=Nightly backup

[Timer]
OnCalendar=*-*-* 03:30

[Install]
WantedBy=timers.target
systemctl daemon-reload
systemctl enable --now nightly-backup.timer

OnCalendar is in the local time of the agent machine, and the window is in the local time of the broker. If the two differ, add the zone of the broker, such as OnCalendar=*-*-* 03:30 UTC. Do not add Persistent=true to the timer: a missed run would start later, outside the window, and that gives an alarm. Each use needs no tap and sends a silent notice. nodkey run writes a new token to the file before it runs the command. If it cannot save the file, the command does not run (exit 77).

Things to know:

  • The client never retries a request with a token. An old token ends the grant with an alarm, because it looks like a stolen token. So if a run failed in the middle (for example the network broke after the broker sent the answer), the next run has an old token. Then make a new grant: one tap.
  • Alarms come with sound. These end the grant: an old token, more uses in a window than allowed, and the token of another machine's grant. These only alarm, and the grant stays: a valid token outside the windows, an unknown token, and a window that ended with no use.
  • warn_days (default 7) before the end, a warning with a Renew button comes, with sound. Renew makes a new grant of the same length, from now, with the same secrets and windows. The job keeps its token file: its next use gets the token of the new grant. A grant can be renewed once, and a grant that is not longer than warn_days gets no warning.
  • The message of the grant has a Revoke button: it ends the grant at once. nodkey preapprove --cancel ID, as the job user, ends it too. A Deny of another request does not end a standing grant.
  • Send /grants to the bot, in your private chat with it: it lists each standing grant, with the machine, the secrets, the windows, the last use and the end. Once a week, at summary, the same list comes as a silent message. The list never shows a token.
  • A broker restart drops all standing grants, as it drops scheduled grants. The stop notice lists them. Then make new grants. Keeping them over a restart is issue #111.

12. Check the audit log

The broker writes each request and each decision to the audit log, /var/lib/nodkey/audit.jsonl. Each entry holds the hash of the entry before it, and the broker writes the hash of each entry to the journal: audit anchor SEQ:HASH LOGID. When it opens the log, it writes audit open SEQ:HASH LOGID, the last entry that it found. LOGID is the SHA-256 of line 1 of the log (DESIGN.md section 9). Check the log with every anchor since the log began, as root:

journalctl -u nodkey -o cat | nodkey audit verify /var/lib/nodkey/audit.jsonl --anchors -

A good log gives:

ok: 1234 entries (torn lines: 0)
anchors: 1234 checked, all match; the newest is entry 1234
log ID: 5c2e0b8f4a1d7e3396b0c1f2a8d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f6
open lines: 12 checked, all match the anchor before them; new log starts: 1
last entry: 1234:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

It exits with 0 when the log passed, and with 65 when it found a problem. Then it prints the line number, never the text of the line. A changed entry shows as an anchor that does not match. A log that someone cut shows as an anchor whose entry is missing. An entry that someone added, removed or changed while the broker was stopped shows as an open line that does not match the last anchor before it. The check lists every open line that does not match, each on its own line. It does not stop at the first one.

  • Entries after the newest anchor are not proven.

  • An open line that does not match can also come from a crash: the entry was on disk, but the power went before its anchor reached the journal. nodkey audit verify cannot tell. Look at the log around that entry and at the time of the crash. If you know that the gap is harmless, accept it with the head of that audit open line (see below).

  • If journald removed its oldest lines (SystemMaxUse= or its age limits), the first line of the log in the journal can be an audit open line with no anchor before it, and the check fails. Keep the journal for as long as the log, or copy it to another machine. If you know why the lines are gone, you can accept that open line (see below).

  • Anchor lines without a log ID come from an earlier version. Each one takes the log ID of the first audit open line with a log ID after it, so old anchors of earlier logs do not fail the check of a newer log. You do not need --since for that. Only if old anchors of two logs come before the same open line, the check of that log can fail: with "two anchors for entry N have different hashes", with an anchor whose hash does not match its entry, or with an anchor whose entry is not in the log. Then check the log with only the lines since it began, with --since.

  • If the journal holds lines of another log, nodkey audit verify prints other log in the journal: ID …, up to entry N; if no person moved it away, it was deleted or replaced. If you moved the log away, this is expected: the broker started a new log at entry 1. If you did not, someone deleted or replaced the log. Keep the old file. It does not fail the check, and its exit code is still 0. A moved log is normal. A script can look for the line other log in the journal.

  • To accept a gap, give the head of the audit open line that does not match, as SEQ:HASH. You can repeat the flag. For example:

    journalctl -u nodkey -o cat | nodkey audit verify /var/lib/nodkey/audit.jsonl \
  --anchors - --accept-open 41:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

    Use it only after you found out why the open line does not match, for example a power cut. You vouch for the gap: it is not proven, and it does not count in anchors: N checked. Entry 41 must still have that hash. An open line with no anchor before it can be accepted only when it is the first audit line in the journal copy. The result then has the line accepted gaps: 1 open lines that do not match, accepted with --accept-open. If no failing open line has that head, --accept-open is an error (exit 65); remove it. The same error comes when that open line has no anchor before it and is not the first audit line: then check with the journal since the log began (--since).

  • jq shows the entries in short, for example the last ten:

    tail -n 10 /var/lib/nodkey/audit.jsonl |
  jq -c '{seq, event, client, secrets, key, hosts, answer, cause, code} | with_entries(select(.value != null))'

13. Upgrades and key changes

A new version of nodkey

The broker tells you when a newer release is out, unless [updates] says check = false (section 7). It asks GitHub 10 minutes after its start, and then once a day. For each newer release, it writes a line in its journal, and sends you one silent notice in each start:

ℹ️ nodkey: A newer release is out: nodkey 0.3.0. The broker runs nodkey 0.2.0. To update the broker, run as root on the broker host: nodkey update. Then restart it: systemctl restart nodkey. The restart ends all grants. Update the agent machines too: run nodkey update as root. What is new: nodkey update --notes. Release notes: https://github.com/code6x/nodkey/releases/tag/v0.3.0

If the check or the notice fails, the journal says why, and the broker tries again after 1 hour. On a terminal, nodkey version says it too, on each machine. A build from the source never asks.

If the release fixes a security problem, the notice is different. It starts with SECURITY UPDATE, it makes a sound, and it comes again at each daily check until the broker runs that release or a newer one. It lists at most 3 advisory IDs. On a terminal, nodkey version prints a red SECURITY UPDATE line (no colour when NO_COLOR is set), also when the security release is older than the newest release. The flag comes from GitHub's answer and nodkey does not check it. So read the checked notes before you decide. An agent machine that is older than a security release prints a SECURITY UPDATE line when it uses the broker, and you get a silent notice that names that client.

  1. Read the changes. nodkey update --notes, as root, downloads the release and checks it as nodkey update does, but installs nothing. Then it prints the changes since your version, security fixes first. If there is none, it says No security fixes after nodkey X.Y.Z, up to nodkey A.B.C. and lists the changes under # CHANGES. nodkey changes --since 0.2.0 prints the same from the changelog that is built into the nodkey that you run. A normal nodkey update prints the changes after it installs. A host on nodkey 0.2.0 has no --notes: read the release notes on GitHub, and update once.

  2. On the broker host, as root, set VERSION to the new version, such as 0.2.0, and install it:

    VERSION=0.2.0
nodkey update --version "$VERSION"

    It runs the installer that is built into the installed nodkey, and checks the new release in the same way (section 2). It also takes an older version. nodkey 0.1.0 has no nodkey update: update it with the installer of section 2, with --version "$VERSION". Or get the new version by hand, check it, and install it with install -o root -g root -m 0755 nodkey /usr/local/bin/nodkey.

  3. Get the example unit of the same version from its tag, as in section 8, but save it as nodkey.service, not over your unit. After a build from the source, use the unit in your copy of the repository instead. Compare it with your unit, and take over the changes:

    curl -fL -o nodkey.service "https://raw.githubusercontent.com/code6x/nodkey/refs/tags/v$VERSION/contrib/systemd/nodkey.service"
diff -u /etc/systemd/system/nodkey.service nodkey.service

  4. Restart the broker:

    systemctl daemon-reload
systemctl restart nodkey
journalctl -u nodkey -n 30

  5. On each agent machine, in a root shell as in section 2, install the same version, such as with nodkey update --version 0.2.0. Then restart nodkey ssh-agent there.

A restart ends all grants. Rarely, right after a restart, Telegram reports a polling conflict, and the broker stops with exit code 76: a poll of the old broker was still on its way (DESIGN.md section 7). If no other program uses the token, start the broker again with systemctl start nodkey.

Until every machine runs the new version, nodkey tells you. Each client sends its version to the broker, and the broker sends its own back. nodkey 0.1 sends no version.

  • On the agent machine, nodkey run, nodkey ssh-grant and nodkey ssh-agent print a warning on stderr. It says which side is older, and what to do there:

    nodkey: warning: the broker runs nodkey 0.2.0, which is older than nodkey 0.3.0 on this machine; to update the broker, run "nodkey update" as root on the broker host

    With a broker of nodkey 0.1, the warning says that the broker runs an older nodkey that does not send its version, and points to this section: nodkey 0.1 has no nodkey update.

  • The broker writes a line in its journal, once in each start for each client that runs another version, or that sends none. It also sends you one silent notice for such a client:

    ℹ️ nodkey: The client laptop says that it runs nodkey 0.2.0, which is older than nodkey 0.3.0 on the broker. To update it, run there as root: nodkey update

    For a client of nodkey 0.1, the notice points to this section too. If the notice cannot go out, such as while Telegram cannot be reached after a power cut, the journal says so, and the next request of that client a minute later or more tries again. Rarely, the same notice comes more than once.

The steps above make both go away. A machine that still runs another version works as before, unless the two versions do not understand each other's requests: then the request fails, for example with the answer of the broker is not valid. The version of a client is only what the client says: the broker allows and denies nothing because of it. A build from the source, such as one with the version dev or a commit, gets no warning and no notice.

New versions of bws and bw

Install them in the same way as the first time (section 5): from the official release, checked, owned by root. Then check that the next request for a secret works. The login of bw stays in the home directory of the store.

The 1Password SDK and the Infisical SDK are part of the nodkey binary, so they change with nodkey (A new version of nodkey).

A new TLS key for the broker

Make a new key and its credential as in section 3, and restart the broker. Then put the new pin into broker_pin on each client, and restart nodkey ssh-agent there. Until then, the clients refuse the broker.

A new key for a client, or a client that must go

To give a client a new key, make it with nodkey keygen in a new file, put its pin into the [client.NAME] section, run nodkey broker reload as root, and change tls_key in the client config. Remove the old key file. Restart nodkey ssh-agent too: it reads the config and the key only when it starts.

To block a client, for example after a strange request, remove its [client.NAME] section, and its name from every clients list, and run nodkey broker reload as root. Its key no longer works, also on a connection that is open, and its grants end. The grants of the other clients stay.

A new bot token

Make a new token with /revoke in BotFather. If BotFather says the bot is invalid, send its user name with the @, for example @my_nodkey_bot, or use /mybots, choose the bot, then API Token, then Revoke current token. Then make the credential again as in section 4, and restart the broker. Do this at once when the token may have leaked. When Telegram no longer takes the old token, the broker stops with exit code 77, and systemd does not start it again until you do.

A new 1Password token

The permissions of a service account cannot change. So for a new token, or a change of vault or permission, make a new service account first, with read_items only, on the nodkey vault only, and with an expiry. Make the credential op-token again, restart the broker, check that the next request works, and then delete the old service account in 1Password. Put the date of the expiry in your calendar: after it, every request for the store is denied.

A new Infisical client secret

Make a new client secret for the machine identity in Infisical, with a TTL. Make the credential infisical-secret again, restart the broker, check that the next request works, and then delete the old client secret in Infisical. Put the date of the TTL in your calendar: after it, every request for the store is denied.

A new bws token or bw master password

For bws, make a new access token for the machine account, make the credential again, restart the broker, and then revoke the old token in Secrets Manager.

For bw, change the master password of the separate account, make the credential again, and restart the broker. If bw is logged out after the change, log in again as in section 5, step 5.

A new SSH key

Make a new key and its credential as in section 6, with a new name, such as ssh-deploy-2. Add its public key to the target hosts. Change file of the [key.NAME] section and the LoadCredentialEncrypted= line of the unit. Then run systemctl daemon-reload, so that systemd reads the changed unit, and restart the broker. At the end, remove the old public key from the target hosts.

14. When something does not work

First run nodkey doctor on the agent machine, as the user of the agent, and sudo nodkey doctor broker on the broker host. They only read: they change nothing and show no secret value. Each check prints one line. A check that fails says what is wrong, why it matters, how to fix it, and which part of the docs to read. A check is skip when one that it needs failed. The exit code is 0 only when every check is ok. For a script, add --json. On a machine with no access to GitHub, add --offline to nodkey doctor.

$ nodkey doctor
ok   version: nodkey 0.3.0; the newest release
ok   config: /home/agent/.config/nodkey/client.toml
FAIL broker: cannot connect to the broker at 192.0.2.10:7743: connect: connection refused
     why: nodkey run, ssh-agent, ssh-grant and preapprove need the broker
     fix: check that the broker runs (systemctl status nodkey on the broker host), broker and broker_pin in the client config, and that the pin of this client's TLS key is in the broker config
     see: docs/setup.md section 14
skip broker version: needs broker
ok   ssh: OpenSSH 9.6
ok   ssh agent: the socket of nodkey ssh-agent

The broker logs each problem in the journal: journalctl -u nodkey. An error names the file, the section or the key, never a secret value.

what you see what it means, and what to do
The broker stopped with exit code 76, and an alert says that another program reads the updates of the bot. Another program polls with the bot token, or a webhook is set. The token may be stolen. Make a new token (section 13), then start the broker. Right after a restart, see section 13.
The broker stopped with exit code 77. Telegram does not take the bot token, such as after /revoke. Make the credential with the new token, then start the broker.
The installer or nodkey update ends with nothing was installed (exit 1). The last line says which step failed. After a network error, try again later. After a failed SHA-256, attestation or version check, do not install that release in another way: someone may have changed it. Report it (SECURITY.md).
The installer or nodkey update cannot download: a TLS or certificate error, behind a proxy. The installer removes SSL_CERT_FILE, SSL_CERT_DIR and CURL_CA_BUNDLE, so curl and gh trust only the certificates of the system. If the proxy inspects TLS, put its CA into the system store: on Debian, copy it to /usr/local/share/ca-certificates/NAME.crt and run update-ca-certificates. Then every program on the host trusts the proxy with its TLS traffic: on the broker host, that includes the calls to Telegram, which hold the bot token. nodkey update passes on only HTTPS_PROXY, https_proxy, NO_PROXY and no_proxy, not ALL_PROXY or all_proxy: gh does not read these. Set HTTPS_PROXY.
The installer says that group or others can write to a directory or a file, or that a path is owned by user. As root, the installer runs programs from these places, so only root may be able to change them, and each directory above them. Fix the owner or the mode, such as with chmod go-w /usr/local/bin, or choose another place with --bin-dir, --gh or TMPDIR. nodkey update does not pass TMPDIR on and always uses /tmp: to use another TMPDIR, run install.sh as root. A Debian system that was set up long ago may still have /usr/local/bin writable by the group staff.
systemd does not start the broker: systemctl status nodkey shows status=243/CREDENTIALS. systemd cannot decrypt a credential, so the broker does not run, and nothing is handed out. Check that each file of a LoadCredentialEncrypted= line is there, and was made with that name. After a firmware update or a change of the Secure Boot settings, the TPM2 chip may refuse credentials that are bound to PCR 7 (section 3). If you did not update the firmware or change Secure Boot, find out why first: someone may have changed how the machine starts. Then make each credential again, as in section 13: a new TLS key (and its pin on each client), the bot token, a new bws token, the bw master password, and new SSH keys.
The journal says that the bot may not send to this chat. You did not tap Start in the chat with the bot, or you blocked it.
The client prints nodkey: warning: ... with the versions of this machine and the broker. They run different versions of nodkey. Update the older side, as the line says (section 13). Requests can fail until then, for example with the answer of the broker is not valid.
A silent notice says that a client runs another version of nodkey, or sent no version. Update nodkey on the older side, as the notice says (section 13). The broker sends one such notice for each client in each start, rarely more than one, and the journal has a line about it.
a store could not start The broker runs without that store, denies requests for its secrets, and tries again with each request. The error names the store and the step. For bw: check the login (section 5). For bw, the error ends with the check env -i -C HOME HOME=HOME PATH=PATH PROGRAM status: run it as the broker's user, never bw unlock, bw lock or bw logout. For bws, it points to section 5: the token goes in the environment, never in an argument.
The broker does not start, and the error says unknown key ... the key may be from a newer release. The config has a key that this nodkey does not know. It is a typo, or the key is from a newer release (section 13). The error names the version that runs.
The client says config: ... no such file or directory and points to section 11 (exit 69). There is no client config. Make it as in section 11.
A command prints one line that ends with see "nodkey CMD --help" (exit 64). You used the command wrongly. The line says how. Run nodkey CMD --help for the usage.
The broker refuses a file: the owner, the mode, or a directory that group or others can write to. The broker checks every file that it reads, and every directory on the way (DESIGN.md section 14). Fix the owner or the mode that the error names.
cannot listen on ... The listen address is not on the broker host yet, such as before the VPN is up. systemd tries again every 5 seconds.
The client says the broker's TLS key is not the pinned key (exit 69). broker_pin in the client config is not the pin of the broker's key. The broker logs its pin at each start.
The client says that the broker refused the TLS connection (exit 69). The broker does not know the key of this client. The broker logs refused a client key that no [client.*] section pins, with the pin. Put the pin into a [client.NAME] section, and restart the broker.
The client says cannot connect to the broker (exit 69). Check the address in the client config, the firewall, and that the broker runs.
The client says the broker answered 503 Service Unavailable (exit 69). The broker could not finish the request: Telegram, a store or the audit log failed, or the broker was starting or stopping. Its journal says why: the error says journalctl -u nodkey.
The client says denied or denied by a limit of the broker (exit 77) at once, without a message. A check or a limit denied the request: a secret or key that this client may not use, or the pause of 60 seconds after a Deny. The journal of the broker says which.
ssh asks for a password, or says agent refused operation. nodkey ssh-agent prints why. Check SSH_AUTH_SOCK, that the host is in the broker's known_hosts, and that ssh does not use a host certificate (section 11).