Skip to content

design.md

Latest commit

 

History

History
269 lines (218 loc) · 13.4 KB

design.md

File metadata and controls

269 lines (218 loc) · 13.4 KB
Raw

Design

Why the pieces are the way they are, what is deliberately not done, and where this is fragile.

Why two detectors

A forced command= in authorized_keys only runs once the client opens a session channel. A client that authenticates and disconnects, or runs ssh -N (including ssh -N -L ...), never opens one — so the forced command never runs and, with it alone, nothing at all would be reported. That is not an exotic case: confirming a stolen key works, without doing anything with it yet, is exactly what a careful intruder does first.

The auth-log watcher closes that gap. The canary account is used by nothing else, so every Accepted publickey for <account> line in sshd's log is a hit by definition, whatever the client does next.

Keeping both is not redundancy for its own sake — they fail differently:

Forced command Auth-log watcher
Depends on sshd's command=, the account's shell sshd's log wording, journald, the service running
Knows the label from its own argument the key fingerprint
Survives a reworded log line a broken authorized_keys, a wrong shell

During this project's own deployment, a nologin shell silently killed the forced command — and the watcher still reported every hit. That is the argument for two, in one incident.

Both call send_alert(), which rate-limits per label, so a normal login trips both and still produces one mail.

Why a real login is never possible

Not one mechanism — every path is closed, and command= is only the last one.

Path Why it is closed
Password over SSH PasswordAuthentication no, and the account's password field is ! (locked)
Keyboard-interactive KbdInteractiveAuthentication no
Some other key authorized_keys is root-owned in a root-owned .ssh, in a root-owned home. The account cannot add, edit, replace or rename it
The bait key sshd runs $SHELL -c "/opt/canary/trigger.sh <label>". The client's requested command lands in SSH_ORIGINAL_COMMAND and is only ever logged — never executed, never passed to a shell
PTY, port/agent/X11/socket forwarding restrict in the key, plus DisableForwarding yes and PermitTTY no in the Match block
su to the account Locked password; only root could, and root has no reason to
sudo No sudo rule grants it
Local console Locked password
Cron as the account Only the selftest line; the crontab is owned by root

The load-bearing rows are the first two. Even if command= vanished, nobody reaches the account without the private key — and whoever has that key is precisely who you want to hear about.

trigger.sh never evaluates SSH_ORIGINAL_COMMAND. It is logged after control characters are stripped, and HTML-escaped in the mail. Do not change that.

Why the shell is /bin/bash

/usr/sbin/nologin is the intuitive choice and it is wrong here: sshd runs a forced command through the login shell, so nologin means the trigger never runs, nothing is logged, no mail is sent, and the client is told This account is currently not available. The canary is dead and announces itself in the same breath.

The protections nologin was standing in for are the locked password, disabled password authentication, and the absence of any sudo rule — all verified by bin/install-server.sh before it will install anything.

Why not make trigger.sh the login shell instead of using command=?

Tempting: enforcement would live in /etc/passwd (root-owned) rather than in authorized_keys. But:

  • The label dies. sshd invokes the shell as shell -c "<client command>", so trigger.sh would receive -c as $1 instead of its label. Every key would report the same wrong label, and knowing which planted copy fired is the main thing the alert is for.
  • The selftest cron breaks. cron uses the account's passwd shell, so the monthly run would execute trigger.sh instead of the alert.
  • Debugging gets unpleasant — every su -s walks into the trap.

Instead, authorized_keys is made root-owned, which puts enforcement outside the account's reach without giving up the label.

Why no pty

Denying the pty makes the ssh client print

PTY allocation request failed

before the session hangs. It is tempting to grant restrict,pty to remove that line, on the theory that a silent hang is better camouflage. It was briefly done here, and then reverted, because the argument does not survive being looked at:

The message is not a tell. It says the account is restricted — which is exactly what a backup or deploy service account is. Anyone who has used a forced-command deploy key, git-shell, or an rsync-only account has seen it. It corroborates the cover story rather than breaking it. Whatever is unusual about the session is the silent three-minute hold afterwards, and that is there with or without a pty.

A pty is real surface, small but not zero. It allocates a /dev/pts entry and a line discipline, which means the client can send control characters: Ctrl-C delivers SIGINT to the process group and ends CANARY_HOLD_SECONDS early. It also consumes a finite kernel resource per held session, giving a flood of connections something to exhaust that they otherwise could not. And it exercises more code in sshd and the kernel than refusing the request does.

It grants nothing useful in exchange: no shell is started, trigger.sh never reads stdin, and everything else restrict blocks stays blocked. So the trade is measurable surface against a cosmetic gain that is not even clearly a gain. Keep no-pty — that is, plain restrict in the key and PermitTTY no in the Match block.

Why no ForceCommand in sshd_config

It looks like the obvious second layer. It is not: ForceCommand overrides the per-key command= including its argument, so every key collapses to one unlabeled trigger. The four forwarding directives in the Match block do not have this problem, because they are independent of which command runs.

Smaller decisions

  • trigger.sh silences stdout and stderr on its first line. sshd forwards a forced command's output to the client over the session channel, so a bash diagnostic, a Python traceback naming these paths, or an SMTP error would land on the intruder's terminal. Everything reports through logger(1) instead. Do not add an echo.
  • The alert is sent before the hold, never after. CANARY_HOLD_SECONDS wastes the intruder's time and buys a reaction window; it must never gate the notification.
  • Mail delivery is synchronous. The process exists to send one mail and exit. A background thread would only add a way to fail silently.
  • The rate limit fails open. Any problem reading or writing /var/lib/canary results in the mail being sent, never suppressed. It caps mail, never logging: every hit reaches syslog regardless.
  • Delivery failures are loud in the log. send_email() returns an error string rather than raising, and the caller logs ALERT DELIVERY FAILED. An exception would reach the client; a swallowed error would reach nobody.
  • Standard library only, no shared code with other projects. An earlier version imported a neighbouring app's mail module. It bought one function and cost a copy that drifts, a dependency on that app's file permissions, and a config surface where a missing key silently produced a broken SMTP default. Twenty-five lines of smtplib are cheaper.

Portability and limits

Developed and tested on Ubuntu 24.04, OpenSSH 9.6p1, Python 3.12.3.

Hard requirements — the design does not work without these:

  • systemd + journald. The watcher runs journalctl -u <unit> -f. On a host with only rsyslog you would need to re-point it at /var/log/auth.log; the parsing itself would not change. The forced-command detector does not care.
  • OpenSSH ≥ 7.2 for restrict. Older versions need the explicit no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding list — which is exactly the list that silently misses AllowStreamLocalForwarding.
  • OpenSSH ≥ 6.8 for SHA256 fingerprints in the log. Older versions log MD5 and the watcher's fingerprint→label lookup finds nothing (it still alerts, as unknown-fingerprint).
  • Python ≥ 3.7 (subprocess text=, f-strings, EmailMessage).
  • /dev/log for syslog. Without it, logging degrades to nothing rather than crashing — deliberate, but you lose the log trail.

Distribution-specific, easy to adjust:

  • CANARY_SSHD_UNIT — ssh.service on Debian/Ubuntu, sshd.service on RHEL/Fedora/Arch. Configurable; verify with systemctl list-units --type=service | grep -i ssh.
  • trigger.sh calls binaries by absolute path (/usr/bin/logger, /bin/date, /usr/bin/timeout, …) on purpose — never trust a PATH near a forced command. The paths are the Debian/Ubuntu layout; on a distro without usrmerge some will differ and the trigger will fail loudly in syslog.
  • adduser --system in the setup docs is Debian/Ubuntu. useradd -r is the equivalent elsewhere.
  • passwd -S prints L on Debian/Ubuntu and LK on RHEL; the installer accepts both.

The fragile part. canary_authwatch.py finds hits by matching sshd's English log line:

Accepted publickey for <account> from <ip> port <n> ssh2: ED25519 SHA256:<fp>

That wording is not an API. It has been stable for many years and sshd does not localise its logs, but a future OpenSSH could reword it and the watcher would go quiet with no error anywhere — the worst failure mode there is. Two things reduce the exposure:

  1. The forced-command detector does not parse anything, so a reworded log line costs you the auth-only case, not all detection.
  2. bin/selftest.sh makes a real connection and asserts that both detectors reported it. It is the only check that catches this. The monthly cron on the host cannot — it proves the alert path, not the detection path.

Filtering the journal by unit rather than by syslog tag is deliberate for the same reason: on OpenSSH ≥ 9.8 that line is written by the sshd-session child rather than by sshd, so a tag filter would have broken on upgrade while a unit filter does not.

Verifying that forwarding is really blocked

Worth knowing before you go looking in the wrong place: sshd does not log a refused forwarding channel at the default LogLevel INFO. Measured on OpenSSH 9.6p1 — a refused -R, -L or -D leaves no trace in the server's journal beyond the ordinary Accepted publickey line. The evidence is on the client, and the alert is what tells you server-side.

# remote forward: fails immediately, one command
ssh -o ControlMaster=no -o ControlPath=none -o IdentitiesOnly=yes \
    -o ExitOnForwardFailure=yes -N -R 19999:127.0.0.1:22 \
    -i keys/canary_<label> <account>@<host>
# -> "Error: remote port forwarding failed for listen port 19999", exit 255

# SOCKS proxy: the realistic pivot attempt. Terminal 1:
ssh -o ControlMaster=no -o ControlPath=none -o IdentitiesOnly=yes \
    -N -D 19999 -i keys/canary_<label> <account>@<host>
# Terminal 2:
curl -s --max-time 5 --socks5 127.0.0.1:19999 http://example.com; echo $?
# -> exit 97 (CURLE_PROXY, "proxy handshake error"): ssh accepted the local
#    SOCKS connection, sshd then refused to open the channel.
# Terminal 1 prints: channel N: open failed: administratively prohibited

Both use -N, so no session channel is opened and trigger.sh never runs — the alert comes from the auth-log watcher alone. That makes this a good test of the second detector as well as of the forwarding block.

Deliberately not done

  • No fail2ban-style blocking. The point is to learn that a secret leaked, not to stop this one connection. Blocking would also tell the intruder they were detected.
  • No shell emulation or fake filesystem. That is a honeypot: much more code, much more attack surface, and it answers a different question.
  • No inbound network service. The only listener involved is the sshd you already run.
  • No alerting on failed authentication. Every internet-facing host sees constant scanner traffic; only a successful login with a key that should not exist is a signal.

Known gaps

  • Once a connection is up, everything after authentication is silent. You get one alert, when the key is used. An intruder who stays connected and keeps probing — a refused SOCKS or port forward for every request they push through it — generates no further alert, and no server-side log line either: measured on OpenSSH 9.6p1, a refused forwarding channel produces zero entries in the journal at LogLevel INFO, while the client sees channel N: open failed: administratively prohibited each time. So the number of alerts tells you nothing about how hard someone is trying. The first alert is the one that matters; treat it as "this key is burned", not as a live activity feed.
  • A stolen key used against a different host is invisible. The trap is on this host only.
  • Detection lag equals mail delivery. No push, no paging. If minutes matter, point CANARY_NOTIFY_EMAIL at a gateway that pages you.
  • The intruder learns the account and host from the bait itself — that is what makes it bait. They learn nothing more: no shell, no files, no forwarding.
  • journalctl -f can drop messages under journald's rate limit if the host is flooded with log traffic. Not observed in practice, but it is a way for a hit to be missed; the forced-command detector is unaffected.