Fix CLI error-handling papercuts and make the gate pass on macOS (#163)

## Summary

Manual QA across the command surface (run via parallel subagents)
surfaced a cluster of **user-input errors mislabeled as internal "report
a bug" failures**, plus a few smaller inconsistencies. While verifying
the fixes through `scripts/check.sh`, two **macOS-only gate failures**
also turned up and are fixed here so the gate is green on both macOS and
Linux.

## CLI fixes

| Command | Before | After |
|---|---|---|
| `webhooks listen --port` | out-of-range port → `Unexpected error …
report a bug` | `min=0/max=65535` → clean exit-2 validation error |
| `init <tmpl> file/sub` | raw `[Errno 20] Not a directory` mid-scaffold
| rejected up front as a clean usage error |
| `stream --redact-pii-sub` | bad value leaked a pydantic dump +
`errors.pydantic.dev` URL | now a `PIISubstitutionPolicy` enum → clean
`[hash\|entity_name]` choice error |
| `clip --out-dir <file>` | "doesn't exist" (misleading) | "is not a
directory" |
| `config path` | failed on a corrupt `config.toml` — the one command
you'd use to find the broken file | reports the location anyway
(deferred parse error; real commands still re-raise it; a bad `--env`
still wins) |
| `llm` (bad key) | exit 1 + raw 401 dict body | exit-4 `auth_failure`
like `transcribe`; entitlement/proxy 403s keep their exit-1 passthrough
|
| account commands | two confusingly-different no-credentials messages |
reworded to explain *why* account data needs a browser session vs an API
key |
| `telemetry enable` | "✓ Telemetry enabled" even when
`DO_NOT_TRACK`/`AAI_TELEMETRY_DISABLED` overrides it | appends a note
that the env kill-switch keeps it off |

## Gate fixes (`scripts/check.sh`) — macOS parity

- **`brew audit`**: Homebrew 6+ disabled `brew audit [path …]`; a
formula must be audited *by name*. The step now copies
`Formula/assembly.rb` into an ephemeral local tap and audits it by name
(works on macOS + Linuxbrew, old + new brew), with cleanup that survives
`set -e`.
- **"no new escape hatches"**: the baseline count used `git grep -E`
while the working tree used `rg`, which disagree on `\b` — macOS's ERE
engine silently ignores it, so a *pre-existing* `time.sleep` made the
working count exceed the baseline and failed the gate on macOS only.
Both sides now use one matcher, `git grep -P` (PCRE), via shared
`hatch_base`/`hatch_work` helpers (`--untracked` counts newly-added
files the way `rg` did).
- Also fixes `test_validate_out_rejects_the_input_via_hard_link` to
tolerate case-insensitive filesystems (macOS).

## Verification

`./scripts/check.sh` → **All checks passed** (exit 0), including:
- 2556 tests pass
- 100% patch coverage on all changed source files
- mutation gate: 16/16 mutants killed on changed lines
- brew audit, escape-hatch gate, build + twine — all green

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Alex Kroman <alex@assemblyai.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: 3 people

aai_cli/app/context.py

@@ -26,6 +26,10 @@ class AppState:
     profile: str | None = None
     env: str | None = None
     quiet: bool = False
+    # Set by the root callback when config.toml is unreadable during environment
+    # resolution: most commands re-raise it (run_command), but `config path` — the
+    # command you reach for to *find* the broken file — tolerates it.
+    deferred_config_error: CLIError | None = None
 
     def resolve_profile(self) -> str:
         """The profile to act on: explicit --profile, else the active profile.
@@ -64,10 +68,12 @@ def resolve_session(self) -> tuple[int, str]:
             # never satisfy these endpoints — they authenticate with the browser
             # session, not the API key — so spell out the only fix that works.
             raise NotAuthenticated(
-                "These commands need a browser login. Run 'assembly login' (without --api-key).",
+                "Account commands read account-level data and authenticate with your "
+                "browser-login session, which this profile doesn't have.",
                 suggestion=(
-                    "Run 'assembly login' to sign in via your browser — an API key alone "
-                    "can't access account commands."
+                    "Run 'assembly login' to sign in via your browser. Unlike transcription "
+                    "commands (which work with an API key), account data like balance, usage, "
+                    "and keys needs that session."
                 ),
             )
         # Registered like the API key in config.resolve_api_key: -v/-vv diagnostics
@@ -197,16 +203,34 @@ def run_command(
     *,
     json: bool = False,
     auto_login: bool = True,
+    tolerate_unreadable_config: bool = False,
 ) -> None:
-    """Execute a command body, mapping CLIError to clean output + exit code."""
+    """Execute a command body, mapping CLIError to clean output + exit code.
+
+    `tolerate_unreadable_config` lets a command (only `config path`) run even when the
+    root callback deferred a corrupt-config error, so it can still report the file's
+    location; every other command re-raises that error here.
+    """
     state: AppState = ctx.obj
     json_mode = output.resolve_json(explicit=json)
+    deferred = state.deferred_config_error
+    if deferred is not None and not tolerate_unreadable_config:
+        # The root callback couldn't read config.toml. Surface that for ordinary
+        # commands (which depend on it) the same way the callback used to — emit and
+        # exit, without telemetry/update-check, both of which would just re-parse it.
+        _fail(deferred, json_mode=json_mode)
     try:
-        # Inside the try so telemetry sees the raw CLIError (and its error_type)
-        # before it's folded into a typer.Exit below.
-        with telemetry.track(ctx.command_path):
+        if deferred is not None:
+            # `config path` opted in (tolerate_unreadable_config): it reports a
+            # contents-independent location, so run just the body and skip the
+            # telemetry/update-check wrappers that re-parse the broken config.
             fn(state, json_mode)
-        update_check.maybe_notify(json_mode=json_mode)
+        else:
+            # Inside the try so telemetry sees the raw CLIError (and its error_type)
+            # before it's folded into a typer.Exit below.
+            with telemetry.track(ctx.command_path):
+                fn(state, json_mode)
+            update_check.maybe_notify(json_mode=json_mode)
     except NotAuthenticated as err:
         if not auto_login or not _should_auto_login(err):
             _fail(err, json_mode=json_mode)

aai_cli/app/init_exec.py

@@ -141,6 +141,24 @@ def _install_step(
     ], will_launch
 
 
+def _reject_file_ancestor(target: Path) -> None:
+    """Reject a target that descends through an existing file (e.g. ``somefile/app``).
+
+    ``scaffold`` calls ``target.mkdir(parents=True)``, which raises a raw
+    ``NotADirectoryError`` mid-scaffold when a parent component is a regular file —
+    surfacing as an "Unexpected error … report a bug" line for what is really a bad
+    path. Catch it up front as a clean usage error instead.
+    """
+    for ancestor in target.parents:
+        if ancestor.exists():
+            if not ancestor.is_dir():
+                raise UsageError(
+                    f"{ancestor} is not a directory, so {target} can't be created.",
+                    suggestion="Pick a target whose parent directories are real directories.",
+                )
+            return
+
+
 def _resolve_target(
     directory: str | None, chosen: str, *, here: bool, force: bool
 ) -> tuple[Path, bool]:
@@ -155,6 +173,7 @@ def _resolve_target(
     target = _resolve_dir(directory, chosen, here=here)
     if target.exists() and not target.is_dir():
         raise UsageError(f"{target} exists and is not a directory.")
+    _reject_file_ancestor(target)
     conflict = scaffold.target_conflict(target)
     if conflict and not force:
         raise CLIError(

aai_cli/commands/clip/_exec.py

@@ -193,11 +193,19 @@ def _transcript_segments(
 
 
 def _validate_out_dir(out_dir: Path | None) -> None:
-    if out_dir is not None and not out_dir.is_dir():
+    if out_dir is None or out_dir.is_dir():
+        return
+    # Distinguish a missing path from one that exists but isn't a directory: the old
+    # "doesn't exist" wording was misleading when --out-dir pointed at a regular file.
+    if out_dir.exists():
         raise UsageError(
-            f"--out-dir doesn't exist: {out_dir}",
-            suggestion="Create it first, or point --out-dir at an existing directory.",
+            f"--out-dir is not a directory: {out_dir}",
+            suggestion="Point --out-dir at a directory, not a file.",
         )
+    raise UsageError(
+        f"--out-dir doesn't exist: {out_dir}",
+        suggestion="Create it first, or point --out-dir at an existing directory.",
+    )
 
 
 def _validate_selection(opts: ClipOptions) -> None:

aai_cli/commands/config.py

@@ -112,7 +112,9 @@ def body(_state: AppState, json_mode: bool) -> None:
             # unwrapped (`cd "$(assembly config path | xargs dirname)"`).
             output.emit_text(str(file))
 
-    run_command(ctx, body, json=json_out)
+    # The location is independent of the file's contents, so report it even when the
+    # config is unreadable — this is the command you'd use to go fix the broken file.
+    run_command(ctx, body, json=json_out, tolerate_unreadable_config=True)
 
 
 @app.command(

aai_cli/commands/stream/__init__.py

@@ -3,6 +3,7 @@
 from pathlib import Path
 
 import typer
+from assemblyai import PIISubstitutionPolicy
 from assemblyai.streaming.v3 import Encoding, NoiseSuppressionModel, SpeechModel
 
 from aai_cli import command_registry, help_panels, options
@@ -210,7 +211,7 @@ def stream(
         help="Comma-separated PII policies",
         rich_help_panel=help_panels.OPT_GUARDRAILS,
     ),
-    redact_pii_sub: str | None = typer.Option(
+    redact_pii_sub: PIISubstitutionPolicy | None = typer.Option(
         None,
         "--redact-pii-sub",
         help="Replace redacted PII with: hash or entity_name",

aai_cli/commands/stream/_exec.py

@@ -13,6 +13,7 @@
 from dataclasses import dataclass
 from pathlib import Path
 
+from assemblyai import PIISubstitutionPolicy
 from assemblyai.streaming.v3 import Encoding, NoiseSuppressionModel, SpeechModel
 
 from aai_cli import code_gen
@@ -67,7 +68,7 @@ class StreamOptions:
     filter_profanity: bool | None
     redact_pii: bool | None
     redact_pii_policy: str | None
-    redact_pii_sub: str | None
+    redact_pii_sub: PIISubstitutionPolicy | None
     webhook_url: str | None
     webhook_auth_header: str | None
     llm_prompt: list[str] | None
@@ -111,7 +112,7 @@ def base_flags(self) -> dict[str, object]:
             "voice_focus_threshold": self.voice_focus_threshold,
             "redact_pii": self.redact_pii,
             "redact_pii_policies": config_builder.split_csv(self.redact_pii_policy),
-            "redact_pii_sub": self.redact_pii_sub,
+            "redact_pii_sub": config_builder.enum_value(self.redact_pii_sub),
             "inactivity_timeout": self.inactivity_timeout,
             "webhook_url": self.webhook_url,
             "prompt": self.prompt,

aai_cli/commands/telemetry.py

@@ -92,11 +92,24 @@ def enable(
 
     def body(_state: AppState, json_mode: bool) -> None:
         config.set_telemetry_enabled(enabled=True)
-        output.emit(
-            {"telemetry_enabled": True},
-            lambda _d: output.success("Telemetry enabled."),
-            json_mode=json_mode,
-        )
+        # An env kill-switch (AAI_TELEMETRY_DISABLED / DO_NOT_TRACK) outranks the stored
+        # choice, so persisting "enabled" wouldn't actually turn telemetry back on while
+        # it's set — say so instead of an unqualified success.
+        source = telemetry.consent_source()
+        env_var = source.removeprefix("env:") if source.startswith("env:") else None
+
+        def render(_d: dict[str, bool]) -> object:
+            line = output.success("Telemetry enabled.")
+            if env_var is None:
+                return line
+            return output.stack(
+                line,
+                output.hint(
+                    f"Note: {env_var} is set, which keeps telemetry off until you unset it."
+                ),
+            )
+
+        output.emit({"telemetry_enabled": True}, render, json_mode=json_mode)
 
     run_command(ctx, body, json=json_out)
 

aai_cli/commands/webhooks/__init__.py

@@ -37,7 +37,11 @@
 def listen(
     ctx: typer.Context,
     port: int = typer.Option(
-        8989, "--port", help="Local listener port (the first free port from here)"
+        8989,
+        "--port",
+        min=0,
+        max=65535,
+        help="Local listener port (the first free port from here)",
     ),
     forward_to: str | None = typer.Option(
         None, "--forward-to", help="Re-POST each delivery to this URL (e.g. your local app)"

aai_cli/core/llm.py

@@ -5,7 +5,7 @@
 from typing import TYPE_CHECKING
 
 from aai_cli.core import environments
-from aai_cli.core.errors import APIError, UsageError
+from aai_cli.core.errors import APIError, UsageError, auth_failure
 
 if TYPE_CHECKING:
     from openai import OpenAI
@@ -116,12 +116,18 @@ def _client(api_key: str) -> OpenAI:
 )
 
 
+def _is_entitlement_denial(exc: object) -> bool:
+    """True when a gateway 401/403 reads as a plan-entitlement block rather than a
+    bad key or an intercepting proxy."""
+    text = f"{exc} {getattr(exc, 'body', None) or ''}".lower()
+    return any(hint in text for hint in _ENTITLEMENT_HINTS)
+
+
 def _denial_suggestion(exc: object) -> str:
     """Pick the suggestion for a gateway 401/403: point at billing only when the
     response actually mentions the plan entitlement, otherwise at key/network —
     a corporate-proxy 403 must not send users to the billing page."""
-    text = f"{exc} {getattr(exc, 'body', None) or ''}".lower()
-    if any(hint in text for hint in _ENTITLEMENT_HINTS):
+    if _is_entitlement_denial(exc):
         return _PAID_PLAN_SUGGESTION
     return _ACCESS_DENIED_SUGGESTION
 
@@ -159,9 +165,14 @@ def complete(
         )
     except (openai.AuthenticationError, openai.PermissionDeniedError) as exc:
         # The gateway returns 401/403 for an invalid key, a proxy block, and a
-        # plan entitlement block ("no access to LLM Gateway"), so surface its
-        # actual message and pick the suggestion from what it says — only an
-        # entitlement message should point at billing.
+        # plan entitlement block ("no access to LLM Gateway"). A plain 401
+        # (AuthenticationError) with no entitlement hint is just a rejected key, so
+        # surface the same clean exit-4 auth_failure transcribe gives instead of
+        # echoing the gateway's raw 401 body. A 403 (proxy or entitlement) keeps the
+        # gateway's own message and picks the suggestion from what it says — only an
+        # entitlement message should point at billing, never a corporate-proxy 403.
+        if isinstance(exc, openai.AuthenticationError) and not _is_entitlement_denial(exc):
+            raise auth_failure() from exc
         raise APIError(
             f"LLM Gateway access denied: {exc}",
             suggestion=_denial_suggestion(exc),

aai_cli/main.py

@@ -190,8 +190,19 @@ def main(
     try:
         environments.set_active(state.resolve_environment())
     except CLIError as err:
-        output.emit_error(err, json_mode=json_mode)
-        raise typer.Exit(code=err.exit_code) from None
+        if err.error_type != "invalid_config":
+            output.emit_error(err, json_mode=json_mode)
+            raise typer.Exit(code=err.exit_code) from None
+        # A corrupt config.toml can't tell us the profile's stored env. Defer the error
+        # (run_command re-raises it for real commands) and fall back to the explicit
+        # --env or the default, so `assembly config path` can still locate the file.
+        # An explicit bad --env still wins — surface it now rather than the deferred one.
+        state.deferred_config_error = err
+        try:
+            environments.set_active(environments.resolve(state.env, None))
+        except CLIError as env_err:
+            output.emit_error(env_err, json_mode=json_mode)
+            raise typer.Exit(code=env_err.exit_code) from None
     active_env = environments.active()
     _LOG.debug("environment: %s (%s)", active_env.name, active_env.api_base)
     for warning in (conflict_warning, state.env_override_warning()):