httpalso layout discovery short-circuits on Right False, breaks fsck for non-first layouts

[yarikoptic]

After commit 5d1baa5 "httpalso: Fix bugs in handling content not being
present on the remote" (released in 10.20260316), httpalso layout
discovery no longer falls back when the first probed layout returns
Right False (HTTP 404). fsck --from <httpalso> falsely reports content
missing when the content lives under a non-first layout.

> git -C main annex fsck --from=httpalso_bad   # content stored as flat baseurl/<key>
fsck file.dat (httpalso_bad)
  ** Failed to verify content of file.dat (httpalso_bad):
  expected to be present, but its content is missing

A control httpalso remote with content stored at the first-tried
hashDirLower(2) layout works fine. The HTTP access log shows exactly one
HEAD request to the first layout (404) — no fallback HEAD to other layouts.

Root cause

5d1baa5 widened the downloader return type in keyUrlAction from
Either String () to Either String a. The existing iterator in
Remote/HttpAlso.hs:177-185:

go' (k, urlcontent) = \case
    [] -> ...
    (mklayout:rest) -> do
        r <- ...
        case r of
            Right _ -> ...   -- ← short-circuits on Right False too
            Left  _ -> go' (k, urlcontent) rest

was correct when a = () (only success was representable), but
checkKey' at Remote/HttpAlso.hs:143-145 returns Right Bool, so
Right False (definitive "content absent at this layout") now also
short-circuits and prevents fallback.

Suggested fix: distinguish layout-probe failure (try next layout) from
content-absence (definitive answer), e.g.:

case r of
    Right True  -> ...                 -- found
    Right False -> go' (k, urlcontent) rest  -- try next layout
    Left  _     -> go' (k, urlcontent) rest

Reproducer (POSIX shell, exits non-zero when bug fires)

#!/bin/sh
# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko <yaroslav.o.halchenko@dartmouth.edu>
# SPDX-License-Identifier: MIT
#
# Reproducer for the git-annex httpalso layout-discovery bug introduced in
# commit 5d1baa5026 ("httpalso: Fix bugs in handling content not being
# present on the remote", first released in 10.20260316).
#
# Claim: keyUrlAction in Remote/HttpAlso.hs treats a confident "Right False"
# (HTTP 404 -- content not present at the tried URL) the same way as a
# successful "Right True" (content present): it stops iterating the
# supported layouts at the first definitive answer. Therefore if content
# actually lives under a non-first layout, checkpresent locks onto the
# first layout's 404 and reports the content as absent even though it is
# reachable at a later layout's URL.
#
# Strategy:
#   * primary remote: a "directory" special remote (purely so that
#     `git annex initremote --sameas` is satisfied). It does NOT contain
#     the actual data.
#   * publish a single annex key on a local HTTP server under the FLAT
#     baseurl/<keyfile> layout (the SECOND outer entry in
#     supportedLayouts). The FIRST outer entry (hashDirLower /
#     hashDirMixed at HashLevels 2) therefore returns HTTP 404 ->
#     Right False.
#   * verify with a control: also stage the same data under the first
#     layout in a SEPARATE HTTP root + httpalso remote, and confirm
#     fsck succeeds there.
#   * verify with the experiment: against the flat-only HTTP root,
#     a fixed httpalso would iterate to the flat layout and locate the
#     content; the buggy one stops at the first Right False.
#
# Exits non-zero when the bug is observed.

set -eux
PS4='> '

TMP="$(mktemp -d)"
# pick a port unlikely to collide with anything else, parameterised on PID
PORT_BAD=$(( 30000 + ($$ % 15000) ))
PORT_OK=$(( PORT_BAD + 1 ))

# 1) Source repo: stage one annexed file, capture its key.
git init -q -b main "$TMP/src"
cd "$TMP/src"
git config user.email t@t
git config user.name t
git annex init -q src
printf 'hello-httpalso\n' > hello.txt
git annex add -q hello.txt
git commit -q -m add
KEY="$(git annex lookupkey hello.txt)"
test -n "$KEY"
CONTENT="$(git annex contentlocation "$KEY")"

# 2a) BAD layout root: publish the key ONLY under the flat baseurl/<key>
#     layout. Layout #1 (hashDirLower(2)) will 404.
SRV_BAD="$TMP/srv-bad"
mkdir -p "$SRV_BAD"
cp "$TMP/src/$CONTENT" "$SRV_BAD/$KEY"

# 2b) OK layout root: publish the key under hashDirLower(2)/<key>/<key>
#     (the very first layout supportedLayouts tries).
SRV_OK="$TMP/srv-ok"
# Compute the hashDirLower(2) path the way git-annex does. The Haskell
# code: baseurl </> toInternalGitPath (hashDirLower (HashLevels 2) k) </>
# keyFile k </> keyFile k.
#
# Instead of replicating the hash math here, fish it out of git-annex
# itself via examinekey.
HASHDIR2_LOWER="$(git annex examinekey --format='${hashdirlower}' "$KEY")"
KEYDIR="$SRV_OK/${HASHDIR2_LOWER}${KEY}"
mkdir -p "$KEYDIR"
cp "$TMP/src/$CONTENT" "$KEYDIR/$KEY"
ls -R "$SRV_OK" | head -20

# 3) Start two HTTP servers; wait for each.
cd "$SRV_BAD"
python3 -m http.server "$PORT_BAD" --bind 127.0.0.1 >"$TMP/http_bad.log" 2>&1 &
HTTP_BAD_PID=$!
cd "$SRV_OK"
python3 -m http.server "$PORT_OK"  --bind 127.0.0.1 >"$TMP/http_ok.log"  2>&1 &
HTTP_OK_PID=$!
i=0
until curl -fsS "http://127.0.0.1:$PORT_BAD/$KEY" -o /dev/null \
   && curl -fsS "http://127.0.0.1:$PORT_OK/${HASHDIR2_LOWER}${KEY}/$KEY" -o /dev/null; do
    i=$((i+1))
    if [ $i -ge 80 ]; then
        echo "one or both servers did not start"
        cat "$TMP/http_bad.log" "$TMP/http_ok.log"
        kill $HTTP_BAD_PID $HTTP_OK_PID 2>/dev/null
        exit 2
    fi
    sleep 0.1
done

# 4) Consumer repo with a directory primary and TWO httpalso aliases:
#    httpalso_bad -> flat-layout server, httpalso_ok -> hashDir-layout server.
git init -q -b main "$TMP/dst"
cd "$TMP/dst"
git config user.email t@t
git config user.name t
git annex init -q dst
git config annex.security.allowed-http-addresses all
git config annex.security.allowed-ip-addresses all

DIRREMOTE="$TMP/dirstore"
mkdir -p "$DIRREMOTE"
git annex initremote -q dirstore type=directory directory="$DIRREMOTE" encryption=none

git annex initremote -q httpalso_ok \
    --sameas=dirstore \
    type=httpalso url="http://127.0.0.1:$PORT_OK/" \
    encryption=none

git annex initremote -q httpalso_bad \
    --sameas=dirstore \
    type=httpalso url="http://127.0.0.1:$PORT_BAD/" \
    encryption=none

# Bring in the file pointer and the git-annex branch.
git remote add src "$TMP/src"
git fetch -q src
git checkout -q src/main -- hello.txt
git annex sync -q --no-push --no-pull || true

# Declare the key as present on the shared (dirstore/httpalso_*) UUID.
DIR_UUID="$(git config remote.dirstore.annex-uuid)"
test -n "$DIR_UUID"
git annex setpresentkey "$KEY" "$DIR_UUID" 1

# 5) CONTROL: fsck via httpalso_ok must succeed (first layout matches).
set +e
git annex fsck --from httpalso_ok hello.txt > "$TMP/fsck_ok.log" 2>&1
FSCK_OK_RC=$?
set -e
echo "=== fsck_ok.log (control) ==="; cat "$TMP/fsck_ok.log"; echo "============================"

# 6) EXPERIMENT: fsck via httpalso_bad. With the bug, this fails because
#    the first layout 404s, returns Right False, and the iterator stops.
set +e
git annex fsck --from httpalso_bad hello.txt > "$TMP/fsck_bad.log" 2>&1
FSCK_BAD_RC=$?
set -e
echo "=== fsck_bad.log (experiment) ==="; cat "$TMP/fsck_bad.log"; echo "================================="

# Show which URLs httpalso actually requested on each server.
echo "=== http_bad.log ==="; cat "$TMP/http_bad.log"; echo "===================="
echo "=== http_ok.log  ===";  cat "$TMP/http_ok.log";  echo "===================="

# Shut down HTTP servers (no trap per convention).
kill $HTTP_BAD_PID $HTTP_OK_PID 2>/dev/null || true
wait $HTTP_BAD_PID $HTTP_OK_PID 2>/dev/null || true

# 7) Verdict.
#    Control should not show a failure; if it does, our setup is wrong
#    and we cannot make a claim.
if [ $FSCK_OK_RC -ne 0 ]; then
    echo "INDETERMINATE: control fsck (against the hashDir-layout server) failed; setup is wrong."
    exit 2
fi
if grep -qE 'fsck: [1-9]|expected to be present, but its content is missing' "$TMP/fsck_ok.log"; then
    echo "INDETERMINATE: control fsck did not see content; setup is wrong."
    exit 2
fi

# Experiment failing the same way the bug predicts.
if [ $FSCK_BAD_RC -ne 0 ] || grep -qE 'fsck: [1-9]|expected to be present, but its content is missing' "$TMP/fsck_bad.log"; then
    echo "BUG REPRODUCED: httpalso layout iterator stopped at first Right False (HTTP 404) and did not fall back to the flat layout where content lives."
    exit 1
fi
echo "NOT REPRODUCED: httpalso located content despite first-layout 404."
exit 0

See reproducer-httpalso-layout-lockin.sh — runs python3 -m http.server,
stages two httpalso remotes (one with content at hashDirLower(2)/... as a
control, one with content at flat baseurl/<key>), runs fsck --from= on
each. Control passes (exit 2 if it fails), experiment fails (exit 1) → bug
reproduced.

Regression evidence

Before commit 5d1baa5 the iterator's downloader returned Either String ().
Only Right () was representable; the Right _ short-circuit was correct.
The reproducer's fsck --from=httpalso_bad (content stored at flat
baseurl/<key>) would have succeeded by falling back through layouts on
Left _.

5d1baa5026 widened the return type to Either String a. In checkKey'
this a is Bool — and Right False (definitive "absent at this layout")
now also short-circuits in the iterator, preventing fallback.

The commit's stated purpose was "httpalso: Fix bugs in handling content not
being present on the remote" — i.e. it intentionally wanted to distinguish
absent-from-this-layout from error. But the iterator at lines 177-185 was not
updated to match the wider return type. The release notes for 10.20260316
describe only the intended fix:

httpalso: Fix bugs in handling content not being present on the remote.

The breaking of layout fallback is not mentioned and is clearly an
unintended side effect.

Version

Verified with: git-annex 10.20260421
Confirmed present in: 10.20260525 (Remote/HttpAlso.hs untouched in the interim)