Skip to content

Commit ff6a1cf

Browse files
kim-emKim Morrisonclaude
authored
test: cross-check every compiled clmul path (#9296)
hex-gf2's SPEC requires that tests exercise each compiled wrapper path of the carry-less multiply. Nothing did. The path is chosen by preprocessor guards, so a build runs exactly one of them, and the Lake target passes only -O3: x86-64 builds compile the portable fallback and never the intrinsic, while Apple silicon compiles vmull_p64 and never the fallback. Each machine tested one path and neither tested the other. Add HexGF2/ffi/clmul_selftest.c and scripts/ci/check_clmul_paths.sh, which compiles it against clmul.c twice -- plain, and with the flag that enables the host's intrinsic -- and cross-checks the intrinsic against the portable reference on 100000 deterministic pairs. The portable build is checked against known answers so it is not vacuous, and a build that asked for an intrinsic but did not get one fails rather than passing quietly. It runs in the existing lint job, taking a few seconds of cc rather than a new runner. To make the paths individually addressable, clmul.c now exports hex_clmul_portable, hex_clmul_intrinsic, and hex_clmul_uses_intrinsic instead of keeping the implementations static inline. HEX_CLMUL_NO_LEAN drops the export wrapper so the self-test compiles the arithmetic alone, needing neither Lean's headers nor its runtime. The dispatcher's behaviour is unchanged. Record in the SPEC how the requirement is met, and by which half: conformance/HexGF2/CrossCheck.lean already compares the extern against Hex.pureClmul, which covers the wrapper against the logical definition; this script covers the wrapper's paths against each other. Co-authored-by: Kim Morrison <kim@lean-fro.org> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 15a68c4 commit ff6a1cf

5 files changed

Lines changed: 225 additions & 13 deletions

File tree

.github/workflows/ci.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ jobs:
4545
run: python3 scripts/release/check_manual_split.py
4646
- run: python3 -m unittest scripts/release/test_sync_released.py
4747
- run: python3 scripts/release/check_trust_surface.py
48+
- name: Cross-check every compiled clmul path (HexGF2/SPEC/hex-gf2.md)
49+
run: bash scripts/ci/check_clmul_paths.sh
4850
- run: python3 scripts/check_phase4.py
4951
- name: Check Phase 7 chapters and anchored tutorials (PLAN/Phase7.md)
5052
run: |

HexGF2/SPEC/hex-gf2.md

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,28 @@ runtime CPU detection): x86-64 `__PCLMUL__` uses
7272
`vmull_p64`; otherwise it runs a portable shift-and-XOR mirroring
7373
`Hex.pureClmul`. Correctness of the intrinsic paths is trusted, same
7474
as the GMP externs in hex-arith. Tests must exercise each compiled
75-
wrapper path and the pure-Lean body (built without the extern
76-
attached) to catch divergence.
75+
wrapper path and the pure-Lean body to catch divergence.
76+
77+
Because the choice is made at compile time, one build runs exactly one
78+
path, and the Lake target passes only `-O3`, so ordinary builds compile
79+
the portable fallback. Covering the requirement therefore needs two
80+
compilations rather than one:
81+
82+
- `scripts/ci/check_clmul_paths.sh` compiles `HexGF2/ffi/clmul_selftest.c`
83+
against `clmul.c` twice, plain and with the flag that enables the host's
84+
intrinsic (`-mpclmul` on x86-64, `-march=armv8-a+crypto` on aarch64),
85+
and cross-checks the intrinsic against the portable reference on 100000
86+
deterministic pairs. A build that asked for an intrinsic and did not get
87+
one fails rather than passing quietly.
88+
- `conformance/HexGF2/CrossCheck.lean` compares the extern against
89+
`Hex.pureClmul` over a pseudorandom stream, which is the Lean half: it
90+
checks the compiled wrapper against the logical definition the proofs
91+
use.
92+
93+
`clmul.c` exposes `hex_clmul_portable`, `hex_clmul_intrinsic`, and
94+
`hex_clmul_uses_intrinsic` so both halves can address the paths
95+
individually; `HEX_CLMUL_NO_LEAN` drops the export wrapper so the
96+
self-test needs no Lean runtime.
7797

7898
**GF(2^n) elements.** Elements of `GF(2^n)` are polynomials of degree
7999
< n over F_2, reduced modulo an irreducible of degree n. This library

HexGF2/ffi/clmul.c

Lines changed: 35 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,32 @@
1+
/* `HEX_CLMUL_NO_LEAN` compiles only the arithmetic, without the Lean export
2+
wrapper and so without the Lean runtime. The self-test uses it: it wants to
3+
compare the two implementations, not to link a Lean binary. */
4+
#ifndef HEX_CLMUL_NO_LEAN
15
#include <lean/lean.h>
6+
#endif
27
#include <stdint.h>
38

9+
/* Which implementation this translation unit selects is a compile-time
10+
decision made by the guards below, not runtime CPU detection.
11+
`HEX_CLMUL_HAVE_INTRINSIC` records that decision so the self-test can report
12+
which path a given build exercised, and so a build that expected an
13+
intrinsic can tell that it did not get one. */
414
#if defined(__PCLMUL__) && (defined(__x86_64__) || defined(_M_X64))
15+
#define HEX_CLMUL_HAVE_INTRINSIC 1
16+
#define HEX_CLMUL_INTRINSIC_X86 1
517
#include <immintrin.h>
6-
#endif
7-
8-
#if defined(__ARM_FEATURE_CRYPTO) && (defined(__aarch64__) || defined(_M_ARM64))
18+
#elif defined(__ARM_FEATURE_CRYPTO) && (defined(__aarch64__) || defined(_M_ARM64))
19+
#define HEX_CLMUL_HAVE_INTRINSIC 1
20+
#define HEX_CLMUL_INTRINSIC_ARM 1
921
#include <arm_neon.h>
22+
#else
23+
#define HEX_CLMUL_HAVE_INTRINSIC 0
1024
#endif
1125

12-
static inline void hex_clmul_portable(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo) {
26+
/* The portable reference, mirroring `Hex.pureClmul`, which is the logical
27+
definition every proof reasons about. Exported rather than `static` so the
28+
self-test can compare it against an intrinsic path inside one binary. */
29+
void hex_clmul_portable(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo) {
1330
uint64_t out_hi = 0;
1431
uint64_t out_lo = 0;
1532
for (unsigned bit = 0; bit < 64; ++bit) {
@@ -27,30 +44,36 @@ static inline void hex_clmul_portable(uint64_t a, uint64_t b, uint64_t* hi, uint
2744
*lo = out_lo;
2845
}
2946

30-
#if defined(__PCLMUL__) && (defined(__x86_64__) || defined(_M_X64))
31-
static inline void hex_clmul_intrinsic(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo) {
47+
#if HEX_CLMUL_HAVE_INTRINSIC
48+
#ifdef HEX_CLMUL_INTRINSIC_X86
49+
void hex_clmul_intrinsic(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo) {
3250
__m128i lhs = _mm_set_epi64x(0, (long long)a);
3351
__m128i rhs = _mm_set_epi64x(0, (long long)b);
3452
__m128i prod = _mm_clmulepi64_si128(lhs, rhs, 0x00);
3553
*lo = (uint64_t)_mm_cvtsi128_si64(prod);
3654
*hi = (uint64_t)_mm_extract_epi64(prod, 1);
3755
}
38-
#elif defined(__ARM_FEATURE_CRYPTO) && (defined(__aarch64__) || defined(_M_ARM64))
39-
static inline void hex_clmul_intrinsic(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo) {
56+
#else
57+
void hex_clmul_intrinsic(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo) {
4058
poly64_t lhs = (poly64_t)a;
4159
poly64_t rhs = (poly64_t)b;
4260
poly128_t prod = vmull_p64(lhs, rhs);
4361
*lo = (uint64_t)prod;
4462
*hi = (uint64_t)(prod >> 64);
4563
}
4664
#endif
65+
#endif
66+
67+
/* Whether this build compiled an intrinsic path at all. */
68+
int hex_clmul_uses_intrinsic(void) {
69+
return HEX_CLMUL_HAVE_INTRINSIC;
70+
}
4771

72+
#ifndef HEX_CLMUL_NO_LEAN
4873
LEAN_EXPORT lean_obj_res lean_hex_clmul_u64(uint64_t a, uint64_t b) {
4974
uint64_t hi;
5075
uint64_t lo;
51-
#if defined(__PCLMUL__) && (defined(__x86_64__) || defined(_M_X64))
52-
hex_clmul_intrinsic(a, b, &hi, &lo);
53-
#elif defined(__ARM_FEATURE_CRYPTO) && (defined(__aarch64__) || defined(_M_ARM64))
76+
#if HEX_CLMUL_HAVE_INTRINSIC
5477
hex_clmul_intrinsic(a, b, &hi, &lo);
5578
#else
5679
hex_clmul_portable(a, b, &hi, &lo);
@@ -60,3 +83,4 @@ LEAN_EXPORT lean_obj_res lean_hex_clmul_u64(uint64_t a, uint64_t b) {
6083
lean_ctor_set(pair, 1, lean_box_uint64(lo));
6184
return pair;
6285
}
86+
#endif

HexGF2/ffi/clmul_selftest.c

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
/* Cross-check of the carry-less multiply implementations in clmul.c.
2+
*
3+
* `HexGF2/SPEC/hex-gf2.md` requires that tests exercise each compiled wrapper
4+
* path, not merely whichever one the host happens to select. A build picks its
5+
* path with preprocessor guards, so a single build can only ever run one of
6+
* them; `scripts/ci/check_clmul_paths.sh` therefore compiles this twice, once
7+
* plain and once with the flags that enable an intrinsic, and this program
8+
* reports which path it got.
9+
*
10+
* When an intrinsic is compiled it is compared against the portable reference
11+
* on every vector, which is the divergence the SPEC is worried about. When it
12+
* is not, the portable path is still checked against known-answer vectors, so
13+
* a plain build is not vacuous.
14+
*
15+
* The Lean side of the same obligation is `conformance/HexGF2/CrossCheck.lean`,
16+
* which compares the extern against `Hex.pureClmul` over a pseudorandom stream.
17+
*/
18+
#include <stdint.h>
19+
#include <stdio.h>
20+
21+
void hex_clmul_portable(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo);
22+
int hex_clmul_uses_intrinsic(void);
23+
#if defined(HEX_CLMUL_SELFTEST_EXPECT_INTRINSIC)
24+
void hex_clmul_intrinsic(uint64_t a, uint64_t b, uint64_t* hi, uint64_t* lo);
25+
#endif
26+
27+
/* Known answers, computed independently of this C code: the first three are
28+
hand-derived, the rest come from `Hex.pureClmul` evaluated in Lean. */
29+
struct known { uint64_t a, b, hi, lo; };
30+
static const struct known KNOWN[] = {
31+
{ 0u, 0u, 0u, 0u },
32+
{ 1u, 1u, 0u, 1u },
33+
{ 2u, 2u, 0u, 4u },
34+
{ 3u, 3u, 0u, 5u },
35+
{ 0xFFu, 0xFFu, 0u, 0x5555u },
36+
{ 0x8000000000000000ull, 2u, 1u, 0u },
37+
{ 0x8000000000000000ull, 0x8000000000000000ull, 0x4000000000000000ull, 0u },
38+
};
39+
40+
/* Deterministic MMIX linear congruential generator, the same one the Lean
41+
cross-check uses, so the two halves sweep comparable input shapes. */
42+
static uint64_t next(uint64_t* state) {
43+
*state = *state * 6364136223846793005ull + 1442695040888963407ull;
44+
return *state;
45+
}
46+
47+
int main(void) {
48+
int intrinsic = hex_clmul_uses_intrinsic();
49+
printf("clmul self-test: intrinsic path %s\n",
50+
intrinsic ? "compiled" : "not compiled (portable fallback)");
51+
52+
#if defined(HEX_CLMUL_SELFTEST_EXPECT_INTRINSIC)
53+
if (!intrinsic) {
54+
fprintf(stderr,
55+
"clmul self-test: expected an intrinsic build, but the guards "
56+
"selected the portable path; the flags did not take effect\n");
57+
return 2;
58+
}
59+
#endif
60+
61+
for (unsigned i = 0; i < sizeof(KNOWN) / sizeof(KNOWN[0]); ++i) {
62+
uint64_t hi, lo;
63+
hex_clmul_portable(KNOWN[i].a, KNOWN[i].b, &hi, &lo);
64+
if (hi != KNOWN[i].hi || lo != KNOWN[i].lo) {
65+
fprintf(stderr,
66+
"clmul self-test: portable disagrees with known answer at "
67+
"vector %u: got (%llx, %llx), want (%llx, %llx)\n",
68+
i, (unsigned long long)hi, (unsigned long long)lo,
69+
(unsigned long long)KNOWN[i].hi,
70+
(unsigned long long)KNOWN[i].lo);
71+
return 1;
72+
}
73+
}
74+
75+
#if defined(HEX_CLMUL_SELFTEST_EXPECT_INTRINSIC)
76+
uint64_t state = 0x243F6A8885A308D3ull;
77+
for (unsigned i = 0; i < 100000u; ++i) {
78+
uint64_t a = next(&state);
79+
uint64_t b = next(&state);
80+
uint64_t phi, plo, ihi, ilo;
81+
hex_clmul_portable(a, b, &phi, &plo);
82+
hex_clmul_intrinsic(a, b, &ihi, &ilo);
83+
if (phi != ihi || plo != ilo) {
84+
fprintf(stderr,
85+
"clmul self-test: intrinsic and portable disagree on "
86+
"(%llx, %llx): portable (%llx, %llx), intrinsic (%llx, %llx)\n",
87+
(unsigned long long)a, (unsigned long long)b,
88+
(unsigned long long)phi, (unsigned long long)plo,
89+
(unsigned long long)ihi, (unsigned long long)ilo);
90+
return 1;
91+
}
92+
}
93+
printf("clmul self-test: intrinsic agrees with portable on 100000 pairs\n");
94+
#else
95+
(void)next;
96+
#endif
97+
98+
printf("clmul self-test: OK\n");
99+
return 0;
100+
}

scripts/ci/check_clmul_paths.sh

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#!/usr/bin/env bash
2+
# Exercise every compiled carry-less-multiply path in HexGF2/ffi/clmul.c.
3+
#
4+
# `HexGF2/SPEC/hex-gf2.md` requires that tests exercise each compiled wrapper
5+
# path. A build selects its path with preprocessor guards, so one build runs
6+
# exactly one of them: the Lake target passes only `-O3`, which means ordinary
7+
# builds compile the portable fallback and never the intrinsic. This script
8+
# compiles the self-test a second time with the flags that enable an intrinsic,
9+
# so both paths are checked on every CI run rather than each machine silently
10+
# testing whichever one it happens to select.
11+
#
12+
# Extends the existing single job (SPEC/CI.md "no parallelism in CI"); it is a
13+
# few seconds of `cc`, not a new runner.
14+
set -euo pipefail
15+
16+
cd "$(dirname "$0")/../.."
17+
18+
CC="${CC:-cc}"
19+
WORK="$(mktemp -d)"
20+
trap 'rm -rf "$WORK"' EXIT
21+
22+
SRC="HexGF2/ffi/clmul.c"
23+
TEST="HexGF2/ffi/clmul_selftest.c"
24+
25+
# `HEX_CLMUL_NO_LEAN` drops the Lean export wrapper, so the self-test compiles
26+
# the arithmetic alone and needs neither Lean's headers nor its runtime.
27+
COMMON=(-DHEX_CLMUL_NO_LEAN -O2)
28+
29+
build_and_run() {
30+
local label="$1"; shift
31+
local out="$WORK/selftest_$label"
32+
echo "--- $label ---"
33+
if ! "$CC" "${COMMON[@]}" "$@" "$SRC" "$TEST" -o "$out" 2>"$WORK/err_$label"; then
34+
if [ "$label" = "portable" ]; then
35+
cat "$WORK/err_$label" >&2
36+
echo "check_clmul_paths: the portable build must compile" >&2
37+
exit 1
38+
fi
39+
echo "check_clmul_paths: $label did not compile on this host; skipping"
40+
sed 's/^/ /' "$WORK/err_$label" | head -5
41+
return 0
42+
fi
43+
"$out"
44+
}
45+
46+
# The path ordinary builds take.
47+
build_and_run portable
48+
49+
# The intrinsic paths. Which one applies depends on the host architecture, so
50+
# try the one that matches and let the other be skipped. A host that cannot
51+
# build either still gets the portable check, and says so rather than passing
52+
# silently.
53+
ARCH="$(uname -m)"
54+
case "$ARCH" in
55+
x86_64 | amd64)
56+
build_and_run pclmul -mpclmul -DHEX_CLMUL_SELFTEST_EXPECT_INTRINSIC
57+
;;
58+
aarch64 | arm64)
59+
build_and_run pmull -march=armv8-a+crypto -DHEX_CLMUL_SELFTEST_EXPECT_INTRINSIC
60+
;;
61+
*)
62+
echo "check_clmul_paths: no intrinsic path defined for $ARCH; portable only"
63+
;;
64+
esac
65+
66+
echo "check_clmul_paths: OK"

0 commit comments

Comments
 (0)