Native ARM64/NEON support (via SIMDe) - #931
Open
BenjaminDEMAILLE wants to merge 3 commits into
Open
Conversation
Vendor the SIMD Everywhere (SIMDe) header tree, used to translate the x86 SSE/AVX intrinsics to ARM NEON on aarch64/arm64 builds. Upstream: https://github.com/simd-everywhere/simde (MIT) Pinned: v0.8.2, commit 71fd833d9666141edcd1d3c109a80e228303d8d7 Only the simde/ header directory and COPYING are vendored (no tests/docs), so a plain `git clone && make` works with no submodule step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build hifiasm on aarch64/arm64 (Linux and Apple Silicon) in addition to x86. All changes are gated behind `defined(__aarch64__) || defined(__arm__)`, so the x86 build is byte-for-byte unchanged (same flags, same intrinsics). - Makefile: detect the target via `uname -m`. On aarch64/arm64 drop the invalid `-msse4.2 -mpopcnt` (the latter is a hard error there), pass `-march=armv8-a+simd`, and add `-Ithird_party/simde`. `ARCH`/`ARCH_FLAGS` stay overridable (e.g. `make ARCH_FLAGS="-mcpu=neoverse-v2"`). No popcnt intrinsic is used anywhere, so dropping `-mpopcnt` needs no code change. - Levenshtein_distance.h: on ARM include the SIMDe SSE4.2 + AVX2 headers instead of <emmintrin.h>/<nmmintrin.h>/<smmintrin.h>/<immintrin.h> (only SSE __m128i code is actually used; <immintrin.h> was included but no AVX intrinsic is called). GCC's x86 _mm_srli_epi32 accepts a runtime shift count; SIMDe/NEON requires a constant, so the two variable-count uses go through a small HA_SRLI_EPI32_VAR macro that expands to the identical intrinsic on x86. - ksw2_extz2_sse.c: provide the SSE2/SSE4.1 kernel via SIMDe and define __SSE2__/__SSE4_1__ so the body compiles on ARM. This file is not in OBJS (its only caller is commented out), so it stays unlinked; it is ported so it builds cleanly if ever re-enabled, with no change to either binary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expand CI from a single x86_64 Linux build into a matrix that builds on Linux x86_64, Linux aarch64, macOS x86_64, and macOS arm64. Add a smoke job that assembles a tiny deterministic synthetic HiFi read set on both Linux arches and asserts that the x86_64 and aarch64 assembly graphs are identical. `-f0` disables the 16 GiB default bloom filter so the smoke test fits in a standard runner's RAM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native ARM64/NEON support (via SIMDe)
This makes hifiasm build and run natively on aarch64/arm64 (Linux ARM64 and
Apple Silicon) in addition to x86, with the x86 build byte-for-byte unchanged.
Fixes #288. Supersedes #641: that PR used sse2neon and built fine, but this one
uses SIMDe instead, which also covers
AVX (not just SSE), is header-complete for the intrinsics hifiasm uses, and comes
with a cross-arch CI matrix that proves the x86 and ARM64 assemblies are identical.
Why the current build fails on ARM
The
Makefilepasses-msse4.2 -mpopcnt(the latter is a hard error on aarch64),and
Levenshtein_distance.h/ksw2_extz2_sse.cinclude x86 intrinsic headers(
<emmintrin.h>,<smmintrin.h>,<nmmintrin.h>,<immintrin.h>) that do notexist on ARM.
What changed
All source changes are gated behind
defined(__aarch64__) || defined(__arm__),so on x86 the preprocessor takes the original path and codegen is unchanged.
Makefile: detect the target withuname -m. On aarch64/arm64, drop-msse4.2 -mpopcnt, use-march=armv8-a+simd, and add-Ithird_party/simde.ARCHandARCH_FLAGSremain overridable, e.g.make ARCH_FLAGS="-mcpu=neoverse-v2"ormake ARCH=aarch64. The x86 branch isidentical to before (
-msse4.2 -mpopcnt, emptyINCLUDES).Levenshtein_distance.h: on ARM, include the SIMDex86/sse4.2.handx86/avx2.hheaders instead of the four x86 headers. One subtlety: GCC's x86_mm_srli_epi32accepts a runtime shift count, but the NEON translationrequires a compile-time constant, so the two variable-count uses go through a
tiny
HA_SRLI_EPI32_VARmacro that expands to the identical_mm_srli_epi32on x86 and to the variable-count shift on ARM.
ksw2_extz2_sse.c: provide the SSE2/SSE4.1 kernel via SIMDe and define__SSE2__/__SSE4_1__so the body compiles on ARM. Note this file is not inthe Makefile
OBJSand its only caller is commented out, so it stays unlinked;it is ported (and verified to compile standalone on ARM) so it builds cleanly
if it is ever re-enabled, with no change to either binary.
third_party/simde/: vendored SIMDe, pinned to v0.8.2(commit
71fd833d9666141edcd1d3c109a80e228303d8d7, MIT). Only thesimde/header tree and
COPYINGare included (no tests/docs), so a plaingit clone && makeworks with no submodule step..github/workflows/ci.yaml+test/gen_hifi_reads.py: CI now builds amatrix of Linux x86_64, Linux aarch64, macOS x86_64, and macOS arm64, and a
smoke job assembles a tiny deterministic synthetic HiFi set on both Linux arches
and asserts the x86_64 and aarch64 graphs are identical.
Notes on scope
No popcnt intrinsic (
_mm_popcnt*/__builtin_popcount*) is used anywhere, sodropping
-mpopcnton ARM needs no code change. And although<immintrin.h>wasincluded, no
_mm256_*/AVX intrinsic is actually called; SIMDe'savx2.hisstill included on ARM to preserve the original include surface.
x86 is unchanged
#if; the#elsebranch is the original code.INCLUDES.ksw2_extz2_sse.cstays out ofOBJS, so the link line is unchanged on all arches.Testing
makesucceeds and produces a working binary on all four targets:-msse4.2 -mpopcntpath, no SIMDe)Determinism (the assembled graph is identical across architectures) is checked
two independent ways, both producing a byte-identical 67,868 B primary-contig GFA:
cross-arch graph determinismjob).Locally on Apple Silicon,
hifiasm -o test -t2 reads.fq.gzon a small syntheticHiFi set assembles a single 59,609 bp contig, and repeated runs are byte-identical.
CI run (on my fork): https://github.com/BenjaminDEMAILLE/hifiasm/actions/runs/29027889900
The three Linux/macOS-arm64 build legs, both smoke legs, and the determinism job
are green; the macOS x86_64 leg was still waiting on a (scarce) Intel runner at
the time of writing and is additionally covered by the local cross-compile above.