LDASM x64

Lightweight x86-64 instruction length disassembler, ported by Fyyre from the classic LDASM by ms-rem Design and extended for long mode with VEX, EVEX, and XOP.

This is a length decoder, not a full symbolic disassembler. It answers: "how many bytes is this instruction?" - the common need for hooks, trampolines, code walkers, and reverse-engineering tools.

This port is compatible with MSVC and GCC/Clang.

---

Features

Area	Support
Mode	x86-64 long mode (primary)
Legacy prefixes	Segment, LOCK, REP/REPE/REPNE, 66, 67
REX	40–4F, including REX.W -> mov r64, imm64
Two- / three-byte opcodes	0F, 0F 38, 0F 3A
VEX	2-byte (C5) and 3-byte (C4), maps 0F / 0F38 / 0F3A
EVEX	4-byte (62), maps 0F / 0F38 / 0F3A; mask / {z} / broadcast / compressed disp8*N (still 1 stream byte)
XOP (AMD)	8F when mmmmm ≥ 8, disambiguated from POP r/m
Addressing	RIP-relative, SIB, VSIB (length rules unchanged), ASZ 64/32 via 67

---

API

#include "ldasm.h"
 
size_t SizeOfCode(void *Code, unsigned char **pOpcode);
size_t SizeOfProc(void *Proc);
char   IsRelativeCmd(unsigned char *pOpcode);

• SizeOfCode — returns instruction length in bytes. Optionally sets *pOpcode to the first opcode byte (after legacy/REX prefixes; for VEX/EVEX/XOP this is the lead prefix byte C4/C5/62/8F).
• SizeOfProc — walks instructions until a single-byte RET (C3) or a zero-length decode.
• IsRelativeCmd — non-zero if the opcode table marks a near relative branch form (E8/E9/0F 8x, etc.).

Calling convention: __fastcall on MSVC; no-op macro on GCC/Clang.

---

Build & test

# Length decoder only
gcc -O2 -c ldasm.c -o ldasm.o
 
# Self-test (hand-crafted vectors + NASM blob walk)
nasm -f elf64 test_vectors.asm -o test_vectors.o
gcc -O2 -Wall -Wextra -o test_ldasm test_ldasm.c ldasm.c
./test_ldasm test_vectors.o

Expected: all tests passed (hand-crafted + NASM corpus covering legacy, SSE4, AVX, AVX-512, VSIB, XOP).

---

What changed (from classic 32-bit LDASM -> this x64 port)

Port baseline (x64 + SIMD prefixes)

Compared to classic 32-bit LDASM, this tree already aimed to:

1. Treat 40–4F as REX (not INC/DEC r16/r32).
2. Handle REX.W for 64-bit immediates on B8–BF.
3. Treat C4 / C5 as VEX and 62 as EVEX (not LES/LDS/BOUND).
4. Keep the original opcode flag tables and ModRM/SIB/imm length model (small, table-driven, RE-friendly).

Correctness fixes applied in review

The initial x64/VEX/EVEX wiring had several length bugs. The following were found and fixed:

Critical — wrong instruction lengths

Issue	Effect	Fix
No 0F 38 / 0F 3A three-byte escape handling	SSE4/AES/BMI etc. reported as 2-byte stubs (OpcodeFlagsExt[0x38/3A] was OP_NONE)	Explicit maps: 0F 38 -> ModRM; 0F 3A -> ModRM + imm8
VEX 2-byte (C5) always OP_MODRM	Missed imm8 (vpshufd, vcmpps, vshufps); forced ModRM on vzeroupper (C5 F8 77)	Use OpcodeFlagsExt[opcode] (implied 0F map)
VEX 3-byte (C4) map 0F: OpcodeFlagsExt[*cPtr & 0x7F]	Bit 7 stripped -> e.g. C2->42, imm8 lost on vcmp*	Index full opcode byte
EVEX map 0F always OP_MODRM	Same imm8 miss as VEX for map 1	Use OpcodeFlagsExt when mm == 1
67 treated as 16-bit addressing	Classic LDASM/protected-mode logic; long mode never uses 16-bit ModRM (67 selects 32-bit ASZ)	Unified 32/64 ModRM+SIB+disp rules (RIP-rel / disp32 / disp8 / SIB)

High — long-mode operand / offset sizes

Issue	Effect	Fix
A0–A3 moffs used operand-size (2/4)	Offsets must be 64-bit, or 32-bit with 67	Dedicated moffs path: +8 or +4
Near branches with 66 (E8/E9/0F 8x)	Imm shortened to 2 bytes	In 64-bit mode near branch imm is always rel32 (OP_REL32 -> +4)

Medium — tables & AMD XOP

Issue	Effect	Fix
No XOP disambiguation	8F … with mmmmm ≥ 8 decoded as POP r/m	XOP 3-byte prefix; maps m8/mA -> ModRM+imm8; m9 -> ModRM; else POP
0F B8 = OP_NONE	popcnt (F3 0F B8 /r) length wrong	OP_MODRM
0F 19–1F = OP_NONE	Multi-byte NOP forms need ModRM	OP_MODRM
0F 78–7B too sparse	Some ModRM forms under-counted	Marked OP_MODRM (rare dual-imm AMD forms still best-effort)

Low — portability

Issue	Fix
#include "LDasm.h" (case)	"ldasm.h" for Linux
size_t / __fastcall on GCC	stddef.h + empty __fastcall macro when not MSVC
EVEX map field	P0 & 0x03 (2-bit mm), not 3-bit

Design choices kept (on purpose)

• Length-only, best-effort on #UD / rare encodings (REX before VEX, reserved maps).
• No dynamic allocation, no ISA database — two 256-byte flag tables + prefix walk.
• EVEX compressed displacement still counts as one byte in the stream (mod = 01).
• VSIB gather/scatter: SIB present; length rules same as normal SIB.

---

Example

unsigned char code[] = {
    0xC5, 0xF8, 0x77,             /* vzeroupper           */
    0xC5, 0xF9, 0x70, 0xC1, 0x1B, /* vpshufd xmm0,xmm1,ib */
    0x62, 0xF1, 0x74, 0x48, 0x58, 0xC2, /* vaddps zmm0,zmm1,zmm2 */
};
 
unsigned char *p = code;
unsigned char *opc;
for (int i = 0; i < 3; i++) {
    size_t n = SizeOfCode(p, &opc);
    /* n: 3, 5, 6 */
    p += n;
}

---

Layout

ldasm.h           Public API
ldasm.c           Tables + SizeOfCode / SizeOfProc / IsRelativeCmd
testing/test_ldasm.c      Self-test harness
testing/test_vectors.asm  NASM corpus (legacy / VEX / EVEX / VSIB)
README.md         This file

---

Limitations

1. Assumes a readable buffer — no explicit end-of-buffer check (classic LDASM model). Caller must ensure the region is valid (or catch faults).
2. Not a full decoder — no mnemonics, operands, or validity #UD reporting.
3. Rare / vendor-specific forms may still mis-length (e.g. AMD SSE4a dual-imm EXTRQ, future APX maps).
4. SizeOfProc stops only on single-byte C3, not C2 iw / far CB.
5. IsRelativeCmd uses primary/0F tables only; does not walk VEX/EVEX (branches are not VEX-encoded).
6. 32-bit protected mode is not a supported target; tables and ASZ logic are for long mode.

---

License / attribution

Based on the classic LDASM length-disassembler approach by ms-rem.

This tree is an x86-64 / VEX / EVEX / XOP maintenance and correctness port by Fyyre. Focused on accurate instruction length for reverse engineering and runtime code patching.

If you distribute a derived work, retain appropriate credit for the original LDASM design.

---

Changelog (review pass)

• Fixed 0F 38 / 0F 3A length calculation
• Fixed VEX.C5 / VEX.C4 / EVEX map-0F imm8 and non-ModRM forms (vzeroupper)
• Fixed long-mode address-size (67) and moffs (A0–A3)
• Fixed near-branch imm size under 66
• Added AMD XOP vs POP disambiguation
• Corrected POPCNT and multi-byte NOP table entries
• Portable header for GCC/Clang; Linux-friendly include path
• Added NASM + C self-test suite

Contributions that tighten edge-case lengths (with test vectors) are welcome.