Composite ML-KEM decapsulation returns EC ciphertext validation errors instead of implicit rejection

[Wowblk]

Hi SymCrypt team,

I found a possible mismatch between the documented Composite ML-KEM implicit-rejection behavior and the current implementation.

Summary

SymCryptCompositeMlKemDecapsulate is documented to implicitly reject an invalid but correctly-sized ciphertext by returning SYMCRYPT_NO_ERROR and writing a pseudo-random agreed secret. In the current implementation, if the ML-KEM part is correctly sized but the traditional EC KEM ciphertext is an invalid encoded public key, the EC parsing/secret-agreement error is returned directly to the caller.

This creates a visible error-code oracle for the EC component of a Composite ML-KEM ciphertext.

Documentation

The public header says:

Given an invalid, but correctly-sized, ciphertext, the Composite ML-KEM Decapsulation operation
will "implicitly reject" the ciphertext, by returning success in equal time to a valid
decapsulation operation, with pseudo-random agreed secret output.

So decapsulate will only ever return an error if there are programming errors (e.g. incorrect size),
or something fundamentally goes wrong with the environment (e.g. internal memory allocation fails).

Minimal PoC

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <symcrypt.h>

int main(void)
{
    SYMCRYPT_MODULE_INIT();

    const SYMCRYPT_COMPOSITE_MLKEM_PARAMS params =
        SYMCRYPT_COMPOSITE_MLKEM_PARAMS_MLKEM768_P256;
    PSYMCRYPT_COMPOSITE_MLKEMKEY key = SymCryptCompositeMlKemkeyAllocate(params);
    if (key == NULL) return 1;

    SIZE_T cbCt = 0;
    BYTE ss[32];
    SYMCRYPT_ERROR err = SymCryptCompositeMlKemSizeofCiphertextFromParams(params, &cbCt);
    if (err != SYMCRYPT_NO_ERROR) return 2;

    BYTE *ct = calloc(1, cbCt);
    if (ct == NULL) return 3;

    err = SymCryptCompositeMlKemkeyGenerate(key, 0);
    printf("keygen=0x%08x\n", err);

    err = SymCryptCompositeMlKemEncapsulate(key, ss, sizeof(ss), ct, cbCt);
    printf("encap=0x%08x ciphertext_size=%zu\n", err, cbCt);

    memset(ct + cbCt - 65, 0, 65);
    memset(ss, 0, sizeof(ss));

    err = SymCryptCompositeMlKemDecapsulate(key, ct, cbCt, ss, sizeof(ss));
    printf("decap_invalid_ec_ciphertext=0x%08x\n", err);
    printf("expected_for_implicit_rejection=0x%08x\n", SYMCRYPT_NO_ERROR);

    free(ct);
    SymCryptCompositeMlKemkeyFree(key);
    return err == SYMCRYPT_NO_ERROR ? 0 : 10;
}

Observed result

Using a release dynamic SymCrypt build:

keygen=0x00000000
encap=0x00000000 ciphertext_size=1153
decap_invalid_ec_ciphertext=0x0000800c
expected_for_implicit_rejection=0x00000000

The ciphertext length is correct (1153 bytes for SYMCRYPT_COMPOSITE_MLKEM_PARAMS_MLKEM768_P256), but replacing the final 65-byte P-256 EC public-key encoding with zeros causes decapsulation to return SYMCRYPT_INVALID_BLOB instead of the documented implicit-rejection success path.

Expected result

For a correctly-sized Composite ML-KEM ciphertext, invalid EC KEM ciphertext contents should follow the same implicit-rejection behavior as other invalid ciphertext contents: return SYMCRYPT_NO_ERROR and output a pseudo-random agreed secret, unless the inputs represent a programming error such as an incorrect size.

Thanks!

[brandon-w-zhang]

Hi @Wowblk, thank you for reporting this. Composite ML-KEM per Section 3.5. Error Handling and Explicit Rejection
of the LAMPS draft does explicit rejection, so the behavior is as expected but the documentation is inaccurate. Will fix documentation to reflect this.