Building A Dilithium Signature Verifier With Mutated Byte Layouts

The bug that sent me down a rabbit hole

I recently tried to validate a CRYSTALS-Dilithium signature (a post-quantum signature scheme standardized by NIST) using a small verification harness. Everything looked correct… until it wasn’t.

The odd part: the signatures were valid when produced by the reference implementation, but my verifier rejected them consistently. After a weekend of tinkering, I discovered the real culprit wasn’t the math—it was how the signature bytes were being serialized and parsed.

This post is the niche piece that actually mattered for me: verifying Dilithium signatures when the signature byte layout is “mutated” by an off-by-one packing step (the kind of mistake that happens when you switch between two libraries, or when you flatten multi-field structs into a single byte array).

I’ll show:

how Dilithium signatures are structured at a high level,
what “mutated byte layout” looks like in practice,
and a working Python verifier that detects this class of failure by trying the correct parsing versus a deliberately mutated parsing.

A quick mental model: what Dilithium’s signature is

Dilithium is a signature scheme where a signature is not a single scalar—it’s typically a serialized tuple of multiple polynomials and vectors (the exact details depend on the parameter set, like Dilithium2/3/5).

In code, you mostly treat it as:

signature = bytes
message = bytes
public_key = bytes

The verification algorithm:

parses signature bytes into internal polynomial/vector form,
recomputes checks using the public_key,
compares derived values and decides valid/invalid.

So when verification fails after swapping libraries, parsing rules (how bytes map to those internal structures) are often the cause.

What I mean by “mutated byte layout”

In my case, I had a signature blob that was supposed to be interpreted as:

[part_0 | part_1 | part_2 | ...] concatenated

But my parser effectively shifted one component by a byte boundary due to an off-by-one when splitting the byte array.

That looks like:

correct split: sig[a:b] and sig[b:c]
mutated split: sig[a:b+1] and sig[b+1:c]

The verifier then interprets wrong coefficients—everything becomes garbage, and verification fails.

The key insight: the failure pattern is consistent. If I “try” a plausible mutated parsing, I can detect whether the issue is in byte layout, not in cryptography.

Working code: verify Dilithium with correct and mutated parsing

Below is a complete Python example using the pqcrypto package. It includes a small wrapper to:

generate a Dilithium signature,
serialize it correctly,
then run verification with both:
- the correct signature parsing (baseline),
- a deliberately mutated signature parsing (shift by 1 byte in the middle).

1) Install dependencies

pip install pqcrypto

2) Verify with correct vs mutated signature parsing

import pqcrypto.sign.dilithium2 as dilithium

def sign_message(msg: bytes):
    # Generate a new Dilithium keypair
    public_key, secret_key = dilithium.generate_keypair()

    # Sign the message
    signature = dilithium.sign(secret_key, msg)

    return public_key, secret_key, signature

def verify_signature(public_key: bytes, msg: bytes, signature: bytes):
    # pqcrypto expects signature bytes in the canonical format
    return dilithium.verify(public_key, msg, signature) is None

def mutate_signature_layout(signature: bytes, split_index: int, shift: int = 1):
    """
    Deliberately performs a mutated split/concat:
    - Choose a split_index.
    - Take the left part up to split_index
    - Take the next part starting at split_index+shift
    - Then append the remainder shifted accordingly.

    This mimics an off-by-one serialization bug.
    """
    if split_index < 1 or split_index >= len(signature) - 1:
        raise ValueError("split_index out of reasonable bounds")

    if shift == 0:
        return signature

    # Left part stays the same
    left = signature[:split_index]

    # Middle/right boundaries shift by 'shift'
    # This drops 'shift' bytes from the original boundary and re-attaches the rest.
    mutated = left + signature[split_index + shift:]

    return mutated

def main():
    msg = b"faulty-byte-layout-demo"

    public_key, secret_key, signature = sign_message(msg)

    print(f"Signature length: {len(signature)} bytes")

    ok = verify_signature(public_key, msg, signature)
    print(f"Verification with canonical signature: {ok}")

    # Pick a split point in the middle.
    # In real life you wouldn't know it, but for demonstrating the layout mutation,
    # any mid-boundary split makes the error obvious.
    split_index = len(signature) // 2

    mutated = mutate_signature_layout(signature, split_index=split_index, shift=1)
    print(f"Mutated signature length: {len(mutated)} bytes (likely different from canonical)")

    # Verification should fail because signature bytes are no longer canonical.
    try:
        ok_mut = verify_signature(public_key, msg, mutated)
    except Exception as e:
        # Some libraries throw on malformed length
        ok_mut = False
        print(f"Mutated verification raised exception: {type(e).__name__}: {e}")

    print(f"Verification with mutated signature: {ok_mut}")

if __name__ == "__main__":
    main()

What each block does (and why)

generate_keypair(): creates a Dilithium2 public/private key pair.
sign(secret_key, msg): produces the signature in the library’s canonical byte format.
verify(public_key, msg, signature): returns nothing on success; throws on failure (so I convert to bool by checking the return).
mutate_signature_layout(...):
- simulates the “shift one boundary by 1 byte” serialization bug,
- outputs a signature byte string that no longer matches the canonical structure.
The verification calls:
- canonical signature should pass,
- mutated signature should fail (either returning invalid or throwing due to length/structure mismatch).

The practical debugging workflow I ended up using

Once I had this runnable baseline, I did two things to narrow down the real source of the mismatch in my project:

1) Confirm whether the signature length changes

A mutated split often changes the length of the serialized blob (or makes the parser read different fields). In the example above, shifting by 1 typically changes the signature length, which triggers failure early.

2) Compare canonical vs produced serialization bytes

I printed signature.hex() for:

signature created by my code path,
signature created by the reference path.

When I saw the mismatch concentrated around a “boundary region,” it strongly suggested my code was flattening fields incorrectly.

Why this matters in post-quantum land

Post-quantum schemes like Dilithium are still new enough that projects frequently mix:

different parameter sets (Dilithium2 vs 3 vs 5),
different serialization conventions,
and different library expectations.

In those settings, “cryptography is broken” is often actually “bytes aren’t interpreted the same way.”

This is especially true for hybrid classical-quantum systems too: if the signature bytes are carried inside another protocol frame, a single off-by-one can be introduced when you pack/unpack the container.

Conclusion

I built a small Dilithium2 verifier harness and used it to reproduce a failure mode I’d been seeing: signatures rejected consistently due to a mutated byte layout from an off-by-one serialization/parsing mistake. The key takeaway is that in post-quantum signature verification, most “everything fails” bugs I encountered weren’t mathematical—they were byte-structure mismatches, which you can isolate quickly by comparing canonical vs mutated parsing and observing whether verification fails due to parsing/length/field-boundary issues.