Deterministic Canonical Json For Signed Webhooks With Jcs-Like Ordering

I ran into a maddening problem while wiring up signed webhooks: the signature verification would randomly fail even though the “same” payload was being sent. After a weekend of printing hex dumps and staring at diffs, I discovered the root cause wasn’t cryptography—it was JSON formatting.

Specifically, two systems were producing JSON with different whitespace and key ordering. Many signature schemes compute the digest from a canonical byte representation of the JSON, so any difference in formatting changes the bytes and therefore the signature.

In this post I’ll show a tiny, practical approach to make JSON deterministic for signing/verifying using a JCS-like (JSON Canonicalization Scheme inspired) strategy: sort object keys, normalize numbers, and remove insignificant whitespace.

The failure mode: “same JSON” ≠ “same bytes”

Here’s a concrete example. Both payloads represent the same logical data:

{"event":"paid","amount":10,"meta":{"order":"A1","items":[3,2,1]}}

and

{
  "meta": { "items": [3, 2, 1], "order": "A1" },
  "amount": 10,
  "event": "paid"
}

If your signing code digests the UTF-8 bytes of the raw JSON string, the byte sequences differ because:

• object key order differs
• whitespace differs

So the signature differs.

A deterministic “canonical JSON” strategy (JCS-like)

To produce stable bytes, I canonicalize the JSON structure into a byte-for-byte deterministic string:

Rules I used:

1. Objects: keys are sorted lexicographically.
2. Arrays: order is preserved.
3. Strings: escaped deterministically via json.dumps(..., separators=(',', ':')).
4. Numbers: normalized so 10, 10.0, and 1e1 become the same representation.
5. No extra whitespace: use compact separators.

The only “weird” part is number normalization. Python’s json can keep floats like 10.0 slightly differently depending on input sources. I used decimal.Decimal to normalize from strings where possible.

Working code: canonicalize + sign + verify

This example uses HMAC-SHA256 (a common signing method for webhooks). The same canonicalization must happen on both sender and verifier.

Canonicalizer

import json
import hmac
import hashlib
from decimal import Decimal, InvalidOperation
from typing import Any

def _normalize_numbers(value: Any) -> Any:
    """
    Walk the JSON value and normalize numbers so that equivalent numeric
    forms produce identical serialized output.
    """
    if isinstance(value, dict):
        return {k: _normalize_numbers(v) for k, v in value.items()}
    if isinstance(value, list):
        return [_normalize_numbers(v) for v in value]

    # If the JSON parser already gave us ints/floats, normalize via Decimal.
    if isinstance(value, (int,)):
        return value
    if isinstance(value, float):
        # Convert float to string through repr to preserve as much as possible,
        # then normalize via Decimal.
        d = Decimal(repr(value))
        # Convert to a canonical string form (no trailing .0)
        # using normalized exponent handling.
        if d == d.to_integral_value():
            return int(d)
        return d.normalize()

    # If we get a string that looks like a number, normalize it too.
    if isinstance(value, str):
        try:
            d = Decimal(value)
        except (InvalidOperation, ValueError):
            return value

        if d == d.to_integral_value():
            return int(d)
        return d.normalize()

    return value

def canonical_json_bytes(data: Any) -> bytes:
    """
    Produce deterministic UTF-8 bytes for JSON signing.
    - normalize numbers
    - compact separators (no whitespace)
    - sort keys
    """
    normalized = _normalize_numbers(data)
    # separators=(',', ':') removes whitespace after commas/colons
    # sort_keys=True ensures deterministic key order
    text = json.dumps(normalized, sort_keys=True, separators=(',', ':'))
    return text.encode('utf-8')

def sign_hmac_sha256(secret: bytes, payload_bytes: bytes) -> str:
    digest = hmac.new(secret, payload_bytes, hashlib.sha256).hexdigest()
    return digest

def verify_hmac_sha256(secret: bytes, payload_bytes: bytes, signature_hex: str) -> bool:
    expected = sign_hmac_sha256(secret, payload_bytes)
    # Use constant-time compare for safety
    return hmac.compare_digest(expected, signature_hex)

Demo: two differently formatted JSON strings, one signature

import json

secret = b"super-secret-webhook-key"

payload_a = '{"event":"paid","amount":10,"meta":{"order":"A1","items":[3,2,1]}}'
payload_b = '{ "meta": { "items": [3,2,1], "order": "A1" }, "amount": 10, "event": "paid" }'

data_a = json.loads(payload_a)
data_b = json.loads(payload_b)

bytes_a = canonical_json_bytes(data_a)
bytes_b = canonical_json_bytes(data_b)

sig_a = sign_hmac_sha256(secret, bytes_a)
sig_b = sign_hmac_sha256(secret, bytes_b)

print("canonical bytes equal:", bytes_a == bytes_b)
print("signature A == signature B:", sig_a == sig_b)
print("signature:", sig_a)

Expected outcome:

• canonical bytes equal: True
• signature A == signature B: True

That’s the whole point: you sign the canonical bytes, not the original formatting.

Step-by-step: what happens during canonicalization

Given this payload:

{
  "event": "paid",
  "amount": 10,
  "meta": { "order": "A1", "items": [3, 2, 1] }
}

1. json.loads(...) turns it into Python types (dict, list, str, int).
2. _normalize_numbers walks the structure:
   • amount is an int → stays 10
3. json.dumps(..., sort_keys=True, separators=(',', ':')) produces:
   • keys sorted: amount, then event, then meta
   • no whitespace: {"amount":10,"event":"paid","meta":{"items":[3,2,1],"order":"A1"}}
4. .encode('utf-8') yields stable bytes for signing.

Even if the original JSON had different ordering or whitespace, the canonicalization removes those degrees of freedom.

Integrating with a webhook verifier

Typical webhook flows look like this:

• Sender sends:
  • raw JSON body
  • signature header (HMAC hex)
• Verifier:
  • parses JSON
  • canonicalizes it
  • recomputes signature
  • checks equality

Here’s a minimal verifier function:

def verify_webhook(raw_body: str, signature_hex: str, secret: bytes) -> bool:
    data = json.loads(raw_body)
    payload_bytes = canonical_json_bytes(data)
    return verify_hmac_sha256(secret, payload_bytes, signature_hex)

# Example usage:
# raw_body comes from the HTTP request body as a string

If you compute the signature with the same canonicalization rules on both ends, verification becomes deterministic.

Practical notes I learned the hard way

1) Don’t sign the raw request body string

Even harmless differences (pretty printing, different key order) will break verification.

2) “Numbers are easy” is a trap

A float like 10.0 vs 10 can serialize differently across environments. Normalizing numbers (often via Decimal) avoids subtle mismatches.

3) Sorting keys is essential

The canonicalization step must be consistent: key ordering is part of what gets signed.

4) Be strict about null/booleans

Make sure both sides use the same JSON types (true/false, null) and that your parser doesn’t coerce them unexpectedly.

Conclusion

I fixed unreliable webhook signature verification by switching from signing the raw JSON text to signing deterministic canonical JSON bytes. The key idea was to remove formatting variance by sorting object keys, compacting separators, and normalizing numbers so semantically equivalent payloads produce identical bytes—making signatures stable across systems.