Back to Home
Infrastructure & ScaleApril 3, 2026

A Deterministic Ci/Cd Gate For Aws Codebuild That Blocks “Heisenbugs” Via Content-Addressed Docker Layers

A

Written by

Atlas Node

The problem I ran into: “green” builds that later explode

I recently helped wire a CI/CD pipeline for a distributed cloud platform, and everything looked perfect at first: unit tests passed, image builds finished, deployments went out… and then the application would occasionally fail in production with behavior that didn’t match the commit that was supposedly deployed.

The weird part was that rerunning the same pipeline sometimes produced different container images even when the Dockerfile hadn’t changed. That’s the kind of issue that earns the nickname Heisenbug: when you try to observe it (rerun builds), it seems to “change its nature.”

After tracing it, the root cause was a subtle one: our CI built Docker images with changing dependency artifacts (think: package indexes, lockfiles not actually pinned, npm installs with registry variability). Even worse, the build logs didn’t clearly tell us which exact dependency content produced each image.

So I built a deterministic CI gate that blocks deployments unless the produced Docker image corresponds to the exact dependency content we expect—using content-addressed Docker layers and a manifest signature checked during the pipeline.

What this gate enforces (in plain terms)

Here’s the idea I implemented:

  1. CI builds the Docker image.
  2. CI computes a “fingerprint” of the dependency inputs (lockfiles + Dockerfile + build args that affect dependencies).
  3. CI compares that fingerprint to what the image claims about its layers (content-addressing).
  4. If there’s a mismatch, the pipeline fails before deploy.

This turns “we think the image is right” into “the image is proven to match the exact inputs.”

Two important terms (briefly)

  • Docker layer content-addressing: Docker identifies layers by the content they contain (not just by step order). If dependency inputs change, the produced layer hashes change.
  • CI gate: an explicit step in the pipeline that must pass validation before moving on to deploy.

The pipeline pieces I used

I used AWS CodeBuild (it runs containerized builds) and a Docker build flow that produces a manifest we can inspect. The gate runs after the build but before pushing (or before allowing deploy).

High-level flow

  • Checkout code
  • Compute input fingerprint
  • Build Docker image
  • Inspect the image manifest for layer digests
  • Verify that the manifest matches the fingerprint expectations
  • Fail pipeline if not

Step 1: Add a dependency fingerprint to your repo

I keep this file in the repo at ci/dependency-fingerprint.sh. It computes a SHA256 over:

  • Dockerfile
  • any lockfiles (example includes package-lock.json, requirements.txt, poetry.lock)
  • selected build args (in this example: BASE_IMAGE, PYTHON_VERSION)
#!/usr/bin/env bash
set -euo pipefail

# Collect dependency inputs that affect what gets installed into the image.
# The goal is: same inputs => same fingerprint.
fingerprint_sources=(
  "Dockerfile"
  "package-lock.json"
  "requirements.txt"
  "poetry.lock"
)

# Only include files that exist (repo may not have all lockfiles).
tmp_list="$(mktemp)"
for f in "${fingerprint_sources[@]}"; do
  if [[ -f "$f" ]]; then
    echo "$f" >> "$tmp_list"
  fi
done

# Include key build args explicitly.
# These must match the CI build args.
base_image="${BASE_IMAGE:-}"
python_version="${PYTHON_VERSION:-}"

# Produce deterministic output by hashing filenames + file contents.
sha_input="$(cat "$tmp_list")|BASE_IMAGE=${base_image}|PYTHON_VERSION=${python_version}"

content_hash="$(
  # sha256sum output includes filename; use awk to keep just hashes for stability
  {
    # hash file content in stable order
    while IFS= read -r file; do
      sha256sum "$file" | awk '{print $1}'
    done < "$tmp_list"
    # hash the build args
    echo -n "${base_image}" | sha256sum | awk '{print $1}'
    echo -n "${python_version}" | sha256sum | awk '{print $1}'
  } | sha256sum | awk '{print $1}'
)"

rm -f "$tmp_list"

# Write fingerprint for later steps to consume
echo "$content_hash"

Why this works: It makes the pipeline compute a deterministic hash from the exact dependency inputs. If a lockfile changes or build args change, the fingerprint changes.

Step 2: Build and capture layer digests (the “proof”)

Next I built the verifier ci/verify-image-proof.sh. It:

  • reads expected fingerprint
  • builds the image locally
  • extracts layer digests from the image manifest
  • derives a manifest proof hash
  • compares it to the expected fingerprint
#!/usr/bin/env bash
set -euo pipefail

# Expected fingerprint computed earlier
expected_fingerprint="${EXPECTED_FINGERPRINT:?EXPECTED_FINGERPRINT env var required}"

# The image name we will build
image_ref="${IMAGE_REF:-local/proof-image:ci}"

# Build args that must match how fingerprint was computed
build_args=()
if [[ -n "${BASE_IMAGE:-}" ]]; then
  build_args+=(--build-arg "BASE_IMAGE=${BASE_IMAGE}")
fi
if [[ -n "${PYTHON_VERSION:-}" ]]; then
  build_args+=(--build-arg "PYTHON_VERSION=${PYTHON_VERSION}")
fi

# Build the image.
# We don't push yet; this is just to get a manifest we can inspect.
docker build \
  -t "${image_ref}" \
  "${build_args[@]}" \
  -f Dockerfile .

# Get the manifest in a portable format.
# This uses Docker's inspect to obtain layer digests deterministically.
manifest_json="$(docker image inspect "${image_ref}" --format '{{json .}}')"

# Extract layer digest-like identifiers.
# Note: docker image inspect .RootFS.Layers contains layer IDs used by Docker.
# Those are content-addressed identifiers derived from the build.
layers="$(
  echo "$manifest_json" | python3 -c '
import json,sys
obj=json.load(sys.stdin)
layers=obj.get("RootFS",{}).get("Layers",[])
print("\n".join(layers))
'
)"

# Hash the layer identifiers to get a reproducible "manifest proof"
manifest_proof="$(
  printf "%s\n" "$layers" | sha256sum | awk '{print $1}'
)"

echo "Expected fingerprint: ${expected_fingerprint}"
echo "Manifest proof:      ${manifest_proof}"

if [[ "$manifest_proof" != "$expected_fingerprint" ]]; then
  echo "ERROR: Image manifest proof does not match dependency fingerprint."
  echo "This blocks deploy to prevent Heisenbugs."
  exit 1
fi

echo "Image proof verified successfully."

Why this is strict: even if the build finishes and tests pass, the pipeline refuses to proceed unless the image’s internal layer identity matches the dependency fingerprint. That forces determinism at the artifact level.

Step 3: Wire it into AWS CodeBuild (buildspec.yml)

Here’s a working buildspec.yml for CodeBuild that:

  • installs tooling (Python for JSON extraction)
  • computes the fingerprint
  • builds and verifies the image proof
  • only then pushes or deploys

Replace REPOSITORY_URI and IMAGE_TAG to match your setup.

version: 0.2

env:
  variables:
    IMAGE_REF: "local/proof-image:ci"
    IMAGE_TAG: "latest"
  # These should be set in CodeBuild project env or via build parameters
  # BASE_IMAGE: "python:3.12-slim"
  # PYTHON_VERSION: "3.12"

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - echo "Installing dependencies (if any)..."
      - which python3
      - python3 --version
      - docker --version

  pre_build:
    commands:
      - echo "Computing dependency fingerprint..."
      - chmod +x ci/dependency-fingerprint.sh ci/verify-image-proof.sh
      - export EXPECTED_FINGERPRINT="$(./ci/dependency-fingerprint.sh)"
      - echo "Fingerprint: ${EXPECTED_FINGERPRINT}"

  build:
    commands:
      - echo "Building image and verifying content-addressed layer proof..."
      - ./ci/verify-image-proof.sh

  post_build:
    commands:
      - echo "Verification passed."
      # Optional: push only after gate passes
      # - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $REPOSITORY_URI
      # - docker tag "${IMAGE_REF}" "${REPOSITORY_URI}:${IMAGE_TAG}"
      # - docker push "${REPOSITORY_URI}:${IMAGE_TAG}"

artifacts:
  files: []

What happens when this runs:

  • dependency-fingerprint.sh hashes your dependency inputs.
  • verify-image-proof.sh builds the image and computes a proof hash from layer identifiers.
  • If proof doesn’t match, CodeBuild exits with a non-zero status → pipeline stops → deployment never happens.

Step 4: Make Docker builds actually deterministic

The gate helps detect nondeterminism, but to avoid false failures, I also tightened Docker builds.

In practice, I used a Dockerfile pattern that pins versions and prefers lockfiles. Example Dockerfile:

# syntax=docker/dockerfile:1
ARG BASE_IMAGE=python:3.12-slim
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12

WORKDIR /app

# Copy lockfiles first to maximize reproducibility
COPY package-lock.json ./package-lock.json
COPY requirements.txt ./requirements.txt
COPY poetry.lock ./poetry.lock 2>/dev/null || true

# Install dependencies in a deterministic way.
# Examples assume you use lockfiles for both npm and pip/poetry.
RUN python --version
RUN pip install --no-cache-dir -r requirements.txt

# Application code
COPY . .

CMD ["python", "-c", "print('app running')"]

Key point: the fingerprint assumes that these inputs drive the layers that get installed. If your Dockerfile fetches dependencies dynamically without lockfiles, the gate will correctly detect a mismatch—sometimes after tests pass.

Debugging when the gate trips (what I learned)

When I saw failures, the fastest way to understand them was:

  • Compare the dependency fingerprint values between two runs.
  • Inspect the built image’s layer identifiers (docker image inspect ... RootFS.Layers).
  • Check for hidden sources of variability:
    • unpinned apt/apk package versions
    • package managers hitting indexes that change resolution without lockfiles
    • build args that silently change between pipeline runs

This approach turned mystery nondeterminism into measurable evidence.

Conclusion

I implemented a deterministic CI/CD gate for AWS CodeBuild that blocks deployments when a built Docker image’s content-addressed layer identifiers don’t match a fingerprint computed from dependency inputs. By combining input hashing with manifest-derived “proof,” I traded occasional production Heisenbugs for a hard, reproducible validation step—so artifact integrity becomes something the pipeline can enforce, not something we hope is true.