Ci Cd Pipeline For Deterministic Docker Sboms With In Pipeline Cosign Rekeying

The problem I ran into: “My images are identical, but the SBOMs keep changing”

I was helping wire up a CI/CD pipeline for a hybrid setup where developers build container images in one place, release them in another, and then security tooling generates an SBOM (Software Bill of Materials). An SBOM is basically a machine-readable inventory of what software a container includes (packages, versions, licenses).

Everything looked fine—until I noticed a nasty symptom:

• Two pipeline runs for the “same” commit produced identical Docker image digests.
• But the generated SBOM files differed.
• The downstream security gate treated this as “new/unknown components” and blocked releases.

I dug in and found the root cause: the SBOM generation step was not fully deterministic because it depended on unstable inputs (timestamps, non-reproducible file ordering, and in some cases the signing/verification setup changing between runs).

What finally worked for me was a very specific pattern:

1. Generate SBOM deterministically from a fixed image digest.
2. Produce the SBOM in a stable format.
3. Sign the SBOM in a way that supports safe “rekeying” (rotating keys) without breaking verification.
4. Fail the pipeline loudly if the produced SBOM changes unexpectedly.

This post documents exactly what I built.

---

The setup I used

I used:

• GitHub Actions for CI/CD
• Docker buildx to build multi-platform images
• Syft to generate SBOMs (Syft = a tool that scans container images and produces SBOMs)
• Cosign to sign artifacts (Cosign = a signing tool compatible with Sigstore)
• Rekeying support by keeping verification logic stable while rotating signing keys

A directory structure

.
├── .github
│   └── workflows
│       └── sbom-deterministic.yml
├── Dockerfile
└── .cosign
    ├── cosign.key
    └── cosign.pub

---

A deterministic SBOM strategy (the part people usually miss)

The non-obvious piece: SBOM tools can include metadata that changes run-to-run. To tame this, I made these decisions:

1. Use the image digest as the input, not the tag.

   • Tags like myapp:sha-123 are mutable in practice.
   • A digest like myapp@sha256:... is immutable.

2. Generate SBOM in a stable output format.

   • I used SPDX JSON because it’s widely supported.
   • I pinned tool versions to avoid “same command, different output.”

3. Enforce determinism in the pipeline.

   • I computed a checksum of the generated SBOM.
   • Then I compared it against the checksum recorded from the first generation for that digest.

4. Sign SBOMs, not the images, so the security gate can trust the SBOM content.

   • Rekeying later should not require the pipeline logic to change—only the key material.

---

The working GitHub Actions workflow

This workflow builds an image, generates a deterministic SBOM from its digest, checks stability, and signs the SBOM with Cosign.

Create: .github/workflows/sbom-deterministic.yml

name: sbom-deterministic

on:
  push:
    branches: [ "main" ]
  workflow_dispatch: {}

permissions:
  contents: read
  packages: write
  id-token: write

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: my-org/my-app
  SYFT_VERSION: "1.20.1"
  COSIGN_VERSION: "2.2.4"
  # Stable output format for SBOMs
  SBOM_FORMAT: "spdx-json"

jobs:
  build-and-sbom:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push image (multi-platform)
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          platforms: linux/amd64
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          # Export metadata so we can capture the digest reliably
          labels: |
            org.opencontainers.image.revision=${{ github.sha }}

      - name: Resolve image digest
        id: digest
        shell: bash
        run: |
          set -euo pipefail
          # docker/build-push-action writes digest info to build outputs only in some modes,
          # so I resolve via manifest to ensure immutability.
          IMAGE_REF="${REGISTRY}/${IMAGE_NAME}:${GITHUB_SHA}"
          DIGEST=$(docker buildx imagetools inspect "$IMAGE_REF" --format '{{.Digest}}')
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Install Syft (pinned)
        shell: bash
        run: |
          set -euo pipefail
          curl -sSfL -o syft.tgz \
            "https://github.com/anchore/syft/releases/download/v${SYFT_VERSION}/syft_${SYFT_VERSION}_linux_amd64.tar.gz"
          tar -xzf syft.tgz syft
          sudo mv syft /usr/local/bin/syft
          syft version

      - name: Install Cosign (pinned)
        shell: bash
        run: |
          set -euo pipefail
          curl -sSfL -o cosign "https://github.com/sigstore/cosign/releases/download/v${COSIGN_VERSION}/cosign-linux-amd64"
          sudo mv cosign /usr/local/bin/cosign
          sudo chmod +x /usr/local/bin/cosign
          cosign version

      - name: Generate deterministic SBOM from immutable digest
        id: sbom
        shell: bash
        env:
          DIGEST: ${{ steps.digest.outputs.digest }}
        run: |
          set -euo pipefail

          IMAGE_WITH_DIGEST="${REGISTRY}/${IMAGE_NAME}@${DIGEST}"
          OUT_DIR="sbom-out"
          mkdir -p "$OUT_DIR"
          SBOM_PATH="${OUT_DIR}/sbom-${DIGEST}.spdx.json"

          # Syft scans the image at the provided digest.
          # I pinned output format and kept tool versions fixed to reduce drift.
          syft "$IMAGE_WITH_DIGEST" \
            --output "$SBOM_FORMAT" \
            --file "$SBOM_PATH"

          # Create a checksum that we can use to detect non-determinism.
          SBOM_SHA256="$(sha256sum "$SBOM_PATH" | awk '{print $1}')"

          echo "sbom_path=$SBOM_PATH" >> "$GITHUB_OUTPUT"
          echo "sbom_sha256=$SBOM_SHA256" >> "$GITHUB_OUTPUT"

          echo "Generated SBOM SHA256: $SBOM_SHA256"

      - name: Determinism gate (fail if SBOM changed for same digest)
        shell: bash
        env:
          DIGEST: ${{ steps.digest.outputs.digest }}
          SBOM_SHA256: ${{ steps.sbom.outputs.sbom_sha256 }}
        run: |
          set -euo pipefail

          # This uses GitHub Actions cache keyed by the image digest.
          # The first run stores the SBOM checksum; subsequent runs for the same digest
          # must match or the pipeline fails.
          #
          # Cache key includes digest only, so it maps to the "should be stable" expectation.
          KEY="sbom-${DIGEST}"

          # The cache for this example is implemented via a tiny local file.
          # We fetch it from cache; if missing, it's a "first run" and we store it.
          # For determinism enforcement, mismatch => exit 1.
          CACHE_DIR=".cache"
          mkdir -p "$CACHE_DIR"
          CACHE_FILE="${CACHE_DIR}/checksum.txt"

          # Attempt to restore checksum from cache:
          # Note: caching in GitHub Actions is handled by the cache action.
          echo "Determinism key: $KEY"

      - name: Restore checksum from cache
        id: cache-restore
        uses: actions/cache/restore@v4
        with:
          path: .cache
          key: sbom-${{ steps.digest.outputs.digest }}

      - name: Enforce checksum determinism
        shell: bash
        env:
          SBOM_SHA256: ${{ steps.sbom.outputs.sbom_sha256 }}
          DIGEST: ${{ steps.digest.outputs.digest }}
        run: |
          set -euo pipefail
          CACHE_FILE=".cache/checksum.txt"

          if [[ -f "$CACHE_FILE" ]]; then
            STORED="$(cat "$CACHE_FILE")"
            echo "Stored checksum: $STORED"
            echo "Current checksum: $SBOM_SHA256"
            if [[ "$STORED" != "$SBOM_SHA256" ]]; then
              echo "::error::SBOM checksum mismatch for same image digest."
              echo "::error::This indicates non-deterministic SBOM generation."
              exit 1
            fi
          else
            echo "First time seeing digest; recording checksum."
            echo -n "$SBOM_SHA256" > "$CACHE_FILE"
          fi

      - name: Save checksum to cache
        if: ${{ always() }}
        uses: actions/cache/save@v4
        with:
          path: .cache
          key: sbom-${{ steps.digest.outputs.digest }}

      - name: Sign SBOM with Cosign (key-based signing for rekeying safety)
        env:
          SBOM_PATH: ${{ steps.sbom.outputs.sbom_path }}
          DIGEST: ${{ steps.digest.outputs.digest }}
        shell: bash
        run: |
          set -euo pipefail

          # I sign the SBOM file. The signature is stored in a deterministic location
          # (attached to an OCI registry reference) so verification is stable.
          #
          # For rekeying, verification should trust the stable public key or identity,
          # while CI can rotate the private key without changing the verification policy.
          cosign sign-blob \
            --yes \
            --key .cosign/cosign.key \
            --output-signature "sbom.sig" \
            "$SBOM_PATH"

          echo "Signed SBOM for digest: $DIGEST"

      - name: Upload SBOM artifact
        uses: actions/upload-artifact@v4
        with:
          name: sbom-${{ steps.digest.outputs.digest }}
          path: |
            ${{ steps.sbom.outputs.sbom_path }}
            sbom.sig

What each important block does (and why)

Build and push

• I build and push the image using docker/build-push-action.
• I tag by ${{ github.sha }} for traceability.

Resolve immutable digest

• I resolve the digest via docker buildx imagetools inspect.
• This gives me my-image@sha256:... which won’t drift across runs.

Generate SBOM from digest

• Syft scans the image content at the digest, producing an SPDX JSON SBOM.
• I pinned Syft’s version and set a stable output format.

Determinism gate

• I store a checksum in GitHub Actions cache keyed by the digest.
• If the same digest ever produces a different SBOM checksum, the job fails.

Sign the SBOM

• I sign the SBOM file with Cosign.
• This is where rekeying matters: the pipeline signs with whatever private key is configured, but the rest of the system verifies based on a public key policy.

---

Dockerfile example (tiny, just to make the post runnable)

This is just a placeholder so the workflow has something to build.

# syntax=docker/dockerfile:1
FROM alpine:3.20
RUN apk add --no-cache curl
CMD ["curl", "--version"]

---

How “rekeying” fits in without breaking the pipeline

Rekeying means rotating signing keys (for security hygiene or after incident response). If verification policies are tied to fragile identifiers, pipelines can break.

The pattern that stayed stable in my setup was:

• CI pipeline uses a private key from a secret or filesystem (rotatable).
• Verification uses a stable public key or trusted identity (configurable in the security gate, not inside the pipeline logic).
• SBOM signatures are tied to the artifact itself (the SBOM file), so verification checks match content, not run-time circumstances.

In the workflow above, the signing step is isolated:

cosign sign-blob --key .cosign/cosign.key --output-signature sbom.sig sbom.spdx.json

In practice, I rotated .cosign/cosign.key (or swapped the secret it came from) without touching the SBOM generation, digest resolution, or determinism gate logic.

---

What I saw after implementing this

After adding the digest-based SBOM generation and the checksum determinism gate:

• I stopped getting “same image digest, different SBOM” surprises.
• Security checks stopped blocking releases due to content drift.
• Key rotation no longer required redeploying or rewriting pipeline steps—only updating key material in a controlled place.

---

Conclusion

I learned that “deterministic CI/CD” isn’t just about building the same container digest—it also requires locking down inputs for SBOM generation (image digest), outputs (stable SBOM format), and even behavior over time (a checksum determinism gate). With Syft + Cosign and a digest-keyed stability check, I got SBOMs that stopped changing unexpectedly and signing that could survive rekeying cleanly.