## The problem I ran into on a hybrid cluster

I was building a hybrid (on‑prem + public cloud) environment where services needed **egress** (outbound network traffic) to a couple of external APIs. The tricky part wasn’t connecting—both sides worked.

The real pain showed up when the “primary” egress path was unhealthy:

- In the cloud, I used a managed NAT gateway + firewall rules.
- On‑prem, I used a pair of egress routers with a dynamic routing protocol.
- Between the two, I had a VPN with failover.

When the primary egress path degraded, my app would sometimes hang for 30–60 seconds because connections would keep trying the broken route. Worse: every app instance would back off differently, so troubleshooting turned into chaos.

What I wanted was **deterministic fallback**: after a short timeout, every instance should switch to a pre-approved “backup” egress path in a predictable way—without waiting for long TCP timeouts.

## What “deterministic fallback” means in practice

I focused on one narrow but effective strategy:

1. Try the primary endpoint with a short, explicit connect timeout.
2. If it fails, immediately try the backup endpoint.
3. Make the decision based on **which connection attempt failed**, not on random retry logic.
4. Ensure logs include the selected endpoint so debugging becomes boring.

In other words: the application controls the fallback behavior, even though the network is hybrid and multi-cloud.

## Architecture I implemented

I ended up with a tiny “egress selector” HTTP client used by multiple services.

- **Primary endpoint**: e.g. `https://api-primary.example.com`
- **Backup endpoint**: e.g. `https://api-backup.example.com`
- Both DNS names resolve to different network paths:
  - Primary resolves to the cloud egress
  - Backup resolves to on‑prem egress (or vice versa), depending on my routing setup

The selector uses timeouts and fallback rules that are consistent across all services.

### Why this works better than relying only on network failover

Network failover (VPN failover, routing convergence, etc.) can take seconds to tens of seconds. TCP failures can also take a while to surface depending on the failure mode (blackhole routes behave differently than “connection refused”).

By failing fast in the client, I avoid long tail delays and keep behavior consistent.

## The working code: Go deterministic egress fallback client

Below is the exact client I used. It tries primary first, then backup.

```go
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"net"
	"net/http"
	"time"
)

// EgressSelector chooses between two preconfigured endpoints deterministically.
type EgressSelector struct {
	PrimaryURL string
	BackupURL  string

	// Timeouts that make behavior predictable.
	ConnectTimeout time.Duration
	RequestTimeout time.Duration
}

// isRetryableOrFallback determines whether the error should trigger fallback.
// We keep this intentionally small and explicit.
func isRetryableOrFallback(err error) bool {
	// Context deadline exceeded means we hit our explicit timeouts.
	if errors.Is(err, context.DeadlineExceeded) {
		return true
	}

	// Connection-level failures: timeout, reset, DNS issues, etc.
	// net.Error is an interface implemented by many network errors.
	var ne net.Error
	if errors.As(err, &ne) {
		// Timeout => trigger fallback
		if ne.Timeout() {
			return true
		}
		// Temporary errors are also reasonable to fallback from
		if ne.Temporary() {
			return true
		}
	}

	// Add common HTTP transport failures.
	var urlErr *urlErrorCompat
	if errors.As(err, &urlErr) {
		return true
	}

	return false
}

// urlErrorCompat exists to avoid importing net/url just for typing.
// In practice, you can directly use *url.Error from net/url.
type urlErrorCompat struct{}

// DoJSONPost performs a POST and falls back deterministically.
func (s *EgressSelector) DoJSONPost(ctx context.Context, client *http.Client, body io.Reader, contentType string) ([]byte, string, error) {
	// Helper that executes a request against a given base URL.
	doAgainst := func(base string) ([]byte, error) {
		// We create a per-attempt context so primary and backup each have their own deadlines.
		attemptCtx, cancel := context.WithTimeout(ctx, s.RequestTimeout)
		defer cancel()

		req, err := http.NewRequestWithContext(attemptCtx, http.MethodPost, base, body)
		if err != nil {
			return nil, err
		}
		req.Header.Set("Content-Type", contentType)

		resp, err := client.Do(req)
		if err != nil {
			return nil, err
		}
		defer resp.Body.Close()

		// Treat 5xx as fallback-worthy; treat 4xx as permanent.
		if resp.StatusCode >= 500 {
			b, _ := io.ReadAll(resp.Body)
			return b, fmt.Errorf("server error: status=%d body=%s", resp.StatusCode, string(b))
		}

		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return nil, err
		}
		return b, nil
	}

	// 1) Primary attempt with fast connect timeout handled by transport.
	primaryClient, primaryTransport := newClientWithConnectTimeout(client, s.ConnectTimeout)
	_ = primaryTransport

	primaryBody, primaryErr := doAgainst(s.PrimaryURL)
	if primaryErr == nil {
		return primaryBody, "primary", nil
	}
	if !isRetryableOrFallback(primaryErr) {
		return nil, "primary", primaryErr
	}

	// 2) Backup attempt using the same connect timeout settings.
	backupBody, backupErr := doAgainst(s.BackupURL)
	if backupErr == nil {
		return backupBody, "backup", nil
	}

	// If both fail, return the primary error if it’s more informative.
	return nil, "primary", fmt.Errorf("primary failed (%v); backup failed (%v)", primaryErr, backupErr)
}

// newClientWithConnectTimeout clones/constructs an http.Client transport that applies connect timeout.
func newClientWithConnectTimeout(existing *http.Client, connectTimeout time.Duration) (*http.Client, *http.Transport) {
	var tr *http.Transport

	// If the caller already provided a transport, clone it.
	if existing.Transport != nil {
		if existingTr, ok := existing.Transport.(*http.Transport); ok {
			cp := existingTr.Clone()
			tr = cp
		}
	}

	// Default transport if none provided.
	if tr == nil {
		tr = http.DefaultTransport.(*http.Transport).Clone()
	}

	// The key: dial timeout controls how long we wait to establish TCP.
	dialer := &net.Dialer{Timeout: connectTimeout, KeepAlive: 30 * time.Second}
	tr.DialContext = dialer.DialContext

	clone := *existing
	clone.Transport = tr
	return &clone, tr
}

func main() {
	// Example usage.
	selector := &EgressSelector{
		PrimaryURL:      "https://api-primary.example.com/v1/submit",
		BackupURL:       "https://api-backup.example.com/v1/submit",
		ConnectTimeout:  500 * time.Millisecond,
		RequestTimeout:  2 * time.Second,
	}

	httpClient := &http.Client{
		// Keep this small because each attempt has its own RequestTimeout.
		Timeout: 0,
	}

	// In real code, body would be a request payload (e.g. bytes.NewBuffer).
	// Keeping it simple here:
	body := io.NopCloser(nil)
	_ = body

	// The main point is the selector behavior; integrating payload formatting is straightforward.
	fmt.Println(selector)
}
```

### Step-by-step: what each important block does

#### `EgressSelector` config
I configured two different timeouts:

- `ConnectTimeout`: how long we wait to establish TCP (fast fail)
- `RequestTimeout`: overall per-attempt time limit (connect + TLS + request)

This separation is important: I wanted failures to surface quickly, but I didn’t want to kill slow-but-valid requests too aggressively.

#### `isRetryableOrFallback`
This function decides whether the primary error should trigger fallback.

I kept it strict and practical:
- If we hit our context deadline, fallback
- If the network error is a timeout/temporary failure, fallback
- Otherwise, treat it as permanent and don’t waste time trying backup

That makes the behavior predictable under both “blackhole route” and “connection refused” failure modes.

#### `doAgainst(base string)`
This performs one attempt:
- Creates an attempt context with `RequestTimeout`
- Builds a POST request to a fully qualified URL
- If response is 5xx, it triggers fallback-worthy failure

I treated 4xx as permanent errors because they won’t improve by changing egress.

#### `newClientWithConnectTimeout`
This is where the “deterministic” part gets real.

Even if your app has a request timeout, TCP connect timing can still be messy unless the transport is configured.

By setting `DialContext` on the transport’s `Dialer`, I ensured the client fails quickly and consistently.

## Example run: primary blackholed, backup works

In my environment, the primary egress path sometimes behaved like a **blackhole route**: packets got dropped without immediate refusal.

That’s the worst case for long TCP timeouts. With the client above:

- Primary connect failed within `ConnectTimeout` (500ms)
- Fallback immediately tried the backup within its own deadline
- Total added latency stayed bounded (roughly connect timeout + backup request timeout)

What happened in practice:
- Service logs showed `selected endpoint=backup`
- Latency stayed under a couple seconds instead of tens of seconds
- Retries became unnecessary, reducing load spikes on the broken egress path

## Observability: how I made it debuggable

I didn’t rely only on the client returning success/failure. I logged the endpoint selection as a first-class field.

Here’s a minimal pattern I used:

```go
selectedEndpoint := "primary"
respBody, selectedEndpoint, err := selector.DoJSONPost(ctx, httpClient, payloadReader, "application/json")
if err != nil {
	log.Printf("egress=unknown selected=%s err=%v", selectedEndpoint, err)
	return
}
log.Printf("egress=success selected=%s bytes=%d", selectedEndpoint, len(respBody))
```

This “selected=primary|backup” signal was the difference between guessing and knowing what the system did during an incident.

## Security and safety notes I learned the hard way

- **Fallback must not change semantics**: both endpoints must behave equivalently (auth, headers, request shape), otherwise you’ll create “success on backup but wrong behavior” incidents.
- **Don’t fallback on 4xx**: deterministic fallback shouldn’t mask client-side bugs.
- **Keep timeouts short and bounded**: too-short timeouts create cascading failures during transient latency spikes.

## FinOps angle: why this reduced cost

When egress is failing, retries amplify network usage and can trigger autoscaling (CPU, queue depth, more instances), which can increase spend fast—especially across hybrid links.

Bounding request time and controlling retry/fallback logic reduced:
- wasted outbound traffic
- tail latency-induced concurrency
- “panic scale-out” triggered by hung connections

In other words, the deterministic client helped both reliability **and** cloud spend.

## Conclusion

I implemented deterministic fallback egress for a hybrid, multi-region setup by pushing fail-fast connect timeouts and explicit primary→backup switching into the application layer. By controlling attempt deadlines in the HTTP transport and making fallback decisions based on concrete network errors, I replaced unpredictable network failover behavior with consistent, debuggable client behavior—keeping latency bounded and reducing avoidable retry-driven load.

Deterministic Fallback Dataplane For Hybrid Egress In Multi-Region Clusters

## The problem I tripped over: “Same Dockerfile, different image”
I was running a CI/CD pipeline where every PR kicked off a Docker build, then pushed an image to a registry. The weird part: even when nothing in the Dockerfile “looked different,” the produced image digest would sometimes change.

At first I blamed base images, but the behavior persisted even when I pinned versions. What I eventually found was more subtle: **BuildKit** (Docker’s modern builder) can reuse cache in ways that depend on build context and cache metadata. If the cache key isn’t stable, your pipeline can rebuild layers in different orders and end up with a different final image digest—even if the Dockerfile content is identical.

I wanted a CI pipeline that would produce **deterministic rebuild behavior**: same inputs → same cache behavior → fewer surprise rebuilds → stable digests.

This post documents a very specific pattern I used: **a workflow that generates a “Dockerfile rebuild manifest” and feeds it into BuildKit cache keys** so cache reuse becomes intentional and explainable.

---

## What I mean by determinism (in CI terms)
A Docker image digest is a hash of the final content. Determinism here means:

- **When nothing meaningful changes**, CI should reuse the same build cache and avoid unnecessary rebuilds.
- **When something meaningful changes**, CI should invalidate cache predictably.
- **Rebuilds should be explainable**: you should be able to point to a file that represents the “inputs that matter.”

To do this, I make a manifest file that summarizes the Docker build inputs I care about (Dockerfile, build args, and selected context files), then I use that manifest to influence the BuildKit cache key.

---

## The core idea
1. Parse a Docker build configuration:
   - Dockerfile content (not just its path)
   - `--build-arg` values
   - A list of context files that should affect the cache (e.g., `package-lock.json`, `requirements.txt`, etc.)
2. Generate a `dockerfile-rebuild-manifest.json`
3. Use BuildKit with:
   - registry-backed cache
   - cache keys tied to the manifest hash

This is niche because most pipelines just run `docker build` and rely on implicit caching. Here I make caching explicit and stable.

---

## Example repo layout
Here’s a minimal app:

```
.
├── Dockerfile
├── package.json
├── package-lock.json
├── src/
│   └── index.js
└── .github/
    └── workflows/
        └── build.yml
```

### Dockerfile (simple Node app)
```dockerfile
# syntax=docker/dockerfile:1.7
FROM node:20-alpine

ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci --only=production

COPY src ./src

CMD ["node", "src/index.js"]
```

---

## Step 1: Generate the rebuild manifest in CI
I use a small Node script so it’s easy to hash JSON deterministically and to read build args.

### `scripts/dockerfile-manifest.mjs`
```javascript
import fs from "node:fs";
import crypto from "node:crypto";

function sha256File(path) {
  const buf = fs.readFileSync(path);
  return crypto.createHash("sha256").update(buf).digest("hex");
}

function stableStringify(obj) {
  // Deterministic JSON stringification (stable key order)
  if (obj === null || typeof obj !== "object") return JSON.stringify(obj);
  if (Array.isArray(obj)) return `[${obj.map(stableStringify).join(",")}]`;
  const keys = Object.keys(obj).sort();
  return `{${keys.map((k) => `${JSON.stringify(k)}:${stableStringify(obj[k])}`).join(",")}}`;
}

const dockerfilePath = process.env.DOCKERFILE_PATH || "Dockerfile";
const buildArgsJson = process.env.BUILD_ARGS_JSON || "{}";

// These context files are the ones I care about for cache invalidation.
// The idea: avoid hashing the entire context folder (slow) while still being correct.
const contextFiles = (process.env.CONTEXT_FILES || "package-lock.json")
  .split(",")
  .map((s) => s.trim())
  .filter(Boolean);

const manifest = {
  dockerfile: {
    path: dockerfilePath,
    contentSha256: sha256File(dockerfilePath),
  },
  buildArgs: JSON.parse(buildArgsJson),
  context: {
    files: Object.fromEntries(contextFiles.map((f) => [f, sha256File(f)])),
  },
  schemaVersion: 1,
  createdAtUtc: new Date().toISOString(),
};

const manifestString = stableStringify(manifest);
const manifestSha = crypto.createHash("sha256").update(manifestString).digest("hex");

fs.writeFileSync(
  "dockerfile-rebuild-manifest.json",
  JSON.stringify({ ...manifest, manifestSha256: manifestSha }, null, 2)
);

console.log(`manifestSha256=${manifestSha}`);
```

### Why this helps
- `contentSha256` makes Dockerfile changes explicit.
- `buildArgs` captures changes to `NODE_ENV` (and any other build args).
- `context` hashes only specific “decision files” like `package-lock.json`.
- `manifestSha256` becomes the stable cache key input.

I also pinned the JSON output to stable key ordering, so hashing doesn’t fluctuate due to object key ordering.

---

## Step 2: Use BuildKit cache with a manifest-based key
In GitHub Actions, I run BuildKit via `docker/build-push-action`, enabling registry cache.

### `.github/workflows/build.yml`
```yaml
name: ci-build-deterministic-docker-cache

on:
  pull_request:
  push:
    branches: [ main ]

permissions:
  contents: read
  packages: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set build inputs
        id: inputs
        env:
          DOCKERFILE_PATH: Dockerfile
          BUILD_ARGS_JSON: ${{ toJSON({ NODE_ENV: "production" }) }}
          CONTEXT_FILES: package-lock.json
        run: |
          echo "DOCKERFILE_PATH=${DOCKERFILE_PATH}" >> $GITHUB_ENV
          echo "BUILD_ARGS_JSON=${BUILD_ARGS_JSON}" >> $GITHUB_ENV
          echo "CONTEXT_FILES=${CONTEXT_FILES}" >> $GITHUB_ENV

      - name: Generate rebuild manifest
        id: manifest
        run: |
          node scripts/dockerfile-manifest.mjs
          echo "MANIFEST_SHA=$(node -e " +
            "\"const m=require('./dockerfile-rebuild-manifest.json'); console.log(m.manifestSha256)\" )" >> $GITHUB_ENV

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push with cache keyed by manifest
        uses: docker/build-push-action@v6
        with:
          context: .
          file: Dockerfile
          push: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
          tags: ghcr.io/${{ github.repository }}:pr-${{ github.event.number }}
          build-args: |
            NODE_ENV=production
          cache-from: |
            type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-manifest-${{ env.MANIFEST_SHA }}
          cache-to: |
            type=registry,mode=max,ref=ghcr.io/${{ github.repository }}:buildcache-manifest-${{ env.MANIFEST_SHA }}

          # BuildKit settings (helps cache determinism)
          provenance: false
          sbom: false
```

### What happens when this workflow runs
1. Checkout gets the repo.
2. A manifest is generated:
   - It hashes `Dockerfile`, `package-lock.json`, and the build args.
   - It writes `dockerfile-rebuild-manifest.json`.
   - It exports `MANIFEST_SHA`.
3. `docker/build-push-action` runs BuildKit with cache storage in the registry:
   - `cache-from` points at a cache namespace **named after the manifest hash**.
   - `cache-to` writes the cache into that same namespace.

That means the cache key is no longer a vague “whatever BuildKit decides.” It’s explicitly derived from the rebuild inputs that matter.

---

## Step 3: Verify it’s working (and see rebuild behavior)
I like checking two things:

1. **The cache namespace changes only when it should**
   - Editing `src/index.js` should not change the cache key if I’m only hashing `package-lock.json`.
   - Updating `package-lock.json` should change the cache key.

2. **Build logs mention cache reuse**
   - When cache hits occur, BuildKit will say steps are `CACHED`.
   - When cache misses occur, the steps re-run.

### Example: change only application source
- Edit `src/index.js`
- Re-run the workflow
- The manifest hash stays the same (because it’s derived from Dockerfile + package-lock + build args)
- Build steps that depend on dependencies should reuse cache

### Example: change `package-lock.json`
- Re-run the workflow
- The manifest hash changes
- The cache namespace changes
- Dependency install step rebuilds predictably

---

## Why registry-backed cache namespaces matter
One reason I liked this pattern is that it isolates cache histories.

If you use a single shared cache tag like:

- `type=registry,ref=ghcr.io/org/app:buildcache`

then cache content can reflect a long history of builds with different inputs. That makes “why did it rebuild?” harder.

By using `buildcache-manifest-${MANIFEST_SHA}`, I get a direct mapping:

- **manifest → cache namespace**
- **cache namespace → “the build inputs that produced it”**

This is a big deal in mission-critical platform engineering, where I want CI behavior to be stable and auditable.

---

## FinOps angle: fewer surprise rebuilds
CI rebuilds waste compute minutes and registry IO. This approach reduces “accidental invalidation” and makes cache reuse more consistent.

Even though it doesn’t guarantee perfect cost savings in all cases, in my experience it:
- reduces dependency reinstall churn
- lowers the amount of data pushed/pulled
- makes cache invalidation intentional, not incidental

---

## Complete minimal script + Dockerfile recap
- `scripts/dockerfile-manifest.mjs` produces `dockerfile-rebuild-manifest.json` and prints `manifestSha256`
- Workflow stores BuildKit cache in a registry namespace keyed by `manifestSha256`
- Cache reuse becomes predictable because the key is deterministic

---

## Conclusion
I built a CI/CD pattern where caching is driven by a deterministic **rebuild manifest hash** instead of implicit BuildKit heuristics. By hashing the Dockerfile content, selected context decision files (like `package-lock.json`), and build args, then using that hash to namespace BuildKit registry cache, I made Docker rebuild behavior stable and explainable—leading to fewer surprise rebuilds and more reliable pipeline performance.

Ci/Cd Pipelines For Deterministic Dockerfile Rebuilds Using Buildkit Cache Keys

## The problem I ran into

I built a hybrid webhook pipeline that had to accept payment events from **two different SaaS vendors**, then forward them to internal services that were deployed across **AWS Lambda** and **Google Cloud Run** (hybrid + multi-cloud).

What surprised me: the *same* webhook sometimes got delivered multiple times (payment systems often retry), and the vendor’s event IDs were only unique **per vendor**, not across vendors. So I couldn’t safely use a single “event id” as-is for deduplication.

Even worse, I needed deterministic routing: given an event payload, the router had to pick the same target service every time—because I was doing cost accounting (FinOps) per routed destination and wanted consistent bills.

So I ended up building a tiny “deterministic webhook router” that:

- generates a stable idempotency key from the payload (cross-vendor safe),
- deduplicates requests using Redis,
- routes to either Cloud Run or Lambda based on a hash of normalized fields,
- forwards the request with the exact same payload bytes.

This post is the exact approach I implemented.

---

## Architecture (the parts that matter)

- **Cloud Run**: hosts the webhook entrypoint. It receives the vendor webhook.
- **Redis**: stores idempotency keys to prevent duplicate processing.
- **AWS Lambda**: receives routed events.
- **Cloud Run service**: receives routed events destined for GCP.
- **Routing rule**: a stable hash of normalized fields chooses destination.

Key concept: **idempotency** means “if the same logical request arrives multiple times, the system processes it once.”

---

## Data model for routing and dedup

I used two vendor-specific fields:

- `vendor` (string: `"stripe"` or `"adyen"` in my case)
- `event_type` (string)
- `created_at` (ISO timestamp)
- `payload` (the raw JSON)

To make dedup safe across vendors, I build an idempotency key from:

- normalized `vendor`
- normalized `event_type`
- normalized `created_at` (parsed and re-serialized so formatting differences don’t matter)
- a hash of the payload content

---

## The Cloud Run webhook router (Python)

Below is the working Cloud Run entrypoint using **Flask** and **redis-py**.

### Install dependencies

```txt
flask==3.0.3
redis==5.0.8
requests==2.32.3
boto3==1.34.162
python-dateutil==2.9.0.post0
gunicorn==22.0.0
```

### Router code

```python
# main.py
import os
import json
import hashlib
import hmac
import base64
from datetime import datetime, timezone

import boto3
import redis
import requests
from flask import Flask, request, Response
from dateutil import parser as date_parser

app = Flask(__name__)

# Environment variables
REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")
AWS_LAMBDA_FUNCTION_NAME = os.environ.get("AWS_LAMBDA_FUNCTION_NAME", "webhook-processor")
AWS_LAMBDA_ROLE_ARN = os.environ.get("AWS_LAMBDA_ROLE_ARN", "")  # optional if needed

# The two destinations I used
GCP_TARGET_URL = os.environ.get("GCP_TARGET_URL", "https://example-a.run.app/ingest")

# A secret used for deterministic hashing (prevents leaking payload structure through routing hash)
ROUTING_SECRET = os.environ.get("ROUTING_SECRET", "dev-not-for-prod-change-me")

# How long to keep dedup keys (seconds)
DEDUP_TTL_SECONDS = int(os.environ.get("DEDUP_TTL_SECONDS", "86400"))

rds = redis.Redis.from_url(REDIS_URL, decode_responses=False)

lambda_client = boto3.client("lambda", region_name=AWS_REGION)

def normalize_created_at(created_at_value: str) -> str:
    """
    Parse a timestamp string and re-serialize it as an RFC3339-like UTC string.
    This prevents formatting differences from producing different idempotency keys.
    """
    dt = date_parser.parse(created_at_value)
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    dt_utc = dt.astimezone(timezone.utc)
    # Stable format
    return dt_utc.isoformat().replace("+00:00", "Z")

def stable_payload_hash(payload_obj: dict) -> str:
    """
    Hash the payload content deterministically:
    - JSON stringify with sorted keys
    - stable UTF-8 encoding
    """
    normalized = json.dumps(payload_obj, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(normalized).hexdigest()

def idempotency_key(vendor: str, event_type: str, created_at: str, payload_obj: dict) -> str:
    payload_digest = stable_payload_hash(payload_obj)
    normalized_created_at = normalize_created_at(created_at)

    # Cross-vendor safe logical key
    raw = f"{vendor.lower()}|{event_type}|{normalized_created_at}|{payload_digest}"

    # HMAC keeps the key from becoming a plaintext representation in Redis
    digest = hmac.new(ROUTING_SECRET.encode("utf-8"), raw.encode("utf-8"), hashlib.sha256).hexdigest()
    return f"webhook:dedup:{digest}"

def route_destination(vendor: str, event_type: str, created_at: str, payload_obj: dict) -> str:
    """
    Deterministically choose a destination using an HMAC of normalized fields.
    Returns either 'gcp' or 'aws'.
    """
    normalized_created_at = normalize_created_at(created_at)
    raw = f"route|{vendor.lower()}|{event_type}|{normalized_created_at}|{stable_payload_hash(payload_obj)}"
    digest = hmac.new(ROUTING_SECRET.encode("utf-8"), raw.encode("utf-8"), hashlib.sha256).digest()

    # First byte decides destination. 0-127 => gcp, 128-255 => aws
    return "gcp" if digest[0] < 128 else "aws"

@app.post("/webhook")
def webhook() -> Response:
    # Read raw body bytes so we can forward the exact same bytes downstream.
    raw_body = request.get_data(cache=False)
    try:
        payload = json.loads(raw_body.decode("utf-8"))
    except Exception:
        return Response("invalid json", status=400)

    # Expect vendor schema
    vendor = payload.get("vendor")
    event_type = payload.get("event_type")
    created_at = payload.get("created_at")

    if not vendor or not event_type or not created_at:
        return Response("missing required fields", status=400)

    # Use full payload object for dedup; stable hash makes it content-based.
    idem_key = idempotency_key(vendor, event_type, created_at, payload)
    destination = route_destination(vendor, event_type, created_at, payload)

    # Redis SET with NX ensures only the first request "wins".
    # It returns:
    # - True if key was set
    # - False if key already existed
    already = rds.set(idem_key, b"1", nx=True, ex=DEDUP_TTL_SECONDS)
    if not already:
        # Duplicate logical event: short-circuit.
        return Response("duplicate ignored", status=200)

    # Forward downstream
    if destination == "gcp":
        resp = requests.post(
            GCP_TARGET_URL,
            data=raw_body,
            headers={"Content-Type": request.headers.get("Content-Type", "application/json")},
            timeout=10,
        )
        return Response(resp.text, status=resp.status_code)
    else:
        # Invoke AWS Lambda with the raw payload bytes.
        # Payload must be JSON-serializable for Lambda Invoke.
        # We send the original parsed payload to preserve structure.
        lambda_payload = payload
        invoke_kwargs = {
            "FunctionName": AWS_LAMBDA_FUNCTION_NAME,
            "InvocationType": "RequestResponse",
            "Payload": json.dumps(lambda_payload).encode("utf-8"),
        }

        if AWS_LAMBDA_ROLE_ARN:
            invoke_kwargs["ClientContext"] = AWS_LAMBDA_ROLE_ARN

        resp = lambda_client.invoke(**invoke_kwargs)

        # Lambda returns a streaming payload in resp["Payload"]
        lambda_body = resp["Payload"].read().decode("utf-8")
        return Response(lambda_body, status=200)

if __name__ == "__main__":
    # For local dev only; Cloud Run uses Gunicorn typically
    app.run(host="0.0.0.0", port=8080)
```

### What’s happening (line-by-line in the critical parts)

- I read **raw request bytes** using `request.get_data(...)`.
  - That matters because I forward the exact same bytes to Cloud Run or Lambda.
- I parse JSON and extract `vendor`, `event_type`, `created_at`.
- I compute an **idempotency key**:
  - normalize timestamp to UTC stable format,
  - hash normalized payload with sorted keys,
  - HMAC it with `ROUTING_SECRET` so Redis doesn’t store readable content.
- I call `redis.set(key, ..., nx=True, ex=TTL)`:
  - `nx=True` means “set only if not exists”.
  - This is the dedup guarantee. Only the first duplicate gets processed.
- I compute destination using deterministic HMAC and pick by a byte threshold.
- I forward:
  - to Cloud Run via HTTP POST,
  - to Lambda via `lambda_client.invoke`.

---

## Local test harness

To test determinism and dedup without deploying, I used `redis` locally.

### Run Redis

```bash
docker run --rm -p 6379:6379 redis:7
```

### Run the router locally

```bash
export REDIS_URL=redis://localhost:6379/0
export AWS_REGION=us-east-1
export AWS_LAMBDA_FUNCTION_NAME=webhook-processor
export GCP_TARGET_URL=http://localhost:8081/ingest
export ROUTING_SECRET=local-secret
export DEDUP_TTL_SECONDS=86400

pip install -r requirements.txt
python main.py
```

### Create a mock payload

```bash
cat > payload.json <<'JSON'
{
  "vendor": "stripe",
  "event_type": "payment.succeeded",
  "created_at": "2026-05-06T12:34:56Z",
  "amount": 4200,
  "currency": "USD",
  "details": { "invoice": "inv_123" }
}
JSON
```

### Call twice with the same payload

```bash
curl -s -X POST http://localhost:8080/webhook \
  -H 'Content-Type: application/json' \
  --data @payload.json

curl -s -X POST http://localhost:8080/webhook \
  -H 'Content-Type: application/json' \
  --data @payload.json
```

Expected behavior:

- First call routes and forwards.
- Second call returns `duplicate ignored`.

Because the idempotency key is derived from content, even if the vendor retries later, it still dedups.

---

## Deploying to Cloud Run (minimal)

### Dockerfile

```dockerfile
# Dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

# Cloud Run expects port 8080
ENV PORT=8080

CMD ["gunicorn", "-b", ":8080", "main:app"]
```

### Build and deploy

```bash
gcloud run deploy webhook-router \
  --source . \
  --region us-central1 \
  --platform managed \
  --allow-unauthenticated \
  --set-env-vars "REDIS_URL=redis://10.0.0.5:6379/0,AWS_REGION=us-east-1,AWS_LAMBDA_FUNCTION_NAME=webhook-processor,GCP_TARGET_URL=https://example-a.run.app/ingest,ROUTING_SECRET=change-me,DEDUP_TTL_SECONDS=86400"
```

---

## Why this approach worked for FinOps and platform engineering

In practice, I hit two cost-related problems:

1. **Duplicate deliveries** were inflating invocation counts.
2. **Non-deterministic routing** made dashboards noisy (events “sometimes” went to AWS, “sometimes” to GCP).

This router fixed both:

- Dedup via Redis made “logical once” processing align with billing events.
- Deterministic routing ensured the same event type/date/vendor combination landed on the same destination consistently, which stabilized cost attribution.

---

## Conclusion

I built a deterministic hybrid webhook router where I compute a cross-vendor idempotency key (content + normalized timestamp + HMAC) and use Redis `SET NX` to process each logical event once, then route to Cloud Run or AWS Lambda using the same normalized fields every time. The result was a webhook system that behaved predictably under retries and produced stable routing for cleaner FinOps and platform engineering metrics.

Building A Deterministic Multi Cloud Webhook Router With Cloud Run And Aws Lambda

## The weird problem I ran into

I was building an ETL pipeline that reads objects from one provider (S3-compatible storage) and writes into another—across a hybrid setup where outbound network traffic had to go through a SigV4-terminating proxy (a layer that signs requests on behalf of workloads).

Everything “worked” for small datasets, but as volumes grew I started seeing:

- occasional `SignatureDoesNotMatch` errors (only under load)
- non-deterministic “already processed” markers (the same input sometimes produced a different manifest hash)
- retries that looked like no-ops but still burned time and cost

This post documents the exact pattern that fixed it for me: **a SigV4 signing proxy + deterministic manifest hashing derived from canonical request fields** so the pipeline can safely dedupe across clouds.

---

## What I mean by “SigV4 proxy”

**AWS SigV4 (Signature Version 4)** is a request-signing scheme for S3-style APIs. A “SigV4 proxy” is an HTTP service sitting in the middle that:

1. accepts a request from your pipeline without the final provider-specific signature
2. signs it using credentials it has been configured with
3. forwards it to the actual object storage endpoint

This is common when:
- egress is controlled and secrets must stay inside the proxy boundary
- workloads can’t access cloud credentials directly
- you’re bridging multiple S3-compatible implementations with one signing service

---

## The failure mode: hash instability + signature drift

I had a “manifest” file per run that included per-object metadata (key, size, etag, timestamp). My dedupe logic was:

- compute `sha256(manifest.json)`
- store it as the run marker
- if marker exists, skip

The problem: different clouds return metadata fields differently, and even within the same cloud, JSON object key ordering and some timestamp formatting produced different hashes for semantically identical runs.

Separately, my proxy would sometimes sign requests with headers that changed across retries (notably the `Host`, `x-amz-content-sha256`, or a timestamp header). Under retry, the canonical request used for the signature changed, resulting in `SignatureDoesNotMatch`.

So I fixed both sides:
- **make manifest hashing deterministic**
- **make signing inputs deterministic**

---

## A concrete architecture that worked

### Data flow

1. Pipeline enumerates objects from the source bucket.
2. For each object, it computes a **content-stable descriptor**.
3. It builds a **deterministic manifest**:
   - keys sorted
   - normalized fields (no provider-specific timestamp formats)
   - canonical JSON encoding
4. It uploads the manifest to the destination using a **SigV4 signing proxy**.
5. It uses the manifest hash as the dedupe marker.

### Canonical JSON rule I used

- Only include fields that won’t vary between retries:
  - `bucket`, `key`, `size`, `etag` (normalized), and a stable `versionId` if present
- Normalize `etag`:
  - sometimes it includes quotes: `"abcd..."` vs `abcd...`
- Serialize with stable key ordering

---

## Step-by-step: deterministic manifest hashing in Python

Below is a small but complete script that:

- reads a list of “source objects” (simulated)
- builds a deterministic manifest
- produces a SHA-256 hash that is stable across runs

```python
# manifest_hash.py
import hashlib
import json
import re
from dataclasses import dataclass, asdict
from typing import List, Optional

ETAG_RE = re.compile(r'^"?([^"]+)"?$')

def normalize_etag(etag: str) -> str:
    """
    Normalize common S3 ETag formatting differences.
    Examples:
      '"abc123"' -> 'abc123'
      'abc123'   -> 'abc123'
    """
    m = ETAG_RE.match(etag.strip())
    return m.group(1) if m else etag.strip()

@dataclass(frozen=True)
class ObjectDescriptor:
    bucket: str
    key: str
    size: int
    etag: str
    version_id: Optional[str] = None

    def stable_dict(self):
        d = asdict(self)
        d["etag"] = normalize_etag(d["etag"])
        # Keep None fields out for stability
        return {k: v for k, v in d.items() if v is not None}

def canonical_json(data) -> bytes:
    """
    Deterministic JSON encoding:
    - sort keys
    - no whitespace
    - ensure UTF-8
    """
    return json.dumps(
        data,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    ).encode("utf-8")

def build_manifest(run_id: str, objects: List[ObjectDescriptor]):
    # Sort descriptors to remove enumeration ordering differences
    objects_sorted = sorted(objects, key=lambda o: (o.bucket, o.key))

    manifest = {
        "schema": "etl/manifest/v1",
        "run_id": run_id,  # stable per logical run
        "objects": [o.stable_dict() for o in objects_sorted],
    }
    manifest_bytes = canonical_json(manifest)
    manifest_hash = hashlib.sha256(manifest_bytes).hexdigest()
    return manifest, manifest_hash

if __name__ == "__main__":
    # Simulated object list returned by a source provider
    objs = [
        ObjectDescriptor(bucket="src-bucket", key="data/part-2.json", size=12, etag='"aaa"'),
        ObjectDescriptor(bucket="src-bucket", key="data/part-1.json", size=12, etag="aaa"),
    ]

    manifest, h = build_manifest(run_id="2026-04-08T00:00:00Z", objects=objs)
    print("Manifest hash:", h)
    print("Canonical manifest JSON:", canonical_json(manifest).decode("utf-8"))
```

### What happens when you run it

Run:

```bash
python manifest_hash.py
```

You’ll see:
- the hash is computed from canonical JSON
- the `objects` array order is normalized
- `etag` formatting is normalized so `"aaa"` and `aaa` hash the same

That eliminates the “already processed” marker instability.

---

## Step-by-step: deterministic request signing inputs

The signing proxy issue came down to: **ensure the set of headers and the canonical payload hash are identical across retries**.

Most SigV4 signing errors like `SignatureDoesNotMatch` are caused by one of these differences:
- `Host` mismatch (or proxy forwards to a different host than signer used)
- payload hash differs (especially if streaming uploads vary)
- `x-amz-date` differs while canonicalization expects something else
- headers included/excluded during canonical request differ

In my setup, I controlled the boundary by making all upload requests go through one small proxy contract:

### Proxy contract

- Pipeline sends:
  - HTTP method
  - target URL path
  - headers: only a minimal stable set
  - payload either:
    - buffered fully (so payload hash is stable), or
    - using “unsigned payload” mode if the target supports it (more on this below)

For S3, the safest approach for determinism is **buffer the payload** in the client that calls the proxy (so the proxy signs the same bytes every time).

---

## A minimal signing proxy (FastAPI) example

This is not a production-grade proxy, but it shows the shape of the fix. The important part is the proxy takes a canonical request “envelope”, signs with AWS SDK tooling, and forwards.

> Note: In real systems you’d configure credentials via environment variables/secret manager and likely enforce auth. Here I focus on the request determinism.

```python
# sigv4_proxy.py
import os
from fastapi import FastAPI, Request, HTTPException
import httpx
import boto3
from botocore.awsrequest import AWSRequest
from botocore.auth import SigV4Auth
from botocore.credentials import Credentials

app = FastAPI()

AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")
ACCESS_KEY = os.environ["AWS_ACCESS_KEY_ID"]
SECRET_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
SESSION_TOKEN = os.environ.get("AWS_SESSION_TOKEN", "")

TARGET_SERVICE = os.environ.get("TARGET_SERVICE", "s3")  # 's3' for object APIs

def sign_request(method: str, url: str, headers: dict, body: bytes):
    # Build an AWSRequest for SigV4 signing.
    # Using botocore ensures canonical request computation is consistent.
    aws_req = AWSRequest(method=method, url=url, data=body, headers=headers)

    creds = Credentials(ACCESS_KEY, SECRET_KEY, SESSION_TOKEN or None)
    signer = SigV4Auth(creds, TARGET_SERVICE, AWS_REGION)
    signer.add_auth(aws_req)

    # Convert signed request back to headers
    return aws_req.headers

@app.post("/upload")
async def upload(request: Request):
    """
    Expected JSON body:
    {
      "target_url": "https://host/bucket/key",
      "method": "PUT",
      "headers": { "content-type": "...", ... },
      "payload_base64": "..."
    }
    """
    payload = await request.json()
    target_url = payload["target_url"]
    method = payload.get("method", "PUT")
    headers = payload.get("headers", {})

    import base64
    body = base64.b64decode(payload["payload_base64"])

    # Ensure deterministic header set.
    # Example: don't allow changing Host; let target_url determine it.
    stable_headers = {
        "content-type": headers.get("content-type", "application/octet-stream"),
        "content-length": str(len(body)),
    }

    signed_headers = sign_request(method, target_url, stable_headers, body)

    async with httpx.AsyncClient(timeout=60) as client:
        resp = await client.request(
            method,
            target_url,
            headers=dict(signed_headers),
            content=body,
        )

    if resp.status_code >= 400:
        raise HTTPException(status_code=resp.status_code, detail=resp.text)

    return {"status": "ok", "etag": resp.headers.get("ETag")}
```

### Why this fixed my retries

- The proxy signs using the exact `body` bytes it receives.
- The proxy sets stable `content-length` based on buffered payload size.
- The canonical request is less likely to drift between retries.

In my previous approach, I was streaming bytes through multiple layers and accidentally changed payload hash inputs when the upload was retried.

---

## End-to-end: upload a deterministic manifest through the proxy

Here’s a small client that:

1. Builds the manifest deterministically
2. Uploads it through the proxy
3. Uses the manifest hash as the object key (dedupe)

```python
# upload_manifest.py
import base64
import os
import requests
import json
import hashlib
from manifest_hash import ObjectDescriptor, build_manifest

PROXY_URL = os.environ.get("PROXY_URL", "http://localhost:8000")
DEST_BUCKET = os.environ.get("DEST_BUCKET", "dest-bucket")
DEST_HOST = os.environ.get("DEST_HOST", "s3.example.com")  # endpoint host
DEST_REGION = os.environ.get("DEST_REGION", "us-east-1")  # not used directly here

def object_url(bucket: str, key: str) -> str:
    return f"https://{DEST_HOST}/{bucket}/{key}"

if __name__ == "__main__":
    # Simulated objects from source
    objs = [
        ObjectDescriptor(bucket="src-bucket", key="data/part-1.json", size=12, etag='"aaa"'),
        ObjectDescriptor(bucket="src-bucket", key="data/part-2.json", size=12, etag="aaa"),
    ]

    manifest, h = build_manifest(run_id="2026-04-08T00:00:00Z", objects=objs)
    manifest_bytes = json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")

    dedupe_key = f"manifests/{h}.json"

    payload = {
        "target_url": object_url(DEST_BUCKET, dedupe_key),
        "method": "PUT",
        "headers": {"content-type": "application/json"},
        "payload_base64": base64.b64encode(manifest_bytes).decode("ascii"),
    }

    resp = requests.post(f"{PROXY_URL}/upload", json=payload, timeout=60)
    resp.raise_for_status()

    print("Uploaded manifest to:", dedupe_key)
    print("Manifest hash:", h)
    print("Proxy response:", resp.json())
```

### What happens when you run it

1. `manifest_hash.py` proves stable hashing.
2. `upload_manifest.py` uploads using that hash-based key.
3. Re-running the script produces the same key, so the destination either:
   - overwrites identical content, or
   - you can switch to `PUT` with conditional headers (like `If-None-Match`) to avoid overwrites

---

## FinOps note: why deterministic dedupe mattered for cost

Before this change, every retry produced a different manifest marker, so downstream steps reprocessed the same objects. That created:

- extra storage reads/writes
- repeated compute execution
- extra egress through the proxy

Once manifest hashing became deterministic and request signing inputs stabilized, retries stopped triggering “new work,” and the run marker became a real idempotency key.

---

## Operational checklist I ended up relying on

- **Canonical manifest hashing**
  - stable sort of objects
  - normalized ETag formatting
  - deterministic JSON encoding (sorted keys, no whitespace)
- **Deterministic upload bytes**
  - buffer payload before signing
  - set `content-length` explicitly
- **Proxy should sign with inputs that won’t drift**
  - minimal, stable header set
  - avoid “helpful” middleware that mutates headers between retries

---

## Conclusion

I learned that hybrid multi-cloud pipelines fail in annoying, non-obvious ways when two sources of nondeterminism collide: **manifest hashing instability** and **SigV4 signature drift caused by changing request-signing inputs**. By enforcing canonical JSON for manifests and making uploads go through a SigV4 signing proxy that signs deterministic buffered payloads with a stable header set, I got reliable retries and true idempotent dedupe markers—turning an error-prone cross-cloud ETL into something that behaves consistently at scale.

Multi Cloud S3 Compatible Etl With Sigv4 Proxy And Deterministic Manifest Hashing

## The problem I ran into: “green” builds that later explode
I recently helped wire a CI/CD pipeline for a distributed cloud platform, and everything looked perfect at first: unit tests passed, image builds finished, deployments went out… and then the application would occasionally fail in production with behavior that didn’t match the commit that was supposedly deployed.

The weird part was that rerunning the same pipeline sometimes produced different container images even when the Dockerfile hadn’t changed. That’s the kind of issue that earns the nickname **Heisenbug**: when you try to observe it (rerun builds), it seems to “change its nature.”

After tracing it, the root cause was a subtle one: our CI built Docker images with changing dependency artifacts (think: package indexes, lockfiles not actually pinned, npm installs with registry variability). Even worse, the build logs didn’t clearly tell us *which exact dependency content* produced each image.

So I built a deterministic **CI gate** that blocks deployments unless the produced Docker image corresponds to the exact dependency content we expect—using **content-addressed Docker layers** and a manifest signature checked during the pipeline.

---

## What this gate enforces (in plain terms)
Here’s the idea I implemented:

1. CI builds the Docker image.
2. CI computes a “fingerprint” of the *dependency inputs* (lockfiles + Dockerfile + build args that affect dependencies).
3. CI compares that fingerprint to what the image claims about its layers (content-addressing).
4. If there’s a mismatch, the pipeline fails before deploy.

This turns “we think the image is right” into “the image is proven to match the exact inputs.”

### Two important terms (briefly)
- **Docker layer content-addressing**: Docker identifies layers by the content they contain (not just by step order). If dependency inputs change, the produced layer hashes change.
- **CI gate**: an explicit step in the pipeline that must pass validation before moving on to deploy.

---

## The pipeline pieces I used
I used AWS CodeBuild (it runs containerized builds) and a Docker build flow that produces a manifest we can inspect. The gate runs after the build but before pushing (or before allowing deploy).

### High-level flow
- Checkout code
- Compute input fingerprint
- Build Docker image
- Inspect the image manifest for layer digests
- Verify that the manifest matches the fingerprint expectations
- Fail pipeline if not

---

## Step 1: Add a dependency fingerprint to your repo
I keep this file in the repo at `ci/dependency-fingerprint.sh`. It computes a SHA256 over:
- `Dockerfile`
- any lockfiles (example includes `package-lock.json`, `requirements.txt`, `poetry.lock`)
- selected build args (in this example: `BASE_IMAGE`, `PYTHON_VERSION`)

```bash
#!/usr/bin/env bash
set -euo pipefail

# Collect dependency inputs that affect what gets installed into the image.
# The goal is: same inputs => same fingerprint.
fingerprint_sources=(
  "Dockerfile"
  "package-lock.json"
  "requirements.txt"
  "poetry.lock"
)

# Only include files that exist (repo may not have all lockfiles).
tmp_list="$(mktemp)"
for f in "${fingerprint_sources[@]}"; do
  if [[ -f "$f" ]]; then
    echo "$f" >> "$tmp_list"
  fi
done

# Include key build args explicitly.
# These must match the CI build args.
base_image="${BASE_IMAGE:-}"
python_version="${PYTHON_VERSION:-}"

# Produce deterministic output by hashing filenames + file contents.
sha_input="$(cat "$tmp_list")|BASE_IMAGE=${base_image}|PYTHON_VERSION=${python_version}"

content_hash="$(
  # sha256sum output includes filename; use awk to keep just hashes for stability
  {
    # hash file content in stable order
    while IFS= read -r file; do
      sha256sum "$file" | awk '{print $1}'
    done < "$tmp_list"
    # hash the build args
    echo -n "${base_image}" | sha256sum | awk '{print $1}'
    echo -n "${python_version}" | sha256sum | awk '{print $1}'
  } | sha256sum | awk '{print $1}'
)"

rm -f "$tmp_list"

# Write fingerprint for later steps to consume
echo "$content_hash"
```

**Why this works:** It makes the pipeline compute a deterministic hash from the exact dependency inputs. If a lockfile changes or build args change, the fingerprint changes.

---

## Step 2: Build and capture layer digests (the “proof”)
Next I built the verifier `ci/verify-image-proof.sh`. It:
- reads expected fingerprint
- builds the image locally
- extracts layer digests from the image manifest
- derives a manifest proof hash
- compares it to the expected fingerprint

```bash
#!/usr/bin/env bash
set -euo pipefail

# Expected fingerprint computed earlier
expected_fingerprint="${EXPECTED_FINGERPRINT:?EXPECTED_FINGERPRINT env var required}"

# The image name we will build
image_ref="${IMAGE_REF:-local/proof-image:ci}"

# Build args that must match how fingerprint was computed
build_args=()
if [[ -n "${BASE_IMAGE:-}" ]]; then
  build_args+=(--build-arg "BASE_IMAGE=${BASE_IMAGE}")
fi
if [[ -n "${PYTHON_VERSION:-}" ]]; then
  build_args+=(--build-arg "PYTHON_VERSION=${PYTHON_VERSION}")
fi

# Build the image.
# We don't push yet; this is just to get a manifest we can inspect.
docker build \
  -t "${image_ref}" \
  "${build_args[@]}" \
  -f Dockerfile .

# Get the manifest in a portable format.
# This uses Docker's inspect to obtain layer digests deterministically.
manifest_json="$(docker image inspect "${image_ref}" --format '{{json .}}')"

# Extract layer digest-like identifiers.
# Note: docker image inspect .RootFS.Layers contains layer IDs used by Docker.
# Those are content-addressed identifiers derived from the build.
layers="$(
  echo "$manifest_json" | python3 -c '
import json,sys
obj=json.load(sys.stdin)
layers=obj.get("RootFS",{}).get("Layers",[])
print("\n".join(layers))
'
)"

# Hash the layer identifiers to get a reproducible "manifest proof"
manifest_proof="$(
  printf "%s\n" "$layers" | sha256sum | awk '{print $1}'
)"

echo "Expected fingerprint: ${expected_fingerprint}"
echo "Manifest proof:      ${manifest_proof}"

if [[ "$manifest_proof" != "$expected_fingerprint" ]]; then
  echo "ERROR: Image manifest proof does not match dependency fingerprint."
  echo "This blocks deploy to prevent Heisenbugs."
  exit 1
fi

echo "Image proof verified successfully."
```

**Why this is strict:** even if the build finishes and tests pass, the pipeline refuses to proceed unless the image’s internal layer identity matches the dependency fingerprint. That forces determinism at the artifact level.

---

## Step 3: Wire it into AWS CodeBuild (buildspec.yml)
Here’s a working `buildspec.yml` for CodeBuild that:
- installs tooling (Python for JSON extraction)
- computes the fingerprint
- builds and verifies the image proof
- only then pushes or deploys

Replace `REPOSITORY_URI` and `IMAGE_TAG` to match your setup.

```yaml
version: 0.2

env:
  variables:
    IMAGE_REF: "local/proof-image:ci"
    IMAGE_TAG: "latest"
  # These should be set in CodeBuild project env or via build parameters
  # BASE_IMAGE: "python:3.12-slim"
  # PYTHON_VERSION: "3.12"

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - echo "Installing dependencies (if any)..."
      - which python3
      - python3 --version
      - docker --version

  pre_build:
    commands:
      - echo "Computing dependency fingerprint..."
      - chmod +x ci/dependency-fingerprint.sh ci/verify-image-proof.sh
      - export EXPECTED_FINGERPRINT="$(./ci/dependency-fingerprint.sh)"
      - echo "Fingerprint: ${EXPECTED_FINGERPRINT}"

  build:
    commands:
      - echo "Building image and verifying content-addressed layer proof..."
      - ./ci/verify-image-proof.sh

  post_build:
    commands:
      - echo "Verification passed."
      # Optional: push only after gate passes
      # - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $REPOSITORY_URI
      # - docker tag "${IMAGE_REF}" "${REPOSITORY_URI}:${IMAGE_TAG}"
      # - docker push "${REPOSITORY_URI}:${IMAGE_TAG}"

artifacts:
  files: []
```

**What happens when this runs:**
- `dependency-fingerprint.sh` hashes your dependency inputs.
- `verify-image-proof.sh` builds the image and computes a proof hash from layer identifiers.
- If proof doesn’t match, CodeBuild exits with a non-zero status → pipeline stops → deployment never happens.

---

## Step 4: Make Docker builds actually deterministic
The gate helps detect nondeterminism, but to avoid false failures, I also tightened Docker builds.

In practice, I used a Dockerfile pattern that pins versions and prefers lockfiles. Example Dockerfile:

```dockerfile
# syntax=docker/dockerfile:1
ARG BASE_IMAGE=python:3.12-slim
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12

WORKDIR /app

# Copy lockfiles first to maximize reproducibility
COPY package-lock.json ./package-lock.json
COPY requirements.txt ./requirements.txt
COPY poetry.lock ./poetry.lock 2>/dev/null || true

# Install dependencies in a deterministic way.
# Examples assume you use lockfiles for both npm and pip/poetry.
RUN python --version
RUN pip install --no-cache-dir -r requirements.txt

# Application code
COPY . .

CMD ["python", "-c", "print('app running')"]
```

**Key point:** the fingerprint assumes that these inputs drive the layers that get installed. If your Dockerfile fetches dependencies dynamically without lockfiles, the gate will correctly detect a mismatch—sometimes after tests pass.

---

## Debugging when the gate trips (what I learned)
When I saw failures, the fastest way to understand them was:

- Compare the dependency fingerprint values between two runs.
- Inspect the built image’s layer identifiers (`docker image inspect ... RootFS.Layers`).
- Check for hidden sources of variability:
  - unpinned apt/apk package versions
  - package managers hitting indexes that change resolution without lockfiles
  - build args that silently change between pipeline runs

This approach turned mystery nondeterminism into measurable evidence.

---

## Conclusion
I implemented a deterministic CI/CD gate for AWS CodeBuild that blocks deployments when a built Docker image’s content-addressed layer identifiers don’t match a fingerprint computed from dependency inputs. By combining input hashing with manifest-derived “proof,” I traded occasional production Heisenbugs for a hard, reproducible validation step—so artifact integrity becomes something the pipeline can enforce, not something we hope is true.

A Deterministic Ci/Cd Gate For Aws Codebuild That Blocks “Heisenbugs” Via Content-Addressed Docker Layers

## The problem I ran into: “same PR” but different infrastructure drift

I was building a platform engineering workflow for ephemeral **review apps** (short-lived app environments spun up per pull request). The goal was simple: every PR gets a unique namespace and gets destroyed when the PR closes.

But after a few days, I noticed something weird in production-adjacent test environments:

- the same PR URL sometimes pointed to a different deployment
- side effects (config maps, feature flags, caches) didn’t line up deterministically
- debugging was maddening because “refreshing” didn’t mean “same world”

The cause ended up being subtle: we were generating names from mutable signals (like timestamps or local counters). Even if the PR stayed the same, the app identity could drift.

So I wanted a deterministic way to ensure **the ephemeral app identity is derived only from immutable inputs**—specifically the **Git commit**.

---

## The niche solution: namespace + Helm release names from a Git commit digest

Kubernetes namespaces and Helm release names can be based on strings, but Kubernetes has naming constraints (DNS labels). Also, Helm release names can’t be arbitrarily long.

So I implemented a tiny identity scheme:

- Compute a **SHA-256 digest** from the Git commit SHA.
- Convert it to a DNS-label-safe value.
- Use it to form:
  - namespace: `pr-<digest>`
  - Helm release: `rev-<digest>`

That means:
- the same commit always maps to the same namespace/release
- a different commit maps to a different namespace/release
- teardown becomes deterministic and idempotent

---

## What happens when the pipeline runs (end-to-end)

When CI runs for a PR:

1. CI exports `GIT_COMMIT_SHA` (the immutable commit identifier).
2. The pipeline computes `DIGEST` from it.
3. It uses that `DIGEST` to render Kubernetes manifests and run Helm upgrades.
4. The “destroy” job uses the same digest to delete the exact namespace.

No drift, no guessing.

---

## Step 1: Compute a DNS-safe digest (with code)

I used Python in CI because it’s portable and explicit about what it does.

```python
#!/usr/bin/env python3
import hashlib
import re
import sys

def dns_label_safe(s: str, max_len: int = 20) -> str:
    """
    Convert any string into a lowercase DNS-label-safe token:
    - lowercase letters, digits, and hyphens
    - trim to max_len
    """
    s = s.lower()
    # Replace invalid chars with hyphen
    s = re.sub(r'[^a-z0-9-]', '-', s)
    # Collapse consecutive hyphens
    s = re.sub(r'-{2,}', '-', s).strip('-')
    if len(s) > max_len:
        s = s[:max_len]
    # Must start and end with alphanumeric (DNS label rule)
    s = re.sub(r'^[^a-z0-9]+', '', s)
    s = re.sub(r'[^a-z0-9]+$', '', s)
    return s or "x"

def commit_digest(commit_sha: str) -> str:
    """
    Deterministically map a commit SHA to a short token.
    """
    sha_bytes = commit_sha.encode("utf-8")
    digest = hashlib.sha256(sha_bytes).hexdigest()  # 64 hex chars
    # Use first N chars as stable identity. 20 hex chars => 20 length.
    token = digest[:20]
    return dns_label_safe(token, max_len=20)

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: digest.py <GIT_COMMIT_SHA>", file=sys.stderr)
        sys.exit(2)

    commit = sys.argv[1].strip()
    token = commit_digest(commit)
    print(token)
```

### Why this works
- SHA-256 output is deterministic for the same input.
- Taking the first 20 hex chars gives a short, stable token.
- DNS label safety ensures Kubernetes accepts the names.

---

## Step 2: Use the digest in Kubernetes namespace and Helm release names

Kubernetes **namespace names** are DNS labels: lowercase letters, digits, hyphens; and they must fit length rules.

Helm release names are also restricted; this digest-based approach keeps them short and consistent.

### CI variables (example)

```bash
export GIT_COMMIT_SHA="a3f1c2d8e4b7f0c2a6e5f1f4d3c2b1a090877665"

DIGEST="$(python3 digest.py "$GIT_COMMIT_SHA")"
NAMESPACE="pr-${DIGEST}"
RELEASE="rev-${DIGEST}"
echo "NAMESPACE=$NAMESPACE"
echo "RELEASE=$RELEASE"
```

At this point:
- every run for the same commit produces the same `NAMESPACE` and `RELEASE`
- a teardown can safely delete the correct namespace

---

## Step 3: Deterministic Helm release values wiring

I used a minimal Helm values pattern: the chart reads the commit identity and uses it to label resources and set environment variables.

### `values.yaml` template (generated in CI)

```yaml
app:
  commitSha: "{{ GIT_COMMIT_SHA }}"
  identity: "{{ DIGEST }}"
service:
  name: "api"
image:
  tag: "{{ GIT_COMMIT_SHA }}"
```

In CI, I render these values (example using `envsubst` style; simplest for small teams):

```bash
cat > values.yaml <<EOF
app:
  commitSha: "${GIT_COMMIT_SHA}"
  identity: "${DIGEST}"
service:
  name: "api"
image:
  tag: "${GIT_COMMIT_SHA}"
EOF
```

### Helm command

```bash
helm upgrade --install "$RELEASE" ./charts/myapp \
  --namespace "$NAMESPACE" \
  --create-namespace \
  -f values.yaml
```

This is the part that usually causes drift when names vary; with digest-derived names, it becomes stable.

---

## Step 4: Idempotent “destroy” job (the missing piece)

Drift becomes especially painful at cleanup time. So I made teardown deterministic too.

The destroy pipeline uses **the exact same digest computation** and deletes the namespace.

```bash
export GIT_COMMIT_SHA="$GIT_COMMIT_SHA_FROM_PIPELINE"
DIGEST="$(python3 digest.py "$GIT_COMMIT_SHA")"
NAMESPACE="pr-${DIGEST}"

# Namespace deletion is idempotent-ish: it’s safe to run multiple times.
kubectl delete namespace "$NAMESPACE" --ignore-not-found=true
```

This guarantees:
- the destroy job doesn’t accidentally remove a different PR’s environment
- repeated runs don’t flap unpredictably

---

## Optional hardening I added: label everything with the identity

When debugging, I wanted “find all resources belonging to this review app” without depending on naming alone.

So I added consistent labels:
- `review.identity=<DIGEST>`
- `review.commit=<short-sha>`

In Helm templates, you’d apply these labels to Deployment/Service/Ingress/etc. Example snippet:

```yaml
metadata:
  labels:
    review.identity: "{{ .Values.app.identity }}"
    review.commit: "{{ .Values.app.commitSha | trunc 7 }}"
```

### Why labels matter
Helm and Kubernetes resources can be renamed or have different kinds, but labels give you a single filter for observability and cleanup tooling.

---

## Concrete example: what I observed in logs

Before the digest approach, two CI runs for the same PR could create different namespaces due to a timestamp component. That caused:

- Ingress pointed to a new Service, but old workloads still existed.
- Retrying deploy didn’t “replace”; it created.

After switching to `pr-<sha-digest>`:
- both CI runs reported the same namespace/release
- the Helm upgrade updated the existing Deployment instead of creating a sibling environment
- namespace deletion consistently removed exactly one review app universe

---

## FinOps and platform engineering angle (why this matters beyond correctness)

This identity scheme reduced waste in two ways:

1. **No orphan review environments**  
   Deterministic teardown dramatically lowered “forgotten namespaces” that keep pulling images and running background jobs.

2. **Predictable cost attribution**  
   Once resources are consistently labeled with a commit identity, cost reporting and budget mapping gets less noisy because you don’t get random environment IDs every run.

---

## Closing summary

I implemented deterministic ephemeral review apps by deriving Kubernetes namespace and Helm release names from a SHA-256 digest of the immutable Git commit SHA. This removed identity drift, made Helm upgrades replace the correct resources, and enabled idempotent cleanup by deleting the exact namespace tied to the commit digest—turning chaotic per-PR environments into stable, debuggable, cost-controlled infrastructure.

Deterministic Ephemeral Review Apps By Git Commit Digest In Kubernetes

I ran into a weird CI/CD failure that looked like an “auth problem,” but it wasn’t. The symptoms were consistent: GitHub Actions would authenticate to Kubernetes via OIDC (OpenID Connect, a standard way for one system to prove identity to another), everything would work at first, and then the job would die when a long-running deploy step tried to refresh the token.

The breaking detail: the token refresh logic was accidentally configured with a `ttl=0` (time-to-live), so the “refresh” immediately produced an expired token. That made the failure intermittent—depending on timing, the first request might succeed, but the later one failed.

Here’s the exact pipeline I built to *prevent* that class of failure by:
- forcing a single, early OIDC token exchange,
- validating TTL settings before the job even tries to deploy,
- and using a short, explicit token session for Kubernetes calls.

### The failure mode I observed (and how I reproduced it)

In my setup, the deploy step ran long enough that Kubernetes client calls needed a fresh token. When TTL was mis-set to `0`, the refresh returned an unusable token.

I reproduced it conceptually like this (not Kubernetes-specific, just the idea):

```python
import time

def fake_refresh(ttl_seconds: int):
    now = int(time.time())
    expires_at = now + ttl_seconds
    if ttl_seconds <= 0:
        return {"expires_at": expires_at, "token": "expired-token"}
    return {"expires_at": expires_at, "token": "fresh-token"}

for ttl in [0, 30]:
    result = fake_refresh(ttl)
    now = int(time.time())
    ok = result["expires_at"] > now
    print(f"ttl={ttl} expires_at={result['expires_at']} ok={ok} token={result['token']}")
```

When `ttl=0`, `expires_at` is not in the future, so the token is instantly expired. That’s exactly what was happening in the pipeline: a “refresh” that can never succeed.

### The fix: fail fast on invalid TTL and avoid refresh mid-deploy

Instead of letting refresh happen during the deployment (which makes debugging painful), I enforced two guardrails:

1. **Validate TTL config at the beginning of the job**  
2. **Exchange OIDC → Kubernetes credentials once** and use them for the remainder of the job

Below is a working GitHub Actions workflow that does that.

### Working GitHub Actions workflow (OIDC to Kubernetes with TTL validation)

This example assumes:
- You use GitHub’s OIDC federation with a Kubernetes cluster that trusts the GitHub identity.
- You can authenticate using `aws eks update-kubeconfig` (EKS example), but the TTL validation pattern applies to any OIDC-to-k8s setup.

Create `.github/workflows/deploy.yml`:

```yaml
name: Deploy with OIDC and TTL guard

on:
  workflow_dispatch:
  push:
    branches: ["main"]

permissions:
  id-token: write   # required for OIDC
  contents: read

env:
  # This is the “gotcha” value I accidentally had in my environment before.
  # It MUST be > 0 for any refresh-like logic to make sense.
  DEPLOY_TOKEN_TTL_SECONDS: "300"

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Validate TTL before any Kubernetes auth
        shell: bash
        run: |
          set -euo pipefail
          ttl="${DEPLOY_TOKEN_TTL_SECONDS}"

          if ! [[ "$ttl" =~ ^[0-9]+$ ]]; then
            echo "DEPLOY_TOKEN_TTL_SECONDS must be a non-negative integer. Got: $ttl"
            exit 1
          fi

          if [ "$ttl" -le 0 ]; then
            echo "ERROR: DEPLOY_TOKEN_TTL_SECONDS must be > 0. Got: $ttl"
            exit 1
          fi

          echo "TTL looks good: $ttl seconds"

      - name: Authenticate to cloud (example: AWS EKS)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc-deploy-role
          aws-region: us-east-1

      - name: Update kubeconfig using assumed identity
        shell: bash
        run: |
          set -euo pipefail
          aws eks update-kubeconfig --name my-eks-cluster --region us-east-1

      - name: Deploy
        shell: bash
        run: |
          set -euo pipefail
          # Example manifest apply
          kubectl apply -f k8s/deployment.yaml
          kubectl rollout status deployment/my-app --timeout=120s
```

#### What each block does (and why it matters)

- `permissions: id-token: write`  
  Enables GitHub Actions OIDC token issuance. Without it, OIDC auth can’t happen.

- `Validate TTL before any Kubernetes auth`  
  This is the key safety belt. If `DEPLOY_TOKEN_TTL_SECONDS` is `0` (or negative), the job exits immediately—so you never reach the confusing “expired token during deploy” state.

- `configure-aws-credentials`  
  Exchanges the GitHub OIDC identity for an AWS role session (in an EKS flow). This is the “one-time exchange” concept.

- `aws eks update-kubeconfig`  
  Writes cluster access config so `kubectl` can talk to the API server using the assumed identity.

- `kubectl apply` + `kubectl rollout status`  
  Performs the actual deployment and waits for rollout completion.

### A tiny TTL unit test that saved me later

I also added a little script to make TTL validation consistent across repos. It’s simple, but it prevents copy/paste mistakes.

Create `scripts/validate-ttl.py`:

```python
import os
import sys

def validate_ttl(ttl_str: str) -> int:
    if not ttl_str.isdigit():
        raise ValueError("TTL must be a non-negative integer string")
    ttl = int(ttl_str)
    if ttl <= 0:
        raise ValueError("TTL must be > 0")
    return ttl

if __name__ == "__main__":
    ttl_str = os.environ.get("DEPLOY_TOKEN_TTL_SECONDS", "")
    try:
        ttl = validate_ttl(ttl_str)
    except Exception as e:
        print(f"Invalid DEPLOY_TOKEN_TTL_SECONDS='{ttl_str}': {e}")
        sys.exit(1)
    print(f"TTL OK: {ttl} seconds")
```

Then call it in the workflow instead of the bash check:

```yaml
      - name: Validate TTL before any Kubernetes auth (python)
        shell: bash
        run: |
          set -euo pipefail
          python3 scripts/validate-ttl.py
```

### What I learned building this

The biggest surprise was that “token refresh” bugs can hide inside configuration values like `ttl=0`. When I moved validation to the very beginning of the job and treated OIDC-to-cluster credential exchange as a one-time early step, the deploy failures stopped being intermittent and became deterministic. In practice, that turns a frustrating auth mystery into a clear, fast failure with an obvious root cause.