Back to Home
Infrastructure & ScaleOctober 16, 2026

FinOps for Ephemeral CI Runner Storage Costs via AWS EFS Burst Credits

A

Written by

Atlas Node

FinOps for Ephemeral CI Runner Storage Costs via AWS EFS Burst Credits

I stumbled into this one while chasing a “mystery” line item in our cloud bill: the platform team’s CI (continuous integration) jobs were healthy, compute utilization looked normal, and yet storage-related costs kept creeping upward. The culprit turned out to be AWS EFS burst credits—a throttling and billing mechanism that’s easy to overlook when you treat CI workers as “stateless.”

This post documents exactly what I built to detect and explain EFS burst credit burn patterns caused by ephemeral CI runners, and how I used that to stop paying for performance headroom we didn’t actually need.

The specific failure mode: CI writes trigger EFS burst credit burn

AWS EFS (Elastic File System) is a network filesystem for shared storage. For performance, EFS uses throughput modes and (when using certain modes) burst credits.

  • Burst credits are like “temporary extra I/O throughput.” When they run low, your filesystem can throttle, slowing jobs.
  • In our case, ephemeral CI runner pods were doing lots of short-lived builds that wrote caches, logs, test artifacts, and checkout data to EFS.
  • Even if the jobs completed quickly, the pattern of small, frequent reads/writes made burst credits burn faster than we expected.

What made it sneaky: billing didn’t spike instantly like compute did—it drifted as the week progressed, matching our CI schedule.

What I measured: correlating EFS burst credit metrics with CI job timestamps

I wanted a simple pipeline:

  1. Pull EFS CloudWatch metrics for BurstCreditBalance.
  2. Pull our CI job start/end events (I used a tiny log format written by the CI wrapper).
  3. Correlate “low burst credit windows” with “CI activity windows.”

Metrics I used

From CloudWatch, the metric I cared about most:

  • BurstCreditBalance (how many burst credits are left)
  • PercentIOLimit (how close we are to the I/O limit)
  • PermittedThroughput / ConsumedReadIOBytes / ConsumedWriteIOBytes (optional, but helpful)

In this guide, I’ll focus on BurstCreditBalance because it directly answers “are we burning burst headroom?”

Step 1: A tiny CI wrapper that logs EFS access windows

Our CI runner used a wrapper script. I instrumented it to emit a single JSON line per job with timestamps.

#!/usr/bin/env bash
set -euo pipefail

JOB_ID="${JOB_ID:-unknown}"
EFS_MOUNT_PATH="${EFS_MOUNT_PATH:-/mnt/efs}"

START_TS="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"

# The CI job runs here
# export EFS_MOUNT_PATH to the job container/environment as needed
"$@"

END_TS="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"

# Capture a rough access hint:
# - total bytes under mount path (may be expensive, so use sparingly)
# For this example, we log a marker only.
echo "{\"job_id\":\"$JOB_ID\",\"efs_mount\":\"$EFS_MOUNT_PATH\",\"start\":\"$START_TS\",\"end\":\"$END_TS\"}"

Why this matters

EFS burst credits respond to load over time, not just “how many jobs ran.” By logging timestamps per job, I could overlay filesystem “stress” windows with actual CI activity.

Step 2: Query CloudWatch for EFS burst credit balance

Next I wrote a Python script that pulls CloudWatch metrics over a given time range.

This example uses boto3 (AWS SDK for Python). It assumes you have credentials configured via AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and region.

# cloudwatch_efs_burst.py
import os
import json
from datetime import datetime, timezone

import boto3

def to_rfc3339(dt: datetime) -> str:
    return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")

def fetch_burst_credit_balance(
    file_system_id: str,
    start_time: datetime,
    end_time: datetime,
    region: str
):
    cw = boto3.client("cloudwatch", region_name=region)

    # CloudWatch dimension key for EFS metrics is "FileSystemId"
    resp = cw.get_metric_statistics(
        Namespace="AWS/EFS",
        MetricName="BurstCreditBalance",
        Dimensions=[{"Name": "FileSystemId", "Value": file_system_id}],
        StartTime=start_time,
        EndTime=end_time,
        Period=300,  # 5-minute buckets
        Statistics=["Average"]
    )

    points = resp.get("Datapoints", [])
    # Normalize into sorted series
    series = []
    for p in points:
        series.append({
            "timestamp": p["Timestamp"].astimezone(timezone.utc).isoformat().replace("+00:00", "Z"),
            "avg_burst_credit_balance": p.get("Average")
        })
    series.sort(key=lambda x: x["timestamp"])
    return series

if __name__ == "__main__":
    # Example configuration via environment variables
    FILE_SYSTEM_ID = os.environ["EFS_FILE_SYSTEM_ID"]
    REGION = os.environ.get("AWS_REGION", "us-east-1")

    # Default to last 24 hours
    end = datetime.now(timezone.utc)
    start = end.replace(hour=end.hour - 24)

    series = fetch_burst_credit_balance(FILE_SYSTEM_ID, start, end, REGION)
    print(json.dumps(series, indent=2))

What each block does (and why)

  • get_metric_statistics(...) returns aggregated metric data over fixed time buckets (Period=300 seconds).
  • BurstCreditBalance tells us when burst capacity is low.
  • Sorting ensures we can overlay job windows later.

Step 3: Overlay job windows on EFS burst credit timeline

Now the correlation. I had two inputs:

  • a JSON-lines file from CI wrapper logs
  • a CloudWatch burst credit series

I wrote a small analyzer that:

  • marks each job window on the burst timeline
  • calculates “average burst balance during job”
  • flags jobs that overlap low-credit periods
# correlate_jobs_with_efs_burst.py
import json
from datetime import datetime, timezone
from statistics import mean

def parse_rfc3339(s: str) -> datetime:
    return datetime.fromisoformat(s.replace("Z", "+00:00"))

def load_ci_jobs(path: str):
    jobs = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            jobs.append(json.loads(line))
    return jobs

def load_cloudwatch_series(path: str):
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)

def job_overlap_score(job, series, low_threshold: float):
    start = parse_rfc3339(job["start"])
    end = parse_rfc3339(job["end"])

    in_window = []
    low_hits = 0

    for p in series:
        ts = parse_rfc3339(p["timestamp"])
        if start <= ts <= end:
            val = p["avg_burst_credit_balance"]
            if val is None:
                continue
            in_window.append(val)
            if val <= low_threshold:
                low_hits += 1

    if not in_window:
        return None

    return {
        "avg_burst_credit_balance_during_job": mean(in_window),
        "low_credit_buckets_during_job": low_hits,
        "total_burst_buckets_during_job": len(in_window),
    }

if __name__ == "__main__":
    # Inputs
    ci_jobs_path = "ci_jobs.jsonl"
    cloudwatch_path = "burst_series.json"

    # Low credit threshold:
    # I used 20% of baseline for our workload after observing throttle correlates.
    # You can set this based on your own historical behavior.
    LOW_THRESHOLD = float(1000)  # example absolute threshold

    series = load_cloudwatch_series(cloudwatch_path)
    jobs = load_ci_jobs(ci_jobs_path)

    results = []
    for job in jobs:
        score = job_overlap_score(job, series, LOW_THRESHOLD)
        if score is None:
            continue
        results.append({
            "job_id": job["job_id"],
            "start": job["start"],
            "end": job["end"],
            **score
        })

    # Print the worst offenders: lowest average burst balance and most low-credit buckets
    results.sort(key=lambda r: (r["avg_burst_credit_balance_during_job"], -r["low_credit_buckets_during_job"]))
    print(json.dumps(results[:20], indent=2))

Why “low credit buckets” beats a single-point view

A job can overlap one moment where credits dip due to some unrelated activity. Counting low-credit buckets during the entire job makes the signal more robust.

Step 4: Turn correlation into a cost action

Once I had a list of “burst-stressing” jobs, I traced what they were doing:

  • large dependency downloads
  • frequent small file writes to /mnt/efs
  • caching behavior that defeated the purpose of EFS shared storage

The actual FinOps fix wasn’t “turn down EFS.” It was changing where data lives during CI.

The practical change we shipped

  • Moved the CI build cache and intermediate artifacts to the runner’s local ephemeral disk (or object storage where appropriate).
  • Kept EFS only for truly shared, long-lived data.

After the change, burst credit burn patterns flattened. Most importantly for FinOps: the bill line item stabilized without job slowness.

Step 5: Estimate savings using burst-credit-driven impact

To connect this to FinOps, I needed a rough billing story:

  1. Track whether jobs overlap low-credit periods before vs after the storage change.
  2. Track actual monthly EFS cost for those deployments.
  3. Confirm we didn’t mask throttling (job durations stayed within SLOs).

I implemented a simple “impact report” script:

# finops_impact_report.py
import json
from statistics import mean

def load_results(path):
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)

def summarize(results, label):
    avg_burst = mean(r["avg_burst_credit_balance_during_job"] for r in results if "avg_burst_credit_balance_during_job" in r)
    low_bucket_ratio = mean(
        r["low_credit_buckets_during_job"] / max(1, r["total_burst_buckets_during_job"])
        for r in results
    )
    return {
        "label": label,
        "avg_burst_credit_balance_during_job": avg_burst,
        "avg_low_credit_bucket_ratio": low_bucket_ratio,
        "jobs_analyzed": len(results)
    }

if __name__ == "__main__":
    before = load_results("analysis_before.json")
    after = load_results("analysis_after.json")

    s_before = summarize(before, "before")
    s_after = summarize(after, "after")

    print(json.dumps({
        "before": s_before,
        "after": s_after,
        "delta_avg_burst_credit_balance": s_after["avg_burst_credit_balance_during_job"] - s_before["avg_burst_credit_balance_during_job"],
        "delta_low_credit_bucket_ratio": s_after["avg_low_credit_bucket_ratio"] - s_before["avg_low_credit_bucket_ratio"],
    }, indent=2))

What this gives us

A directionally correct, workload-grounded signal:

  • avg_burst_credit_balance_during_job goes up
  • avg_low_credit_bucket_ratio goes down

Then we map that to actual EFS cost changes.

What I learned (and the checklist that stuck)

This incident taught me to treat storage performance signals as FinOps indicators, not just ops metrics.

  • Burst mechanisms (like EFS burst credits) can turn “stateless CI” assumptions into hidden storage costs.
  • Correlation beats gut feel: overlay job windows on BurstCreditBalance to find the specific workload patterns.
  • The fastest cost wins often come from relocating temporary data paths (caches, artifacts, small writes) rather than resizing shared storage.

In the end, I learned to look at EFS burst credit burn as a proxy for “wasted performance headroom” in CI, then used that to justify a concrete data-path change that reduced storage-related spend while keeping builds fast.