Why we added Blacksmith to GitHub Actions: 1,527 jobs, performance, and cost

We do not force the same CI stack into client and internal environments. Many client pipelines begin in GitLab or CodeCommit and follow CodePipeline → CodeBuild → ECR/ECS or CodePipeline → CodeBuild → CodeDeploy/EC2. Keeping source, IAM, network access, and deployment history inside AWS is often the point.

Applications built by 801 PLANET start elsewhere: GitHub → GitHub Actions → Blacksmith → ECR/ECS. GitHub Actions remains the workflow control plane; Blacksmith executes the jobs. AWS access comes from short-lived GitHub OIDC credentials rather than long-lived access keys.¹

This is not a recommendation to standardize every environment on Blacksmith. It records what we observed in a real account, how much support the public performance claims have, and where CodeBuild or a self-managed BuildKit service becomes the better choice. The supplied Findy Tools review is members-only beyond its title and public author information, so we did not use it as quantitative evidence.²

Both paths deploy to AWS, but their control planes differ. In the upper lane, CodePipeline accepts changes from GitLab or CodeCommit and invokes CodeBuild. The container branch pushes an image to ECR for ECS to pull, while the instance branch hands the release to CodeDeploy and EC2.

In the lower lane, GitHub Actions controls the workflow and Blacksmith executes the BuildKit job. Docker layers are reused from cache close to the runner. GitHub OIDC assumes a short-lived IAM role, the image is pushed to ECR, and ECS pulls that image. The bars on the right show the compressed ECR sizes of four images checked under the same commit tag.

The two pipelines start from different constraints

Blacksmith does not replace GitHub Actions as a CI product. It keeps GitHub workflows and actions, then changes the runs-on target to a Blacksmith runner. The official quickstart also says that Blacksmith is limited to GitHub organizations and is unavailable for personal repositories.³

runs-on: blacksmith-2vcpu-ubuntu-2404

The small migration surface is valuable. It disappears when GitLab or CodeCommit is the source of truth. In those environments, CodeBuild or an existing GitLab Runner usually leaves a simpler configuration and audit path.

What six months of production data tells us

From January 23 to July 23, 2026, the unfiltered organization view in Blacksmith CI Analytics recorded 1,527 jobs. Median job duration was 255 seconds, or 4m 15s. The p90 was 5m 40s, p95 6m 46s, p99 12m 45s, and maximum 27m 22s.

This screen cannot prove that Blacksmith made us a specific multiple faster. We did not preserve an equivalent pre-migration baseline that reran the same commits on GitHub-hosted runners. The 13.75% failure rate is not a Blacksmith incident rate, either. It combines failed tests, application errors, workflow mistakes, and deliberate failures across the organization. It is an operational signal for finding the repositories and jobs to fix.

Over the same selected period, Docker Analytics recorded 1,487 builds, a 9.28% failure rate, and 14.8 GB of average cache use. Most builds clustered under one minute, with a long tail extending to roughly seven minutes.

Blacksmith's dashboard breaks down job duration, failure rate, caches, and repository cost in one place.⁴ It makes long p95 and p99 jobs and repositories with expanding caches easier to find than workflow logs alone. For us, observability was part of the managed-runner decision rather than a cosmetic add-on.

Three levels of evidence behind the speed claims

"Up to 40x faster Docker builds" is not a guarantee for every workload. Blacksmith's documentation describes it as a 2x to 40x improvement reported by customers.⁵ Separating the test conditions and source incentives produces a more useful picture.

The shared mechanism is straightforward. When an ephemeral runner disappears, BuildKit's workspace and the dependency caches created by RUN --mount=type=cache disappear with it. Blacksmith persists repository-scoped Docker layers and makes them available to later runners. A self-managed version requires a remote buildkitd, persistent storage, garbage collection, concurrency control, isolation, and recovery.

The gain can be large when cache reuse and single-core CPU performance dominate. It may be small or negative when a workflow spends its time loading a large image back into the runner with --load, transferring to a remote registry, or waiting on integration tests. Compare p50, p95, cache hit rate, and cost per successful job on the same commits, not the largest marketing multiple.

Final image size belongs in the runner comparison

The four final images in ECR ranged from 79.9 to 330.5 MB compressed. The files and production dependencies retained in each runner stage explain most of that spread.

On July 23, 2026, we queried four internal ECR repositories in the Seoul region with read-only access. We inspected DescribeImages.imageSizeInBytes and the OCI manifest layer list for the same commit tag in every repository. Account IDs and repository names remain private. These are compressed registry sizes, not the uncompressed number shown by local docker images or the disk footprint of a running container.⁹

All four Dockerfiles use multi-stage builds. They copy the lockfile and workspace package.json files before source code, then run pnpm install. That ordering lets an application-only change reuse the dependency layer. The root .dockerignore also removes node_modules, existing dist and .next output, infrastructure code, documentation, and .github from the build context. Docker recommends the same pattern: put expensive, stable steps early and keep the context small.¹⁰

The final runner stage determines the result. The web image copies Next.js standalone and static output and weighs 81.3 MB. The API keeps production dependencies, compiled output, and Prisma files and reaches 245.2 MB. The single-JS image adds only dist/main.js to the Node.js runtime and weighs 79.9 MB. The orchestrator retains production dependencies and outputs from its Temporal and database workspaces and reaches 330.5 MB. Multi-stage builds exclude builder tooling in every case, yet the runtime boundary still creates a spread of more than 4x.¹¹

Our workflows use docker/build-push-action with push: true, sending the result directly to ECR instead of loading a large image back into the runner with --load. A dependency change can still invalidate and transfer a layer larger than 100 MB. A nearby Blacksmith cache cannot repair poor application-layer boundaries or shrink a heavy runtime image.

Runner time alone is not enough for the pilot. Alongside cold- and warm-cache duration for the same commits, record compressed ECR size, the largest changed layer, push time, and time until the new ECS task becomes ready. A fast cache and a small deployable image answer different questions.

Separate unit-rate math from the invoice

As of July 23, 2026, a standard GitHub-hosted Linux 2-core runner is $0.006 per minute, while a Blacksmith Ubuntu x64 2 vCPU runner is $0.004 per minute.¹²¹³ The AWS CodeBuild pricing example uses $0.005 per minute for general1.small.¹⁴

If the same 10,000 billed minutes were charged before plan allowances, cache, and logging, the compute-only comparison would look like this:

This isolates rate cards; it is not a TCO ranking. CPU generation, memory, minute rounding, actual runtime, network, and cache behavior differ. CodeBuild also offers a Docker image server with a shared layer cache, billed while the server runs and at a lower cache-at-rest rate during cooldown.¹⁴ At Blacksmith's current cache price, the observed 14.8 GB average is a simple $7.40 per month equivalent.

Our invoices were smaller. From March through July 2026, the five charges were $4.31, $6.26, $22.13, $4.65, and $9.97. They total $47.32, or $9.46 per month on average.

The invoice window covers March through July, while Analytics covers January through July. The periods do not match. Median duration is not mean duration, so we also did not estimate total usage as 1,527 × 255 seconds. The defensible conclusion is limited: this was our actual spend at the current scale.

Where CodeBuild and Blacksmith diverge

CodeBuild projects can attach a VPC ID, subnets, and security groups to reach private RDS, ElastiCache, internal ECS, and private artifact repositories.¹⁵ Blacksmith offers a dedicated static egress IP, but the current public rate is $100 per IP each month.¹⁶ When private network access is mandatory, network design and security review should precede runner-minute math.

Blacksmith's security page states SOC 2 Type 2 and GDPR compliance, ISO 27001 data centers, and a fresh Firecracker microVM per job. Its GitHub App requests permissions needed to edit workflows and mint JIT runner tokens.¹⁷ Those claims are inputs to a vendor review, not a replacement for one.

Why Blacksmith fit our own stack

Our code review and source flow already live in GitHub, and we frequently build Docker images for ECR and ECS. For a small team, buying a managed execution layer was more sensible than operating Actions Runner Controller or an EC2 runner pool, including scaling, patching, disks, and BuildKit garbage collection. GitHub OIDC opens AWS access briefly, with role sub conditions tied to the repository and branch.

That choice does not put Blacksmith into every client pipeline. We keep CodePipeline and CodeBuild where GitLab, CodeCommit, private VPC access, and AWS service roles define the environment. The opposite design — bringing source and build fully inside a private platform — appears in our NKS modernization pipeline with GitLab and Jenkins.

Blacksmith fit our stack for three reasons: it retained the GitHub Actions files, removed Docker cache operations, and put jobs, caches, failures, and cost in one dashboard. Six months of data shows that this configuration runs in production. It does not prove a universal speed multiplier.

Run a two-week pilot before adopting it

1. Select two Docker-heavy workflows and record baseline p50, p95, queue time, success rate, and monthly cost.
2. Rerun the same commits on GitHub-hosted and Blacksmith runners, separating cold-cache and warm-cache results.
3. Set a Docker cache ceiling with max-cache-size-mb, then monitor cache hit rate and monthly GB together. Record compressed ECR size and the largest changed layer as well.⁵
4. Review the AWS OIDC aud and sub, GitHub App permissions, pinned third-party actions, and static-IP requirements.
5. Do not assume a free allowance is a hard stop. Confirm budgets and shutdown behavior. A 2026 public account described receiving a $1,081 invoice after usage continued beyond the free allowance.¹⁸

Changing one workflow line is easy. The decision begins afterward. Measure whether reviewers wait less, and whether the cost of getting one successful change through CI without runner operations falls.

Frequently asked questions

What is Blacksmith

Blacksmith is a managed runner service for GitHub Actions. GitHub retains workflow control; Blacksmith supplies isolated execution, co-located dependency caching, persistent Docker layer caching, and CI Analytics. Teams adopt it by changing the runs-on label in workflows inside a GitHub organization.

Can Blacksmith replace AWS CodeBuild

Not universally. Blacksmith fits when GitHub Actions is the control plane and the team wants to remove runner operations. CodeBuild is usually simpler when GitLab or CodeCommit, CodePipeline, private VPC access, and AWS service roles define the system. They are execution layers attached to different control planes.

Can Blacksmith push images to ECR and deploy ECS

Yes. A GitHub Actions workflow assumes a narrowly scoped AWS IAM role through OIDC, then receives permissions to push to ECR and update ECS. Avoid long-lived AWS access keys in GitHub Secrets, and constrain the role's sub condition to approved repositories and branches.

Can a team reproduce Blacksmith's speed itself

The core mechanism is reproducible. A remote BuildKit daemon with persistent storage and nearby cache lets ephemeral runners reuse build state. The team then owns garbage collection, concurrent builds, isolation, patching, recovery, and cost visibility. Blacksmith's value is less the cache idea than operating that execution layer.

If you need to measure a GitHub CI bottleneck or draw the boundary between CodeBuild, managed runners, and self-managed BuildKit, our Cloud & Infrastructure work starts with the current workflow and AWS network. For the AWS-internal traffic path around ECR, continue with AWS's invisible costs.

References

We rechecked product behavior and pricing on July 23, 2026. The 801 PLANET figures come from the attached organization-wide dashboards and payment history. Repository-level source data and a pre-migration A/B baseline are outside this article.