Where ECS deployment time leaked — why Early Success Criteria was not a one-setting fix

In our previous article, we cut Docker builds to the 30-second range with Blacksmith caching and wrote that database migration and ECS stabilization remained the bottlenecks. A five-minute production deployment still spent three and a half minutes starting new ECS tasks and cleaning up the old ones. The build had little room left, so we examined how ECS decided that a deployment was complete.

AWS had just introduced Early Success Criteria for rolling deployments.¹ ECS can mark a deployment successful when a configured proportion of new tasks is healthy and can move old-task cleanup outside the deployment lifecycle. We measured whether the same change would help our pipeline, starting in development. One setting was not enough.

ECS consumed three and a half minutes of a four-minute deployment

We measured the api and web services for one client. Both use Fargate rolling deployments. Their ALB target groups use a five-second health-check interval, two successful checks, and a five-second deregistration delay. After pushing an image, the GitHub Actions workflow updates the service with amazon-ecs-deploy-task-definition. A custom Monitor step polls describe-services and posts progress to Slack.

The ECS events from one pre-change production api deployment, a two-task service, looked like this.

The new tasks were ready for traffic after about one minute. The remaining two and a half minutes went to stopping old tasks and waiting for ECS to call the service stable. The one-task development services had the same shape. Across the previous six deployments, the Monitor step took 180–198 seconds for api and 165–200 seconds for web. Baselines measured again that day were 197 and 189 seconds.

What Early Success Criteria actually moves forward

By default, ECS completes a rolling deployment only after four conditions are met: the target revision reaches 100% of desired count and all tasks are healthy; no deployment circuit breaker or CloudWatch alarm triggers a rollback; any alarm bake time elapses; and the source revision tasks are cleaned up.² Early Success Criteria changes the target revision's required healthy percentage and the timing of source revision cleanup.

• healthyPercent: ECS completes the deployment when this percentage of target-revision tasks is healthy. The required task count is rounded up. ECS launches any remaining tasks later through regular service scaling. The value must be between the service's minimumHealthyPercent and 100.
• sourceServiceRevisionCleanup: With BLOCKING, ECS cleans up the old tasks before declaring success. With DEFERRED, ECS declares success first and attempts the cleanup asynchronously for up to two weeks.

For the one-task services and the conservative production setup, we kept healthyPercent at 100 and used DEFERRED only to separate old-task cleanup. We measured a 50% setup separately on the two-task services.

The three changes

Enabling the setting did not shorten CI

We set healthyPercent to 100 and cleanup to DEFERRED on the development api and web services, then deployed each service twice. Our local AWS CLI 2.35.7 did not yet expose the earlySuccessCriteria parameter, so we called update_service with boto3 1.43.89. Because the request replaces the complete deploymentConfiguration, we read the current values and merged them to preserve the circuit breaker configuration.

cur = ecs.describe_services(cluster=cluster, services=[svc])["services"][0]["deploymentConfiguration"]
ecs.update_service(
    cluster=cluster, service=svc,
    deploymentConfiguration={
        "deploymentCircuitBreaker": cur["deploymentCircuitBreaker"],
        "maximumPercent": cur["maximumPercent"],
        "minimumHealthyPercent": cur["minimumHealthyPercent"],
        "strategy": "ROLLING",
        "bakeTimeInMinutes": cur.get("bakeTimeInMinutes", 0),
        "earlySuccessCriteria": {
            "enable": True,
            "healthyPercent": 100,
            "sourceServiceRevisionCleanup": "DEFERRED",
        },
    },
)

The CI wait barely moved. Monitor took 181 and 181 seconds for api, and 183 and 150 seconds for web, which was hard to distinguish from the baseline. The service deployment API showed a different result. In describe-service-deployments, api fell from 139 seconds to 111 and 104 seconds; web fell from 106 seconds to 101 and 84 seconds. Each record included the status reason "Service deployment met early success criteria." ECS finished early, but the workflow could not see that state.

The workflow watched the wrong signal. Monitor waited for deployments[].rolloutState from describe-services to become COMPLETED. That field changed only after ECS cleaned up the old tasks. Early Success Criteria advances the status of the service deployment resource; the legacy deployments array follows later. For the same deployment, the service deployment finishedAt was 15:56:19, while rolloutState became COMPLETED at 15:57:16, a 57-second gap. The developer guide also directs operators to DescribeServiceDeployments and to the SUCCESSFUL filter in ListServiceDeployments.²

We changed Monitor to select only service deployments created near the start of the current run. It now exits on SUCCESSFUL, fails on rollback or stop states, and keeps rolloutState as a fallback.

DEPLOY_STARTED_ISO=$(date -u -d "@$((MONITOR_START - 120))" +%Y-%m-%dT%H:%M:%SZ)
while true; do
  SD_STATUS=$(aws ecs list-service-deployments \
    --cluster "$ECS_CLUSTER" --service "$ECS_SERVICE" \
    --created-at "after=$DEPLOY_STARTED_ISO" \
    --query 'serviceDeployments[0].status' --output text 2>/dev/null || echo "")
  [ "$SD_STATUS" = "SUCCESSFUL" ] && exit 0
  case "$SD_STATUS" in
    ROLLBACK_*|STOPPED|STOP_REQUESTED) exit 1 ;;
  esac
  # Keep the existing rolloutState polling as a fallback.
  sleep 5
done

The second trap: a call without permission failed silently

The third deployment, with the revised workflow, still took 191 seconds for api and 168 seconds for web. The logs showed no error. The deployment role lacked the required IAM permission, and 2>/dev/null hid the error. The same command returned SUCCESSFUL locally, so the command was correct and the credentials were the only difference.

The GitHub OIDC deployment role's inline policy contained four ECS actions: DescribeServices, DescribeTaskDefinition, RegisterTaskDefinition, and UpdateService. A tightly scoped least-privilege policy blocked the new API. We allowed ListServiceDeployments on the service ARN and DescribeServiceDeployments on the service-deployment/<cluster>/<service>/* ARN. Terraform manages both the role and the ECS service settings, so we moved the manual change into Terraform code to prevent the next apply from reverting it.

Results after fixing all three points

After adding the permission, we ran a fourth deployment. The key before-and-after numbers make the result clear.

The run-by-run table shows why the completion signal and permission were both necessary.

The complete workflow fell from 275 to 189 seconds for api and from 233 to 159 seconds for web, reductions of 31% and 32%. The service deployments themselves completed in 117 and 98 seconds, both through early success. Roughly 100 of the remaining 120 seconds went to Fargate task launch, application startup, and health checks. Early Success Criteria cannot remove that time.

The production api and web services each run two tasks. We began conservatively with healthyPercent 100, DEFERRED, and minimumHealthyPercent 100. This setup separates only the old-task cleanup wait.

Both conservative runs completed through early success: 141 and 149 seconds for api, and 123 and 133 seconds for web. Immediately after deployment, both services showed running 2 / desired 2; after deferred cleanup, no source-revision tasks remained. Because healthyPercent stayed at 100, the only removed interval was source-task cleanup. That alone cut Monitor by about 70 seconds for api and 55 seconds for web relative to the baseline median. The full workflow moved from roughly 300 seconds to 227–246 seconds for api, and from roughly 270 seconds to 196–205 seconds for web.

We applied the same sequence to the remaining services that day. The one-task production and development counsel services and the development admin service used healthyPercent 100 with DEFERRED. Monitor fell from 164–181 to 106 seconds for production counsel, from 164–197 to 127 seconds for development counsel, and from 164–180 to 105 seconds for development admin. Removing only cleanup wait cut 30–40% from these one-task services.

Values by task count

The applied values and measured results are easier to compare in one table.

After measuring the conservative setup twice, we changed minimumHealthyPercent and healthyPercent to 50 for one more deployment. Monitor reached 135 seconds for api and 112 seconds for web, another 20–40 seconds faster than the conservative setup. The service deployments completed early in 119 and 104 seconds.

Adoption sequence and verification

We now use this order to avoid repeating the same mistakes.

1. Check the SDK. If aws ecs update-service --generate-cli-skeleton does not contain earlySuccessCriteria, update the CLI or use a current boto3 release or the console.

2. Configure the service. Read and merge the existing deploymentConfiguration, then add earlySuccessCriteria. A healthyPercent below minimumHealthyPercent returns InvalidParameterException.²

3. Change the workflow. Observe list-service-deployments instead of rolloutState from describe-services or wait services-stable.

4. Update Terraform and IAM. Allow ecs:ListServiceDeployments on the service ARN and ecs:DescribeServiceDeployments on the service-deployment/... ARN, then add the permissions to Terraform. If the provider does not expose earlySuccessCriteria, reapply the setting with a separate script after Terraform runs.

5. Verify the service deployment first. Look for "met early success criteria" in describe-service-deployments.statusReason. If it appears but CI time does not fall, the workflow signal or IAM permission is the likely cause.

Deployment pipelines can spend more time deciding that work is complete than building it. Through our Cloud & Infrastructure service, 801 PLANET measures these waits and removes bottlenecks across the delivery path.

References

We checked the ECS announcement and documentation on September 8, 2026. Development measurements came from same-day GitHub Actions runs, ECS service events, and describe-service-deployments responses. Production measurements came from three same-day deployments of the main branch: two conservative runs and one 50% run.

Terraform implementation note

At the time of implementation, Terraform AWS Provider 5.100 did not expose earlySuccessCriteria on aws_ecs_service. We kept minimumHealthyPercent and IAM permissions in Terraform and used a repository boto3 script to reapply the early-success setting after any apply that changed deployment_* fields.