processing-s3-uploads-with-step-functions

Step Functions Workflow: Route S3 Uploads to Lambda or Fargate

Overview

This skill deploys an event-driven workflow using AWS CLI. When a file is uploaded to an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine checks the file size and routes processing to either a Lambda function (files ≤ 6 MB) or a Fargate task (files > 6 MB).

The architecture includes:

• An S3 bucket with EventBridge notifications enabled
• An EventBridge rule that triggers Step Functions on S3 object creation
• A Step Functions state machine with a Choice state for routing
• A Lambda function for processing small files
• An ECS Fargate task for processing large files
• A VPC with two subnets, internet gateway, and security group
• An ECR repository for the Fargate container image
• Scoped IAM roles for Lambda, Step Functions, and ECS tasks

Use this skill when:

• You need to process S3 uploads with different compute based on file size
• You want a serverless workflow that can handle both small and large files
• You need Step Functions orchestration with Lambda and Fargate

Do not use this skill when:

• All files are small enough for Lambda (use S3 → Lambda directly)
• You need real-time streaming (use Kinesis)
• You don't need file-size-based routing

Prerequisites

1. AWS CLI v2 — Installed and configured. Verify with aws sts get-caller-identity.
2. Python 3.12 — For the Lambda function runtime.
3. Docker — For building and pushing the Fargate container image.

Parameters

• bucket_name (required): Name for the S3 bucket (globally unique, lowercase, 3-63 characters)
• region (required): AWS region for all resources
• ecr_repo_name (required): Name for the ECR repository
• state_machine_name (required): Name for the Step Functions state machine
• kms_key_arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with aws kms create-key --description "Key for CloudWatch Logs encryption" --region {region}

Constraints for parameter acquisition:

• You MUST ask for all required parameters upfront in a single prompt
• You MUST support multiple input methods (direct input, file path, URL)
• You MUST confirm successful acquisition of all parameters before proceeding
• You MUST validate that bucket_name follows S3 naming rules

Procedures

Step 0: Verify Dependencies

Constraints:

• You MUST verify the following tools are available: aws-cli, python3 (3.12+), docker
• You MUST inform the user about any missing tools with a clear message
• You MUST ask if the user wants to proceed despite missing tools
• You MUST respect the customer's decision to abort at any point
• You MUST explain to the customer what step is being executed, why, and which tool is being called

Step 1: Retrieve AWS Account ID

Constraints:

• You MUST retrieve the account ID with: aws sts get-caller-identity --query 'Account' --output text
• You MUST store the result as {account_id} for use in all subsequent steps
• You MUST abort if credentials are not configured

Step 2: Get the Default VPC and Networking

Constraints:

• You MUST retrieve the default VPC ID with: aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text --region {region}
• If no default VPC exists, inform the user they must create one with aws ec2 create-default-vpc --region {region} or provide a VPC ID manually
• You MUST retrieve two subnet IDs from the default VPC: aws ec2 describe-subnets --filters Name=vpc-id,Values={vpc_id} --query 'Subnets[0:2].SubnetId' --output text --region {region}
• You MUST create a security group in the default VPC: aws ec2 create-security-group --group-name fargate-sg --description "Security group for Fargate tasks" --vpc-id {vpc_id} --region {region}
• You MUST configure security group egress rules to allow only HTTPS and DNS outbound. First revoke the default allow-all egress rule: aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]' --region {region} Then add scoped rules: aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0 --region {region} and aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0 --region {region}
• You MUST recommend VPC endpoints for S3 and CloudWatch Logs for production workloads to avoid internet-routed traffic and eliminate the need for broad egress rules
• You MUST capture {vpc_id}, {subnet1_id}, {subnet2_id}, and {sg_id} for use in later steps

Step 3: Create the ECR Repository

Constraints:

• You MUST create the repository with: aws ecr create-repository --repository-name {ecr_repo_name} --region {region}
• You MUST capture the repositoryUri from the response

Step 4: Build and Push the Container Image

Constraints:

• You MUST verify Docker is installed by running docker --version. If Docker is not installed, instruct the user to install it from https://docs.docker.com/get-docker/ and abort until it is available
• You MUST authenticate Docker with ECR: aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {account_id}.dkr.ecr.{region}.amazonaws.com
• The Dockerfile and processor code are in scripts/Dockerfile and scripts/fargate_processor.py
• You MUST build and push the image from the scripts directory:

  cd scripts
  docker build --platform linux/amd64 -t {ecr_repo_name} .
  docker tag {ecr_repo_name}:latest {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  docker push {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  cd ..

Step 5: Create IAM Roles

Follow the detailed instructions in references/iam-roles.md to create all IAM roles (Lambda, ECS task execution, ECS task, Step Functions, and EventBridge roles).

• You MUST wait at least 10 seconds for IAM role propagation

Step 6: Create the Lambda Function

Constraints:

• The function code is in scripts/lambda_function.py
• You MUST be in the skill root directory before packaging and creating the function
• You MUST package it with: python3 -c "import zipfile,io; z=io.BytesIO(); f=zipfile.ZipFile(z,'w'); f.writestr('lambda_function.py', open('scripts/lambda_function.py').read()); f.close(); open('/tmp/lambda_function.zip','wb').write(z.getvalue())"
• You MUST create the function with:

  aws lambda create-function \
      --function-name sfn-file-processor \
      --runtime python3.12 \
      --handler lambda_function.lambda_handler \
      --role arn:aws:iam::{account_id}:role/sfn-lambda-role \
      --zip-file fileb:///tmp/lambda_function.zip \
      --timeout 60 \
      --architectures x86_64 \
      --region {region}

• You MUST verify the function was created with: aws lambda get-function --function-name sfn-file-processor --region {region}

Step 7: Create the CloudWatch Log Group

Constraints:

• You MUST create the log group for Fargate: aws logs create-log-group --log-group-name /StepFunctionFargateTask --region {region}
• You MUST encrypt the log group with a KMS key: aws logs associate-kms-key --log-group-name /StepFunctionFargateTask --kms-key-arn {kms_key_arn} --region {region}

Step 8: Create the ECS Cluster and Task Definition

Follow the detailed instructions in references/ecs-task-definition.md to create the ECS cluster and register the Fargate task definition.

• You MUST capture the task definition ARN from the response

Step 9: Create the S3 Bucket with EventBridge Notifications

Constraints:

• You MUST create the bucket with: aws s3api create-bucket --bucket {bucket_name} --region {region} --create-bucket-configuration LocationConstraint={region}
• You MUST NOT include --create-bucket-configuration if region is us-east-1
• You MUST enable EventBridge notifications on the bucket: aws s3api put-bucket-notification-configuration --bucket {bucket_name} --notification-configuration '{"EventBridgeConfiguration": {}}' --region {region}
• You MUST enable default encryption on the bucket: aws s3api put-bucket-encryption --bucket {bucket_name} --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}' --region {region}

Step 10: Create the Step Functions State Machine

Constraints:

• The state machine definition is in scripts/statemachine.asl.json
• You MUST create a working copy and replace all placeholders:

  sed -e 's|${LambdaFunction}|arn:aws:lambda:{region}:{account_id}:function:sfn-file-processor|g' \
      -e 's|${Cluster}|arn:aws:ecs:{region}:{account_id}:cluster/sfn-cluster|g' \
      -e 's|${TaskDefinition}|{task_definition_arn}|g' \
      -e 's|${Subnet1}|{subnet1_id}|g' \
      -e 's|${Subnet2}|{subnet2_id}|g' \
      -e 's|${SecurityGroup}|{sg_id}|g' \
      scripts/statemachine.asl.json > /tmp/statemachine.asl.json

• You MUST create the state machine with:

  aws stepfunctions create-state-machine \
      --name {state_machine_name} \
      --definition file:///tmp/statemachine.asl.json \
      --role-arn arn:aws:iam::{account_id}:role/sfn-state-machine-role \
      --type STANDARD \
      --region {region}

• You MUST capture the stateMachineArn from the response

Step 11: Create the EventBridge Rule

Constraints:

• You MUST create the EventBridge rule to trigger on S3 object creation:

  aws events put-rule \
      --name s3-to-stepfunctions \
      --event-pattern '{
        "source": ["aws.s3"],
        "detail-type": ["Object Created"],
        "detail": {
          "bucket": {
            "name": ["{bucket_name}"]
          }
        }
      }' \
      --region {region}

• You MUST add the state machine as a target:

  aws events put-targets \
      --rule s3-to-stepfunctions \
      --targets '[{
        "Id": "StepFunctionsTarget",
        "Arn": "{state_machine_arn}",
        "RoleArn": "arn:aws:iam::{account_id}:role/sfn-eventbridge-role"
      }]' \
      --region {region}

Step 12: Configure Monitoring

Constraints:

• You MUST create a Dead Letter Queue for failed EventBridge invocations: aws sqs create-queue --queue-name s3-to-stepfunctions-dlq --region {region}
• You MUST update the EventBridge target to attach the DLQ:

  aws events put-targets \
      --rule s3-to-stepfunctions \
      --targets '[{
        "Id": "StepFunctionsTarget",
        "Arn": "{state_machine_arn}",
        "RoleArn": "arn:aws:iam::{account_id}:role/sfn-eventbridge-role",
        "DeadLetterConfig": {
          "Arn": "arn:aws:sqs:{region}:{account_id}:s3-to-stepfunctions-dlq"
        }
      }]' \
      --region {region}

• You MUST create a CloudWatch alarm for Step Functions execution failures: aws cloudwatch put-metric-alarm --alarm-name sfn-execution-failures --metric-name ExecutionsFailed --namespace AWS/States --statistic Sum --period 300 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --evaluation-periods 1 --dimensions Name=StateMachineArn,Value={state_machine_arn} --region {region}

Step 13: Validate

Constraints:

• You MUST test with a small file (< 6 MB) to verify Lambda processing:

  echo 'test data' > /tmp/small-file.txt
  aws s3 cp /tmp/small-file.txt s3://{bucket_name}/small-file.txt --region {region}

• You MUST wait 15 seconds then check the Step Functions execution: aws stepfunctions list-executions --state-machine-arn {state_machine_arn} --region {region}
• You MUST verify the execution succeeded and routed to Lambda
• You MUST provide a summary of all created resources including: VPC ID, subnet IDs, security group ID, ECR repo URI, ECS cluster ARN, task definition ARN, Lambda function ARN, state machine ARN, bucket name, and EventBridge rule name

Troubleshooting

EventBridge rule not triggering

• Verify EventBridge notifications are enabled on the bucket: aws s3api get-bucket-notification-configuration --bucket {bucket_name}
• Verify the rule exists: aws events describe-rule --name s3-to-stepfunctions --region {region}
• Check that the target has the correct state machine ARN and role

Step Functions execution fails at Fargate task

• Verify the container image exists in ECR: aws ecr describe-images --repository-name {ecr_repo_name} --region {region}
• Check that the subnets have internet access (route table with IGW)
• Verify the security group allows outbound traffic
• Check CloudWatch Logs at /StepFunctionFargateTask

Lambda invocation fails

• Check CloudWatch Logs: aws logs tail /aws/lambda/sfn-file-processor --region {region}
• Verify the Step Functions role has lambda:InvokeFunction permission

IAM PassRole errors

• The Step Functions role must have iam:PassRole for both the ECS execution role and task role ARNs

Fargate task stuck in PROVISIONING

• Verify the subnets have auto-assign public IP enabled
• Verify the internet gateway is attached and route table has 0.0.0.0/0 route

Security Considerations

• Fargate tasks with public IPs are exposed to the internet. Revoke the default allow-all egress rule and configure scoped egress: aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]' then add aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0 and aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0. For production, consider using VPC endpoints for S3 and CloudWatch Logs instead of internet-routed traffic.
• Scan container images for vulnerabilities before pushing to ECR. Enable ECR image scanning with: aws ecr put-image-scanning-configuration --repository-name {ecr_repo_name} --image-scanning-configuration scanOnPush=true --region {region}
• Use IAM roles for credentials — never hardcode access keys in container code.
• Enable encryption at rest for the S3 bucket: aws s3api put-bucket-encryption --bucket {bucket_name} --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'
• Enable CloudWatch Logs encryption for Fargate container logs: aws logs associate-kms-key --log-group-name /StepFunctionFargateTask --kms-key-arn <KMS_KEY_ARN>
• Configure a Dead Letter Queue on the EventBridge rule for failed invocations
• Set up CloudWatch alarms on Step Functions execution failures for operational visibility

Version information

• AWS CLI: 2.x
• Python runtime: 3.12
• Last validated: 2026-04-27

Additional Resources

• Step Functions developer guide
• EventBridge S3 events
• Fargate task definitions
• ECR pushing images
• Step Functions Fargate integration