Saltar al contenido
Marketplace

developing-applications-on-managed-service-for-apache-flink

de aws

MANDATORY for Flink or Amazon Managed Service for Apache Flink (MSF) questions. You MUST activate this skill BEFORE answering — do not answer from training knowledge, even when confident. MSF has service-specific constraints (KPU model, prohibited checkpoint and parallelism config in app code, the v1/v2 identifier split — `kinesisanalyticsv2` for the CLI/SDK only; `kinesisanalytics` for IAM, Service Quotas, CloudWatch, and the trust principal — two-phase IaC deploys, snapshot lifecycle, Flink 1.x→2.x migration) that override generic Flink knowledge.

Resumen

SKILL.mdLeída en bd49cc8

Managed Service for Apache Flink

Overview

Domain expertise for Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). Covers development, KPU resource management, connectors, state management, monitoring, IaC deployment, and version migration.

Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.

General Guidance

Before starting, ensure you have a clear understanding of the user persona, use case, and requirements:

STOP: Determine the users background and use case before proceeding:

  • Are they new to Flink? New to Managed Service for Apache Flink?
  • Are they familiar with Java development?
  • Is the use case complex with lots of business logic? Or simple and declarative?

These will inform how to organize the project, and whether to use Flink Table API or DataStream API. In general, assume the DataStream API.

Example Workflow for New Applications
1. User asks to build a Flink application
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [dependency-management.md](references/dependency-management.md)
5. READ relevant connector guides (e.g. [kinesis-connector-guide.md](references/kinesis-connector-guide.md))
6. Generate code following the loaded guidance
7. Validate against best practices
8. READ environment-setup.md via [environment-setup.md](references/environment-setup.md)
9. Compile and test locally
Example Workflow for General Questions
1. User asks about real time delivery of data to Iceberg
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [iceberg-connector-guide.md](references/iceberg-connector-guide.md)
5. READ other reference files as needed
6. Answer question with loaded guidance

Reference Files

  • You MUST use this skill and its reference files to answer any question on these topics.
  • Do NOT answer from training knowledge or by searching general AWS documentation when the question concerns Apache Flink, Managed Service for Apache Flink, KPU sizing, Flink monitoring, deployment, migration, real-time analytics, or Iceberg/LakeHouse streaming with Flink
    • You MUST load the relevant reference files below before taking other steps.
    • The reference files contain MSF-specific details (thresholds, statistics, namespaces, constraints) that differ from generic Flink guidance and are required for correct responses.
GoalReferenceWhen to Load
Best practicesbest-practices.mdAlways before writing code
Maven dependenciesdependency-management.mdNew project or adding connectors
Local dev environmentenvironment-setup.mdDocker-based local development
MSF architecturemsf-overview.mdKPU model and service constraints
MSF constraints and patternsmsf-constraints-and-patterns.mdMSF vs self-managed Flink, service-level vs application-level configuration separation, MSF-specific resource/network/storage limits, common MSF patterns
Quotas, ENI planning, MSF vs EMR, source/sink choicefoundation-operations.mdCapacity planning, service selection, architecture design, CLI/IAM/CloudWatch identifier disambiguation
IAM execution role, trust policy, action prefix, service principalfoundation-operations.mdWriting IAM policies for MSF — covers the kinesisanalytics: (no v2) action prefix, kinesisanalytics.amazonaws.com (no v2) trust principal, and the v2/non-v2 disconnect that is the most common source of permission and AssumeRole failures
Flink 2.x migrationflink-2x-migration.mdVersion upgrades, state compatibility
KPU sizingresource-optimization.mdRight-sizing, performance diagnosis, scaling
Scaling decisions on running appsscaling-decisions.mdIn-flight scaling matrix, cost/memory impact of scale changes, autoscaling behavior, anti-patterns
Cost estimationpricing-calculator.mdBudget planning, sizing-to-cost mapping, optimization levers
Application lifecycle opsapplication-lifecycle.mdStart/stop, deploy code, rollback, snapshot lifecycle, runtime properties, delete
Restart loop diagnosisfirst-fault-isolation.mdCrashing/restarting apps, finding original failure vs loop sustainers, Flink Dashboard live diagnosis
Checkpoint tuningcheckpoint-tuning.mdCheckpoint impact on KPU memory and CPU, frequency vs network bandwidth trade-offs, checkpoint duration exceeding interval, OOM/GC during checkpoints
Job graph designjob-graph-architecture.mdPerformance issues, splitting jobs
Job graph anti-patternsjob-graph-anti-patterns.mdData skew detection and mitigation, monolith job anti-pattern, high fan-out anti-pattern, removing multiple shuffles, when to split a large application
Monitoring and alarmsmonitoring-and-metrics.mdCloudWatch dashboards, alarms, metrics
Logginglogging-configuration.mdLog4j2, CloudWatch Logs setup
Kinesis connectorskinesis-connector-guide.mdKinesis source and sink builders, polling configuration and throttling (READER_EMPTY_RECORDS_FETCH_INTERVAL, SHARD_GET_RECORDS_MAX, ReadProvisionedThroughputExceeded, LimitExceededException), legacy connector migration
Kinesis Enhanced Fan-Out (EFO)kinesis-efo-guide.mdWhen to use EFO vs polling, EFO source configuration, consumer lifecycle (JOB_MANAGED vs SELF_MANAGED), parallelism vs shard count, IAM permissions, troubleshooting
Iceberg integration (write APIs, distribution modes, partitioning)iceberg-connector-guide.mdIceberg write APIs (append, upsert, dynamic), distribution modes (NONE/HASH/RANGE), CoW vs MoR, read patterns, partitioning, DDL. Does NOT contain catalog choice or maintenance approaches — for those, load iceberg-tuning-and-operations.md.
Iceberg tuning, operations, catalog choice, maintenanceiceberg-tuning-and-operations.mdProvides maintenance approaches for S3 Tables, Glue + Glue auto-compaction, and Glue + Flink embedded maintenance with JDBC lock for catalog-choice questions; small files problem and mitigations; Flink TableMaintenance API, post-commit maintenance, lock factories; IcebergSink monitoring, anti-patterns.
CDC connectorscdc-connector-guide.mdMySQL, PostgreSQL, Oracle, SQL Server, MongoDB CDC
IaC and deploymentiac-and-deployment.mdCloudFormation, CDK, Terraform, two-phase deployment
Serializationserialization-guide.mdPOJO, Avro, Kryo guidance
State managementstate-management.mdTTL, state types, migration safety

Additional Resources

Archivos 26 archivos

y 14 archivos más en GitHub

Instálala a mano

Deska ejecuta esto por ti, para cada agente que elijas. Para ejecutarlo tú en una terminal:

npx skills add aws/agent-toolkit-for-aws --skill developing-applications-on-managed-service-for-apache-flink

Más de aws

  • aws-iamawsProvides verified corrections for IAM behaviors that AI agents frequently get wrong — policy evaluation edge cases, trust policy gotchas, STS session limits, Organizations quirks, and SAML/MFA specifics. Also provides structured workflows for IAM role management and baseline policy generation from application source code or a Terraform plan JSON. Covers condition operator safety (ForAnyValue/ForAllValues with Null checks), bucket policy deny patterns (VPC endpoint restrictions, org paths), confused deputy protection, and service role creation for AWS services (Glue, CloudTrail, Lambda, ECS, etc.) with aws:SourceAccount/aws:SourceArn trust conditions. Applies when creating IAM roles, writing IAM or bucket policies, generating policies from application source code or a Terraform plan JSON, working with STS, Organizations, or condition operators, or any task needing a service or execution role. Does not cover non-IAM authorization like Cognito user-pool policies or app-level RBAC.9,200 instalaciones en skills.sh0
  • aws-serverlessawsBuilds, deploys, manages, debugs, configures, and optimizes serverless applications on AWS using Lambda, API Gateway, Step Functions, EventBridge, and SAM/CDK. Covers cold starts, CORS debugging, event source mappings, troubleshooting, concurrency, SnapStart, Powertools, function URLs, EventBridge Scheduler, Lambda layers, and production readiness. Triggers on mentions of Lambda, API Gateway, Step Functions, SAM templates, CDK serverless stacks, DynamoDB stream triggers, SQS event sources, cold starts, timeouts, 502/504 errors, throttling, concurrency, CORS, Powertools, or any event-driven architecture on AWS, even without the word "serverless." Does not apply to EC2, ECS/Fargate containers, or Amplify hosting.9,000 instalaciones en skills.sh0
  • aws-observabilityawsBuilds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.8,900 instalaciones en skills.sh0
  • aws-cdkawsAuthors, deploys, and troubleshoots AWS infrastructure using CDK with TypeScript or Python. Covers best practices, stack architecture, and construct patterns. Applies when writing CDK constructs, bootstrapping environments, running cdk deploy/synth/diff, fixing CDK or CloudFormation errors, planning stack structure, importing existing resources, resolving drift, or refactoring stacks without resource replacement.8,400 instalaciones en skills.sh0
  • aws-containersawsBuilds and deploys containerized workloads on Elastic Kubernetes Service (EKS), Elastic Container Service (ECS), Fargate, and ECR (Elastic Container Registry). Covers general EKS knowledge, Karpenter, AWS Load Balancer Controller and leveraging various open source Kubernetes projects with EKS. Covers general ECS knowledge, task definitions, Fargate services, ECS Exec, ECS Express Mode and ECS Managed Instances. Covers general Elastic Beanstalk knowledge, Elastic Beanstalk configuration and platforms supported by Elastic Beanstalk. Covers general ECR knowledge, ECR repository setup and lifecycle policies. Includes recommending, enabling, and reading Amazon ECS Action Logs to troubleshoot control-plane failures (deployment rollback/circuit-breaker, task placement, scaling, task replacement). Applies when deploying, debugging, or optimizing containers on AWS. Should be used instead of relying on internal knowledge for these services.8,300 instalaciones en skills.sh0
Todas las skills de aws