The workshop is built on GitBook, so you can actually ask the LLM questions and it’ll surface answers from the workshop docs.

The source code deploys a web app that walks you through questions about your call center design (menus, intents, tools) and generates the contact flows and backing infrastructure.

I built a retirement fund sample: callers can check their balance, ask about the next payment date, order a 1099, or request special forms. The tool scaffolded the Bedrock agent, wired up the tools, and generated the Lambda stubs. Not bad for a few minutes of clicking through a form.

Where it broke

The generated CloudFormation had the wrong Lambda handler entry point. Easy fix once you know what you’re looking for, but it took a debugging pass to find it.

If you’re starting fresh, paste the error and the CloudFormation into an LLM and it’ll find it in seconds.

Why look at it

Even if you never deploy it, read the prompts. The way it structures the agent definition and maps intents to tools is a concrete example of how a production Connect + Bedrock setup actually gets wired together. Good working examples of that are hard to come by.

For POCs and small call centers, it’s a solid starting point. For a long-running production project, you’d outgrow the scaffolding fast. But for standing up a demo or validating an architecture, it does the job.

My blog workflow: I have a vague idea, I run /post in my blog repo, and Claude drafts something. Then I proofread it. The problem is the output is loaded with AI tells — too many transition phrases, filler words, and inflated sentence structures that aren’t how I talk or write.

“It’s worth noting that…”, “This is particularly useful when…”, “Let’s dive in.” None of that is me.

The avoid-ai-writing skill audits generated text and rewrites it to remove those patterns. You can run it in detect-only mode to see what’s wrong, or have it edit in place. There’s also a voice profile option if you want to tune the output toward casual, professional, blunt, etc.

I set it to run on my drafts after /post generates them. The diff is usually significant.

The pipeline hadn’t run in days, but that didn’t matter. My job was to figure out who actually made the change.

Here’s what I think happened:

• Sally is building a new feature in dev — not ready for production
• John needs to test a hotfix, so he exports the prod flow and imports it into dev
• John validates in dev, replicates the fix in prod, moves on
• Sally comes back and finds all her work gone

Sally can restore from the flow’s version history. But who caused the overwrite?

The Easy Way (I Should Have Known This)

Right there on the Contact Flow List page is a link: “View Historical Changes”. You can filter by date and time and see exactly who made each change.

And on the flow details page, if you select an older version and switch to the Details tab, it shows you the user who saved that version.

I went straight to CloudTrail. The answer was sitting in the UI the whole time. I’ve spent way too much time in code and not enough in the Connect console.

The CloudTrail Way

If you need an API-based approach — or if you’re already in CloudTrail for other reasons — here’s how to read the events.

Changes Made via API or CLI

1
2
3
4
5
6

{
    "userIdentity": {
        "type": "AssumedRole",
        "arn": "arn:aws:sts::999999999999:assumed-role/developer-sso-role/user@example.com"
    }
}

The email is embedded in the role ARN as the Role Session Name. Easy.

Changes Made via the Connect Console

1
2
3
4
5
6

{
    "userIdentity": {
        "type": "AssumedRole",
        "arn": "arn:aws:sts::999999999999:assumed-role/AWSServiceRoleForAmazonConnect_DwT0YgEuWX9uaWsPKbOu/78adbca8-17ff-47dd-b354-2405d9371018"
    }
}

The Role Session Name here is an internal Connect user ID, not an email. To resolve it:

1

aws connect describe-user --user-id 78adbca8-17ff-47dd-b354-2405d9371018 --instance-id <your-instance-id>

Emergency Access

If describe-user returns nothing, the user logged in with Emergency Access. You can’t identify them directly from the flow change event, but you can search CloudTrail for who used the feature:

1
2
3
4
5
6
7
8

{
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA3LF2EZF67TQBAWQ23:user@example.com"
    },
    "eventSource": "connect.amazonaws.com",
    "eventName": "AdminGetEmergencyAccessToken"
}

The principalId contains the email of whoever requested emergency access.

It feels familiar, but it’s the wrong tool for this job.

The Problem

When your Lambda returns a response object — any response object — Amazon Connect sees a successful invocation. There’s no Lambda error. No CloudWatch alarm fires. No metric spikes. From the platform’s perspective, everything is fine.

Now your contact flow has to do the detective work. You add a “Check Contact Attributes” block, branch on $.External.statusCode, wire up paths for 200, 400, 404, 500… and suddenly a flow that should be simple is a maze.

And let’s be honest — how often does the IVR actually handle all those codes differently? Usually there’s a “happy path” for 200 and everything else hits a generic error message. You’ve added complexity without adding value, and you’ve buried your errors where no one can see them.

What to Do Instead

Let the Lambda throw.

If there’s an unhandled exception or a validation error, don’t swallow it. You can still use a global try/catch to log structured details before you give up:

1
2
3
4
5
6
7
8
9
10
11
12
13

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger({ serviceName: 'customer-lookup' });

export const handler = async (event: ConnectContactFlowEvent) => {
  try {
    // your logic here
    return await lookupCustomer(event);
  } catch (err) {
    logger.error('Unhandled error', { err });
    throw err; // let it propagate
  }
};

The Lambda fails. CloudWatch logs the structured error. Your Lambda error metrics reflect reality. CloudWatch Insights lets you query across invocations to see exactly what’s going wrong and how often.

Your contact flow stays clean: there’s a success branch and an error branch. The error branch handles the situation gracefully — a fallback message, a queue transfer, whatever makes sense — and you move on.

The One Legitimate Exception

Sometimes you genuinely need to branch on whether a record exists, and that’s not an error — it’s a valid business case. A caller might or might not have an account. That’s fine.

Return a boolean:

1
2
3

return { found: true, ...customerData };
// or
return { found: false };

Check $.External.found in your flow and take the appropriate branch. That’s a business decision, not an error condition, and it shouldn’t be modeled as one.

A Convention in the Wrong Place

Status codes are an API Gateway convention. They exist so HTTP clients can interpret responses without reading the body. Amazon Connect isn’t an HTTP client — it doesn’t need that layer of indirection, and adding it just obscures what your system is actually doing.

Throw on errors. Return meaningful data on success. Your flows will be simpler and your CloudWatch logs will actually tell you when something is broken.

I write a lot of technical documentation in Markdown — architecture docs, design proposals, runbooks. It’s fast, version-controlled, and plays well with Mermaid diagrams. But when it’s time to hand something to a client, a raw markdown export looks amateurish.

Here’s how I now generate polished, branded PDFs without leaving VS Code.

The Setup

Install the Markdown Preview Enhanced extension for VS Code. It renders Markdown in a live preview pane and can export to PDF via Chrome.

Let Claude Generate the Stylesheet

Open a Claude Code session and ask it to generate a stylesheet based on the client’s website:

“Generate a Markdown Preview Enhanced stylesheet based on the branding at acmecorp.com. Save it to .crossnote/style.less”

Claude fetches the site, pulls the primary colors, fonts, and heading styles, and writes a stylesheet to .crossnote/style.less — the folder MPE uses for custom styles.

Preview and Export

Open your Markdown file, open the MPE preview pane (Ctrl+Shift+V), and you’ll see the document rendered in the client’s colors. When you’re ready, right-click in the preview pane and select Export → PDF.

You get a PDF with client-matching typography, colors, and properly rendered Mermaid diagrams.

Per-Client Stylesheets

The .crossnote folder lives in the repo, so commit stylesheets per project or client:

1
2
3
4

.crossnote/
  style.less         ← active stylesheet
  acme-style.less
  globex-style.less

Swap stylesheets when switching projects. The whole team gets consistent, on-brand output.

Download a sample PDF

I still love the technical work — debugging gnarly issues, designing systems, writing code. But leading a team of exceptional developers on cutting-edge projects is just as challenging, if not more. Like any hard technical problem, I’m approaching it with study and practice.

These lessons apply at every level of leadership — a senior mentoring a junior, a team lead guiding senior devs. But as a director, getting them right is vital. The more I free up my own time, the more I have for the meetings, one-on-ones, interviews, and the work that only I can do.

Let People Get Bruised

The hardest part is letting others stumble. Use the Socratic method — prompt people to discover solutions rather than handing them the answer. Let them learn to solve their own problems.

As an individual contributor, this was a challenge. In this role, it’s mandatory.

Stop Being the Hero

I often spot something I could “just fix myself” in 15 minutes. Using tools like Claude Code, it may even be fun! Resist.

We work with serverless applications that contain infrastructure, backend, frontend, and testing code. A simple change can surface deployment issues, runtime bugs, rendering problems, or test failures. Your “15 minute fix” quietly becomes 3 hours.

A better approach: spend those 15 minutes with Claude Code in the target repo drafting a user story. Iterate until it’s clear, then put it on the backlog for a developer to pick up. Your job is to multiply the team, not be a bottleneck.

Vibe Code Your Specs

I tend to write short user stories. A few sentences is enough for me to come back in a few weeks and know exactly what it means. Other developers don’t have the project and industry context sitting in my head. A short doc means a meeting for context transfer. A detailed doc is time consuming to write.

So I started using Claude Code to draft design documents from the target repo or source materials. The output is more detailed and accurate than what I’d normally produce on my own. I iterate until it says what I need and nothing else — AI tools love to drop in extra fluff like future enhancements, troubleshooting, and scaling sections. If it’s not relevant, cut it. Reading docs takes brain power too.

This ties into a broader trend. Spec Driven Development is gaining traction with tools like Kiro, OpenSpec, and the BMAD method. A good spec pays off twice — once for the human developer reading it, and again as input for an agentic coding tool that someone else monitors. The spec becomes the handoff, not a meeting.

When I joined CXBuilder back in April as a Senior Architect, I was drawn to the scale and complexity of the work — designing active/active global resiliency for one of the world’s largest Amazon Connect deployments. Six months in, I’m now leading the engineering organization.

I’m excited about this next chapter and the opportunity to shape both the technical direction and the team building it.

Amazon Connect quietly introduced a game-changing feature that can eliminate this waste entirely—but their announcement barely scratched the surface.

I wrote a technical guide on the CXBuilder blog:

Stop Making Callers Wait →

We solved this for a client and I wrote up the solution on the CXBuilder blog — how to perform reliable backend health checks without destroying caller experience or overwhelming your systems.

Scaling Amazon Connect Health Checks →

I’m leading the technical architecture for one of the world’s largest Amazon Connect deployments — designing active/active global resiliency at a scale that pushes the boundaries of what’s possible with contact center infrastructure.

CXBuilder tackles the complex integrations that standard solutions can’t handle. It’s the kind of work where you need both technical precision and creative problem-solving, and I’m excited to keep pushing the envelope on what’s achievable with serverless architectures and Amazon Connect at global scale.

Fortunately, HubSpot Forms API provides a onFormReady callback!

Code Sample

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

const HubSpotContactForm = () => {
  const { getClaims } = useAmplify();
  const [email, setEmail] = useState<string>('');

  useEffect(() => {
    getClaims().then((d) => setEmail(d?.email));
  }, []);

  useEffect(() => {
    if (email) {
      const script = document.createElement('script');
      script.src = 'https://js.hsforms.net/forms/v2.js';
      document.body.appendChild(script);

      script.addEventListener('load', () => {
        const hbspt = (window as any).hbspt;
        if (hbspt) {
          hbspt.forms.create({
            portalId: 'TODO',
            formId: 'TODO',
            target: '#hubspotForm',
            onFormReady: function ($form: any) {
              const emailInput = $form[0];
              emailInput.value = email;
              emailInput.dispatchEvent(new Event('input', { bubbles: true }));
            },
          });
        }
      });
    }
  }, [email]);

  return (
    <Container
      header={
        <Header
          variant="h2"
          description="Submit any questions or feedback here and we will get back to you shortly"
        >
          Contact Form
        </Header>
      }
    >
      <div id="hubspotForm"></div>
    </Container>
  );
};

I generated the static site using Hugo, and I am relying on HubSpot for the contact form.

Fortunately, HubSpot Forms API provides a onFormSubmit callback!

Code Sample

1
2
3
4
5
6
7
8
9

hbspt.forms.create({
  region: 'na1',
  portalId: 'TODO',
  formId: 'TODO',
  onFormSubmit: function ($form) {
    // Use gtag to track conversion
    gtag_report_conversion();
  },
});

1
2
3
4
5
6
7
8

{
  "SchemaVersion": "1.0",
  "Sequence": 2,
  "InvocationEventType": "INVALID_LAMBDA_RESPONSE",
  "CallDetails": { ... },
  "ErrorType": "ActionExecutionThrottled",
  "ErrorMessage": "This account has exceeded the number of bot references allowed by the PSTN Audio service"
}

This error was a surprise because I have deployed similar apps in older accounts, without being throttled.

The Service Quota

After doing some searching, I found this documentation about a new service quota

• Name: Amazon Chime SDK SIP trunking and voice - StartBotConversation Amazon Lex bots
• Default: 0
• Adjustable: Yes
• Description: The maximum number of Amazon Lex bots you can use with the PSTN Audio StartBotConversation action in the current AWS Region.

I assume this quota is pretty new, because I could not find it in the Service Quota console, or list it using the CLI.

I do not know yet if the quota will limit how many distinct bots I can use, or how many bot sessions. Ping me if you know the answer!

Timeline (ETA 12 Days)

I submitted a support case to get the limit increased. I will keep this timeline updated until the issue is resolved.

• 2024-11-15: Submitted case titled: Need to increase service quote. Option is documented but not available in AWS Console.
• 2024-11-18: Response from support requesting use case description and what region I expect to use.
• 2024-11-19: Response from support: The request has been forwarded to the service team, expect a response in 7-10 days.
• 2024-11-27: The service quota has been updated and applied.

Conclusion

If developing an Chime SIP Media Application, include time for the following service quota increases:

• Amazon Chime SDK SIP trunking and voice - StartBotConversation Amazon Lex bots
  • Default: 0
  • ETA: 12 days
• Amazon Chime SDK SIP trunking and voice - provisioned phone numbers
  • Default: 5
  • ETA: 10 days
• Other Quotas

Recently I built a CDK web application that could provision other CDK apps. React app sends a message to API GW, which invokes a lambda, which triggers a CDK deployment.

The default Lambda runtime cannot deploy CDK. You have to use a custom Docker image. There are plenty of online guides on how to do this, so I am only covering the highlights.

Deployer Stack

It takes a while to build the docker container, so you should define a deployer stack separate from your web app.

It is also handy to add the following script to your package.json:

"cdk:deployer": "npm run cdk -- --app=\"npx ts-node --prefer-ts-exts bin/deployer.ts\""

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

import { StackProps, Stack, Duration, CfnOutput, Aws } from 'aws-cdk-lib';
import { PolicyStatement } from 'aws-cdk-lib/aws-iam';
import {
  Architecture,
  DockerImageCode,
  DockerImageFunction,
} from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';
import { PROJECT_ROOT } from '../../constants';

interface DeployerStackProps extends StackProps {
  prefix: string;
}
/**
 * Deploys the cdk deployer lambda.
 * Implemented as own stack with export because it takes a while to build and deploy.
 * You should redeploy this when your CDK code changes
 */
export class DeployerStack extends Stack {
  readonly deployFunctionArn: CfnOutput;

  constructor(scope: Construct, id: string, props: DeployerStackProps) {
    const { prefix } = props;
    super(scope, id, { stackName: `${prefix}-deployer`, ...props });

    const deploy = new DockerImageFunction(this, 'Deploy', {
      code: DockerImageCode.fromImageAsset(PROJECT_ROOT),
      timeout: Duration.minutes(15),
      memorySize: 4096,
      initialPolicy: [
        // Scope this down to whatever you need.
        new PolicyStatement({
          resources: [
            `arn:aws:iam::${Aws.ACCOUNT_ID}:role/cdk-hnb659fds-*-role-${Aws.ACCOUNT_ID}-*`,
          ],
          actions: ['sts:AssumeRole'],
        }),
      ],
      environment: {
        // Required to use npx
        HOME: '/tmp',
      },
    });

    this.deployFunctionArn = new CfnOutput(this, 'DeployFunctionArn', {
      value: deploy.functionArn,
      exportName: `${prefix}-deployer-function-arn`,
    });
  }
}

Notice the deployer-function-arn export. You will want to import this into your other app.

Dockerfile

This docker file will copy your entire application, and set the lambda entry point.

1
2
3
4
5
6
7
8
9

FROM public.ecr.aws/lambda/nodejs:20

WORKDIR ${LAMBDA_TASK_ROOT}

# Copy in pre-installed/built resources
COPY . .

# Entry point
CMD ["dist/lib/DeployerStack/lambda/index.handler"]

Note: I found a few examples online which use a different base image (node:18-bookworm), which require additional steps including installing aws-lambda-ric, but I ran into some issues.

• After a certain amount of pulls from the Docker registry, you will have to sign up. For AWS work, ECR is fast and automatic.
• The public.ecr.aws/lambda/nodejs:20 image is already configured for running node apps. Total build time dropped from ~15 to 4.5 minutes.

Deployer Lambda

Here is the lambda entry point. Notice the handler accepts stack props. Since my app uses API GW with a 15 second timeout, I trigger the trigger this second lambda.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import { randomUUID } from 'crypto';
import { App } from 'aws-cdk-lib';
import * as fs from 'fs-extra';
import {
  MyMiniAppStack,
  MyMiniAppStackProps,
} from '../../constructs/MyMiniAppStack';
import { exec } from '@lsw-apps/lsw-dev-util';

/**
 * Entry point for the deployer lambda
 */
export async function handler(childStackProps: MyMiniAppStackProps) {
  console.log('deploying', childStackProps);

  // Synthesize a Cloud Assembly somewhere in /tmp
  const assemblyDir = `/tmp/cdk.out.${randomUUID()}`;

  const app = new App({ outdir: assemblyDir });
  new MyMiniAppStack(app, 'MyMiniApp', childStackProps);
  app.synth();

  try {
    // Deploy the assembly
    await exec(
      `npx cdk deploy --app ${assemblyDir} --all --require-approval=never`,
      true
    );
  } finally {
    // Clean up.
    await fs.remove(assemblyDir);

    // TODO: Publish error to SNS topic?
  }
}

Limitations

• Lambda has a 15 minute timeout. If you expect your app to take longer to run, you can use CodeBuild.
  • Alternatively, once CloudFormation has started deploying the changeset, you can kill the process and exit the lambda.
• You should add SNS notifications to the CloudFormation stack so that you can update your provisioning app with success/failure.

Using Codebuild to Build the Deployer

Building a Docker container on a MacBook Pro is crazy slow. The following CodeBuild took ~5 minutes to build the deployer image and deploy it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

import { Stack, StackProps } from 'aws-cdk-lib';
import {
  BuildEnvironmentVariableType,
  BuildSpec,
  ComputeType,
  LinuxBuildImage,
  Project,
  Source,
} from 'aws-cdk-lib/aws-codebuild';
import { PolicyStatement, Role } from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';

interface CodeBuildStackProps extends StackProps {
  prefix: string;
  defaultBranch: string;
}
export class CodeBuildStack extends Stack {
  constructor(scope: Construct, id: string, props: CodeBuildStackProps) {
    const { prefix, defaultBranch } = props;
    super(scope, id, {
      stackName: prefix,
      ...props,
    });

    const project = new Project(this, 'Project', {
      // Have to manually add app credential in the UI
      source: Source.gitHub({
        owner: 'ibliskavka',
        repo: 'my-provisioning-app',
        branchOrRef: defaultBranch,
      }),
      projectName: prefix,
      environment: {
        buildImage: LinuxBuildImage.STANDARD_7_0,
        computeType: ComputeType.MEDIUM,
      },
      environmentVariables: {
        // This token is required to pull private npm packages from GH
        NPM_TOKEN: {
          type: BuildEnvironmentVariableType.SECRETS_MANAGER,
          value: 'github-npm-token',
        },
      },
      buildSpec: BuildSpec.fromObject({
        version: 0.2,
        phases: {
          build: {
            commands: [
              'echo $NPM_TOKEN',
              'npm config set //npm.pkg.github.com/:_authToken=$NPM_TOKEN',
              'npm i',
              'npm run deploy:deployer', // Remember this script from earlier?
            ],
          },
        },
      }),
    });

    const { account, region } = Stack.of(this);
    // Allows CodeBuild to execute CDK commands
    project.addToRolePolicy(
      new PolicyStatement({
        actions: ['sts:AssumeRole'],
        resources: [
          `arn:aws:iam::${account}:role/cdk-hnb659fds-deploy-role-${account}-${region}`,
          `arn:aws:iam::${account}:role/cdk-hnb659fds-image-publishing-role-${account}-${region}`,
          `arn:aws:iam::${account}:role/cdk-hnb659fds-file-publishing-role-${account}-${region}`,
        ],
      })
    );
  }
}

This is a fairly new certification from AWS focusing on MLOps and GenAI workloads on AWS. I highly recommend these practice exams on Udemy.

CDK Construct

The DataLake I am building requires that all data be encrypted with KMS, and deployed through CDK. This construct deploys the SAM template and configures all the necessary permissions. The OOTB role was too permissive, so I injected my own.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171

import { aws_sam as sam, Stack } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { Bucket } from 'aws-cdk-lib/aws-s3';
import { IKey } from 'aws-cdk-lib/aws-kms';
import {
  IGrantable,
  ManagedPolicy,
  PolicyStatement,
  Role,
  ServicePrincipal,
} from 'aws-cdk-lib/aws-iam';
import { ITable } from 'aws-cdk-lib/aws-dynamodb';
import { Constants } from '../../constants';
import { Function, IFunction } from 'aws-cdk-lib/aws-lambda';

export interface AthenaConnector {
  grantRead: (grantee: IGrantable) => void;
}

interface DynamoConnectorProps {
  prefix: string;

  /**
   * What ddb tables it has access to
   */
  tables: ITable[];

  /**
   * What encryption key to use
   */
  encryptionKey: IKey;
}
/**
 * Provides Athena access to Dynamo tables
 * Source: https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:292517598671:applications/AthenaDynamoDBConnector
 * NOTE: You must manually register the connector: https://docs.aws.amazon.com/athena/latest/ug/connect-data-source-serverless-app-repo.html
 * NOTE: You must manually add this environment variable to the lambda: spill_put_request_headers={"x-amz-server-side-encryption" : "aws:kms"}
 */
export class DynamoConnector extends Construct implements AthenaConnector {
  readonly bucket: Bucket;
  readonly lambda: IFunction;

  constructor(scope: Construct, id: string, props: DynamoConnectorProps) {
    super(scope, id);
    const { prefix, tables, encryptionKey } = props;

    const name = `${prefix}-ddb-connector`;

    this.bucket = new Bucket(this, 'Bucket', {
      bucketName: name,
      encryptionKey,
      // Reduce KMS Costs
      bucketKeyEnabled: true,
    });
    // Ensure all bucket data is encrypted
    this.bucket.addToResourcePolicy(
      new PolicyStatement({
        sid: 'Deny Unencrypted Objects',
        actions: ['s3:PutObject'],
        effect: Effect.DENY,
        resources: [this.arnForObjects('*')],
        principals: [new AnyPrincipal()],
        conditions: {
          StringNotEquals: {
            's3:x-amz-server-side-encryption': 'aws:kms',
          },
        },
      })
    );

    const role = new Role(this, 'Role', {
      roleName: name,
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromAwsManagedPolicyName(
          'service-role/AWSLambdaBasicExecutionRole'
        ),
      ],
    });

    // KMS Permissions
    encryptionKey.grantEncryptDecrypt(role);
    role.addToPolicy(
      new PolicyStatement({
        actions: ['kms:GenerateRandom'],
        // GenerateRandom does not use any account-specific resources, such as KMS keys.
        resources: ['*'],
      })
    );

    // S3 Permissions
    role.addToPolicy(
      new PolicyStatement({
        actions: ['s3:ListAllMyBuckets'],
        resources: ['*'],
      })
    );
    role.addToPolicy(
      new PolicyStatement({
        actions: [
          's3:GetObject',
          's3:ListBucket',
          's3:GetBucketLocation',
          's3:GetObjectVersion',
          's3:PutObject',
          's3:PutObjectAcl',
          's3:DeleteObject',
          's3:GetLifecycleConfiguration',
          's3:PutLifecycleConfiguration',
        ],
        resources: [this.bucket.bucketArn, this.bucket.arnForObjects('*')],
      })
    );

    // DDB Permissions
    tables.forEach((t) => {
      // Allows connector to perform queries
      t.grantReadWriteData(role);
    });

    // Glue/Athena Permissions
    role.addToPolicy(
      new PolicyStatement({
        actions: [
          'athena:GetQueryExecution',

          'glue:GetTableVersions',
          'glue:GetPartitions',
          'glue:GetTables',
          'glue:GetTableVersion',
          'glue:GetDatabases',
          'glue:GetTable',
          'glue:GetPartition',
          'glue:GetDatabase',

          // Allows Athena Query Explorer to show schema. Required on first run for Athena to discover schemas
          // Can probably be commented out later to avoid oversharing data
          'dynamodb:ListTables',
          'dynamodb:DescribeTable',
          'dynamodb:ListSchemas',
          'dynamodb:Scan',
        ],
        resources: ['*'],
      })
    );

    const { region } = Stack.of(scope);
    new sam.CfnApplication(this, 'App', {
      location: {
        applicationId: `arn:aws:serverlessrepo:${region}:292517598671:applications/AthenaDynamoDBConnector`,
        semanticVersion: '2024.25.1',
      },
      parameters: {
        LambdaRole: role.roleArn,
        // This is the name of the lambda function that will be created. This name must satisfy the pattern ^[a-z0-9-_]{1,64}$
        AthenaCatalogName: name,
        DisableSpillEncryption: 'false',
        SpillBucket: this.bucket.bucketName,
        KMSKeyId: this.bucket.encryptionKey.keyId,
        LambdaMemory: '512',
      },
    });

    this.lambda = Function.fromFunctionName(this, 'Connector', name);
  }

  grantRead(grantee: IGrantable) {
    this.bucket.grantRead(grantee);
    this.lambda.grantInvoke(grantee);
  }
}

Manual Steps

• You must manually register the connector with Athena
• To use KMS you must manually add this environment variable to the lambda: spill_put_request_headers={"x-amz-server-side-encryption" : "aws:kms"}

Drawbacks

This worked perfectly for small, homogeneous and relatively flat tables, but you may run into problems if:

• You have nested schemas
  • The connector could not infer nested object schemas well and would fail
• You have a lot of columns. I had a flattened table with ~1,000 columns, whenever I ran SELECT * the query failed. When I queried specific columns it succeeded.
  • Under the hood, the connector creates Projection Expressions. ~1000 columns created massive expressions that were rejected by DynamoDB.

Alternative for Large Tables

I still use the connector in a few places, but for the really wide table I went with another approach.

I am using DDB for operational processes during the day. The reporting data does not need to be realtime, and could be rebuilt nightly.

• A nightly Lambda triggers a DDB Table Export to S3 ($.50/GB)
• When the export drops, another Lambda
  • Unmarshalls the JSON data and stores it in a S3 staging prefix
  • Invokes an Athena Query to INSERT * INTO optimized_table SELECT * FROM json_table
    • json_table is a Glue table definition over the staged unmarshalled JSON data
    • optimized_table is a Glue table definition using Parquet, Snappy, and Partitioning
    • NOTE: The query also performs some deduping, which I omitted for brevity
• Reporting is done on optimized_table which is super compressed and fast.

Other Tips

• If your data is strictly transactional, you may be able to delete the DDB source data to reduce costs
  • If you have a lot of data, the Table Scan + BatchDelete operation may be too slow. In my case, the Lambda was timing out after 15 minutes
  • Solution: Delete and recreate the table using the SDK.

Troubleshooting

Athena Dynamo DB Connector returns 0 rows

Re-run your query with LIMIT 10, if the query succeeds check your lambda logs. You may be getting a permission denied when writing to the spill bucket.

Solution: Manually add this environment variable to the lambda:

• spill_put_request_headers={"x-amz-server-side-encryption" : "aws:kms"}

Athena Dynamo DB Connector Error: Invalid ProjectionExpression

Selecting certain fields works
SELECT foo, bar FROM "default"."datalake" limit 10;

Selecting all fields fails. I am guessing this is because connector puts all the columns into the ProjectionExpression.

SELECT * FROM "default"."datalake" limit 10;

1

GENERIC_USER_ERROR: Encountered an exception[software.amazon.awssdk.services.dynamodb.model.DynamoDbException] from your LambdaFunction[arn:aws:lambda:{region}:{account}:function:ddb-connector] executed in context[S3SpillLocation{bucket='ddb-connector', key='athena-spill/xxx', directory=true}] with message[Invalid ProjectionExpression: Expression size has exceeded the maximum allowed size; (Service: DynamoDb, Status Code: 400, Request ID: xxx)]

Then I decided to move my app from us-east-2 to us-east-1, so I ran terraform destroy and go walk my dog.

Lo and behold - that was a Noob move. Terraform deleted my state bucket and table too early, and the operation failed.

There is probably a way to make Terraform delete those resources last, but I think its safest to simply create them outside of your IaC. Here is a quick CLI script to make it simple.

Create Terraform State Resources Commands

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

aws s3api create-bucket --bucket my-tf-state
aws s3api put-bucket-versioning --bucket my-tf-state --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket my-tf-state --server-side-encryption-configuration '{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }
  ]
}'
aws dynamodb create-table \
    --table-name my-tf-lock \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST

The provider can do simple low-security tasks like associateLambda, or complex tasks like createInstance, which requires access to security-sensitive resources like kms and iam.

During a recent security review, we discovered that the same role policy was being used across all provider instances. This meant that if we used a low-security operation, such as associateLambda, the role would be granted access to high-security resources like kms and iam.

Solution 1 - Inject a Pre-Built Role

For the current project, we resolved the issue by introducing an optional role prop. This allowed the developer to select specific IAM permissions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// PSEUDO-CODE

class ConnectProvider {
  role: IRole;

  constructor(props: {role?: IRole}){
    if(!props.role){
      // Configures the default (overly permissive) permissions
      this.role = new Role(...);
    } else {
      // Uses the injected role
      this.role = props.role;
    }

    // Custom resource handler
    this.handler = new Function(... {role: this.role})
  }
}

const role = new Role(...);
role.addToPrincipalPolicy(
  new PolicyStatement({
    effect: Effect.ALLOW,
    actions: [
      'connect:AssociateLambdaFunction',
      'connect:DisassociateLambdaFunction'],
    resources: [instanceArn]
  })
);
const provider = new ConnectProvider({role});
provider.associateLambda(...)

Pros

• We were able to quickly patch the current app

Cons

• Each dependent app would have to be updated manually. We have A LOT!
• The app developer must know exactly which IAM permissions are required.

Solution 2 - Dynamically Generate the Role

I updated the custom resource constructs to dynamically build up the policy based on which resources are used, so I could roll out the update in a backward-compatible way.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

// PSEUDO-CODE

class ConnectProvider {
  role: IRole;

  constructor(props: {role?: IRole}){
    if(!props.role){
      this.role = new Role(...);
    } else if (props.role instanceof Role){
      // Convert to IRole to avoid manipulating the role
      this.role = Role.fromArn(props.role.roleArn)
    } else {
      this.role = props.role;
    }

    // Custom resource handler
    this.handler = new Function(... {role: this.role})
  }

  // Users call helper functions to create the custom resource
  associateLambda(id, instanceArn, lambda){
    if(this.role instanceof Role){
      // Dynamically update self-managed role
      this.role.addToPrincipalPolicy(
        new PolicyStatement({
          effect: Effect.ALLOW,
          actions: [
            'connect:AssociateLambdaFunction',
            'connect:DisassociateLambdaFunction'],
          resources: [instanceArn]
        })
      );
    }

    return new CustomResource({
      serviceToken: this.handler.functionArn,
      properties: {
        instanceArn,
        functionArn: lambda.functionArn
      }
    });
  }
}

Pros

• No manual intervention is needed for dependent apps. Simply upgrade the NPM package and redeploy.

Cons

• Resource deletion does not work properly.
  • If you had a custom resource like associateLambda, everything works fine because the role policy is updated before the resource is created.
  • But if you remove the custom resource in a future release, CloudFormation will update the role policy first (and remove the associated permission) before cleaning up the resource.
  • As a result, you encounter a permission error when cleaning up the associateLambda resource
• Circular dependencies
  • If you used the provider to createInstance and then used the instance ARN in another construct like associateLambda you will encounter a circular reference
  • Details
    • Invoke createInstance and get instance ARN
    • Invoke associateLambda using instance ARN
      • Instance ARN is used in the dynamic policy, resulting in a circular reference

Solution 3 - Mix of both

In the end, I decided to use a combination of both solutions. I created a ConnectProviderRoleBuilder to make it easier for developers to build the role.

Additionally, I also updated the ConnectProvider to automatically use the builder if a role is not provided.

This means that we can update existing apps without any manual intervention. If the app encounters the issues described in Solution 2 during ongoing development, the team can use the ConnectProviderRoleBuilder to generate an appropriate role quickly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

// PSEUDO-CODE

class ConnectProviderRoleBuilder {
  role: IRole;

  /**
    * Tracks if the provider was used to create an instance.
    * If so, we cannot limit role permissions to a specific instance
    * due to circular dependency.
    */
  private createdInstance: boolean = false;

  constructor(props: {existingRole?: IRole}){
    if(!props.role){
      this.role = new Role(...)
    } else if(props.role instanceof Role){
      // Ensures role is not manipulated by the builder
      this.role = Role.fromArn(props.existingRole.roleArn)
    } else {
      this.role = props.existingRole;
    }
  }

  /**
  * Create an instance ARN for permission filtering
  * If the provider was used to create the instance the ARN will be
  * `instance/*` to avoid circular dependency error
  * Assumes this provider will operate on a single instance.
  */
  instanceArn(instanceId: string): string {
    if (this.createdInstance) {
      // We can't reference the instanceId (circular ref)
      return `arn:aws:connect:${region}:${account}:instance/*`;
    } else {
      return `arn:aws:connect:${region}:${account}:instance/${instanceId}`;
    }
  }

  allow(actions, resources){
    if(this.role instanceof Role){
      // Only add permissions if the role is being managed by the construct.
      this.role.addToPrincipalPolicy(
        new PolicyStatement({
          effect: Effect.ALLOW,
          actions,
          resources
        })
      );
    }
  }

  // Helpers to add policy statements
  allowAssociateLambda(instanceId, ...functionArns){
    this.allow([
      'connect:AssociateLambdaFunction',
      'connect:DisassociateLambdaFunction'],
      [this.instanceArn(instanceId)]
    );

    // Update lambda resource policy to allow connect invoke

    // ...
  }
  allowCreateInstance(){
    this.createdInstance = true;
    this.allow(...)
    // ...
  }
  // ...
}

class ConnectProvider {
  builder: ConnectProviderRoleBuilder;
  role: IRole;
  constructor(props: {role?: IRole}){
    this.builder = new ConnectProviderRoleBuilder({role: props.role})
    this.role = this.builder.role;

    this.handler = new Function(... {role: this.role})
  }

  associateLambda(instanceId, lambda){
    this.builder.allowAssociateLambda(instanceId, lambda.functionArn)
    return new CustomResource({
      serviceToken: this.handler.functionArn,
      properties: {
        instanceId,
        functionArn: lambda.functionArn
      }
    })
  }
}

const myLambda: IFunction;

// Pre-build the role
const builder = new ConnectProviderRoleBuilder()
builder.allowAssociateLambda(instanceId, myLambda.functionArn)

const provider = new ConnectProvider({role: builder.role})
provider.associateLambda(instanceId, myLambda)

Conclusion

The simplest solution would have been to simply force the developer to inject a role but it would have created unnecessary developer friction because:

• My app used to deploy fine, but now I have to manually create a new role.
• I have no idea what is happening under the hood and which permissions are required, resulting in even more friction.

This solution was certainly more work, but it solved the problem with the least effort from the downstream developers.

No, go build secure and elegant tools!

I wanted something very easy to use, which has a lot of features and uses my core tools (AWS, Node, CDK, Markdown).

For this blog, I settled on Hexo, a static site generator. I have also used the more popular Hugo which solves the same problem.

For hosting, I am using AWS S3 with CloudFront and Route53. Deployed using CDK.

This setup is pretty easy if you are comfortable with Node and CDK. If you are not, you can refer to my blog repo on GitHub.

Prerequisites

• AWS Account
• AWS CLI Access
• Familiarity with NodeJS
• Familiarity with CDK is a plus

Highlights

• Init a new Hexo app
• npm install aws-cdk aws-cdk-lib
• Create a cdk/index.ts entry file
  • Change the string parameters to match your environment
  • Assuming Route53 domain and ACM Cert already exist. If not, comment out those resources to use the CloudFront URL.
• Create a cdk.json file
• Create the helper scripts in package.json
  • The publish script parameters (BucketName & DistributionId) will not be available until you run the deploy
  • More info on CDK package scripts

Extensions

Be sure to install a VS Code extension like Grammarly! I thought I was a pretty good writer, but after I installed it I had to go back and touch almost every post!

In the CloudFormation days, this would have been a significant overhaul if your app was not already set up for it. But with CDK, this is pretty easy with the Role.customizeRoles method.

While working on such a project, we bumped into this weird error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Error: Resolution error: Resolution error: PolicySynthesizer at 'PolicySynthesizer' should be created in the scope of a Stack, but no Stack found.
Object creation stack:
  at stack traces disabled.
Object creation stack:
  at Execute again with CDK_DEBUG=true to capture stack traces
    at _lookup (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/stack.js:1:3005)
    at _lookup (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/stack.js:1:3178)
    at Function.of (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/stack.js:1:2736)
    at Object.produce (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resource.js:1:4264)
    at Reference.resolve (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resource.js:1:4877)
    at DefaultTokenResolver.resolveToken (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resolvable.js:1:1401)
    at resolve (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/private/resolve.js:1:2711)
    at Object.resolve [as mapToken] (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/private/resolve.js:1:1079)
    at TokenizedStringFragments.mapTokens (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/string-fragments.js:1:1475)
    at DefaultTokenResolver.resolveString (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resolvable.js:4:362)

The above error doesn’t provide any meaningful info, but if you follow the instructions like so:

CDK_DEBUG=true npm run cdk -- synth

You get a much more meaningful error

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Error: Resolution error: Resolution error: PolicySynthesizer at 'PolicySynthesizer' should be created in the scope of a Stack, but no Stack found.
Object creation stack:
  at new Intrinsic (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/private/intrinsic.js:1:942)
  at new Reference (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/reference.js:1:599)
  at new <anonymous> (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resource.js:1:4806)
  at mimicReference (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resource.js:1:4802)
  at CacheTable.getResourceArnAttribute (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/resource.js:1:4185)
  at new Table (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/aws-dynamodb/lib/table.js:1:18313)
  at new CacheTable (/home/ibliskavka/git/data-lakedata/packages/caching/src/constructs/CacheTable.ts:20:5)
  ... (truncated)
Object creation stack:
  at Function.string (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/core/lib/lazy.js:1:953)
  at CacheTable.combinedGrant (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/aws-dynamodb/lib/table.js:1:13211)
  at CacheTable.grantReadWriteData (/home/ibliskavka/git/data-lakedata/node_modules/aws-cdk-lib/aws-dynamodb/lib/table.js:1:6157)
  ... (truncated)

The CacheTable.grantReadWriteData stack trace deals with permissions (which we are customizing), so I traced it down to this bit of code

1
2
3
4
5
6
7
8

export class CachingLambdaRole extends Role {
  constructor(scope: Construct, id: string, props: CachingLambdaRoleProps) {
    super(scope, id, {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
    });
    props.table.grantReadWriteData(this);
  }
}

When I replaced props.table.grantReadWriteData(this) with policy statements in the role inlinePolicy the error went away and I was able to deploy.

TL;DR;

1. Enable CDK_DEBUG
2. Check if your stack trace uses the grant API. If so, try replacing it with policy statements.

More Info

I was able to find only a few articles about this error message and none of them dealt with customizeRoles. I suspect this may be a bug in the CDK code because my app uses the grant API elsewhere in the code and it works as-is.

All I ask is that you attribute bliskavka.com as the original source of the article. If it’s on the web, please provide a link back to the original article on bliskavka.com.

I think that’s fair. Free content for your website in exchange for a link back.

Thanks!
Ivan Bliskavka
Author

Syndication policy idea attribution: Universe Today ;)

Retrospective

• Milestones (PR & body weight)

  • 12/30/21 - 212.5 @ ~163lb
  • In March ‘22, my friend Noel commented that if I wanted to hit my goal should be eating even if I had no appetite.
  • 6/16/22 - 215 @ ~175lb
  • 11/28/22 - 225 @ ~176lb
  • 3/16/23 - 230 @ ~176lb
  • 6/5/23 - 235 @ ~176lb
  • 9/7/23 - 240 @ ~176lb
  • 10/3/23 - 242.5 @ ~176lb
  • 10/26/23 - 250 @ ~179lb

• I tried to maintain a lean physique which slowed down my gains

• I have been doing 100+ pushups every day since last August. I think has kept me near my peak performance, so I hit PRs more often.

  • Previously my PRs were 3-6 months apart because I would slack off for a bit after each milestone

• It’s super important to keep a log and a consistent routine

• It’s less important to work near your maximum weight when not working towards a PR.

  • Instead do 5 sets at 5 reps at 80% of max. Increase reps or weight slowly over a couple of weeks.

The Team

Project Description

Habitat for Humanity, a renowned non-profit organization dedicated to building homes and better communities, faces several operational challenges that your hackathon team can help address:

Targeting New Audiences/Real-Time Inventory Management: Implement a solution that updates the online inventory in real-time, ensuring that customers are aware of available items and reducing discrepancies.

Background

Before starting any design work, our team decided to take a field trip to the Broward Restore, which was 30 minutes away.

The store manager Mark, was very gracious and spent 30 minutes talking about the store, the mission, the volunteers, and the amazing items that pass through his store.

Some interesting points

• Each store operates like a franchise and typically manages its own outreach, website, etc.
• Mark just came back from a conference with other ReStores where they shared ideas and approaches.
• Some stores (Los Angeles) have an excellent social media presence including viral videos on TikTok.
• Independent stores can have their own website, but there is no central platform for posting or managing inventory.
• The Broward store does not have a website, but they have a Facebook page.
• They move a lot of furniture, and the entire sales floor is typically replaced in a couple of weeks. Because each item is unique, maintaining a website inventory is a lot of overhead.

Donation Receiving

Mark said that 54% of donations are corporate. Typically the item is scratched, dented, or discontinued.

He typically uses Google Lens to scan the item to get a feel for what the item sells for. From this price, he discounts about 50%.

Proposal

• A SaaS-like platform that is owned and managed by the Habitat for Humanity “Corporate”
• Each store is a “tenant” on the platform and can manage its own inventory.
• The inventory is streamlined using a web app, on a mobile phone
  • Receiving takes a photo and adds a quality rating.
  • The app uploads a photo to the cloud and uses image recognition to identify the product.
  • If the product is IDed, meta-data and trending price is returned to the app.
    • Price is automatically discounted based on quality rating: 3-50%, 2-60%, 1-70%
  • If the item is not IDed, the app prompts for metadata and price.
  • Idea: If the current user is not the manager, the item will need to be approved by the manager.
    • Once approved, the item automatically shows up on the website and in the Point of Sale system.
  • A QR Code sticker is printed and attached to the item. (Bluetooth printer)
  • When the item is sold, the sticker is scanned, and it is automatically removed from the website.
• The SaaS platform dynamically builds a storefront for each tenant
  • Adds the ability to search all stores in the vicinity.
  • Customers can buy online and pick up in the store.
    • Orders not picked up within X days are a donation.
    • Online customer contact data can be imported into Outreach programs.

Technical Architecture

With 900 stores across the country and hundreds of thousands of customers, we need to consider scalability. We chose to build using managed services in the cloud for resilience, durability, and scalability.

Result

Unfortunately we didn’t win. Here are some presentation takeaways for future reference.

• Going first was a disadvantage. We didn’t know what to expect, and probably the judges didn’t either.
  • Even if our presentation was a 10/10, there is no way a judge would assign that to the first presentation without seeing all the other stuff. And there is no way they remembered ours after 16 presentations.
  • [Correction 10/3/23]: Going first was a challenge, we were up for it, and we did great!
• We should have figured out how to screen share an iPhone. Seeing it on a phone would be more impressive.
• Personally, I was more focused on the practicality of the idea and our ability to execute (in 8-12) hours
  • Since only 2 of the judges were asking technical questions, I assume the others were more business-minded. They weren’t judging on “can these people do it”, they were judging on “as described, will this idea impact a lot of people?”
  • Go for the big idea - not on the technical feasibility.
• Reverse Image Search using an AI model, we should have thrown in that extra jargon.

Links

• Demo Site
• App Recording
• Code Repo

This simplified my code, reduced costs, and improved performance.

The average API calls took 50-100ms! …but the start-up time was 5 seconds!!!

I was using a lambda warmer, so most clients didn’t hit the 5-second latency, but I wondered if I could reduce it.

TL;DR; Store an uncompressed JSON file in a Lambda Layer and rebuild it periodically

Before

1. A nightly process reads data from Dynamo DB and generates a zip file that looks like this:

   • cache.zip
     • tableA.json
     • tableB.json

2. The zip file is stored in S3

3. When the lambda starts, it loads the zip file from S3, and loads it into memory (Duration: 4-5 seconds)

   1	let cache: Record<string, ITable>;

4. All future requests are served from the in-memory cache (50-100ms)

Attempt 1

Instead of loading from S3, store the zip file in a lambda layer, and rebuild the layer every night.

Result: This reduced the start-up by 1 second.

Attempt 2

The layer improved the load time, so it was a good start, but I felt like it should be faster.

I added some basic timers to my warmup function and discovered that almost 3 seconds were spent parsing the 217 files from the zip file.

I updated the code to parse the table files in parallel but it only shaved off 500ms.

Attempt 3

Armed with the timing information, I understood that my load issue was computationally constrained - decompressing zip files requires CPU cycles.

I refactored my lambda layer cache to be a single cache.json file that looks like this

1
2
3
4

{
  "tableA": [],
  "tableB": []
}

The lambda can load the data using readFileSync('/opt/cache.json') and parse it immediately.

Presto: Start-up load time is 500ms - a 10x improvement!

Conclusion

If your data is relatively static you can cache it as a flat file in a Lambda layer to reduce app complexity, costs, and startup times. Use a Lambda to periodically query a new data set and publish a new Lambda Layer Version.

When using layers, your maximum package size is 250MB. By compressing your data files you can cache A LOT of data, but you will experience decompression latency.

Sign up here

Details

This is an IN-PERSON event

AI from the big three providers can make it a snap to quickly create what you need before trying to roll it yourselves.

This month: Ivan Bliskavka, a 7x Amazon Web Services Certified, will show us Amazon Lex and Connect to see what you can do with these services.

Amazon Lex is a Natural Language Understanding chat/voice bot that can simplify a complex inbound phone menu system by matching common phrases (and their variants) with user intents.

Summary

• Create an Amazon Connect instance and claim a phone number.
• Create a simple touch-tone menu that can route the call to the appropriate agent.
  • i.e.: Press 1 for purchasing, press 2 for technical support, and press 3 for the billing department.
• Enhance the menu by replacing it with an Amazon Lex Bot, which understands natural language to route to the appropriate agent.
  • i.e.: “Connect me to purchasing”, “I want to buy ${x}”, or “I need a quote on ${x}” will route to the purchasing department.
  • i.e.: “I need help”, “My ${x} is not working”, or “This computer is stupid” will route to the helpdesk.

5 years of AWS experience adds up. I only studied about 5 hours for the test using these Udemy practice exams.

This script stages the CloudFormation template and assets in a regional bucket. You can share the bucket with other accounts via Bucket Policy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Pass stage as command line parameter
STAGE=$1

# Load package version
VERSION=$(jq -r .version package.json)

# Listing all regions that support Amazon Connect
declare -a regions=("us-east-1" "us-west-2" "ap-southeast-1" "ap-southeast-2" "ap-northeast-1" "ca-central-1" "eu-central-1" "eu-west-2")

for REGION in "${regions[@]}"
do

   if [ $STAGE == "dev" ] && [ $REGION != "us-east-1" ]; then

    # Only deploy to us-east-1 in dev
    echo "$STAGE: Skipping $REGION"

   else

    BUCKET="my-app-${STAGE}-${REGION}"
    echo "$STAGE: Publishing $BUCKET"

    sam package -t template.yaml --s3-bucket $BUCKET --s3-prefix $VERSION --output-template-file .aws-sam/packaged.yaml --region $REGION
    aws s3 cp .aws-sam/packaged.yaml s3://$BUCKET/$VERSION/template.yaml --region $REGION

   fi
done

That it! Now you can deploy your Serverless application in any region, using only the CloudFormation console.

I usually don’t jump on the latest tech right away because there is just too much new stuff all the time! It’s been almost 4 months since ChatGPT 4 was released and having seen some very impressive examples, I am ready to jump in and learn something new.

I work in the Contact Center space, and we build voice and chat self-service bots for clients, so I have some vested interest in this technology.

Reading List

ChatGPT for Dummies - Pam Baker

I found this book to be an excellent introduction. It offered a good background in layman’s terms and was very easy to follow. It’s also very explicit about the limitations of the models. GPT is great at generating content, but it may be lies ;)

Developing Apps with GPT-4 and Chat GPT - Oliveier Caelen and Marie-Alice Blete

This was a great follow-up to the “For Dummies” book. The authors dive deeper into the differences between the OpenAI models and how they affect pricing.

As a developer, I found the code samples useful. The authors showed examples of using Completion and Insertion commands and covered some cost control topics.

The ChatGPT Gold Rush: Profiting from the AI Revolution Online: Prompt Engineering Mastery with ChatGPT - Mark Adelson

This book might be useful if you are looking to use ChatGPT to create a bunch of ChatGPT content for your blog.

The title looks like clickbait for a reason (it is). The book is full of lists that include a heading, and a small paragraph. Along with a list of tools and a prompt dump.

I felt like some sections were written by ChatGPT.

Modern Generative AI with ChatGPT and OpenAI Models - Valentina Alto

This is my favorite book so far. It covers some history, ethics, and limitations. The book also has code and use case examples.

Chapter 10 had a use case which is right up my alley. Using an Azure OpenAI instance to analyze call center transcripts.

Prompt

1
2
3
4
5
6
7
8
9
10
11

{{transcript}}
"Extract the following information from the above text:
Name and Surname
Reason for calling
Policy Number
Resolution
Initial Sentiment
Reason for Initial Sentiment
Final sentiment
Reason for final sentiment
Contact center improvement"

Output

What’s Next?

I am going to read a few more books on the topic. Blogs and the internet may have more current content, but I like the curated feel of a book.

In the meantime, I’m making lists for potential Proof of Concepts. This technology is amazing.

I already have 5 years of experience, my Solution Architect Pro, and Data Analytics certificates so this one wasn’t very difficult.

Study Guide

I watched the video course and did the following practice exams. Preparation took about 2 weeks.

• Video Course - Stephane Maarek and Riyaz Sayyad
• Practice Exam - Stephane Maarek and Abhishek Singh
• Practice Exam - Jon Bonso

Error

1
2

10:54:08 AM | CREATE_FAILED | AWS::Lambda::EventSourceMapping | *
Resource handler returned message: "The event source arn * and function * provided mapping already exists. Please update or delete the existing mapping with UUID *.

Resolution

Check if your CDK Metadata path has changed. Changing the stack or construct logical ID will change the metadata path.

If the logical ID change was intentional, manually delete the mapping and redeploy.

More Info

When the path changes CDK attempts to replace the mapping. Since CloudFormation must create a new resource before deleting the old, this threw a mapping already exists error.

In my case, I changed the stack logical ID, which triggered a replacement on the event source mapping.

This feature is awesome if you are pre-synthing CDK apps to CloudFormation Templates.

My first attempt was to use Asset but it used the file hash for the file name, which could change over time and is not user-friendly.

After playing around with a few methods, I discovered that you can do this.

1
2
3
4
5
6
7
8
9

class MyStack extends Stack{
  constructor(...){
    this.synthesizer.addFileAsset({
      sourceHash: 'RELEASE_NOTES',
      fileName: join(rootDir, 'RELEASE_NOTES.md'),
      packaging: FileAssetPackaging.FILE
    })
  }
}

In the above sample, the sourceHash is used to create the file name: RELEASE_NOTES.md. Since I publish templates under their version prefix, the file hash is unimportant to me!

More Context

We generate pre-synthed templates for very complex applications which are:

• shared with clients using a S3 bucket policy
• installed by a non-developer using CloudFormation QuickStart URLs
• capable of automatically updating itself by checking the bucket for a newer version and initiating a cloudformation update

Including a RELEASE_NOTES.md allows the template consumer to make a more informed decision on when to upgrade.

Cheers!

To be honest - it was peer pressure. My wife has always wanted to try skydiving, and our friends gifted her a ticket on her birthday.

So that question is settled - If my friends were jumping out of a perfectly good airplane, I would too 🤣.

It was a great experience! The facility is cozy and everyone was super friendly. They also had great coffee and an awesome selection of laptop stickers!

I spoke about my journey from being a plumber to being a Sr. Architect at the leading Amazon Connect integrator, there were jokes, laughter, and most importantly - questions!

Questions

How long did it take before you finally felt confident as a developer?

About 3 years - but you can get there sooner! The first 2 years I worked alone and didn’t have anyone to compare to, so I assumed I wasn’t very good but I kept studying and kept at it.

The 3rd year I worked with a team of sub-contractors on a project that got scrapped - so I figured that wasn’t a very good metric.

In my fourth year, I moved to another company with a full internal dev team - and I realized that in many respects I was ahead of guys with 10-15 years of experience. They knew A LOT, but much of that tech was already dead. I knew a lot MORE than them about the current tech.

Don’t forget that as a boot camp graduate, your education may be more current than the other devs. You still have to get experience, but I promise you are doing much better than your imposter syndrome is telling you.

What makes a good impression for new employees?

Typically the onboarding material includes reading, and assignments to get you familiar with the toolset. If you are spinning your wheels for 2 days on something - and the answer is in the reading, this is a strong negative.

1. You did not read the assignment and you just wasted 2 days because you are afraid to ask for help.
2. I would rather you do the reading, do some googling, and when you are stuck for 4-8 hours, reach out for help. This work requires communication - maybe the requirements were unclear - being able to clear that up on your own initiative is very impressive!

Corollary - don’t ask for help for every little thing. Do your due diligence. Also, take notes - don’t ask the same thing over and over again.

What type of questions do you usually ask at an interview?

I like to gauge a candidate’s skill level, and then dive a little deeper based on their response.

• For a Jr role, I may ask what is the difference between an RDBMS and a NoSQL DB.
  • Followup: When would you choose one over the other and why?
• For a Mid role, I would ask them to name the 3 types of testing (Unit, Integration, Manual).
  • What is the difference?
  • How would you design your code to make testing easier? (expecting mocks & dependency injection)
• For a Sr role, I would have a conversation about design patterns.
• I always love to ask about personal or favorite projects. You have to love this work to be exceptional at it. If you don’t have any code you are proud of - it’s a turn-off.
• What is the most impactful project you have participated in? What was the impact - social, financial, etc? You have to be able to speak about your work in terms of return-on-investment

What tips do you have for interviewing?

Be collaborative. If you are asked design questions they may be intentionally vague - ask the interviewer clarifying questions.

If the question requires a succinct factual response, and you don’t know the answer immediately, you can ask questions to get partial credit.

For example:

• I am not familiar with the term X, would you provide a short description and I may be able to able to answer the rest of the question.
• I have never used that in my work, but I think I know what it does based on your description - can I try to guess?

Remember not to B.S. during a technical interview. I prefer someone says honestly “I don’t know the answer, but am willing to try anyway if you give me some hints”.

In the real world, you will rarely have all the facts. You will have to reach out to the client who reported the bug and ask them to do a screen share so that you can see what is happening. Asking good questions is part of the job.

Would you recommend I get cloud certs?

I work in an AWS shop. I would take an AWS certified bootcamp grad over a non-certified grad every day. It’s probably less relevant if the company is not using AWS, but it still shows personal initiative and a willingness to learn.

If you are applying to work for an AWS Partner, they have to have a minimum amount of AWS Certified engineers to maintain their partner status. Having a cert in this case is extremely useful.

Understanding your job economics

During my talk, I mentioned a couple of concepts and a student asked me to elaborate on them.

Understanding the economics of your business

I work in the call center space - for each minute a customer is on a call it costs the company $1. If you have thousands of agents and tens of thousands of calls per month, these figures start to add up!

Reducing the average call handle time for a call by 1% for a sufficiently large business can reduce costs by millions over several years. If I can contribute that improvement consistently - my career is made.

Understanding the economics of your position

Typically IT is a cost center, not a profit center. You cost money. If you are doing your job well, you cost a lot of money.

You should NOT be doing work that a computer can do faster, better, and more consistently. For this reason, you should be on the lookout for automation, DevOps, scripts, or processes that reduce the amount of repetitive/manual work that you do, so you can focus on real innovations.

Example: If some nightly job runs for 3 hours - don’t bother spending 3 days speeding it up. It starts at 12:00, finishes by 3:00, and the first person that needs it is at 8:00. There is a 5-hour buffer - but you just cost the company 3 days for no economic benefit.

Not everything needs to be a script. If some work only happens once a month and takes 15 minutes - a How-To guide may be more appropriate. Next time you have to do the task, you can do it in 10 minutes, or simply hand it off to the new guy :)

If you can make yourself (and others) more productive through automation and/or documentation you will also go far.

Conclusion

It was fun to reflect on my journey and share some personal experiences with budding new developers!

Boca Code is an intense 40/hrs a week, for 10 weeks, software development bootcamp. Having spoke with the class and founders - I highly recommend them!

1
2
3
4
5
6
7
8
9
10
11
12

{
  "scripts": {
    "build": "tsc --noEmit",
    "cdk": "npm run build && cdk",
    "cdk:pipeline": "npm run build && cdk --app 'npx ts-node --prefer-ts-exts bin/pipeline.ts'",
    "test": "jest",
    "diff": "npm run cdk -- diff",
    "synth": "npm run cdk -- synth --quiet",
    "deploy": "npm run cdk -- deploy --all --require-approval never",
    "deploy:pipeline": "npm run cdk:pipeline -- deploy --all --require-approval never"
  }
}

This configuration allows us to create convenient scripts like deploy/diff/synth, but we still have the ability to pass in additional parameters like:

npm run deploy -- --no-rollback --profile default

I can also define multiple entry points

• cdk: Interact with the stack directly.
• cdk:pipeline: Deploy a CDK Pipeline which is able to patch the CDK app when new changes are pushed.

A pipeline execution can be slow, so being able to circumvent the pipeline in a dev/sandbox environment is extremely useful.

• The large app had ~50 assets, so a Publish Assets CodeBuild was generated for each asset (UGLY).
• Often the publish step took longer than the deploy because the CodeBuilds were queued and only 5-10 ran at one time.
• One of our client accounts hit a CodeBuild soft limit :(

Solution

After some Google-fu, I found the solution on the cdk.dev Slack channel!

1
2
3
4

new CodePipeline(this, 'Pipeline', {
  // Set this to false!
  publishAssetsInParallel: false,
});

By default, publishAssetsInParallel is on, creating the issues listed above. After I turned it off I had immediate gratification!

Results

• Only 1 CodeBuild was generated to publish assets (prettier).
• Deployment time for the ~50 asset project was reduced by 20 minutes.
• On another project, the self-mutate step was reduced from ~60 to 6 minutes!

Your mileage may vary, but it’s a very simple change that you can validate quickly. Good Luck!

I already had a lot of experience with Kinesis, Athena, and RedShift but there is A LOT of material in this exam. I highly recommend practice exams by Stephane Maarek because they explain why you got something wrong, and you can adjust before taking the real thing!

The file contains settings like Cognito Pool ID, region, and/or branding information.

I used to write my own Custom Resource for building this file but recently discovered a simpler CDKV2 way! Credit: aws-cdk-v2-three-tier-serverless-application/

I am copy-pasting the relevant code for future reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import {
  AwsCustomResource,
  AwsCustomResourcePolicy,
  PhysicalResourceId,
} from 'aws-cdk-lib/custom-resources';
import { PolicyStatement } from 'aws-cdk-lib/aws-iam';

const myConfig = {
  prefix: 'some-static-value',
  userPoolId: userPool.userPoolId,
  // Call toString() on tokens
  region: Aws.REGION.toString(),
  etc,
};

new AwsCustomResource(this, 'SpaConfig', {
  logRetention: RetentionDays.ONE_DAY,
  onUpdate: {
    action: 'putObject',
    parameters: {
      Body: JSON.stringify(myConfig),
      Bucket: websiteBucket.bucketName,
      CacheControl: 'max-age=0, no-cache, no-store, must-revalidate',
      ContentType: 'application/json',
      Key: 'config.json',
    },
    physicalResourceId: PhysicalResourceId.of('config'),
    service: 'S3',
  },
  policy: AwsCustomResourcePolicy.fromStatements([
    new PolicyStatement({
      actions: ['s3:PutObject'],
      resources: [websiteBucket.arnForObjects('config.json')],
    }),
  ]),
});

Pitfall

I have a config.json in the SPA deploy assets also, so with each rebuild and redeploy, CloudFormation was replacing and overwriting the config.json from SPA assets. I added this to the myConfig object above to force the custom resource to execute with each cdk deploy

1

buildTime: new Date().toISOString();

Summary

1. Create a client-specific staging bucket
2. Share the bucket with the client account via the Bucket Policy
3. Synth the stack to the staging bucket
4. Share template URL with client
5. The client can install using the URL in CloudFormation web console with their user credentials

App Staging Bucket Policy

1
2
3
4
5
6
7
8
9
10
11
12

{
  "Sid": "MyClient",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::DEV_ACCOUNT_ID:root",
      "arn:aws:iam::PROD_ACCOUNT_ID:root"
    ]
  },
  "Action": ["s3:GetObject", "s3:GetObjectVersion"],
  "Resource": "arn:aws:s3:::app-staging-bucket/*"
}

Usage

1. Install CDK Assets npm i -D cdk-assets

2. Customize the stack synthesizer to use your custom staging bucket

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

   const app = new cdk.App(); new MyApp(app, 'template', { someParam: 'someValue', synthesizer: new DefaultStackSynthesizer({ fileAssetsBucketName: 'app-staging-bucket', // Use a custom role which has access to the asset bucket fileAssetPublishingRoleArn: 'my-client-staging-role', // Consider using a build date or version bucketPrefix: '2.4.1', // The client account does not need to be bootstrapped generateBootstrapVersionRule: false, }), }); app.synth();

3. Run cdk synth to generate your assets.

4. Modify cdk.out/template.assets.json to make the template file name more predictable

   • find the entry with sourcePath=template.template.json
   • modify its objectKey to something like 2.4.1/template.json
   • (you should probably write some code to automate this)

5. Run cdk-assets -v -p ./cdk.out/template.assets.json publish

6. Share your template URL with the client. It will look something like:
   https://app-staging-bucket.s3.amazonaws.com/2.4.1/template.json

7. Client can install the app using the CloudFormation web console.

Simpler Template Output

Not sure what the side effects of these are, but this produces a simpler template with less CDK metadata.

cdk synth --path-metadata false --version-reporting false

cdk.json

1
2
3
4
5

{
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": false
  }
}

Conclusion

This has been very helpful for creating installers that are accessible to non-developers and usable in beginner AWS environments. I hope it saved you some head-scratching!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

StartQueryFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: src/lambda/search.start
    Policies:
      - S3ReadPolicy:
          BucketName: !Ref DataBucket
      - S3CrudPolicy:
          BucketName: !Ref AthenaResultsBucket
      - AthenaQueryPolicy:
          WorkGroupName: !Ref AthenaWorkGroup
      - Statement:
        - Effect: Allow
          Action:
          - glue:GetTable
          Resource:
          - !Sub arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog
          - !Sub arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/${GlueDatabase}
          - !Sub arn:aws:glue:${AWS::Region}:${AWS::AccountId}:table/${GlueDatabase}/*

GetResultFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: src/lambda/search.results
    Policies:
      - S3CrudPolicy:
          BucketName: !Ref AthenaResultsBucket
      - AthenaQueryPolicy:
          WorkGroupName: !Ref AthenaWorkGroup

Cheers!

Don’t judge me on the single AZ. I am running a single task in Fargate and only need one instance.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

const { vpc, az, region, account } = props;

const fileSystem = new FileSystem(this, 'Efs', {
  vpc,
  performanceMode: PerformanceMode.GENERAL_PURPOSE,
  vpcSubnets: {
    subnetType: ec2.SubnetType.PUBLIC,
    onePerAz: true,
    availabilityZones: [az],
  },
});

const accessPoint = new AccessPoint(this, 'AccessPoint', {
  fileSystem: fileSystem,
});

const task = new ecs.FargateTaskDefinition(this, 'Task', {
  cpu: 256,
  memoryLimitMiB: 512,
});

const volumeName = 'efs-volume';

task.addVolume({
  name: volumeName,
  efsVolumeConfiguration: {
    fileSystemId: fileSystem.fileSystemId,
    transitEncryption: 'ENABLED',
    authorizationConfig: {
      accessPointId: accessPoint.accessPointId,
      iam: 'ENABLED',
    },
  },
});

const container = task.addContainer('Container', {
  image: ecs.ContainerImage.fromAsset('./container'),
  portMappings: [{ hostPort: 80, containerPort: 80 }],
});

container.addMountPoints({
  containerPath: '/mount/data',
  sourceVolume: volumeName,
  readOnly: false,
});

task.addToTaskRolePolicy(
  new iam.PolicyStatement({
    actions: [
      'elasticfilesystem:ClientRootAccess',
      'elasticfilesystem:ClientWrite',
      'elasticfilesystem:ClientMount',
      'elasticfilesystem:DescribeMountTargets',
    ],
    resources: [
      `arn:aws:elasticfilesystem:${region}:${account}:file-system/${fileSystem.fileSystemId}`,
    ],
  })
);

task.addToTaskRolePolicy(
  new iam.PolicyStatement({
    actions: ['ec2:DescribeAvailabilityZones'],
    resources: ['*'],
  })
);

I hope this saves someone a headache!

{dynamo table name} already exists in stack {current stack arn}

Upon further investigation it turned out that one of our internal CDK modules for building the dynamo tables had changed how it was naming the construct, resulting in a different CloudFormation Logical ID. Since we were explicitly naming the table, the stack update could not complete, because the table already exists.

This app is a self-contained NPM module, so I wasn’t able to modify the construct, but I came up with a workaround.

1. Back up the dynamo table data
2. Copy the raw stack YAML from CloudFormation and save it on your computer
3. Delete the table resource and references from the template
4. Update the stack with the new template using the CloudFormation web console
5. Verify that the table was deleted (and not retained) by CloudFormation
6. Redeploy the CDK app, which will create a new table
7. Restore the table data

I had used the typical index.tsx import and I could see the CSS output, but for some reason, it did not work.

1
2

// index.tsx
import '@aws-amplify/ui/dist/style.css';

After much Googling, it turns out that Electron React Boiler plate does some webpack-ing magic under the hood.

The correct way to add the css is to import it in App.global.css:

1
2

/* App.global.css */
@import '~@aws-amplify/ui/dist/style.css';

Consulting requires you to work within the client’s parameters. Some clients have internal standards and want you to deliver your white-label CDK app as CloudFormation. Call me old fashioned but…

I dont expect Apple to rewrite their products in TypeScript because that is my current favorite.

Joking aside, in consulting this is a pretty common ask, so you must be prepared to deal with it.

On a related note, some AWS Clients have compliance or security restrictions that make it very difficult to get CLI access to deploy using CDK.

Fortunately, you can synth CDK apps into plain old CloudFormation, package it to an S3 bucket, and deploy it from the CloudFormation web console.

TL;DR;

Check out the demo project on GitHub.

Initial Product Stack

Our demo stack will create a bucket, and name it based on the account, region, and client name.

1
2
3
4
5
6
7
8
9
10
11
12

interface MyProductProps extends StackProps {
  client: string;
}
export class MyProduct extends Stack {
  constructor(scope: Construct, id: string, props: MyProductProps) {
    super(scope, id);
    const bucketName = `${props.env.account}-${props.env.region}-${props.client}`;
    new Bucket(this, 'Bucket', {
      bucketName: bucketName,
    });
  }
}

The env Field

Typically you will pass an env field to your CDK stack props like this:

1
2
3
4
5
6
7
8

const app = new cdk.App();
new MyProduct(app, 'my-product', {
  env: {
    account: '123456879123',
    region: 'us-east-1',
  },
  client: 'foo',
});

And it’s very easy to access account and region from props like this:

1

const bucketName = `${props.env.account}-${props.env.region}-${props.client}`;

The synthesized template will look like this:

1

BucketName: '123456789123-us-east-1-foo'

This is very intuitive and works great for CDK deploys, but IS NOT PORTABLE. The synthesized template will contain hard-code account and region values.

The above props are evaluated at synth-time. We want the account and region values to be evaluated at deploy-time.

Introducing Stack.of()

In plain CloudFormation you wouldn’t hard-code the account and region information, you would use pseudo-functions like: !Ref AWS::AccountId and !Ref AWS::Region which get evaluated at deploy-time.

Let’s rewrite our stack params and exclude the optional env field.
Also, let’s rewrite our stack to use Stack.of when env is not available.

1
2
3
4
5
6
7
8
9
10
11
12

export class MyProduct extends Stack {
  constructor(scope: Construct, id: string, props: MyProductProps) {
    super(scope, id);

    const stack = props.env || Stack.of(this);
    const bucketName = `${stack.account}-${stack.region}-${props.client}`;

    new Bucket(this, 'Bucket', {
      bucketName: bucketName,
    });
  }
}

If you look at the synthesized CloudFormation template, the result should look familiar:

1

BucketName: !Sub '${AWS::AccountId}-${AWS::Region}-foo'

Bonus: The above approach works whether env is passed in or not, so we should use it by default.

CloudFormation Parameters and Tokens

The other major requirement for portable apps is CloudFormation Parameters. So far, we have passed in our client name as a string. This is very convenient for CDK deploys so we want to keep this format, but let’s rewrite our stack to use CloudFormation parameters so that we can have a deploy-time parameter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

export class MyProduct extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);
  }

  // Separate build step from constructor to allow inheriting stack to add properties.
  protected build(props: MyProductProps) {
    const stack = props.env || Stack.of(this);
    const bucketName = `${stack.account}-${stack.region}-${props.client}`;

    new Bucket(this, 'Bucket', {
      bucketName: bucketName,
    });
  }
}

export class MyPortableProduct extends MyProduct {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    // Add CloudFormation parameter
    const client = new CfnParameter(this, 'Client', {
      type: 'String',
      description: 'Used for naming',
    });

    // Build the base stack, using the client parameter as a string token
    this.build({
      client: client.valueAsString,
    });
  }
}

We just extended our product stack to make it portable, with just a slight modification in the base stack! If you were to synthesize MyPortableStack, your bucketName would look something like this:

1

BucketName: !Sub '${AWS::AccountId}-${AWS::Region}-${Client}'

Warning!

valueAsString generates a token. Tokens don’t represent your data, so don’t perform string manipulations or conditionals with tokens. I will cover how to deal with this in another post.

Synthesizing a Template

So far we have been making our CDK app portable, but we want CDK to generate a plain CloudFormation template.

Create a new bin/product.ts file

1
2
3
4
5
6
7
8

const app = new cdk.App();
new MyPortableProduct(app, 'my-product');

const output = app.synth();
const outStack = output.stacks[0];

const templatePath = path.resolve('./cdk.out/template.yaml');
fs.writeFileSync(templatePath, YAML.stringify(outStack.template));

Create a new synth script in package.json. Notice this script excludes metadata and version reporting, this is not required for plain CloudFormation and makes our output template a lot cleaner.

1
2
3
4
5

{
  "scripts": {
    "synth:product": "tsc && npx cdk --app 'npx ts-node --prefer-ts-exts bin/product.ts' --path-metadata false --version-reporting false synth --quiet"
  }
}

Run the script npm run synth:product

Your cdk.out/template.yaml should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Parameters:
  Client:
    Type: String
    Description: Used for naming
Resources:
  Bucket83908E77:
    Type: AWS::S3::Bucket
    Properties:
      BucketName:
        Fn::Join:
          - ''
          - - Ref: AWS::AccountId
            - '-'
            - Ref: AWS::Region
            - '-'
            - Ref: Client
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain

CloudFormation Logical Ids

If you are upgrading an existing CloudFormation or SAM app to CDK, you will notice that the bucket logical ID (Bucket83908E77) is auto-generated.

If our legacy template logical ID was Bucket, this would force the bucket to be recreated if you updated the stack.

We must update the build step to use overrideLogicalId to specify our logical ID.

1
2
3
4

const bucket = new Bucket(this, 'Bucket', {
  bucketName: bucketName,
});
(bucket.node.defaultChild as CfnBucket).overrideLogicalId('Bucket');

Now the template has our expected logical ID, and we can update our stack without losing our data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Parameters:
  Client:
    Type: String
    Description: Used for naming
Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName:
        Fn::Join:
          - ''
          - - Ref: AWS::AccountId
            - '-'
            - Ref: AWS::Region
            - '-'
            - Ref: Client
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain

Conclusion

This is a lot of extra work if you are building internal AWS apps, but if you are upgrading or building a product that will be installed in client accounts, you need the flexibility to support different deployment mechanisms.

Check out the GitHub Repo for full project setup.

I was trying to move files and kept getting denied. Then I simply tried renaming files and also got an access denied. I was able to upload and delete files though.

I looked at CloudTrail, and there was no obvious access denied.

I enabled S3 full control s:*, but I was still getting the same error.

I then tried to rename and move files using the CLI, which worked just fine.

Finally, I opened the IAM visual editor and created an entirely new policy for the user, which worked. After inspecting the policy, I noticed some permissions were scoped to the * resource, and not scoped to any ARN. After I added those to my CloudFormation, the user was able to move files using the web console.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# S3 CRUD policy
- Effect: Allow
  Action:
    - s3:GetObject
    - s3:GetObjectAcl
    - s3:ListBucket
    - s3:GetBucketLocation
    - s3:GetObjectVersion
    - s3:PutObject
    - s3:PutObjectAcl
    - s3:GetLifecycleConfiguration
    - s3:PutLifecycleConfiguration
    - s3:DeleteObject
  Resource:
    - !Sub arn:${AWS::Partition}:s3:::${DataBucket}
    - !Sub arn:${AWS::Partition}:s3:::${DataBucket}/*

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Additional S3 Permissions
- Effect: Allow
  Action:
    - s3:ListStorageLensConfigurations
    - s3:ListAccessPointsForObjectLambda
    - s3:GetAccessPoint
    - s3:PutAccountPublicAccessBlock
    - s3:GetAccountPublicAccessBlock
    - s3:ListAllMyBuckets
    - s3:ListAccessPoints
    - s3:ListJobs
    - s3:PutStorageLensConfiguration
    - s3:ListMultiRegionAccessPoints
    - s3:CreateJob
  Resource: '*'

After some testing, turns out that s3:ListAllMyBuckets permission is required to be able to move and rename files using the web UI!

1
2
3
4