Using CDK as CloudFormation template generator

I always thought that CDK is just a fancy CloudFormation template generator. That's why I recently proposed to one of my clients to generate parametrized CloudFormation template for their simple use-case in CDK. Boy was I naive.

Why considering CloudFormation at all?

TL;DR - Portability.

If you automated anything on AWS using CDK you might ask why would anyone want to use CloudFormation anymore? The main reason is simplicity and portability.

As a TypeScript/CDK developer I don't use CloudFormation directly. I uderstand it as crucial part of the infrastructure automation but I would never want to write down any config in yaml or json file. I am all about the code. I want to write it and then run it.

But when you want to create something for your customers who are used to ClickOps you want to go as simple as possible. The greatest part about CloudFormation is that you only need single yaml or json file which you can easily upload to AWS Console and you are good to go. You don't need to install any toolchain, you don't need to run anything, all you need is to have a file, upload it and then just click around.

Describing the infrastructure via CloudFormation template sound like a dream. The best part is that you can even parametrize it.

Why generate CloudFormation templates with CDK?

TL;DR - Flexibility.

Yes, I know, I can just describe exactly the same I've described in previous section but now I would need to point out the advantages of CDK.

If you are using CDK you are already generating the CloudFormation templates. 🙃

The biggest advantage of CDK vs CloudFormation is its flexibility. You type code which is dynamic. You can dynamically generate templates which will be deployed later based on different configurations and conditions. This is something which helped us not to loose our minds since 2019 because it came with such a nice code abstractions around simple config files. But as always with each abstraction we are further from the source - in this case each abstraction make us to forget about how CloudFormation actually works.

Runtime vs deployment time evaluation

The crucial difference between CDK and CloudFormation is how the thing you write down is being interpreted. There are two "times" which the code/template is being interpreted in.

Runtime - whenever you run cdk synth (cdk synth is also run as part of cdk deploy) your code is being evaluated and the CloudFormation template (and other things) are being generated based on your code logic.

Deployment time - this is the time when the CloudFormation service processes your template - doesn't matter if you've pushed it to CloudFormation through CDK or you've uploaded it through AWS console. The template is already generated and there is not much CDK can do here anymore.

Most of the time I don't think twice about this distinction becaue I don't use CloudFormation templates directly. Whatever needs to be dynamic I make it dynamic in runtime. But since we want to use plain CloudFormation template there is no such thing as runtime in this case so we need to make things dynamic in deployment time.

CloudFormation dynamic parts

Before CDK was introduced in 2019 the CloudFormation needed to be somehow dynamic. Otherwise you would need to write template for each use case. CloudFormation contains two ways of making things dynamic.

CloudFormation parameters - before you deploy the template you can pass various parameters to it and the deployment behave based on these. For example you can pass number of instances to be deployed which will be passed as number and interpreted in deployment time.

Intristic functions - you can use various functions which are dynamically evaluated within the deployment time. You can use various conditions, mapping, splitting, joining etc.

Using CloudFormation parameters

Simple usage (not so simple)

Usually you want different configuration for your production and development resources. For instance you want your database to be resilient in the production so you enable multiAz in RDS. On the other hand, in development you might want to save some money by disabling this feature.

This should be easy, right? Just introduce simple boolean CloudFormation parameter and we are done.

When you look into CloudFormation parameters documentation you can see that there is no such thing as boolean parameter. So we need to come up with some workaround here.

This is what I ended up with:

import { App, CfnCondition, CfnParameter, Fn, Stack } from "aws-cdk-lib";
import { Vpc } from "aws-cdk-lib/aws-ec2";
import {
  DatabaseInstance,
  DatabaseInstanceEngine,
  MysqlEngineVersion,
} from "aws-cdk-lib/aws-rds";

const app = new App();
const stack = new Stack(app, "MyStack");
declare const vpc: Vpc;

const multiAzParameter = new CfnParameter(stack, ",multiAz", {
  type: "String",
  description:
    "Specifies if the database instance is a multiple Availability Zone deployment.",
  default: "true",
  allowedValues: ["true", "false"],
});

const multiAzCondition = new CfnCondition(stack, "MultiAzCondition", {
  expression: Fn.conditionEquals(multiAzParameter.valueAsString, "true"),
});

const database = new DatabaseInstance(stack, "Database", {
  engine: DatabaseInstanceEngine.mysql({
    version: MysqlEngineVersion.VER_8_4_8,
  }),
  vpc,
  multiAz: Fn.conditionIf(
    multiAzCondition.logicalId,
    true,
    false,
  ) as unknown as boolean,
});

As you can see in order to add a boolean parameter which is passed to a template you need to do lot of nasty things inside your code. You need to make sure that your String parameter is correctly translated to boolean value in the deployment time not in the runtime.

CloudFormation complex behaviours

Unfortunatelly it is not easy to describe complex conditional behaviours inside CloudFormation templates.

As I come from the CDK world I thought it would be easy to create a parameter like vpcId with default value of createNew and then easily add condition based on this value - for createNew I would like to create new VPC, for any other value I would love to try to use the existing VPC. Something like this:

import { App, CfnParameter, Stack } from "aws-cdk-lib";
import { Vpc } from "aws-cdk-lib/aws-ec2";

const app = new App();
const stack = new Stack(app, "MyStack");

const vpcIdParameter = new CfnParameter(stack, "vpcId", {
  type: "String",
  default: "createNew",
});

const vpc =
  vpcIdParameter.valueAsString === "createNew"
    ? new Vpc(stack, "Vpc")
    : Vpc.fromVpcAttributes(stack, "Vpc", {
        vpcId: vpcIdParameter.valueAsString,
        availabilityZones: ["euc1-az1", "euc1-az2", "euc1-az3"],
      });

note

If you want to use CfnParameter for referencing VPC you need to use Vpc.fromVpcAttributes() function as Vpc.fromLookup() does not support token resolution and you will get following error:

ValidationError: All arguments to Vpc.fromLookup() must be concrete (no Tokens)
    at path [MyStack] in aws-cdk-lib.Stack

Sounds great, right? Not so fast though. This won't work unfortunatelly.

The problem is that whenever we run cdk synth the final template is being rendered. This means that if we don't provide any parameters, the default parameter is used and the template with default behaviour is rendered.

Click to show generated YAML file.

Parameters:
  vpcId:
    Type: String
    Default: createNew
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAA/zPSMzIx1DNQTCwv1k1OydbNyUzSqw4uSUzO1nFOywtILErMTS1JLdIJSi3OLy1KTo1W0lKKrdXJy09J1csq1i8zMtUz0zNUzCrOzNQtKs0rycxN1QuC0ADx7nisWwAAAA==
    Metadata:
      aws:cdk:path: MyStack/CDKMetadata/Default
    Condition: CDKMetadataAvailable
Conditions:
  CDKMetadataAvailable:
    Fn::Or:
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - af-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-3
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-3
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-4
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-northwest-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-south-2
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-3
          - Fn::Equals:
              - Ref: AWS::Region
              - il-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - me-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - me-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - sa-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-2
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-1
      - Fn::Equals:
          - Ref: AWS::Region
          - us-west-2

As you can see there is no condition here regarding our VPC creation. CDK got the default parameter as we haven't provided any in the time of the synth and it rendered the template where our parameter now is useless.

In this case the only solution I come up with is to generate two separate CloudFormation templates - one for using the existing VPC and one for creating new VPC. The example code would look like this:

import { App, CfnParameter, Stack } from "aws-cdk-lib";
import { SubnetType, Vpc } from "aws-cdk-lib/aws-ec2";

const app = new App();
const stack = new Stack(app, "MyStack");

const existingVpc = stack.node.tryGetContext("existingVpc") === "true";

if (existingVpc) {
  const vpcIdParameter = new CfnParameter(stack, "vpcId", {
    type: "String",
    default: "createNew",
  });
  const availabilityZonesParameter = new CfnParameter(
    stack,
    "availabilityZones",
    {
      type: "CommaDelimitedList",
    },
  );

  const vpc = Vpc.fromVpcAttributes(stack, "Vpc", {
    vpcId: vpcIdParameter.valueAsString,
    availabilityZones: availabilityZonesParameter.valueAsList,
  });

  // your code using existing VPC goes here
} else {
  const vpc = new Vpc(stack, "Vpc", {
    restrictDefaultSecurityGroup: false,
    natGateways: 0,
    subnetConfiguration: [
      {
        cidrMask: 24,
        name: "ingress",
        subnetType: SubnetType.PUBLIC,
      },
    ],
  });

  // your code using newly created VPC goes here
}

tip

When you use CDK just for generating the CloudFormation templates you probably want to set restrictDefaultSecurityGroup: false, in your Vpc construct.

When this parameter is set to true (default value) the CDK generates Lambda function which is deployed as CustomResource which deletes all inbound/outbound rules from the default security group within VPC.

In this case we can generate two separate CloudFormation templates for our two use-cases.

When we want to generate template where user can provide existing VPC ID and availability zones we provide context like this:

cdk synth MyStack --context existingVpc=true

Click to show generated YAML file

Parameters:
  vpcId:
    Type: String
    Default: createNew
  availabilityZones:
    Type: CommaDelimitedList
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAA/zPSMzIx1DNQTCwv1k1OydbNyUzSqw4uSUzO1nFOywtILErMTS1JLdIJSi3OLy1KTo1W0lKKrdXJy09J1csq1i8zMtUz0zNUzCrOzNQtKs0rycxN1QuC0ADx7nisWwAAAA==
    Metadata:
      aws:cdk:path: MyStack/CDKMetadata/Default
    Condition: CDKMetadataAvailable
Conditions:
  CDKMetadataAvailable:
    Fn::Or:
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - af-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-3
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-3
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-4
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-northwest-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-south-2
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-3
          - Fn::Equals:
              - Ref: AWS::Region
              - il-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - me-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - me-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - sa-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-2
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-1
      - Fn::Equals:
          - Ref: AWS::Region
          - us-west-2

When we want to create new VPC we can run

cdk synth MyStack --context existingVpc=false

# or without context

cdk synth MyStack

Click to show generated YAML file.

Resources:
  Vpc8378EB38:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyStack/Vpc
    Metadata:
      aws:cdk:path: MyStack/Vpc/Resource
  VpcingressSubnet1Subnet059F22C6:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      CidrBlock: 10.0.0.0/24
      MapPublicIpOnLaunch: true
      Tags:
        - Key: aws-cdk:subnet-name
          Value: ingress
        - Key: aws-cdk:subnet-type
          Value: Public
        - Key: Name
          Value: MyStack/Vpc/ingressSubnet1
      VpcId:
        Ref: Vpc8378EB38
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet1/Subnet
  VpcingressSubnet1RouteTable804C7A26:
    Type: AWS::EC2::RouteTable
    Properties:
      Tags:
        - Key: Name
          Value: MyStack/Vpc/ingressSubnet1
      VpcId:
        Ref: Vpc8378EB38
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet1/RouteTable
  VpcingressSubnet1RouteTableAssociation5DF54E70:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: VpcingressSubnet1RouteTable804C7A26
      SubnetId:
        Ref: VpcingressSubnet1Subnet059F22C6
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet1/RouteTableAssociation
  VpcingressSubnet1DefaultRoute4188A546:
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: VpcIGWD7BA715C
      RouteTableId:
        Ref: VpcingressSubnet1RouteTable804C7A26
    DependsOn:
      - VpcVPCGWBF912B6E
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet1/DefaultRoute
  VpcingressSubnet2SubnetF175D754:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      CidrBlock: 10.0.1.0/24
      MapPublicIpOnLaunch: true
      Tags:
        - Key: aws-cdk:subnet-name
          Value: ingress
        - Key: aws-cdk:subnet-type
          Value: Public
        - Key: Name
          Value: MyStack/Vpc/ingressSubnet2
      VpcId:
        Ref: Vpc8378EB38
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet2/Subnet
  VpcingressSubnet2RouteTable8B0E23A7:
    Type: AWS::EC2::RouteTable
    Properties:
      Tags:
        - Key: Name
          Value: MyStack/Vpc/ingressSubnet2
      VpcId:
        Ref: Vpc8378EB38
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet2/RouteTable
  VpcingressSubnet2RouteTableAssociation39E1BA3A:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: VpcingressSubnet2RouteTable8B0E23A7
      SubnetId:
        Ref: VpcingressSubnet2SubnetF175D754
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet2/RouteTableAssociation
  VpcingressSubnet2DefaultRoute18203BC3:
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: VpcIGWD7BA715C
      RouteTableId:
        Ref: VpcingressSubnet2RouteTable8B0E23A7
    DependsOn:
      - VpcVPCGWBF912B6E
    Metadata:
      aws:cdk:path: MyStack/Vpc/ingressSubnet2/DefaultRoute
  VpcIGWD7BA715C:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: MyStack/Vpc
    Metadata:
      aws:cdk:path: MyStack/Vpc/IGW
  VpcVPCGWBF912B6E:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId:
        Ref: VpcIGWD7BA715C
      VpcId:
        Ref: Vpc8378EB38
    Metadata:
      aws:cdk:path: MyStack/Vpc/VPCGW
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAA/81S0WrCQBD8Fu9RzrSG1oe82RQk0KKo+FCRsrls9PRyF+72IiHk34smVtov8OnmhtnZWZgwCF/GwfMAzm4kstNIyTRoVgTixOHsvlGEQbMpxbZhFh1ZKegdc/CKVii8lVTPrPEli3JQDjnTQDMgPEPtWMSGjDPnU40UG53LvbdA0mgWbRsmZGY/wZ16mYYC/0ys6/JCLHyqpGDtrt3xONebRcw7anVVbRsGFUgFqVSS6i+jby5VKZKsx5ddb8qI27ICys4kKef6A7wWBxaR9ciZLKtJ/E8Ozsm9TspqMs0yi87NdWyxP4UNWcsfJMXDBHmMFF1luqJc0NJ4wjWkCu/8nZs6Z4S8GvyKLyDRhFbjrdd9C/vflAjEoUBNLV+iM94KbLk2GQZH91SFr8EkGA+OTsqR9ZpkgcGye38AtKw2v3kDAAA=
    Metadata:
      aws:cdk:path: MyStack/CDKMetadata/Default
    Condition: CDKMetadataAvailable
Conditions:
  CDKMetadataAvailable:
    Fn::Or:
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - af-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-northeast-3
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-south-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-2
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-3
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - ap-southeast-4
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - ca-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - cn-northwest-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-central-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-north-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-south-2
      - Fn::Or:
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-1
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-2
          - Fn::Equals:
              - Ref: AWS::Region
              - eu-west-3
          - Fn::Equals:
              - Ref: AWS::Region
              - il-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - me-central-1
          - Fn::Equals:
              - Ref: AWS::Region
              - me-south-1
          - Fn::Equals:
              - Ref: AWS::Region
              - sa-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-1
          - Fn::Equals:
              - Ref: AWS::Region
              - us-east-2
          - Fn::Equals:
              - Ref: AWS::Region
              - us-west-1
      - Fn::Equals:
          - Ref: AWS::Region
          - us-west-2
Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]

Fortunatelly, this was the most complex use-case I've come across. I needed to generate two separate files which behave differently which was not that bad afterall.

Conclusion

Using CDK for generating standalone CloudFormation templates is not as straightforward as I thought. As always understanding how things works under the hood helps a lot.

The resulting CDK code is not as clean as I would expect but trust me it is easier to manage than yaml/json files. At least for me.

What is your experience with CDK/CloudFormation?