Create AMIs through AWS CloudFormation and use them to create the infrastructure within the same template.

Motivation

We have a devops project structure used to deploy our infrastructure to AWS environments. We have increased the automation level of this along the way and we did a presentation of this to an organization level team. One of their questions was “If we are to deploy our infrastructure using our project what should be done as steps?”. So our initial steps involved installing Jenkins and Sonar and then installing some plugins on jenkins and configuring them. But after that it was just creating pipelines and executing them which is really smooth. So the motivation came to automate the installation and configuration of jenkins. So first steps were to find out ways to configure jenkins and install and configure plugins to it programmatically. Here I used Jenkins Configuration as a code project. After that there were two requirements.

1. Create Jenkins and Sonar AMIs and use them for installations.
2. Create Jenkins worker AMIs and use it to configure Jenkins EC2 plugin.

I researched how to create AMIs from the CloudFormation template itself. Found two ways again.

1. Use a Lambda backed custom resource to create AMI. Lambda will create the golden AMI.
2. Use SSM Automation to create AMI.

Issue with the first one is that flow needs to be under 15 minutes and there are other limitations. Issue with the second is triggering automation from CloudFormation is not straight forward and even if I do it needs to wait till its finished. So my solution is to solve these issues.

Solution

Solution is a combination of above solutions. These are the steps involved in my CloudFormation template.

1. Create Automation Documents to define flow to create AMIs
2. Within this flow the last step should be to invoke the callback lambda function to callback CloudFormation stack.
3. Create a lambda backed custom resource to invoke SSM Automation which will start an execution. This lambda will just fire and forget and will not call back CloudFormation about status.
4. When execution is done it will trigger the callback function to notify Cloudformation to indicate the custom resource status.

Here is my full CloudFormation template with all other necessary components and configurations to configure my CICD infrastructure[1].

Conclusion

Here I used this solution with SSM Automation. But the same solution should be possible with services like Step Functions like services. Idea is that one lambda will trigger the workflow and then in the end we embed a call to the workflow itself to trigger a callback function to callback about the status to the CloudFormation service.

References

[1] https://github.com/danushkaf/cloudformation_sample_templates/blob/master/cicd-infrastructure/jenkins-cicd-infra-with-ami-creation.yaml

[2] https://github.com/jenkinsci/configuration-as-code-plugin