CodePipeline, which is a visual workflow tool that will allow you to orchestrate your CI/CD within AWS.
So with CodePipeline, you can show sources such as hey, my code is in CodeCommit or we have a Docker image in ECR, or my code is in Amazon S3. Or even external tools such as Bitbucket or GitHub. Then you can move on to the build phase, which is, hey, now that we have this code, let’s build it. So CodeBuild, Jenkins, CloudBees, TeamCity are all options.
Then once you have the build phase, you can have the test phase. So again, testing your code. So CodeBuild or Device Farm if you have an app, for example, an iOS, an Android app, or any third-party tools you want.
Then once the code is tested, you want to deploy it. So CodeDeploy, Beanstalk, CloudFormation, ECS, S3, all these options can be handled by CodePipeline.
And Invoke to invoke a Lambda function or a step function. And then when you have all these building blocks, you can build stages. So each stage can have sequential actions and/or parallel actions.
You can do a lot of things. Here’s a very simple example. Build, test, deploy onto staging, then load testing to make sure staging is doing fine. And then once the load testing is done, maybe deploy again to production.
Now, there’s also a way for you to define manual approval at any stage in your pipeline, really allowing you, for example, just before deploying to prod to have someone review the results of the load testing, for example, if it was there. And make sure it says, yeah, looks good. Green light, deploy to prod.
Artifacts
Now, how does CodePipeline work inside? So let’s assume we have a source, a build and a deploy phase. So source is CodeCommit. Build is CodeBuild, even though we haven’t seen CodeBuild yet, and Deploy is CodeDeploy even though we haven’t seen CodeDeploy yet.
So each pipeline can create artifacts. Artifacts is whatever is created out of the pipeline and the artifacts are going to be stored in S3 buckets to be passed onto the next stage. And this is how the next stage is going to be able to do what it needs to do.
So let’s do a concrete example. Developer is going to push some code into CodeCommit, right? And then CodeCommit is going to be orchestrated by CodePipeline, which is going to extract all the code and create an artifact out of it and place that artifact into an S3 bucket.
Now, when CodeBuild is invoked, the same artifacts that were extracted are going to be inputted directly into CodeBuild, and that’s why CodeBuild doesn’t need to have straight access into CodeCommit. Actually, it’s CodePipeline that will be pushing the code to CodeBuild through Amazon S3.
Then when CodeBuild is building the code, it’s going to create some deployment artifacts. So these artifacts are going to be stored yet again in your S3 bucket by CodePipeline, and CodePipeline will push these artifacts yet again to CodeDeploy and CodeDeploy says hey, I have these artifacts, I need to deploy them.
So as you can see, these stages interact with each other through Amazon S3, and this is why we have artifacts in CodePipeline.
Troubleshooting
Now, some troubleshooting for CodePipeline.
So if you need to have a look at all these things, for example, you need to look at like CodePipeline action or stage execution state changes, you can use CloudWatch Events, EventBridge to have a look at them. So for example, you can create events for failed pipelines, events for cancelled stages and then receive an email notification, for example.
And then if CodePipeline has a failure at a stage, you will see it visually and you can get information through the console.
And if there is no way for CodePipeline to perform a specific action, for example, to invoke some code in CodeBuild or to pull the code from CodeCommit, then check the IAM service role of CodePipeline and make sure it has right IAM permissions.
Also, if you need to have a look at maybe some denied API calls within your infrastructure, you can use CloudTrail, which is a service used to audit AWS API calls. So let’s say for CodePipeline, it will make a lot more sense once we go to the hands on.
Events vs Webhooks vs Polling
So let’s talk about some few additional points about CodePipeline.
The first one is that you have the concept to start a pipeline of events, webhooks and polling. So let’s have a look at all of them and see which one is best in which situation. So events is going to be the preferred way in CodePipeline and this says to start a pipeline whenever we have an event. So for example, when we have CodeCommit, for example, we know that on a new commit, an event will happen in EventBridge, and the EventBridge rule associated with that event can trigger and start a CodePipeline. This is the preferred way, and it’s not only CodeCommit, but could be any kind of events within AWS.
With GitHub, which is not part of AWS, how do you have event driven starting of CodePipeline? Well, to do so, you’re going to use what’s called a CodeStarSourceConnection. It’s just a fancy name for a GitHub application that is going to connect GitHub into AWS. And then from this, you’re going to trigger CodePipeline.
So these things are event driven and they are very fast because as soon as the event happens, CodePipeline is going to be triggered.
An older way of triggering CodePipeline is using webhook. So if you choose that option, then CodePipeline is going to expose an HTTP endpoint and that endpoint can be triggered by a script, whatever you want.
If that script sends a payload to CodePipeline on that webhook, then CodePipeline is going to start.
And finally, you can have CodePipeline poll the source, in which case you have regular checks, for example, from CodePipeline onto GitHub, but that is not recommended because it’s not as efficient as events.
So events are the default and recommended way to start a CodePipeline.
Action Types Constraints for Artifacts
Next, we have this table on CodePipeline that you don’t have to learn.
I’m going to just help you decrypt it and help you understand what it means. So we have owner, action type, provider, and then a valid number of input artifacts and output artifacts. The last two columns we don’t need to look at. We’re going to look at the first three.
So the owner can be either AWS and that means that it’s an action related to an AWS service. It could be a third party. That’s when action is related to a third party, folks such as GitHub or Alexa Skills Kit. Or custom, and this is once you have something like Jenkins.
So remember AWS for AWS and third party and custom. Now the action type corresponds to different stages of your pipeline.
For example, you have the source for S3, ECR, GitHub, CodeCommit. We have the build for CodeBuild and Jenkins. We have the test for CodeBuild, Device Farm, and Jenkins.
We have approval and the action type is called manual. We’ll see this in the next slide as well. That’s important. Invoke for Lambda and Step Functions, and deploy for S3, CloudFormation, CodeDeploy, Elastic Beanstalk, AppWorks, ECS, and Service Catalog.
Manual Approval Stage
So let’s go over the details of how manual approval works.
So when you have a CodePipeline and you have a manual approval, the important part is that the owner is AWS because it says related to within AWS.
It’s a capability offered by an AWS service and the action is going to be manual because it says a manual approval. In which case when you have a manual approval, what’s going to happen is that you can trigger an SNS topic which is in turn you can send in emails to a user and the user will have an IAM user on AWS.
And then you will have to approve this stage. And to approve this stage, it needs permissions.
And the permissions the user must have are twofold. Number one is get pipeline because while the user must be able to get to your pipeline to actually view it and to find that manual approval step, so we want the get pipeline star type of action.
And then we want the put approval result action on the approval action itself because we want to be able to say yes or no, we approve or we deny.
So that’s all you need to know. But it’s important to understand the IAM user permissions as well as the fact that the step owner is AWS and the action of this step is manual.
CloudFormation as a Target
So let’s talk about using CloudFormation as a target for CodePipeline.
So CloudFormation is a deploy action and it can be used to deploy any kind of AWS resources and including Lambda functions, either as pure CloudFormation or using the CDK framework or using SAML.
So the idea is that from CodePipeline, you have your code in CodeCommit. The code could be directly the template itself. You can create a change set from CloudFormation, then a manual approval to make sure that the change set is what we want it to be. And then execute the change sets. This is one option.
You can just go ahead also and deploy things right directly without doing a manual approval. So on top of supporting CloudFormation in one region, you support CloudFormation’s StackSets to deploy across multiple AWS accounts and AWS regions.
On top of things, you can configure multiple settings, such as the stack name, the change set name, the template input, the parameters if you want to override them, the IAM role, the action mode, and so on.
CloudFormation Integration
So let’s have a look at an architecture and a use case for using CodePipeline with CloudFormation.
So we have CodePipeline and we build our app in CodeBuild and we create a template.yaml file out of it. Next, we use CloudFormation to actually deploy the infrastructure in our app and we use the CREATE_UPDATE mode to create or update an existing stack. So our entire stack will be deployed. For example, this is an ALB with an auto scaling group and EC2 instances.
And then we have a new stage to test the application using CodeBuild. So CodeBuild can run, for example, test against the ALB by using the HTTP protocol and make sure our application is functioning as expected. It could be a functional test, it could be load testing, it could be whatever you want.
If all of this is working, that means that our application version is working as expected and we can, for example, delete the test infrastructure using the DELETE_ONLY action for CloudFormation to delete the stack.
And then another CloudFormation action, this time to deploy the prod infrastructure in prod by using again the CREATE_UPDATE action, which in this time is going to actually update our prod infrastructure.
So hope that makes sense, but that makes it very easy for you to create infrastructure and destroy them as part of your CI/CD pipeline.
CloudFormation as a Target
So let’s summarize the different action modes.
So the first one is about change sets. So we can create or replace a change set or we can execute a change set. This is usually when you want to have a manual approval in this to verify the change sets before applying it.
Or if you don’t want to have a manual approval, then you just want to deal with stacks. So you can create or update a stack, delete a stack or replace a failed stack. All of these are action modes in CodePipeline for CloudFormation.
You can also override template parameters. So in this case, you just specify a JSON object to override these parameter values. And this object can be part of the input artifact that you give to CodePipeline.
And then you say, okay, which parameter gets replaced by what? So all the parameters must be present in a template and you can have a static override. So use a template configuration file or dynamic to use parameters to pass them into your template.
Bottom line is just to make it as simple as possible, you can override the parameters if you wanted to at runtime using CodePipeline.
So that’s it for CodePipeline in CloudFormation.
Best Practices
So, let’s have a few more slides on CodePipeline.
So, the first one is around best practices of CodePipeline. So, in this example, we have one CodePipeline, one CodeDeploy, and from CodeDeploy, we do parallel deployments into multiple deployment groups. So, you would wanna do this, for example, if you wanted to deploy into multiple environments at a time, or multiple deployment groups. So, instead of creating two CodeDeploy deployments, you would create one with multiple groups.
There is also a way for you to speed things up in CodePipeline using Parallel Actions. So, if you have, for example, CodeCommit, and you wanna run two kinds of build, or some build and some test, or whatever you want, then instead of having one after the other, if you specify the same RunOrder value, then they will be run in parallel.
So, this is very helpful when building, for example, this is very helpful when deploying into multiple regions, and so on. Also, you can deploy to pre-prod before deploying to prod, and have a manual approval before deploying to prod. This is a very common pattern in CodePipeline.
EventBridge
Any kind of events happening in CodePipeline are going to be detected in EventBridge. So, this is for any state’s execution changes, or, for example, to intercept failures.
So, if we have this CodePipeline and CodeBuild fails, we can, for example, intercept it in EventBridge, and then invoke a Lambda function to, for example, diagnose the code, or trigger an SNS notification to notify some users by email, for example.
Invoke Action
If you wanted to run an API call from CodePipeline, the best way for you would be to invoke, and to invoke, you would use Lambda functions, and Lambda functions can be invoked, and from there we can run whatever code we want, such as calling a REST API.
There is no way to have CodePipeline directly do this, so invoking a Lambda function does extend the capability of CodePipeline by a lot.
Also, if you wanted to start step functions, you can now do this thanks to the Invoke Action. So, we can start a state machine within the pipeline. So, step functions allow us to do whatever we want, such as, for example, inserting items into a DynamoDB table, or starting a task in ECS, and having a graph, a workflow to show all these things happening one after the other. And so, it’s possible thanks to step functions.
Then we can, for example, once the step functions as executed to go and start a CodeBuild build, and, for example, CodeBuild could have a script to retrieve some value out of some DynamoDB table. Again, I don’t have the precise example in mind, but it just gives you an idea of how we can combine step functions into a CodePipeline.
And finally, well after CodeBuild, you deploy into CodeDeploy.
Multi Region
There’s also a way for you to do multi-region deployments with CodePipeline. So, it’s actually the fact that any or many actions in your pipeline can be in different regions.
So, for example, the use case for this would be to deploy a Lambda function through CloudFormation into multiple regions, all from one CodePipeline. So, to make this work, is very important for you to remember that S3 Artifact Stores must be defined for your CodePipeline for each region where you have actions. So, Artifact Stores is where the artifacts of CodePipeline are being stored.
And, of course, we know there’s one for your main CodePipeline wherever it is located. But in case you have actions in multiple regions, you must define these Artifact Stores.
So, if you are doing so, then, of course, your CodePipeline service role must have read and write access into every artifact buckets to be able to read and write the artifacts.
And if you’re using the console to define your CodePipeline, by default you’re going to be using the default CodePipeline buckets. So, they’re the ones that are shown in the bottom graph.
But if you are using, for example, the CLI, or the SDK, to create your CodePipeline, then you must create these artifact buckets ahead of time, and make sure CodePipeline has proper permissions. Very important for you to understand this.
And then the magical thing is that whenever we have a cross-region action, we just reference the input artifacts we want, and CodePipeline is going to automatically copy them from one region to the next, which makes it super easy to configure the cross-region actions.
CloudFormation Multi Region
To make this a little bit more simple, a little bit more clear, I have created a diagram.
So, let’s assume we want to deploy a Lambda function into two regions. The way we’re going to do this, that first of all, we define our CodePipeline, we have multiple regions, so we will have multiple Artifact Store.
The first one is eu-west-1, this is where our pipeline is located. So, the code is in CodeCommit, it gets sent to CodeBuild, and CodeBuild is going to build our code, and also create the output yaml file that will be used by CloudFormation to deploy the Lambda function. And so, because we are deploying into multiple regions, we need to create multiple templates to make sure we have one template per region we’re deploying to.
So, once this template is done, then we can have a CloudFormation action in the region of our pipeline, which is going to reference as an input the template we have created from before from CodeBuild, and deploy a Lambda function.
This is just a normal pipeline we’ve seen in one specific region. But because we are deploying cross region, we can also define a CloudFormation action in another region. So, this one is us-east-2. And so, because we have multiple regions now for CodePipeline, we must create an Artifact Store in us-east-2 as well, an S3 bucket.
And this Artifact Store is going to get the input artifacts copied automatically by CodePipeline so that, for example, my template-us-east-2.yaml file, is copied, and then what’s going to happen is that CloudFormation is going to take that file as an input, and then use that file automatically from the correct Artifact Store.
So, from a configuration perspective, we just told CodePipeline to define a CloudFormation action in another region, and that the input artifact was this template-us-east-2.yaml, the location of which is going to be automatically handled and copied behind the scenes by CodePipeline.
And so, therefore, a Lambda function in another region will be deployed.
