A reasonable approach to CI/CD pipelines

January 31, 2022

Stacks

The term "stack" is used here as a deployed instance of the application. I'm using the term stack since I'm primarily working on AWS infrastructure with the AWS Cloud Development Kit (CDK) or CloudFormation.

You want the stack to be configurable and be deployable with a few commands to make it trivial to spin up new ones and tearing down old ones.

An approach that led to a great developer experience for me has been:

• A personal stack for each developer
  • These are tied to a branch that a developer works off of on their own. The fact that they're not being used by anyone else allows the owner of the stack to freely deploy as often as they like without affecting another stack or requiring a code review
    • This is especially helpful whn working on breaking changes
  • Once the developer feels comfortable with the change, they can open a pull request into the development branch and allow the reviewer to test out the changes on their instance
• A development stack
  • This is a shared stack that all developers use to validate changes themselves before sharing them with the stakeholders. Think of it as a pre-staging environment
  • When all of the developers feel comfortable shipping this feature, they'll push to the staging environment
• A staging stack
  • This is the environment that the core stakeholders will test the new changes before sharing them with the broader audience
  • This stack should have the same settings as the production stack so you can have confidence that if it works here, it'll work in production
  • Bonus: if you can copy production data - replacing the user data with fake data for even more confidence
  • Once the core stakeholders have tested, deploy to production
• One or more production stacks, depending on your availability needs
  • It's always good to do another round of testing as best as you can here, but avoid modifying data in this environment. At this point you should be confident that your application works from all of the rounds of testing.

CI/CD pipeline

You'll want to automate the deployment of these stacks so you can ensure that your tests, linter, formatter, security scans, etc. always succeed before a deployment occurs. It also prevents the "works on my machine" problem when deploying from your own computer.

There are plenty of products and documentation on how to do this and I won't cover that here to keep the scope of this post short. If you have no idea where to start and are using AWS CDK to deploy your resources, I recommend the CDK Pipeline construct and CodePipeline. I also recommend setting each pipeline to point to a specific branch in your code repository so you can push to git and let it do the work for you.

AWS Amplify also has a great CI/CD pipeline that's even more managed for you if you're already using their ecosystem.

Bonus tip: reduce your blast radius

The full topic of this could be its own post. Just think about what happens if something fails and do your best to reduce the amount of things that can go wrong. An example of a bad scenario would be having your developer, staging and production stacks all in one AWS account.

A better scenario would be splitting the development, staging and production accounts. The development account can be a little more permissive than the other accounts to ensure you're not slowed down too much. But you'll want staging and production to be very locked down, and mostly read-only to avoid deleting or modifying resources unintentionally.

Happy coding! SL