Looking for some resources on best practices for migrating existing AWS infrastructure to Terraform.

barf, 11 months ago

I've done this once or twice - with and without some other existing IaC in place - it's not the worst thing in the world with some tooling and a good approach. How exactly you should do things depends on the situation, primarily on whether you're talking about a single product or many products and services. The (now excessively long) opinionated post below is probably better suited for the latter case - it was primarly learned from pulling an entire SaaS business into Terraform, lots of infra that followed some similar base patterns across many products and just as many services under each.

One thing to accept about this process is that it's a one-off. It does not need to be repeatable, there does not need to be a magic bullet. There aren't any magic bullets anyway, so the naive approach is one of the better options. You only have to setup all the base infra for prod one time, you might as well do it the slow and methodical way.

I can't speak to the new import block, I have only ever used the CLI. I'd personally stay away from it until Hashicorp says it's fully ready.

Some of the tooling out there (terraformer, terracognita, etc) can help, though IMO it's a bad stopgap as you're going to need to actually modularize everything eventually anyway. And if you need to do that, pulling well-used and tested modules from the registry is probably your better bet. These tools can be helpful for IDing what needs to be handled and in some cases produce plenty usable output, so it's worth a download and run to see.

In most situations, you have many of a thing deployed (EC2 instances, Step Functions, Glue everything, etc), so if you can narrow things down you can come up with a decently simple strategy to get 90% done.

What I'd recommend, roughly in order:

• Use the AWS Tag Editor to get a comprehensive list of resources in an account.
  • Select whatever regions are used.
  • Select All supported resource types
  • Click Search Resources and wait a bit - depending on how many resources are in the account, a bit could be many minutes.
  • Export everything to CSV, it's much easier to deal with it this way
• Use that export to ID what you need to create Terraform config for.
• Break things down by resource type and figure out the main modules you need
• Work through an import/plan process for something with each major module
  • plan the module
  • See the resources it's trying to create, figure out the import syntax
  • Repeat until the plan shows no changes
  • Keep notes!

After that point you should have a roughly step-by-step way to import large chunks of your infrastructure, just copy/paste and replace with new resource identifiers for your import steps. Plan often, keep track of what you have imported and haven't down to the security group. Add tags to all imported resources so you can more easily ID what might have been missed.

For CI/CD I'd recommend Atlantis. It isn't the greatest solution in the world, but it does the job. On a smaller team I'd probably just use Github Actions workflows or the equivalent.

While I use Terragrunt for multi-env situations, a vanilla Terraform approach using the backend-config flag to configure environments works plenty well. I can't speak to Terraform Cloud, though I understand it is pretty good these days.

adduc, 11 months ago

Terraform 1.5.0 introduced support for the import block, which you may find useful to associate existing resources with resources declared in terraform.

KingWizard, 11 months ago

Talk about perfect timing, this was just released yesterday!

Anomandaris, 11 months ago

I've never actually used it myself, but terraformer might be able to help here.

Or at least give some ideas on how you want to tackle the problem.

KingWizard, 11 months ago

This is great. Looks like this would greatly help with at least getting started and pulling the various resources I need into Terraform. I'll have to play around with this.