Emmanuel Akinfulubi
Free courses Cloud, DevOps and AWS, from zero

Infrastructure as Code all 4 written

Why click-ops fails at scale, and real Terraform: providers, state, modules, and the plan/apply/destroy loop.

28 Why click-ops fails at scale

Every command run in this course through Part 8 shares one property: it does something, immediately, with no step in between "I typed this" and "it happened" where a person or a machine could say wait, is that right? That gap is fine at the scale of a course. At the scale of a real account, with real customers and real money, that gap is where real incidents come from.

Failure one: a destructive command with nothing to preview it

The AWS CLI does exactly what it is told, exactly when it is told. There is no built-in pause between typing a delete and it taking effect -- which is correct behaviour for a tool, and exactly the property that makes a typo dangerous.

A typo, and nothing between typing it and it happening

ran at build time 
aws s3 mb s3://prod-customer-uploads > /dev/null
echo "hello" | aws s3 cp - s3://prod-customer-uploads/important-file.txt > /dev/null
echo "a real bucket, with a real object a customer is depending on:"
aws s3 ls s3://prod-customer-uploads/

echo
echo "an engineer, meaning to clean up a DIFFERENT, similarly-named test bucket,"
echo "runs this instead -- there is no preview, no confirmation, nothing to catch it:"
aws s3 rm s3://prod-customer-uploads/important-file.txt
delete_exit=$?

echo
echo "checking whether it is still there:"
after=$(aws s3 ls s3://prod-customer-uploads/)
echo "bucket contents now: [$after]"

check "the command ran with no dry-run, no plan, no confirmation step" "$delete_exit" "0"
check "the object is simply gone, the instant the command was typed" "$after" ""

What it printed, when the build ran it

a real bucket, with a real object a customer is depending on:
2026-09-15 21:20:03          6 important-file.txt

an engineer, meaning to clean up a DIFFERENT, similarly-named test bucket,
runs this instead -- there is no preview, no confirmation, nothing to catch it:
delete: s3://prod-customer-uploads/important-file.txt

checking whether it is still there:
bucket contents now: []
ok   the command ran with no dry-run, no plan, no confirmation step
ok   the object is simply gone, the instant the command was typed

This is not a hypothetical. Almost every serious "we deleted the wrong thing" outage begins exactly like this: a correct command, aimed at the wrong resource, run by someone with real permissions and good intentions. The tool did its job perfectly. The process around the tool had no step that could have caught it.

Failure two: two things that were "created the same way," silently drifting apart

Click-ops has no artifact -- no file describing what a resource is supposed to look like, only the resource itself, after the fact. Two engineers, or one engineer on two different days, "doing the same setup twice" can produce two configurations that look identical until something specific is compared, by hand, after the fact.

Two buckets, 'the same,' until you actually diff them

ran at build time 
echo "two buckets that are SUPPOSED to be configured identically, created by hand:"
aws s3 mb s3://web-assets-east > /dev/null
aws s3api put-bucket-versioning --bucket web-assets-east --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket web-assets-east \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws s3 mb s3://web-assets-west > /dev/null
aws s3api put-bucket-versioning --bucket web-assets-west --versioning-configuration Status=Enabled
# a real, easy-to-make slip: RestrictPublicBuckets typed as false here, not true.
# nothing about running this command warns that it does not match the other bucket.
aws s3api put-public-access-block --bucket web-assets-west \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=false

echo
echo "both were 'set up the same way' -- confirming that requires manually reading BOTH back:"
east=$(aws s3api get-public-access-block --bucket web-assets-east --output json)
west=$(aws s3api get-public-access-block --bucket web-assets-west --output json)
echo "east: $east"
echo "west: $west"

east_restrict=$(echo "$east" | grep -o '"RestrictPublicBuckets": [a-z]*' | grep -o '[a-z]*$')
west_restrict=$(echo "$west" | grep -o '"RestrictPublicBuckets": [a-z]*' | grep -o '[a-z]*$')

check "the two buckets, meant to be identical, actually differ on one real security setting" \
  "$([ "$east_restrict" != "$west_restrict" ] && echo yes)" "yes"

What it printed, when the build ran it

two buckets that are SUPPOSED to be configured identically, created by hand:

both were 'set up the same way' -- confirming that requires manually reading BOTH back:
east: {
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}
west: {
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": false
    }
}
ok   the two buckets, meant to be identical, actually differ on one real security setting

Notice what did NOT happen above. No error. No warning. Both commands returned success, because both were individually valid -- west's weaker setting is not a mistake AWS can see, only a mistake relative to an intention that existed nowhere but in someone's memory. This is "configuration drift," and click-ops has no defence against it beyond someone remembering to check.

What Infrastructure as Code actually changes

Not the underlying AWS calls -- Terraform, covered starting next lesson, ultimately issues the same kind of API calls just run by hand above. What changes is the presence of a file, in version control, that says what the infrastructure is supposed to be, plus a tool that can compare that file against reality before touching anything:

  • A written record. "What is this bucket supposed to look like" has an answer that exists independent of anyone's memory -- a file, reviewable in a pull request, the same as application code.
  • A preview before action. A plan step that states exactly what will change, shown before anything changes, closing precisely the gap that let the delete above happen with no warning.
  • Drift becomes detectable. Comparing the written record against real infrastructure is a mechanical diff, not a memory exercise -- the same class of mismatch just found by hand above becomes something a tool reports on its own.

What this course means by "real" from here on. Every AWS lesson before this one ran real commands against a real (local, fake) AWS -- that does not change. What changes is the tool issuing those commands: from here, it is Terraform, reading a file, rather than a person typing each call directly. Part 9 builds up to exactly the plan-then-apply discipline described above, checked against real state, not asserted as a slide.

29 Real Terraform: providers, resources and state

A Terraform provider block is a real config file, a resource block is a real desired state, and a plan is a real diff between that file and whatever currently exists. None of that is a metaphor for what AWS actually does -- underneath, an aws_s3_bucket resource still, eventually, calls the same CreateBucket API this course has been calling by hand since Part 7.

A provider block: where, and through what door

The provider block says which cloud, which region, and -- specific to this course -- which endpoint to actually send requests to. Every lesson from here uses the same shape: point the aws provider at this build's own moto instance, the identical stand-in every earlier AWS lesson in this course has used.

provider + resource blocks: a real file, a real plan, nothing created yet

ran at build time 
work=/tmp/cd-course-tf-basics
rm -rf "$work"
mkdir -p "$work"
cd "$work"

cat > main.tf << HCL
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    s3       = "$MOTO_ENDPOINT"
    dynamodb = "$MOTO_ENDPOINT"
  }
}

resource "aws_s3_bucket" "course" {
  bucket = "tf-basics-bucket"
}

resource "aws_dynamodb_table" "course" {
  name         = "tf-basics-table"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "Id"
  attribute {
    name = "Id"
    type = "S"
  }
}
HCL

echo "the exact file just written -- this is the entire 'desired state':"
cat main.tf

# The dependency lock file this course primed once, at the start of the whole
# build, so this init reuses that already-verified provider instead of
# re-fetching and re-checking its signature from scratch on every single lesson.
cp "$TF_SEED_LOCKFILE" .terraform.lock.hcl

echo
echo "--- terraform init ---"
terraform init -no-color

echo
echo "--- terraform plan: what WOULD happen, before anything does ---"
plan_output=$(terraform plan -no-color)
echo "$plan_output"

check "the plan states it would create exactly two resources" \
  "$(echo "$plan_output" | grep -c '2 to add, 0 to change, 0 to destroy')" "1"
check "nothing was actually created yet -- plan is read-only" \
  "$(aws s3api list-buckets --query 'length(Buckets)' --output text)" "0"

What it printed, when the build ran it

the exact file just written -- this is the entire 'desired state':
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    s3       = "http://localhost:5099"
    dynamodb = "http://localhost:5099"
  }
}

resource "aws_s3_bucket" "course" {
  bucket = "tf-basics-bucket"
}

resource "aws_dynamodb_table" "course" {
  name         = "tf-basics-table"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "Id"
  attribute {
    name = "Id"
    type = "S"
  }
}

--- terraform init ---
Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using hashicorp/aws v5.100.0 from the shared cache directory

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

--- terraform plan: what WOULD happen, before anything does ---

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_dynamodb_table.course will be created
  + resource "aws_dynamodb_table" "course" {
      + arn              = (known after apply)
      + billing_mode     = "PAY_PER_REQUEST"
      + hash_key         = "Id"
      + id               = (known after apply)
      + name             = "tf-basics-table"
      + read_capacity    = (known after apply)
      + stream_arn       = (known after apply)
      + stream_label     = (known after apply)
      + stream_view_type = (known after apply)
      + tags_all         = (known after apply)
      + write_capacity   = (known after apply)

      + attribute {
          + name = "Id"
          + type = "S"
        }

      + point_in_time_recovery (known after apply)

      + server_side_encryption (known after apply)

      + ttl (known after apply)
    }

  # aws_s3_bucket.course will be created
  + resource "aws_s3_bucket" "course" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "tf-basics-bucket"
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + cors_rule (known after apply)

      + grant (known after apply)

      + lifecycle_rule (known after apply)

      + logging (known after apply)

      + object_lock_configuration (known after apply)

      + replication_configuration (known after apply)

      + server_side_encryption_configuration (known after apply)

      + versioning (known after apply)

      + website (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
ok   the plan states it would create exactly two resources
ok   nothing was actually created yet -- plan is read-only

This is the gap lesson 28 said Infrastructure as Code closes. terraform plan is exactly the missing preview step -- a full statement of what is about to happen, readable by a person, computed before a single API call actually runs. Nothing above has touched AWS yet except to ask it what already exists.

Apply: the plan, actually executed

terraform apply: the plan, executed for real, confirmed by a different tool

ran at build time 
cd /tmp/cd-course-tf-basics

apply_output=$(terraform apply -auto-approve -no-color)
echo "$apply_output"

echo
echo "confirming via the real aws cli -- NOT via terraform -- that this really happened:"
aws s3api list-buckets --query 'Buckets[].Name' --output json
aws dynamodb list-tables --output json

check "apply reports both resources added" \
  "$(echo "$apply_output" | grep -c '2 added, 0 changed, 0 destroyed')" "1"
check "the bucket genuinely exists, confirmed by aws cli, not by terraform's own say-so" \
  "$(aws s3api list-buckets --query 'length(Buckets)' --output text)" "1"

What it printed, when the build ran it

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_dynamodb_table.course will be created
  + resource "aws_dynamodb_table" "course" {
      + arn              = (known after apply)
      + billing_mode     = "PAY_PER_REQUEST"
      + hash_key         = "Id"
      + id               = (known after apply)
      + name             = "tf-basics-table"
      + read_capacity    = (known after apply)
      + stream_arn       = (known after apply)
      + stream_label     = (known after apply)
      + stream_view_type = (known after apply)
      + tags_all         = (known after apply)
      + write_capacity   = (known after apply)

      + attribute {
          + name = "Id"
          + type = "S"
        }

      + point_in_time_recovery (known after apply)

      + server_side_encryption (known after apply)

      + ttl (known after apply)
    }

  # aws_s3_bucket.course will be created
  + resource "aws_s3_bucket" "course" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "tf-basics-bucket"
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + cors_rule (known after apply)

      + grant (known after apply)

      + lifecycle_rule (known after apply)

      + logging (known after apply)

      + object_lock_configuration (known after apply)

      + replication_configuration (known after apply)

      + server_side_encryption_configuration (known after apply)

      + versioning (known after apply)

      + website (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.
aws_s3_bucket.course: Creating...
aws_dynamodb_table.course: Creating...
aws_dynamodb_table.course: Creation complete after 0s [id=tf-basics-table]
aws_s3_bucket.course: Creation complete after 0s [id=tf-basics-bucket]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

confirming via the real aws cli -- NOT via terraform -- that this really happened:
[
    "tf-basics-bucket"
]
{
    "TableNames": [
        "tf-basics-table"
    ]
}
ok   apply reports both resources added
ok   the bucket genuinely exists, confirmed by aws cli, not by terraform's own say-so

State: Terraform's only memory of what it made

Terraform does not ask AWS "what have I created" before every command -- for a large enough account, that would be unusably slow. It keeps a local file, terraform.tfstate, recording exactly what it believes exists, and trusts that record between commands.

terraform.tfstate: not a log, Terraform's working memory

ran at build time 
cd /tmp/cd-course-tf-basics

echo "the state file -- Terraform's own record, in plain JSON, of what it made:"
head -c 500 terraform.tfstate
echo "... (truncated)"

echo
echo "the same information, through terraform's own lens on that file:"
terraform state list

echo
echo "running apply again, with NOTHING changed -- state lets it know there is nothing to do:"
noop_output=$(terraform apply -auto-approve -no-color)
echo "$noop_output"

check "the state file really lists both resources" \
  "$(terraform state list | wc -l | tr -d ' ')" "2"
check "a second apply with no changes does nothing, because state already matches reality" \
  "$(echo "$noop_output" | grep -c '0 added, 0 changed, 0 destroyed')" "1"

What it printed, when the build ran it

the state file -- Terraform's own record, in plain JSON, of what it made:
{
  "version": 4,
  "terraform_version": "1.16.1",
  "serial": 3,
  "lineage": "e25378c7-e5f2-7850-aabd-c368128ae5fd",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "aws_dynamodb_table",
      "name": "course",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "arn": "arn:aws:dynamodb:us-east-1:123456789012:table/tf-basics-table",
            "... (truncated)

the same information, through terraform's own lens on that file:
aws_dynamodb_table.course
aws_s3_bucket.course

running apply again, with NOTHING changed -- state lets it know there is nothing to do:
aws_dynamodb_table.course: Refreshing state... [id=tf-basics-table]
aws_s3_bucket.course: Refreshing state... [id=tf-basics-bucket]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
ok   the state file really lists both resources
ok   a second apply with no changes does nothing, because state already matches reality

Deleting that memory, and watching what actually breaks

The bucket and the table are still sitting in AWS, completely unaffected. Only Terraform's own record of having made them is about to be destroyed -- and what breaks is not the infrastructure, it is Terraform's belief about the infrastructure.

Lose the state file, and Terraform forgets what it already built

ran at build time 
cd /tmp/cd-course-tf-basics

echo "before: the resources still exist, confirmed independently of terraform:"
aws s3api list-buckets --query 'length(Buckets)' --output text

echo
echo "deleting ONLY terraform's memory of them -- the real infrastructure is untouched:"
rm -f terraform.tfstate terraform.tfstate.backup

echo
echo "asking terraform what it thinks needs to happen now:"
plan_after_loss=$(terraform plan -no-color 2>&1)
echo "$plan_after_loss"

echo
echo "meanwhile, completely unaffected, because state loss never touched AWS itself:"
aws s3api list-buckets --query 'length(Buckets)' --output text

cd /tmp && rm -rf /tmp/cd-course-tf-basics

check "terraform now believes it needs to CREATE things that already exist, having lost its memory" \
  "$(echo "$plan_after_loss" | grep -c '2 to add')" "1"
check "the real infrastructure was never touched by losing the state file" \
  "$(aws s3api list-buckets --query 'length(Buckets)' --output text)" "1"

What it printed, when the build ran it

before: the resources still exist, confirmed independently of terraform:
1

deleting ONLY terraform's memory of them -- the real infrastructure is untouched:

asking terraform what it thinks needs to happen now:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_dynamodb_table.course will be created
  + resource "aws_dynamodb_table" "course" {
      + arn              = (known after apply)
      + billing_mode     = "PAY_PER_REQUEST"
      + hash_key         = "Id"
      + id               = (known after apply)
      + name             = "tf-basics-table"
      + read_capacity    = (known after apply)
      + stream_arn       = (known after apply)
      + stream_label     = (known after apply)
      + stream_view_type = (known after apply)
      + tags_all         = (known after apply)
      + write_capacity   = (known after apply)

      + attribute {
          + name = "Id"
          + type = "S"
        }

      + point_in_time_recovery (known after apply)

      + server_side_encryption (known after apply)

      + ttl (known after apply)
    }

  # aws_s3_bucket.course will be created
  + resource "aws_s3_bucket" "course" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "tf-basics-bucket"
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + cors_rule (known after apply)

      + grant (known after apply)

      + lifecycle_rule (known after apply)

      + logging (known after apply)

      + object_lock_configuration (known after apply)

      + replication_configuration (known after apply)

      + server_side_encryption_configuration (known after apply)

      + versioning (known after apply)

      + website (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

meanwhile, completely unaffected, because state loss never touched AWS itself:
1
ok   terraform now believes it needs to CREATE things that already exist, having lost its memory
ok   the real infrastructure was never touched by losing the state file

This is the entire reason remote state exists. A local terraform.tfstate file, as used in every panel above for simplicity, has a real failure mode: lose the laptop, lose the file, and the next terraform plan genuinely believes it needs to recreate infrastructure that is sitting there fine, exactly as just demonstrated. Real teams store this file in a shared, durable backend (an S3 bucket with locking, or Terraform Cloud) for exactly this reason -- the state file is Terraform's memory, and a team sharing infrastructure needs to share that memory too.

30 Variables and modules: Terraform that scales

A variable is a named input a configuration takes without changing its own source. A module is a directory of configuration, callable by name, the same relationship a function has to the code that calls it. Neither is a new AWS concept -- they are how Terraform avoids the copy-paste that let lesson 28's two "identical" buckets quietly differ.

A variable: one file, values supplied per run

Hardcoding "tf-basics-bucket" the way lesson 29 did means the only way to make a second bucket is to copy the whole block and edit one string by hand -- exactly the manual-and-forgettable step that caused drift before. A variable removes the copying.

One file, two different real buckets, zero copy-pasting

ran at build time 
work=/tmp/cd-course-tf-vars
rm -rf "$work"
mkdir -p "$work"
cd "$work"

cat > main.tf << HCL
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "$MOTO_ENDPOINT"
  }
}

variable "bucket_name" {
  type        = string
  description = "the bucket to create -- supplied at apply time, not hardcoded"
}

resource "aws_s3_bucket" "course" {
  bucket = var.bucket_name
}
HCL

cp "$TF_SEED_LOCKFILE" .terraform.lock.hcl
terraform init -no-color > /dev/null

echo "the SAME file, applied twice, with a different value each time -- no editing main.tf:"
terraform apply -auto-approve -no-color -var="bucket_name=team-a-uploads" > /dev/null
first_result=$(aws s3api list-buckets --query 'Buckets[].Name' --output json)
echo "after run 1: $first_result"

terraform destroy -auto-approve -no-color -var="bucket_name=team-a-uploads" > /dev/null
terraform apply -auto-approve -no-color -var="bucket_name=team-b-uploads" > /dev/null
second_result=$(aws s3api list-buckets --query 'Buckets[].Name' --output json)
echo "after run 2: $second_result"

terraform destroy -auto-approve -no-color -var="bucket_name=team-b-uploads" > /dev/null
cd /tmp && rm -rf "$work"

check "the first run created exactly the bucket named at apply time" \
  "$(echo "$first_result" | grep -c 'team-a-uploads')" "1"
check "the second run, same file, created a completely different bucket" \
  "$(echo "$second_result" | grep -c 'team-b-uploads')" "1"
check "the second run's bucket is not just the first one renamed -- the first is gone by then" \
  "$(echo "$second_result" | grep -c 'team-a-uploads')" "0"

What it printed, when the build ran it

the SAME file, applied twice, with a different value each time -- no editing main.tf:
after run 1: [
    "team-a-uploads"
]
after run 2: [
    "team-b-uploads"
]
ok   the first run created exactly the bucket named at apply time
ok   the second run, same file, created a completely different bucket
ok   the second run's bucket is not just the first one renamed -- the first is gone by then

A module: that same shape, packaged so it cannot silently drift

A module is a directory with its own resource blocks, called by name from elsewhere with module "name" { source = ... }. Calling it twice with different inputs is how Terraform gets two genuinely separate resources from one definition -- instead of lesson 28's two engineers each typing the setup by hand.

One module, called twice, two real buckets with genuinely different config

ran at build time 
work=/tmp/cd-course-tf-modules
rm -rf "$work"
mkdir -p "$work/modules/bucket"
cd "$work"

cat > modules/bucket/main.tf << HCL
variable "bucket_name" {
  type = string
}

variable "environment" {
  type    = string
  default = "dev"
}

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  tags = {
    Environment = var.environment
  }
}

output "bucket_arn" {
  value = aws_s3_bucket.this.arn
}
HCL

echo "the module's own definition -- notice it has no idea what will call it, or how many times:"
cat modules/bucket/main.tf

cat > main.tf << HCL
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "$MOTO_ENDPOINT"
  }
}

module "logs_bucket" {
  source      = "./modules/bucket"
  bucket_name = "course-logs-bucket"
  environment = "production"
}

module "backups_bucket" {
  source      = "./modules/bucket"
  bucket_name = "course-backups-bucket"
}

output "logs_arn" {
  value = module.logs_bucket.bucket_arn
}
output "backups_arn" {
  value = module.backups_bucket.bucket_arn
}
HCL

echo
echo "calling that SAME module twice, with different inputs -- the second call even omits"
echo "environment entirely, relying on the module's own default:"
cp "$TF_SEED_LOCKFILE" .terraform.lock.hcl
terraform init -no-color > /dev/null
terraform apply -auto-approve -no-color > /dev/null

echo
echo "both are real, separate buckets -- confirmed via aws cli, not terraform's own output:"
bucket_count=$(aws s3api list-buckets --query 'length(Buckets)' --output text)
aws s3api list-buckets --query 'Buckets[].Name' --output json

logs_tags=$(aws s3api get-bucket-tagging --bucket course-logs-bucket --output json)
backups_tags=$(aws s3api get-bucket-tagging --bucket course-backups-bucket --output json)
echo "logs bucket tags:    $logs_tags"
echo "backups bucket tags: $backups_tags"

terraform destroy -auto-approve -no-color > /dev/null
remaining_after_destroy=$(aws s3api list-buckets --query 'length(Buckets)' --output text)
cd /tmp && rm -rf "$work"

check "both module calls created real, independent buckets" "$bucket_count" "2"
check "the module call that set environment explicitly used that value" \
  "$(echo "$logs_tags" | grep -c 'production')" "1"
check "the module call that omitted environment used the module's own default instead" \
  "$(echo "$backups_tags" | grep -c '"Value": "dev"')" "1"
check "destroying the root config cleans up everything both module calls created" \
  "$remaining_after_destroy" "0"

What it printed, when the build ran it

the module's own definition -- notice it has no idea what will call it, or how many times:
variable "bucket_name" {
  type = string
}

variable "environment" {
  type    = string
  default = "dev"
}

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  tags = {
    Environment = var.environment
  }
}

output "bucket_arn" {
  value = aws_s3_bucket.this.arn
}

calling that SAME module twice, with different inputs -- the second call even omits
environment entirely, relying on the module's own default:

both are real, separate buckets -- confirmed via aws cli, not terraform's own output:
[
    "course-backups-bucket",
    "course-logs-bucket"
]
logs bucket tags:    {
    "TagSet": [
        {
            "Key": "Environment",
            "Value": "production"
        }
    ]
}
backups bucket tags: {
    "TagSet": [
        {
            "Key": "Environment",
            "Value": "dev"
        }
    ]
}
ok   both module calls created real, independent buckets
ok   the module call that set environment explicitly used that value
ok   the module call that omitted environment used the module's own default instead
ok   destroying the root config cleans up everything both module calls created

What this actually prevents. Lesson 28's two buckets drifted apart because "set it up the same way" was a memory, not a mechanism. A module makes the shape -- versioning on, a specific tagging scheme, whatever a team standardises on -- exist in exactly one place. Every caller inherits that shape automatically; changing it means editing the module once, not re-auditing every bucket that copied it by hand.

The last check above is really checking two things at once. It confirms both that the bucket count after destroy is zero (cleanup worked) AND, by construction, that the two applied buckets truly were distinct resources -- if the second module call had somehow collided with the first instead of creating something new, the tag checks earlier would have shown one bucket with contradictory tags, not two buckets with different, correct ones.

31 Plan, apply, destroy: the loop that keeps you honest

Lesson 28 ended on two failures: a delete with no preview, and two resources that silently drifted apart. plan is the preview. What is left to show is that plan also catches drift AFTER the fact -- not just before an initial apply -- and that infrastructure created by hand, exactly like lesson 28's click-ops buckets, can be brought under Terraform's management without tearing it down and rebuilding it.

Plan catches drift that happened outside Terraform entirely

Nothing stops someone from changing a Terraform-managed resource directly through the AWS CLI or console -- moto does not know or care who is calling it. What Terraform does about that is the point: the next plan compares real, current state against the file, and a hand-made change shows up as a difference, not a mystery.

A hand-made change, caught by the next plan, not by luck

ran at build time 
work=/tmp/cd-course-tf-drift
rm -rf "$work"
mkdir -p "$work"
cd "$work"

cat > main.tf << HCL
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "$MOTO_ENDPOINT"
  }
}

resource "aws_s3_bucket" "course" {
  bucket = "tf-drift-bucket"
  tags = {
    Environment = "production"
  }
}
HCL

cp "$TF_SEED_LOCKFILE" .terraform.lock.hcl
terraform init -no-color > /dev/null
terraform apply -auto-approve -no-color > /dev/null
echo "created, with Environment=production, entirely through terraform"

echo
echo "now someone hand-edits it directly with the aws cli -- no terraform involved at all:"
aws s3api put-bucket-tagging --bucket tf-drift-bucket \
  --tagging 'TagSet=[{Key=Environment,Value=hand-edited-by-mistake}]'
echo "tag changed by hand"

echo
echo "terraform plan, run with no changes to main.tf at all -- does it notice?"
drift_plan=$(terraform plan -no-color)
echo "$drift_plan"

terraform destroy -auto-approve -no-color > /dev/null
cd /tmp && rm -rf "$work"

check "plan detects the hand-made change as something needing correction" \
  "$(echo "$drift_plan" | grep -c '0 to add, 1 to change, 0 to destroy')" "1"
check "plan specifically names the tag that drifted, not just 'something changed'" \
  "$([ "$(echo "$drift_plan" | grep -c 'hand-edited-by-mistake')" -ge 1 ] && echo yes)" "yes"

What it printed, when the build ran it

created, with Environment=production, entirely through terraform

now someone hand-edits it directly with the aws cli -- no terraform involved at all:
tag changed by hand

terraform plan, run with no changes to main.tf at all -- does it notice?
aws_s3_bucket.course: Refreshing state... [id=tf-drift-bucket]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket.course will be updated in-place
  ~ resource "aws_s3_bucket" "course" {
        id                          = "tf-drift-bucket"
      ~ tags                        = {
          ~ "Environment" = "hand-edited-by-mistake" -> "production"
        }
      ~ tags_all                    = {
          ~ "Environment" = "hand-edited-by-mistake" -> "production"
        }
        # (12 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
ok   plan detects the hand-made change as something needing correction
ok   plan specifically names the tag that drifted, not just 'something changed'

This is drift detection, the direct fix for lesson 28's second failure. The two buckets in lesson 28 differed because nobody compared them. Here, the comparison is automatic: run plan at any time, and any gap between the file and reality -- however it got there -- is stated explicitly, with the specific field and value that changed.

Import: bringing click-ops infrastructure under management, without rebuilding it

Not every real account starts from Terraform on day one. A bucket created by hand, exactly like lesson 28's, does not have to be deleted and recreated to become Terraform-managed -- terraform import tells Terraform "this already exists, start tracking it," and nothing about the resource itself changes.

Existing, hand-made infrastructure, adopted without being rebuilt

ran at build time 
work=/tmp/cd-course-tf-import
rm -rf "$work"
mkdir -p "$work"
cd "$work"

echo "a bucket created entirely by hand -- no terraform, no config file, exactly like lesson 28:"
aws s3api create-bucket --bucket hand-created-bucket > /dev/null
echo "already-there" | aws s3 cp - s3://hand-created-bucket/data.txt > /dev/null
echo "created and populated, with zero terraform involvement so far"

cat > main.tf << HCL
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "$MOTO_ENDPOINT"
  }
}

resource "aws_s3_bucket" "manual" {
  bucket = "hand-created-bucket"
}
HCL

cp "$TF_SEED_LOCKFILE" .terraform.lock.hcl
terraform init -no-color > /dev/null

echo
echo "plan BEFORE import: terraform has never heard of this bucket, so it wants to create it:"
before_plan=$(terraform plan -no-color)
echo "$before_plan"

echo
echo "terraform import: tell it the resource already exists, instead of creating a duplicate:"
terraform import aws_s3_bucket.manual hand-created-bucket

echo
echo "plan AFTER import: same file, same real bucket -- now recognised as already matching:"
after_plan=$(terraform plan -no-color)
echo "$after_plan"

echo
echo "and the object that was there before terraform ever ran is still exactly there:"
still_there=$(aws s3 cp s3://hand-created-bucket/data.txt -)
echo "$still_there"

# Terraform (like real AWS) refuses to destroy a bucket that still has objects in
# it -- emptying it first is the same real-world step this course's earlier S3
# lessons already covered, not something special to import.
aws s3 rm s3://hand-created-bucket/data.txt > /dev/null
terraform destroy -auto-approve -no-color > /dev/null
cd /tmp && rm -rf "$work"

check "before import, terraform wants to create a bucket that already exists" \
  "$(echo "$before_plan" | grep -c '1 to add, 0 to change, 0 to destroy')" "1"
check "after import, terraform recognises reality already matches the file" \
  "$(echo "$after_plan" | grep -c 'No changes')" "1"
check "importing never touched the data that was already in the bucket" "$still_there" "already-there"

What it printed, when the build ran it

a bucket created entirely by hand -- no terraform, no config file, exactly like lesson 28:
created and populated, with zero terraform involvement so far

plan BEFORE import: terraform has never heard of this bucket, so it wants to create it:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.manual will be created
  + resource "aws_s3_bucket" "manual" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "hand-created-bucket"
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + cors_rule (known after apply)

      + grant (known after apply)

      + lifecycle_rule (known after apply)

      + logging (known after apply)

      + object_lock_configuration (known after apply)

      + replication_configuration (known after apply)

      + server_side_encryption_configuration (known after apply)

      + versioning (known after apply)

      + website (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

terraform import: tell it the resource already exists, instead of creating a duplicate:
aws_s3_bucket.manual: Importing from ID "hand-created-bucket"...
aws_s3_bucket.manual: Import prepared!
  Prepared aws_s3_bucket for import
aws_s3_bucket.manual: Refreshing state... [id=hand-created-bucket]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.


plan AFTER import: same file, same real bucket -- now recognised as already matching:
aws_s3_bucket.manual: Refreshing state... [id=hand-created-bucket]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

and the object that was there before terraform ever ran is still exactly there:
already-there
ok   before import, terraform wants to create a bucket that already exists
ok   after import, terraform recognises reality already matches the file
ok   importing never touched the data that was already in the bucket

Import does not write the resource block for you. The resource "aws_s3_bucket" "manual" block above was written by hand, matching what the real bucket already looked like, before import ran -- import only links an existing block to an existing resource in the state file. A resource block that does not match reality closely enough still shows up as a change on the very next plan.

Destroy: the same plan/apply discipline, in reverse

terraform destroy is not a special, separate danger button -- it is a plan (what would be removed) and an apply (actually removing it), exactly like every panel already run in this lesson. The preview applies here too.

destroy: a plan and an apply, same as every other change

ran at build time 
work=/tmp/cd-course-tf-destroy
rm -rf "$work"
mkdir -p "$work"
cd "$work"

cat > main.tf << HCL
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
provider "aws" {
  region                      = "us-east-1"
  access_key                  = "test"
  secret_key                  = "test"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "$MOTO_ENDPOINT"
  }
}
resource "aws_s3_bucket" "temp" {
  bucket = "tf-teardown-bucket"
}
HCL

cp "$TF_SEED_LOCKFILE" .terraform.lock.hcl
terraform init -no-color > /dev/null
terraform apply -auto-approve -no-color > /dev/null
before_count=$(aws s3api list-buckets --query 'length(Buckets)' --output text)
echo "buckets before destroy: $before_count"

echo
echo "destroy's own preview -- what would be removed, before anything is:"
destroy_plan=$(terraform plan -destroy -no-color)
echo "$destroy_plan"

terraform destroy -auto-approve -no-color > /dev/null
after_count=$(aws s3api list-buckets --query 'length(Buckets)' --output text)
echo
echo "buckets after destroy: $after_count"

cd /tmp && rm -rf "$work"

check "the destroy plan previewed exactly one removal, before removing anything" \
  "$(echo "$destroy_plan" | grep -c '1 to destroy')" "1"
check "the bucket really existed before the destroy ran" "$before_count" "1"
check "the bucket is genuinely gone after, not just 'marked' gone" "$after_count" "0"

What it printed, when the build ran it

buckets before destroy: 1

destroy's own preview -- what would be removed, before anything is:
aws_s3_bucket.temp: Refreshing state... [id=tf-teardown-bucket]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_s3_bucket.temp will be destroyed
  - resource "aws_s3_bucket" "temp" {
      - arn                         = "arn:aws:s3:::tf-teardown-bucket" -> null
      - bucket                      = "tf-teardown-bucket" -> null
      - bucket_domain_name          = "tf-teardown-bucket.s3.amazonaws.com" -> null
      - bucket_regional_domain_name = "tf-teardown-bucket.s3.us-east-1.amazonaws.com" -> null
      - force_destroy               = false -> null
      - hosted_zone_id              = "Z3AQBSTGFYJSTF" -> null
      - id                          = "tf-teardown-bucket" -> null
      - object_lock_enabled         = false -> null
      - region                      = "us-east-1" -> null
      - tags                        = {} -> null
      - tags_all                    = {} -> null
        # (4 unchanged attributes hidden)

      - grant {
          - id          = "75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a" -> null
          - permissions = [
              - "FULL_CONTROL",
            ] -> null
          - type        = "CanonicalUser" -> null
            # (1 unchanged attribute hidden)
        }

      - versioning {
          - enabled    = false -> null
          - mfa_delete = false -> null
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

buckets after destroy: 0
ok   the destroy plan previewed exactly one removal, before removing anything
ok   the bucket really existed before the destroy ran
ok   the bucket is genuinely gone after, not just 'marked' gone

The whole loop, end to end. Write the file. Plan shows what would happen. Apply makes it happen, and records it in state. Plan again, any time later, catches drift from anything that bypassed Terraform. Import adopts what already existed. Destroy removes it, previewed the same way it was created. Every one of those steps was run for real in this lesson and lesson 29, against the exact same fake AWS every other lesson in this course has used -- none of it was a diagram.

Newsletter

New writing, when it lands.

Occasional notes on shipping software with AI agents. No schedule, no filler, unsubscribe in one click.