CI/CD all 3 written
What a pipeline actually is, a real pipeline file, and the deployment strategies that keep a bad release small.
32 What a pipeline actually is
Every command run by hand in Parts 1-9 of this course could be
put in a file and triggered by an event instead of a keystroke -- that is the entire idea
of a pipeline. This lesson does exactly that: writes a real
.github/workflows/*.yml file, and runs it with a tool that actually executes
it, not one that reads it back to you.
The same commands, now triggered instead of typed
A workflow file names an event to listen for, and a list of steps to run when it
fires. act, used throughout this part, is a real, free tool that reads that
exact file format and runs it in real Docker containers -- the same execution model
GitHub's own hosted runners use, just local and free.
work=$(mktemp -d)
cd "$work"
mkdir -p .github/workflows
git init -q -b main
git config user.name course
git config user.email course@example.com
cat > .github/workflows/ci.yml << 'YAML'
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: echo "running the test suite"
- run: echo "no failures"
YAML
echo "hello" > app.txt
git add -A
git commit -q -m "add app"
echo "running this file for real -- no GitHub account, no push, just the same engine GitHub uses:"
result=$(act push -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm 2>&1)
echo "$result"
cd / && rm -rf "$work"
check "the pipeline actually ran the real step's own output" \
"$([ "$(echo "$result" | grep -c 'running the test suite')" -ge 1 ] && echo yes)" "yes"
check "the pipeline reports the job succeeded" "$(echo "$result" | grep -c 'Job succeeded')" "1"What it printed, when the build ran it
running this file for real -- no GitHub account, no push, just the same engine GitHub uses: time="2026-09-15T21:24:58+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/test] ⭐ Run Set up job [CI/test] 🚀 Start image=node:20-bookworm-slim [CI/test] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/test] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/test] ✅ Success - Set up job [CI/test] ⭐ Run Main echo "running the test suite" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/test] | running the test suite [CI/test] ✅ Success - Main echo "running the test suite" [122.777494ms] [CI/test] ⭐ Run Main echo "no failures" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/1] user= workdir= [CI/test] | no failures [CI/test] ✅ Success - Main echo "no failures" [116.357953ms] [CI/test] ⭐ Run Complete job [CI/test] Cleaning up container for job test [CI/test] ✅ Success - Complete job [CI/test] 🏁 Job succeeded ok the pipeline actually ran the real step's own output ok the pipeline reports the job succeeded
What just happened, mechanically. act read the YAML
file, started a real container from a real image, and ran echo "running the test
suite" inside it exactly as written -- the same two lines a person would type by
hand, just triggered by the tool instead. Nothing about the commands themselves is
different from Parts 1-9; only who presses enter changed.
The trigger is the whole point: an event, not a person
on: [push] is not decoration -- it is the condition under which anything
runs at all. A workflow only reacts to events it explicitly lists, and act's own error for
an unmatched event is a real, useful fact, not a bug to work around.
work=$(mktemp -d)
cd "$work"
mkdir -p .github/workflows
git init -q -b main
git config user.name course
git config user.email course@example.com
cat > .github/workflows/ci.yml << 'YAML'
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: echo "this ran because of a push event, not a person"
YAML
git add -A
git commit -q -m init
echo "simulating the event this file DOES listen for:"
push_result=$(act push -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm 2>&1)
echo "$push_result"
echo
echo "simulating an event this file does NOT listen for:"
pr_result=$(act pull_request -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm 2>&1)
echo "$pr_result"
cd / && rm -rf "$work"
check "the push event triggered the real job" \
"$([ "$(echo "$push_result" | grep -c 'this ran because of a push event')" -ge 1 ] && echo yes)" "yes"
check "an event this workflow does not listen for triggers nothing" \
"$([ "$(echo "$pr_result" | grep -c 'this ran because')" = "0" ] && echo yes)" "yes"
check "act states plainly that no job matched that event, rather than failing silently" \
"$(echo "$pr_result" | grep -c 'Could not find any stages to run')" "1"What it printed, when the build ran it
simulating the event this file DOES listen for: time="2026-09-15T21:25:00+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/test] ⭐ Run Set up job [CI/test] 🚀 Start image=node:20-bookworm-slim [CI/test] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/test] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/test] ✅ Success - Set up job [CI/test] ⭐ Run Main echo "this ran because of a push event, not a person" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/test] | this ran because of a push event, not a person [CI/test] ✅ Success - Main echo "this ran because of a push event, not a person" [118.30941ms] [CI/test] ⭐ Run Complete job [CI/test] Cleaning up container for job test [CI/test] ✅ Success - Complete job [CI/test] 🏁 Job succeeded simulating an event this file does NOT listen for: time="2026-09-15T21:25:01+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" Error: Could not find any stages to run. View the valid jobs with `act --list`. Use `act --help` to find how to filter by Job ID/Workflow/Event Name ok the push event triggered the real job ok an event this workflow does not listen for triggers nothing ok act states plainly that no job matched that event, rather than failing silently
Stages block each other on purpose -- a failure stops the pipeline where it stands
A pipeline with several jobs chained by needs: is not several independent
scripts -- a failure in an earlier one prevents every later one from running at all. This
is the mechanism that stops a broken build from ever reaching deploy.
work=$(mktemp -d)
cd "$work"
mkdir -p .github/workflows
git init -q -b main
git config user.name course
git config user.email course@example.com
cat > .github/workflows/ci.yml << 'YAML'
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: echo "running tests"
- run: exit 1
build:
needs: test
runs-on: ubuntu-latest
steps:
- run: echo "THIS SHOULD NEVER PRINT -- test already failed"
deploy:
needs: build
runs-on: ubuntu-latest
steps:
- run: echo "THIS SHOULD NEVER PRINT EITHER"
YAML
git add -A
git commit -q -m init
echo "a three-stage pipeline where the FIRST stage fails on purpose:"
set +e
result=$(act push -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm 2>&1)
pipeline_exit=$?
set -e
echo "$result"
echo "the pipeline's own exit code: $pipeline_exit"
cd / && rm -rf "$work"
check "the test stage genuinely failed" \
"$([ "$(echo "$result" | grep -c 'Job failed')" -ge 1 ] && echo yes)" "yes"
check "the build stage never ran at all" "$(echo "$result" | grep -c 'THIS SHOULD NEVER PRINT -- test already failed')" "0"
check "the deploy stage never ran either -- it depended on build, which never started" \
"$(echo "$result" | grep -c 'THIS SHOULD NEVER PRINT EITHER')" "0"
check "the whole pipeline reports failure, which is what a real CI system checks before deploying" \
"$([ "$pipeline_exit" -ne 0 ] && echo yes)" "yes"What it printed, when the build ran it
a three-stage pipeline where the FIRST stage fails on purpose: time="2026-09-15T21:25:02+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/test] ⭐ Run Set up job [CI/test] 🚀 Start image=node:20-bookworm-slim [CI/test] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/test] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/test] ✅ Success - Set up job [CI/test] ⭐ Run Main echo "running tests" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/test] | running tests [CI/test] ✅ Success - Main echo "running tests" [139.228338ms] [CI/test] ⭐ Run Main exit 1 [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/1] user= workdir= [CI/test] ❌ Failure - Main exit 1 [144.105551ms] [CI/test] exitcode '1': failure [CI/test] ⭐ Run Complete job [CI/test] Cleaning up container for job test [CI/test] ✅ Success - Complete job [CI/test] 🏁 Job failed Error: Job 'test' failed the pipeline's own exit code: 1 ok the test stage genuinely failed ok the build stage never ran at all ok the deploy stage never ran either -- it depended on build, which never started ok the whole pipeline reports failure, which is what a real CI system checks before deploying
This is the entire mechanism behind "CI blocks bad merges." A branch-protection rule that says "tests must pass before merging" is not special GitHub magic -- it is exactly this: a pipeline reports failure (a real non-zero exit, as checked above), and GitHub simply refuses to allow the merge button to be pressed while the most recent run for that commit reports anything other than success.
What a pipeline is, stated plainly
- A list of commands -- the exact kind already run by hand throughout this course -- stored in a file instead of a terminal history.
- A trigger, naming the event that runs them: a push, a pull request, a schedule, or (as later lessons in this part will use) a manual button.
- Stages that depend on each other, so a failure early stops everything that would have come after it -- not as a safety feature bolted on, but as the direct, mechanical consequence of one job needing another to have succeeded first.
33 A real pipeline file, stage by stage
A build-test-deploy pipeline is the same three ideas as any software process -- produce something, check it, ship it -- written as steps a machine runs in order, stopping the moment one of them fails. This lesson writes that shape once, and runs it for real twice: once clean, once with a deliberate test failure.
Build, then test, in the same job -- the artifact only has to survive one container
Steps in the same job share a filesystem; steps in a different job do not (each job gets its own fresh container). A build step's output is simply a file on disk by the time the test step runs, as long as both live in the same job.
work=$(mktemp -d)
cd "$work"
mkdir -p .github/workflows
git init -q -b main
git config user.name course
git config user.email course@example.com
cat > .github/workflows/ci.yml << 'YAML'
name: CI
on: [push]
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build
run: |
mkdir -p dist
echo "console.log('built app, version 1.0')" > dist/app.js
- name: Test the build
run: node dist/app.js
YAML
git add -A
git commit -q -m init
result=$(act push -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm 2>&1)
echo "$result"
cd / && rm -rf "$work"
check "the build step really produced a file the test step could run" \
"$([ "$(echo "$result" | grep -c 'built app, version 1.0')" -ge 1 ] && echo yes)" "yes"
check "both steps, and the whole job, succeeded" "$(echo "$result" | grep -c 'Job succeeded')" "1"What it printed, when the build ran it
time="2026-09-15T21:25:03+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/build-and-test] ⭐ Run Set up job [CI/build-and-test] 🚀 Start image=node:20-bookworm-slim [CI/build-and-test] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/build-and-test] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/build-and-test] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/build-and-test] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/build-and-test] ✅ Success - Set up job [CI/build-and-test] ⭐ Run Main actions/checkout@v4 [CI/build-and-test] 🐳 docker cp src=/var/folders/kw/d8jyj6d53qj57mb1txzgjdlc0000gn/T/tmp.ihF8iC4UTq/. dst=/var/folders/kw/d8jyj6d53qj57mb1txzgjdlc0000gn/T/tmp.ihF8iC4UTq [CI/build-and-test] ✅ Success - Main actions/checkout@v4 [53.055612ms] [CI/build-and-test] ⭐ Run Main Build [CI/build-and-test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/1] user= workdir= [CI/build-and-test] ✅ Success - Main Build [141.51551ms] [CI/build-and-test] ⭐ Run Main Test the build [CI/build-and-test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/2] user= workdir= [CI/build-and-test] | built app, version 1.0 [CI/build-and-test] ✅ Success - Main Test the build [232.03635ms] [CI/build-and-test] ⭐ Run Complete job [CI/build-and-test] Cleaning up container for job build-and-test [CI/build-and-test] ✅ Success - Complete job [CI/build-and-test] 🏁 Job succeeded ok the build step really produced a file the test step could run ok both steps, and the whole job, succeeded
Secrets: available to the step that needs them, invisible in the log even by accident
A real pipeline needs credentials -- an API key, a deploy token -- without ever printing them anywhere a log viewer could see. This is not a convention engineers have to remember; the runner itself redacts a known secret value from anything a step prints, even a step that carelessly echoes it directly.
work=$(mktemp -d)
cd "$work"
mkdir -p .github/workflows
git init -q -b main
git config user.name course
git config user.email course@example.com
cat > .github/workflows/ci.yml << 'YAML'
name: CI
on: [push]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Use the secret correctly
env:
API_KEY: ${{ secrets.API_KEY }}
run: echo "key is present, length ${#API_KEY}"
- name: A step that makes the mistake of printing it directly
env:
API_KEY: ${{ secrets.API_KEY }}
run: echo "the key is $API_KEY"
YAML
git add -A
git commit -q -m init
result=$(act push -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm \
--secret API_KEY=super-secret-value-123 2>&1)
echo "$result"
cd / && rm -rf "$work"
check "the secret's real length was usable inside the step" \
"$(echo "$result" | grep -c 'length 22')" "1"
check "even a step that tried to print the raw secret got it redacted, not leaked" \
"$(echo "$result" | grep -c 'the key is \*\*\*')" "1"
check "the real secret value never appears anywhere in the captured output" \
"$(echo "$result" | grep -c 'super-secret-value-123')" "0"What it printed, when the build ran it
time="2026-09-15T21:25:06+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/deploy] ⭐ Run Set up job [CI/deploy] 🚀 Start image=node:20-bookworm-slim [CI/deploy] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/deploy] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/deploy] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/deploy] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/deploy] ✅ Success - Set up job [CI/deploy] ⭐ Run Main Use the secret correctly [CI/deploy] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/deploy] | key is present, length 22 [CI/deploy] ✅ Success - Main Use the secret correctly [164.853549ms] [CI/deploy] ⭐ Run Main A step that makes the mistake of printing it directly [CI/deploy] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/1] user= workdir= [CI/deploy] | the key is *** [CI/deploy] ✅ Success - Main A step that makes the mistake of printing it directly [138.797486ms] [CI/deploy] ⭐ Run Complete job [CI/deploy] Cleaning up container for job deploy [CI/deploy] ✅ Success - Complete job [CI/deploy] 🏁 Job succeeded ok the secret's real length was usable inside the step ok even a step that tried to print the raw secret got it redacted, not leaked ok the real secret value never appears anywhere in the captured output
What this redaction does and does not protect against. The runner masks a KNOWN secret value wherever it appears verbatim in output -- it cannot mask a secret that has been transformed first (base64-encoded, reversed, split across two print statements). Part 13 (Security) returns to this with the real, further consequence: a secret is never actually safe once a step has had the chance to print it in any form, which is why access to modify a pipeline file is itself a security boundary, not just a taste preference.
A deploy stage that only exists because the one before it succeeded
Putting needs: test on a deploy job is the entire mechanism behind "we
never deploy a broken build" -- not a policy anyone has to remember to follow, but a
structural fact about which jobs act is willing to start.
run_pipeline() {
local should_fail="$1"
local work=$(mktemp -d)
cd "$work"
mkdir -p .github/workflows
git init -q -b main
git config user.name course
git config user.email course@example.com
local test_step="echo \"tests passed\""
if [ "$should_fail" = "yes" ]; then
test_step="exit 1"
fi
cat > .github/workflows/ci.yml << YAML
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: echo "running tests"
- run: $test_step
deploy:
needs: test
runs-on: ubuntu-latest
steps:
- run: echo "DEPLOYED TO PRODUCTION"
YAML
git add -A
git commit -q -m init
set +e
act push -P ubuntu-latest=node:20-bookworm-slim --pull=false --rm 2>&1
set -e
cd / && rm -rf "$work"
}
echo "=== run 1: tests pass ==="
clean_result=$(run_pipeline no)
echo "$clean_result"
echo
echo "=== run 2: tests fail ==="
broken_result=$(run_pipeline yes)
echo "$broken_result"
check "when tests pass, the deploy step really runs" \
"$([ "$(echo "$clean_result" | grep -c 'DEPLOYED TO PRODUCTION')" -ge 1 ] && echo yes)" "yes"
check "when tests fail, deploy never runs, with the exact same pipeline file" \
"$(echo "$broken_result" | grep -c 'DEPLOYED TO PRODUCTION')" "0"What it printed, when the build ran it
=== run 1: tests pass === time="2026-09-15T21:25:08+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/test] ⭐ Run Set up job [CI/test] 🚀 Start image=node:20-bookworm-slim [CI/test] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/test] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/test] ✅ Success - Set up job [CI/test] ⭐ Run Main echo "running tests" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/test] | running tests [CI/test] ✅ Success - Main echo "running tests" [179.390772ms] [CI/test] ⭐ Run Main echo "tests passed" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/1] user= workdir= [CI/test] | tests passed [CI/test] ✅ Success - Main echo "tests passed" [155.420423ms] [CI/test] ⭐ Run Complete job [CI/test] Cleaning up container for job test [CI/test] ✅ Success - Complete job [CI/test] 🏁 Job succeeded [CI/deploy] ⭐ Run Set up job [CI/deploy] 🚀 Start image=node:20-bookworm-slim [CI/deploy] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/deploy] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/deploy] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/deploy] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/deploy] ✅ Success - Set up job [CI/deploy] ⭐ Run Main echo "DEPLOYED TO PRODUCTION" [CI/deploy] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/deploy] | DEPLOYED TO PRODUCTION [CI/deploy] ✅ Success - Main echo "DEPLOYED TO PRODUCTION" [152.338057ms] [CI/deploy] ⭐ Run Complete job [CI/deploy] Cleaning up container for job deploy [CI/deploy] ✅ Success - Complete job [CI/deploy] 🏁 Job succeeded === run 2: tests fail === time="2026-09-15T21:25:12+01:00" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'" [CI/test] ⭐ Run Set up job [CI/test] 🚀 Start image=node:20-bookworm-slim [CI/test] 🐳 docker pull image=node:20-bookworm-slim platform= username= forcePull=false [CI/test] 🐳 docker create image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker run image=node:20-bookworm-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [CI/test] 🐳 docker exec cmd=[node --no-warnings -e console.log(process.execPath)] user= workdir= [CI/test] ✅ Success - Set up job [CI/test] ⭐ Run Main echo "running tests" [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/0] user= workdir= [CI/test] | running tests [CI/test] ✅ Success - Main echo "running tests" [162.700319ms] [CI/test] ⭐ Run Main exit 1 [CI/test] 🐳 docker exec cmd=[bash -e /var/run/act/workflow/1] user= workdir= [CI/test] ❌ Failure - Main exit 1 [181.693586ms] [CI/test] exitcode '1': failure [CI/test] ⭐ Run Complete job [CI/test] Cleaning up container for job test [CI/test] ✅ Success - Complete job [CI/test] 🏁 Job failed Error: Job 'test' failed ok when tests pass, the deploy step really runs ok when tests fail, deploy never runs, with the exact same pipeline file
Nothing about the deploy job's own file changed between the two runs
above. The only difference was whether the test job -- upstream of it --
succeeded. This is the whole design: a deploy stage does not need its own logic to
"check if it's safe to deploy," because needs: already makes that
unreachable otherwise.
34 Blue-green, canary and rolling deploys
Every strategy below answers the same question -- "how do old
and new versions coexist during a deploy" -- differently. None of them needs a new AWS
service: all three are built entirely from register-targets and
deregister-targets against a target group, the same two calls lesson 21 used
to demonstrate what a load balancer actually does.
Blue-green: an instant, all-or-nothing swap
Blue-green keeps two complete, independent environments and moves all traffic from one to the other in a single cutover -- there is no moment where some requests hit the old version and some hit the new one.
created=$(aws ec2 create-vpc --cidr-block 10.20.0.0/16 --output json)
vpc_id=$(echo "$created" | grep -o '"VpcId": "[^"]*"' | head -1 | cut -d'"' -f4)
tg=$(aws elbv2 create-target-group --name bluegreen-tg --protocol HTTP --port 80 --vpc-id "$vpc_id" --output json)
tg_arn=$(echo "$tg" | grep -o '"TargetGroupArn": "[^"]*"' | head -1 | cut -d'"' -f4)
blue=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --count 1 --output json)
blue_id=$(echo "$blue" | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
aws elbv2 register-targets --target-group-arn "$tg_arn" --targets Id="$blue_id"
echo "blue (current version) is live and serving all traffic: $blue_id"
echo
echo "the new version is built and launched, completely separately -- not registered yet:"
green=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --count 1 --output json)
green_id=$(echo "$green" | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
during=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --output json)
during_ids=$(echo "$during" | grep -o '"Id": "[^"]*"')
echo "while green exists but is not registered, traffic still only reaches: $during_ids"
echo
echo "the cutover -- one moment where traffic moves entirely from blue to green:"
aws elbv2 register-targets --target-group-arn "$tg_arn" --targets Id="$green_id"
aws elbv2 deregister-targets --target-group-arn "$tg_arn" --targets Id="$blue_id"
after=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --output json)
after_count=$(echo "$after" | grep -c '"Id"')
check "before cutover, green existed but received nothing" \
"$(echo "$during_ids" | grep -c "$green_id")" "0"
check "after cutover, exactly one instance is registered -- never both at once" "$after_count" "1"
check "the one instance registered after cutover is green, not blue" \
"$(echo "$after" | grep -c "$green_id")" "1"What it printed, when the build ran it
blue (current version) is live and serving all traffic: i-695b6462aedfa5771 the new version is built and launched, completely separately -- not registered yet: while green exists but is not registered, traffic still only reaches: "Id": "i-695b6462aedfa5771" the cutover -- one moment where traffic moves entirely from blue to green: ok before cutover, green existed but received nothing ok after cutover, exactly one instance is registered -- never both at once ok the one instance registered after cutover is green, not blue
What blue-green actually costs. Both environments have to exist, fully sized, at the same time during the cutover -- double the compute, even if only briefly. What it buys back is the check above: no request is ever served by a half-deployed fleet, and rolling back is exactly the same swap run in reverse.
Canary: a small, known fraction of traffic, on purpose
A canary release does not swap anything -- it sends a deliberately small percentage of requests to the new version while the rest keep hitting the old one, so a real problem shows up on a small slice of traffic before it reaches everyone.
created=$(aws ec2 create-vpc --cidr-block 10.21.0.0/16 --output json)
vpc_id=$(echo "$created" | grep -o '"VpcId": "[^"]*"' | head -1 | cut -d'"' -f4)
sub=$(aws ec2 create-subnet --vpc-id "$vpc_id" --cidr-block 10.21.1.0/24 --availability-zone us-east-1a --output json)
sub_id=$(echo "$sub" | grep -o '"SubnetId": "[^"]*"' | head -1 | cut -d'"' -f4)
alb=$(aws elbv2 create-load-balancer --name canary-alb --subnets "$sub_id" --output json)
alb_arn=$(echo "$alb" | grep -o '"LoadBalancerArn": "[^"]*"' | head -1 | cut -d'"' -f4)
stable_tg=$(aws elbv2 create-target-group --name stable-tg --protocol HTTP --port 80 --vpc-id "$vpc_id" --output json)
stable_arn=$(echo "$stable_tg" | grep -o '"TargetGroupArn": "[^"]*"' | head -1 | cut -d'"' -f4)
canary_tg=$(aws elbv2 create-target-group --name canary-tg --protocol HTTP --port 80 --vpc-id "$vpc_id" --output json)
canary_arn=$(echo "$canary_tg" | grep -o '"TargetGroupArn": "[^"]*"' | head -1 | cut -d'"' -f4)
echo "a listener splitting traffic 90/10 between the stable version and the canary:"
listener=$(aws elbv2 create-listener --load-balancer-arn "$alb_arn" --protocol HTTP --port 80 \
--default-actions "[{\"Type\":\"forward\",\"ForwardConfig\":{\"TargetGroups\":[{\"TargetGroupArn\":\"$stable_arn\",\"Weight\":90},{\"TargetGroupArn\":\"$canary_arn\",\"Weight\":10}]}}]" \
--output json)
echo "$listener"
weights=$(echo "$listener" | grep -o '"Weight": [0-9]*' | grep -o '[0-9]*' | tr '\n' ' ' | sed 's/ $//')
check "the listener really carries two weighted target groups, not one" \
"$(echo "$listener" | grep -c 'TargetGroupArn')" "2"
check "the weights are exactly the 90/10 split configured, not something else" "$weights" "90 10"What it printed, when the build ran it
a listener splitting traffic 90/10 between the stable version and the canary:
{
"Listeners": [
{
"ListenerArn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/canary-alb/d758ec231c53f47a/b64eb4cdae52ae1d",
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/canary-alb/d758ec231c53f47a",
"Port": 80,
"Protocol": "HTTP",
"SslPolicy": "ELBSecurityPolicy-2016-08",
"DefaultActions": [
{
"Type": "forward",
"ForwardConfig": {
"TargetGroups": [
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/stable-tg/60aa5eba9446986a",
"Weight": 90
},
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/canary-tg/4175d8352685ff61",
"Weight": 10
}
],
"TargetGroupStickinessConfig": {
"Enabled": false
}
}
}
],
"AlpnPolicy": []
}
]
}
ok the listener really carries two weighted target groups, not one
ok the weights are exactly the 90/10 split configured, not something else
The number that matters here is the BLAST RADIUS, not the traffic split itself. A 10% canary means a bug in the new version affects roughly one in ten requests, for however long it takes monitoring (Part 12) to notice and someone to shift that weight back to 0 -- not zero requests. Canary reduces the size of a mistake; it does not prevent one from reaching real traffic at all, which is the tradeoff blue-green does not have and canary accepts on purpose, in exchange for a much smaller cost than running two full environments.
Rolling: replace capacity gradually, never dropping below full strength
A rolling deploy swaps instances one (or a few) at a time -- old and new genuinely coexist for the whole rollout, but the fleet's total serving capacity never dips, because a replacement is registered before its predecessor is removed.
created=$(aws ec2 create-vpc --cidr-block 10.22.0.0/16 --output json)
vpc_id=$(echo "$created" | grep -o '"VpcId": "[^"]*"' | head -1 | cut -d'"' -f4)
tg=$(aws elbv2 create-target-group --name rolling-tg --protocol HTTP --port 80 --vpc-id "$vpc_id" --output json)
tg_arn=$(echo "$tg" | grep -o '"TargetGroupArn": "[^"]*"' | head -1 | cut -d'"' -f4)
echo "three v1 instances, all registered and serving:"
v1_ids=""
for i in 1 2 3; do
inst=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --count 1 --output json)
id=$(echo "$inst" | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
aws elbv2 register-targets --target-group-arn "$tg_arn" --targets Id="$id" > /dev/null
v1_ids="$v1_ids $id"
done
before_count=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --query 'length(TargetHealthDescriptions)' --output text)
echo "serving: $before_count instances, all v1"
echo
echo "rolling one v1 out for one v2, one at a time -- register new BEFORE deregistering old:"
min_during_roll=999
for old_id in $v1_ids; do
new_inst=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --count 1 --output json)
new_id=$(echo "$new_inst" | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
aws elbv2 register-targets --target-group-arn "$tg_arn" --targets Id="$new_id" > /dev/null
mid_count=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --query 'length(TargetHealthDescriptions)' --output text)
echo " registered a replacement -- momentarily serving $mid_count (old + new, briefly above normal capacity)"
aws elbv2 deregister-targets --target-group-arn "$tg_arn" --targets Id="$old_id" > /dev/null
after_swap_count=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --query 'length(TargetHealthDescriptions)' --output text)
echo " deregistered the old one -- back to $after_swap_count"
if [ "$after_swap_count" -lt "$min_during_roll" ]; then min_during_roll=$after_swap_count; fi
done
final_count=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --query 'length(TargetHealthDescriptions)' --output text)
echo
echo "final fleet size: $final_count"
check "the fleet started at full strength" "$before_count" "3"
check "capacity never dropped below full strength at any point during the roll" "$min_during_roll" "3"
check "the fleet ends at the same size it started, now entirely replaced" "$final_count" "3"What it printed, when the build ran it
three v1 instances, all registered and serving: serving: 3 instances, all v1 rolling one v1 out for one v2, one at a time -- register new BEFORE deregistering old: registered a replacement -- momentarily serving 4 (old + new, briefly above normal capacity) deregistered the old one -- back to 3 registered a replacement -- momentarily serving 4 (old + new, briefly above normal capacity) deregistered the old one -- back to 3 registered a replacement -- momentarily serving 4 (old + new, briefly above normal capacity) deregistered the old one -- back to 3 final fleet size: 3 ok the fleet started at full strength ok capacity never dropped below full strength at any point during the roll ok the fleet ends at the same size it started, now entirely replaced
The order of the two calls in each swap is the entire mechanism.
register-targets for the replacement runs BEFORE
deregister-targets for the one it replaces -- reverse that order and the
fleet genuinely drops to two instances, briefly under capacity, for every swap. This is
not a subtlety; it is the one design decision that makes "rolling" mean "never fewer
than full strength" instead of "temporarily short-staffed."
Choosing between them
- Blue-green: zero version-mixing, instant rollback, but pays for two full environments at once, even briefly. Best when a bug reaching ANY real traffic is unacceptable.
- Canary: real traffic hits the new version early, on a small, controllable slice -- the fastest way to catch a real production-only bug before it is everyone's problem, at the cost of accepting some real requests will hit it.
- Rolling: no extra capacity needed beyond one or two spare instances at a time, but old and new code run side by side for the whole rollout -- fine for a stateless web tier, genuinely risky for anything where two versions disagreeing about a shared format (a database schema, a message shape) causes real problems.