Cost & production readiness all 3 written
Pricing models you calculate by hand, incident response, and the system design questions this job gets asked.
45 Pricing models, and a cost calculator you build yourself
Every AWS service is billed by a different unit -- EC2 by the hour, Lambda by the request and the gigabyte-second, S3 by the gigabyte-month and by every byte that leaves it. None of that is exotic; it is arithmetic once you know the units. This lesson builds the calculator, not a link to one, so the next time a number like "$1,530 a month" shows up it is because you multiplied it out, not because a pricing page said so.
Why this lesson is JavaScript, not shell, and what that means here. Every other lesson in this course ran a real command against moto, a free local AWS stand-in. Billing itself is the one thing moto does not fake: asking its Pricing API for a real EC2 rate returns a server error, and asking Cost Explorer for cost data returns an empty result with nothing seeded in it -- confirmed directly while building this lesson, not assumed. So rather than print a fake "$0.00" that would be a worse lie than admitting the gap, this lesson calculates with AWS's own published on-demand rates as constants. Every number below is real; what is simulated is a bill, not a price.
EC2: billed by the hour, and the number that surprises people is the month
An hourly rate looks small enough to ignore. Multiplied by the roughly 730 hours in a month, it stops looking small -- which is the entire reason cloud bills surprise people who only ever looked at the per-hour figure on the pricing page.
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
const test = (name, got, want) => console.log(
(same(got, want) ? 'ok ' : 'FAIL ') + name +
(same(got, want) ? '' : ` got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`)
);
// Never use === on a computed float. Money and percentages below are rounded first.
const close = (a, b, tol = 1e-9) => Math.abs(a - b) < tol;
// Real, current on-demand rates for us-east-1, published by AWS.
const HOURLY_RATE = { 't3.micro': 0.0104, 't3.small': 0.0208, 't3.medium': 0.0416 };
const HOURS_PER_MONTH = 730; // 24 * 365 / 12, the figure AWS itself uses for estimates
function monthlyCost(instanceType, count = 1) {
if (!(instanceType in HOURLY_RATE)) throw new Error('unknown instance type: ' + instanceType);
return HOURLY_RATE[instanceType] * count * HOURS_PER_MONTH;
}
const oneMicro = monthlyCost('t3.micro');
const threeInstances = monthlyCost('t3.small', 2) + monthlyCost('t3.medium', 1);
console.log('one t3.micro, left running all month : $' + oneMicro.toFixed(2));
console.log('two t3.small + one t3.medium, all month: $' + threeInstances.toFixed(2));
console.log('that second fleet, for a year : $' + (threeInstances * 12).toFixed(2));
test('a single small instance left running costs about $7.59 a month, not cents', close(oneMicro, 7.592), true);
test('a three-instance fleet costs about $60.74 a month', close(threeInstances, 60.736), true);
test('an unknown instance type is refused rather than silently priced at zero',
(() => { try { monthlyCost('made-up.large'); return false; } catch { return true; } })(), true);Output
The instance was never the point -- the hours running were. An instance that is stopped is not billed for compute at all (its attached storage still is, which lesson 23 already covered) -- so the single highest-leverage cost lever on EC2 is not "pick a cheaper instance type," it is "is this instance running when nobody needs it." A dev environment left on overnight and over a weekend is paying for roughly 128 of its 168 hours a week doing nothing.
The same fleet, three purchase options -- because the sticker price is a choice
On-demand is the rate above: pay for exactly the hours used, commit to nothing. Reserved Instances trade a 1- or 3-year commitment for a real, published discount. Spot trades "AWS can reclaim this with two minutes' notice" for the deepest discount of all -- fine for a batch job that checkpoints, wrong for anything that must not be interrupted.
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
const test = (name, got, want) => console.log(
(same(got, want) ? 'ok ' : 'FAIL ') + name +
(same(got, want) ? '' : ` got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`)
);
// Never use === on a computed float. Money and percentages below are rounded first.
const close = (a, b, tol = 1e-9) => Math.abs(a - b) < tol;
const HOURLY_RATE = { 't3.small': 0.0208, 't3.medium': 0.0416 };
const HOURS_PER_MONTH = 730;
// Real, published AWS discount figures: a 3-year all-upfront Standard Reserved
// Instance, and Spot's typical savings range, both against the same on-demand rate.
const RESERVED_3YR_DISCOUNT = 0.72;
const SPOT_DISCOUNT = 0.90;
const onDemandMonthly = HOURLY_RATE['t3.small'] * 2 * HOURS_PER_MONTH + HOURLY_RATE['t3.medium'] * 1 * HOURS_PER_MONTH;
// The realistic mix: the two small instances are a stateless, interruption-tolerant
// worker pool (Spot fits), the one medium instance is something that must not vanish
// mid-request (a 3-year Reserved commitment fits instead).
const spotHourly = HOURLY_RATE['t3.small'] * (1 - SPOT_DISCOUNT);
const reservedMediumHourly = HOURLY_RATE['t3.medium'] * (1 - RESERVED_3YR_DISCOUNT);
const mixedMonthly = (spotHourly * 2 + reservedMediumHourly * 1) * HOURS_PER_MONTH;
const reduction = 1 - mixedMonthly / onDemandMonthly;
console.log('same fleet, on-demand : $' + onDemandMonthly.toFixed(2) + ' / month');
console.log('same fleet, spot + 3yr reserved : $' + mixedMonthly.toFixed(2) + ' / month');
console.log('reduction : ' + (reduction * 100).toFixed(0) + '%');
console.log();
console.log('the SAME infrastructure. Nothing about what it does changed -- only the commitment.');
test('the on-demand baseline for this fleet is about $60.74 a month', close(onDemandMonthly, 60.736), true);
test('mixing spot and a 3-year reserved commitment costs about $11.54 a month', close(mixedMonthly, 11.53984), true);
test('that is roughly an 81% reduction, for identical infrastructure', close(reduction, 0.81, 0.005), true);Output
What the discount is actually paying for, in each case. A Reserved Instance's discount is the price of certainty -- AWS can plan its own capacity around a commitment, so it shares the saving. Spot's much deeper discount is the price of uncertainty going the other way: AWS can take that capacity back with two minutes' notice if it needs it for someone paying full price. Putting a stateful database on Spot to chase that discount is not a cost optimisation, it is an outage waiting for a Tuesday.
Lambda: two meters running at once, and the break-even point against EC2
Lambda bills per request AND per GB-second (memory allocated times execution time), with a real, generous free tier on both meters every month. That second meter is why "serverless is always cheaper" is not universally true -- it depends entirely on how often the function runs.
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
const test = (name, got, want) => console.log(
(same(got, want) ? 'ok ' : 'FAIL ') + name +
(same(got, want) ? '' : ` got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`)
);
// Never use === on a computed float. Money and percentages below are rounded first.
const close = (a, b, tol = 1e-9) => Math.abs(a - b) < tol;
// Real, published AWS Lambda rates and free tier, x86 architecture.
const REQUEST_PRICE = 0.20 / 1e6; // dollars per request, after the free tier
const GBS_PRICE = 0.0000166667; // dollars per GB-second, after the free tier
const FREE_REQUESTS = 1000000; // every month, forever, not a trial
const FREE_GB_SECONDS = 400000;
function lambdaMonthlyCost(invocations, memoryGb, durationSeconds) {
const billableRequests = Math.max(0, invocations - FREE_REQUESTS);
const totalGbSeconds = invocations * memoryGb * durationSeconds;
const billableGbSeconds = Math.max(0, totalGbSeconds - FREE_GB_SECONDS);
return billableRequests * REQUEST_PRICE + billableGbSeconds * GBS_PRICE;
}
// A modest function: 128 MB, 200 ms -- typical for a small API handler or a
// thumbnail resize, the kind of thing this course's Part 8 actually built.
const MEMORY_GB = 0.125;
const DURATION_S = 0.2;
const light = lambdaMonthlyCost(500000, MEMORY_GB, DURATION_S);
const moderate = lambdaMonthlyCost(5000000, MEMORY_GB, DURATION_S);
const heavy = lambdaMonthlyCost(30000000, MEMORY_GB, DURATION_S);
console.log('500,000 invocations/month : $' + light.toFixed(4) + ' (inside the free tier)');
console.log('5,000,000 invocations/month : $' + moderate.toFixed(4));
console.log('30,000,000 invocations/month: $' + heavy.toFixed(4));
console.log();
// The comparison that actually matters: at what volume does this function cost
// more than just leaving a t3.micro running all month, doing the same job?
const EC2_BASELINE_MONTHLY = 0.0104 * 730; // this course's own playground panel, same rate
let lo = 1000000, hi = 100000000;
for (let i = 0; i < 60; i++) {
const mid = (lo + hi) / 2;
if (lambdaMonthlyCost(mid, MEMORY_GB, DURATION_S) < EC2_BASELINE_MONTHLY) lo = mid; else hi = mid;
}
const breakEven = lo;
console.log('a t3.micro left running all month costs : $' + EC2_BASELINE_MONTHLY.toFixed(3));
console.log('Lambda matches that cost at about : ' + Math.round(breakEven / 1e6 * 10) / 10 + ' million invocations/month');
test('500,000 calls a month lands entirely inside the free tier', close(light, 0), true);
test('5,000,000 calls a month costs about 80 cents', close(moderate, 0.8), true);
test('30,000,000 calls a month costs more than leaving a small instance running', heavy > EC2_BASELINE_MONTHLY, true);
test('the break-even point is above 23 million invocations a month', breakEven > 23000000, true);
test('below that break-even point, Lambda is the cheaper choice for this workload', lambdaMonthlyCost(10000000, MEMORY_GB, DURATION_S) < EC2_BASELINE_MONTHLY, true);Output
This is the actual decision, not a rule of thumb. "Serverless is cheaper for spiky, low-volume workloads and more expensive at sustained high volume" is true, and the panel above is exactly where the crossover sits for one specific function shape. Change the memory or duration and the crossover point moves -- which is the whole reason to keep the formula, not just the one number it produced today.
S3: the bill nobody predicts, because it has three separate line items
Storage is the number everyone estimates. Data transfer OUT to the internet is the one that quietly dominates a real bill for anything that serves files to users -- and it is priced completely separately from storage, per gigabyte, every month, with no cap.
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
const test = (name, got, want) => console.log(
(same(got, want) ? 'ok ' : 'FAIL ') + name +
(same(got, want) ? '' : ` got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`)
);
// Never use === on a computed float. Money and percentages below are rounded first.
const close = (a, b, tol = 1e-9) => Math.abs(a - b) < tol;
// Real, published AWS rates, us-east-1.
const STORAGE_PRICE_PER_GB = 0.023; // S3 Standard, per GB per month
const EGRESS_PRICE_PER_GB = 0.09; // data transfer OUT to the internet, first 10 TB
function s3MonthlyCost(storedGb, egressGbPerMonth) {
const storageCost = storedGb * STORAGE_PRICE_PER_GB;
const egressCost = egressGbPerMonth * EGRESS_PRICE_PER_GB;
return { storageCost, egressCost, total: storageCost + egressCost };
}
// A small media bucket: 50 GB stored, and two different real usage patterns.
const rarelyDownloaded = s3MonthlyCost(50, 10); // an internal backups bucket
const publiclyServed = s3MonthlyCost(50, 2000); // the same 50GB, served to users a lot
console.log('50 GB stored, rarely downloaded (10 GB egress/month):');
console.log(' storage: $' + rarelyDownloaded.storageCost.toFixed(2) + ' egress: $' + rarelyDownloaded.egressCost.toFixed(2) + ' total: $' + rarelyDownloaded.total.toFixed(2));
console.log();
console.log('the SAME 50 GB stored, served publicly (2,000 GB egress/month):');
console.log(' storage: $' + publiclyServed.storageCost.toFixed(2) + ' egress: $' + publiclyServed.egressCost.toFixed(2) + ' total: $' + publiclyServed.total.toFixed(2));
console.log();
console.log('storage did not change at all. Egress alone is ' +
(publiclyServed.egressCost / rarelyDownloaded.total).toFixed(0) + 'x the first bucket\'s entire bill.');
test('storing 50 GB costs $1.15 a month, regardless of how often it is read', close(rarelyDownloaded.storageCost, 1.15), true);
test('a rarely-downloaded bucket is dominated by storage, not transfer', rarelyDownloaded.storageCost > rarelyDownloaded.egressCost, true);
test('the same bucket served publicly is dominated by egress instead', publiclyServed.egressCost > publiclyServed.storageCost, true);
test('egress on the public bucket alone costs $180, far more than storage ever did', close(publiclyServed.egressCost, 180), true);Output
Why a CDN (Part 6 did not cover one, but this is where it belongs) changes this specific number. Amazon CloudFront caches content at edge locations, so a file requested by a thousand nearby users is fetched from S3 once, not a thousand times -- and data transferred from an S3 origin to CloudFront carries no separate transfer charge at all. The egress line item above is exactly the cost a CDN exists to reduce; sizing that tradeoff is arithmetic, not intuition, and it is the same multiplication shown here with a cache hit rate added in.
What a real FinOps tool automates, on top of the same arithmetic
Nothing above needed a dashboard. A real cost-management practice (often called FinOps) adds two things this lesson cannot: your account's ACTUAL usage numbers pulled automatically instead of estimated, and an alert the moment a trend breaks -- neither of which changes the arithmetic itself, only where the input numbers come from.
# AWS Cost Explorer's real API, returning ACTUAL billing data instead of an estimate.
# NOT RUN HERE: moto's Cost Explorer implementation accepts this call but returns an
# empty result set (confirmed directly) because it has no real billing data to
# report -- there is no local stand-in for a real AWS bill, only for the services
# that generate one.
aws ce get-cost-and-usage \
--time-period Start=2026-08-01,End=2026-09-01 \
--granularity MONTHLY \
--metrics BlendedCost \
--group-by Type=DIMENSION,Key=SERVICEmoto's Cost Explorer accepts this call but returns an empty result -- confirmed directly while writing this lesson -- because there is no real account spend behind it to report. Every number in the panels above came from AWS's own published rates instead, multiplied out by hand.
The habit that actually controls a cloud bill
- Tag everything with an owner and a project -- the SINGLE change that makes "why did the bill go up" answerable in minutes instead of days.
- Turn off what nobody is using -- an EC2 instance running nights and weekends for no reason is the same waste in every one of the calculators above, and it is the easiest one to actually fix.
- Match the purchase option to how predictable the workload is -- steady and known: Reserved. Interruption-tolerant: Spot. Unpredictable: on-demand, and accept paying the sticker price for that flexibility.
- Price data transfer on purpose, not storage alone -- the S3 panel above is the general shape: the line item that varies with USAGE, not the one that varies with how much you keep, is usually where a real bill actually moves.
46 On-call, runbooks and blameless postmortems
A page firing is the easy part -- Part 12 already built that. What happens in the minutes after is where most of the real damage either gets contained or doesn't: a runbook that tells you where to actually look, a budget that tells you how much room this incident had before it became a promise broken, and a writeup afterward that finds the fixable thing rather than the person to blame.
A runbook: the same three checks, giving two genuinely different verdicts
A runbook is not a document that describes what might be wrong -- it is a sequence of checks, each one either explaining the symptom or ruling itself out and handing off to the next. Below, the exact same three checks run against two different real failures, and correctly produce two different diagnoses.
# The runbook: process running? port accepting connections? health check passing?
# Each check either explains the symptom, or rules itself out and moves on.
triage() {
local port="$1" pid_file="$2"
local pid
pid=$(cat "$pid_file" 2>/dev/null || echo "")
if [ -z "$pid" ] || ! kill -0 "$pid" 2>/dev/null; then
echo "VERDICT: the process itself is gone -- restart it"
return
fi
if ! curl -s -o /dev/null -m 2 "http://localhost:$port/"; then
echo "VERDICT: process alive but not serving -- check for a deadlock or a crashed listener"
return
fi
status=$(curl -s -o /dev/null -m 2 -w '%{http_code}' "http://localhost:$port/")
if [ "$status" != "200" ]; then
echo "VERDICT: process alive and reachable, but unhealthy -- check application logs, not infrastructure"
return
fi
echo "VERDICT: healthy"
}
work=$(mktemp -d)
cd "$work"
echo "=== real failure A: the process actually crashed ==="
python3 -m http.server 8210 --bind 127.0.0.1 > /dev/null 2>&1 &
pid_a=$!
echo "$pid_a" > pid_a
for i in 1 2 3 4 5 6 7 8 9 10; do curl -s -o /dev/null -m 1 http://localhost:8210/ && break; sleep 0.3; done
kill -9 "$pid_a" 2>/dev/null
wait "$pid_a" 2>/dev/null
sleep 0.3
verdict_a=$(triage 8210 pid_a)
echo "$verdict_a"
echo
echo "=== real failure B: the process is alive, every request just fails ==="
cat > unhealthy.py << 'PY'
import http.server, socketserver
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(500)
self.send_header("Content-Length", "0")
self.end_headers()
def log_message(self, *a): pass
with socketserver.TCPServer(("127.0.0.1", 8211), Handler) as httpd:
httpd.serve_forever()
PY
python3 unhealthy.py > /dev/null 2>&1 &
pid_b=$!
echo "$pid_b" > pid_b
for i in 1 2 3 4 5 6 7 8 9 10; do curl -s -o /dev/null -m 1 http://localhost:8211/ && break; sleep 0.3; done
verdict_b=$(triage 8211 pid_b)
echo "$verdict_b"
kill "$pid_b" 2>/dev/null
wait "$pid_b" 2>/dev/null
cd / && rm -rf "$work"
check "a genuinely crashed process is correctly diagnosed as gone" \
"$(echo "$verdict_a" | grep -c 'process itself is gone')" "1"
check "a genuinely alive-but-failing process correctly rules out the process and the port" \
"$(echo "$verdict_b" | grep -c 'alive and reachable, but unhealthy')" "1"
check "the two real failures produced two DIFFERENT verdicts, not one generic answer" \
"$([ "$verdict_a" != "$verdict_b" ] && echo yes)" "yes"What it printed, when the build ran it
=== real failure A: the process actually crashed === VERDICT: the process itself is gone -- restart it === real failure B: the process is alive, every request just fails === VERDICT: process alive and reachable, but unhealthy -- check application logs, not infrastructure ok a genuinely crashed process is correctly diagnosed as gone ok a genuinely alive-but-failing process correctly rules out the process and the port ok the two real failures produced two DIFFERENT verdicts, not one generic answer
Why the ORDER of the checks is the whole design. Each check is cheaper and more certain than the one after it -- "is the process even running" needs no network call and answers instantly; "is it healthy" needs a real request and a real response. Running them in that order means the checks that rule out the most common causes fastest run first, so a page at 3am spends the least possible time before pointing at the right place.
The error budget: what an SLO's percentage actually allows to fail
"99.9% uptime" sounds close enough to perfect to ignore. Multiplied out over a real month, it is a specific, small number of minutes -- and once an incident's real duration is compared against that number, "how bad was it" stops being a feeling.
echo "an SLO promises a percentage. the error budget is what that percentage allows to FAIL,"
echo "in real minutes, over a real 30-day month:"
python3 -c "
def error_budget_minutes(slo_percent, period_days):
total_minutes = period_days * 24 * 60
allowed_failure_fraction = 1 - slo_percent / 100
return round(total_minutes * allowed_failure_fraction, 2)
for slo in (99.0, 99.9, 99.99):
print(f'{slo}% over 30 days allows {error_budget_minutes(slo, 30)} minutes of downtime')
"
budget_999=$(python3 -c "print(round(30*24*60*(1-99.9/100), 2))")
echo
echo "this month's real incident log -- three separate real outages, in minutes:"
incidents="14 8 22"
total_downtime=0
for m in $incidents; do
total_downtime=$((total_downtime + m))
done
echo "incidents: $incidents -- total downtime: $total_downtime minutes"
pct_used=$(python3 -c "print(round($total_downtime / $budget_999 * 100, 1))")
echo
echo "99.9% monthly budget: $budget_999 minutes"
echo "used so far this month: $total_downtime minutes ($pct_used% of the budget)"
check "the 99.9% monthly error budget really is 43.2 minutes, not a rounder guess" "$budget_999" "43.2"
check "three real incidents this month totalled 44 minutes of downtime" "$total_downtime" "44"
check "that alone already exceeds the entire monthly budget -- the SLO was missed, provably" \
"$(python3 -c "print('yes' if $total_downtime > $budget_999 else 'no')")" "yes"What it printed, when the build ran it
an SLO promises a percentage. the error budget is what that percentage allows to FAIL, in real minutes, over a real 30-day month: 99.0% over 30 days allows 432.0 minutes of downtime 99.9% over 30 days allows 43.2 minutes of downtime 99.99% over 30 days allows 4.32 minutes of downtime this month's real incident log -- three separate real outages, in minutes: incidents: 14 8 22 -- total downtime: 44 minutes 99.9% monthly budget: 43.2 minutes used so far this month: 44 minutes (101.9% of the budget) ok the 99.9% monthly error budget really is 43.2 minutes, not a rounder guess ok three real incidents this month totalled 44 minutes of downtime ok that alone already exceeds the entire monthly budget -- the SLO was missed, provably
What a real error budget changes about the NEXT decision, not the last one. Once the budget for this period is gone, the standard SRE practice is to freeze new feature launches and redirect the team to reliability work until the budget resets -- not as punishment, but because the number just proved that reliability, this month, needs the attention more than the roadmap does. A team with an SLO nobody actually tracks against real incidents does not have an SLO, it has a slogan.
A blameless postmortem: checked to prove it ends at a system, not a person
"Five whys" is not a slogan either -- it is a specific discipline: each answer becomes the next question, and the process is not finished until the answer names something a PROCESS change actually fixes. The check below is not on the incident's facts (this is a constructed example) -- it is on the STRUCTURE of the chain: does the final answer name a fixable gap, or does it stop early at a person.
declare -a CHAIN=(
"the site returned 500s for 8 minutes"
"the new deploy had a config value pointing at a database that no longer existed"
"the config was never validated before being applied"
"there is no automated check that a referenced database actually exists before deploy"
"the deploy pipeline has no pre-flight validation step at all"
)
echo "five whys, each answer becoming the next question:"
for i in "${!CHAIN[@]}"; do
echo "why #$((i+1)): ${CHAIN[$i]}"
done
last_index=$((${#CHAIN[@]} - 1))
first_answer="${CHAIN[0]}"
last_answer="${CHAIN[$last_index]}"
echo
echo "the FIRST answer is only the symptom: \"$first_answer\""
echo "the LAST answer is the fixable gap: \"$last_answer\""
names_a_person=$(echo "$last_answer" | grep -icE '\b(he|she|they|blamed|fault|carelessness)\b')
names_the_pipeline=$(echo "$last_answer" | grep -c 'pipeline')
check "the chain genuinely has five real steps, not stopped at the first symptom" "${#CHAIN[@]}" "5"
check "the final root cause names a missing SYSTEM step, not a person" "$names_a_person" "0"
check "the final root cause is specific enough to become an actual ticket, not a feeling" \
"$names_the_pipeline" "1"What it printed, when the build ran it
five whys, each answer becoming the next question: why #1: the site returned 500s for 8 minutes why #2: the new deploy had a config value pointing at a database that no longer existed why #3: the config was never validated before being applied why #4: there is no automated check that a referenced database actually exists before deploy why #5: the deploy pipeline has no pre-flight validation step at all the FIRST answer is only the symptom: "the site returned 500s for 8 minutes" the LAST answer is the fixable gap: "the deploy pipeline has no pre-flight validation step at all" ok the chain genuinely has five real steps, not stopped at the first symptom ok the final root cause names a missing SYSTEM step, not a person ok the final root cause is specific enough to become an actual ticket, not a feeling
"Blameless" is not the same as "consequence-free." The engineer who wrote the bad config is not disciplined for it, and that is the entire point: if raising your hand about a mistake risks your job, the next person who causes a similar problem hides it instead of reporting it, and the organisation learns nothing until the same gap causes a much larger incident later. The consequence that DOES follow is the fix itself -- in this example, the pipeline gains the validation step it was missing.
What goes in the writeup, in the order a reader actually needs it
- Impact, first. Who was affected, for how long, measured against the error budget above -- not a narrative, a number.
- Timeline, merged from every real source. Deploys, alerts, and on-call actions, sorted by real timestamp into one sequence -- the same merge Part 12's tracing lesson demonstrated across services applies here across systems.
- Root cause, via the five-whys chain above. Stopped at the first answer, a postmortem blames a symptom. Carried to the end, it names something fixable.
- Action items, each with an owner and a real ticket. "We'll be more careful" fixes nothing and is not an action item; "add a pre-flight check that the referenced database exists" is one, and is exactly what this lesson's chain arrived at.
The through-line back to lesson 41. An alert clearing tells you the symptom went away. A runbook tells you where to look while it is still happening. An error budget tells you how much this incident actually cost against a real promise. A blameless postmortem is what turns all three into something that makes the SAME incident less likely next month -- which is the only measure of whether an incident was actually handled, rather than merely survived.
47 The cloud/DevOps system design interview
A senior cloud/DevOps interview almost always ends with some version of "design a system that survives X". The question is never really about drawing the right boxes -- every candidate draws roughly the same boxes. It is about whether you can turn "survives a whole Availability Zone failing" into an actual number of instances, an actual recovery mechanism, and an actual monthly cost, and this lesson does exactly that, with every number coming from a panel that ran.
The brief
"Design the infrastructure for a checkout API. It must keep serving requests even if an entire AWS Availability Zone fails, and you must be able to recover the database if it is destroyed."
That is the whole prompt you will get. The first mistake is drawing a single EC2 instance and a single RDS instance and calling it done -- neither survives anything. The second is over-building: multi-region active-active for a checkout API that gets 450 requests per second at peak is solving a problem nobody asked about. Ask what "survives" actually needs to mean before designing for it: does it mean zero dropped requests during the failure, or does it mean back up within an agreed number of minutes -- because those are two very different, differently priced designs.
State the assumption out loud and move on: "I'll assume 450 requests/second at peak, each instance sustaining about 60 of those, and that a few seconds of failover during an AZ loss is acceptable but data loss is not." An interviewer who disagrees will correct you immediately, and now it is their assumption, not your guess.
Sizing the fleet: peak load, then doubled for the failure it must survive
The naive answer is "enough instances for peak load, spread across two AZs." That is wrong in a specific, checkable way: if each AZ holds exactly half the fleet, losing ONE AZ leaves only half the capacity to serve 100% of peak traffic. Surviving the failure means each AZ alone must be sized for the FULL peak, not half of it.
python3 -c "
import math
PEAK_RPS = 450
RPS_PER_INSTANCE = 60 # one t3.small, sustained -- the same instance type lesson 45 priced
bare_fleet = math.ceil(PEAK_RPS / RPS_PER_INSTANCE)
print('naive sizing (peak load, ignoring AZ failure):', bare_fleet, 'instances')
# the real requirement: EACH az, alone, must handle 100% of peak -- not half the fleet
per_az_needed = math.ceil(PEAK_RPS / RPS_PER_INSTANCE)
total_fleet = per_az_needed * 2
print('capacity required in EACH az to survive losing the other:', per_az_needed)
print('total fleet size, split evenly across two AZs:', total_fleet)
print()
print('the naive fleet has', bare_fleet, 'instances total -- losing one AZ (roughly half)')
print('would leave about', bare_fleet // 2, ', which is short of the', PEAK_RPS, 'RPS peak by design.')
print('the AZ-aware fleet is exactly double the naive one, on purpose.')
"
check "the naive, failure-unaware sizing is 8 instances" \
"$(python3 -c "import math; print(math.ceil(450/60))")" "8"
check "the AZ-aware fleet doubles that, to 16, because each AZ alone must cover full peak" \
"$(python3 -c "import math; print(math.ceil(450/60)*2)")" "16"What it printed, when the build ran it
naive sizing (peak load, ignoring AZ failure): 8 instances capacity required in EACH az to survive losing the other: 8 total fleet size, split evenly across two AZs: 16 the naive fleet has 8 instances total -- losing one AZ (roughly half) would leave about 4 , which is short of the 450 RPS peak by design. the AZ-aware fleet is exactly double the naive one, on purpose. ok the naive, failure-unaware sizing is 8 instances ok the AZ-aware fleet doubles that, to 16, because each AZ alone must cover full peak
Proving the design actually survives the failure, not just describing it
This is the same real mechanism lesson 21 already built -- a load balancer's target
group, and the exact two API calls (register-targets,
deregister-targets) that back both health-check failover AND Auto Scaling.
Here it runs across two real Availability Zones, and the AZ that goes down is proven gone,
not assumed.
created=$(aws ec2 create-vpc --cidr-block 10.9.0.0/16 --output json)
vpc_id=$(echo "$created" | grep -o '"VpcId": "[^"]*"' | head -1 | cut -d'"' -f4)
sub_a=$(aws ec2 create-subnet --vpc-id "$vpc_id" --cidr-block 10.9.1.0/24 --availability-zone us-east-1a --output json)
sub_a_id=$(echo "$sub_a" | grep -o '"SubnetId": "[^"]*"' | head -1 | cut -d'"' -f4)
sub_b=$(aws ec2 create-subnet --vpc-id "$vpc_id" --cidr-block 10.9.2.0/24 --availability-zone us-east-1b --output json)
sub_b_id=$(echo "$sub_b" | grep -o '"SubnetId": "[^"]*"' | head -1 | cut -d'"' -f4)
tg=$(aws elbv2 create-target-group --name checkout-tg --protocol HTTP --port 80 --vpc-id "$vpc_id" --output json)
tg_arn=$(echo "$tg" | grep -o '"TargetGroupArn": "[^"]*"' | head -1 | cut -d'"' -f4)
# two instances in AZ-a, two in AZ-b -- a small stand-in for the 16-instance fleet sized above
az_a_1=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --subnet-id "$sub_a_id" --count 1 --output json | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
az_a_2=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --subnet-id "$sub_a_id" --count 1 --output json | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
az_b_1=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --subnet-id "$sub_b_id" --count 1 --output json | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
az_b_2=$(aws ec2 run-instances --image-id ami-03cf127a --instance-type t2.micro --subnet-id "$sub_b_id" --count 1 --output json | grep -o '"InstanceId": "[^"]*"' | head -1 | cut -d'"' -f4)
aws elbv2 register-targets --target-group-arn "$tg_arn" --targets Id="$az_a_1" Id="$az_a_2" Id="$az_b_1" Id="$az_b_2"
before=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --query 'length(TargetHealthDescriptions)' --output text)
echo "targets registered, split across two real AZs: $before"
echo
echo "simulating an entire AZ-a failure -- deregistering exactly the instances that live there,"
echo "the same outcome a real failed health check produces automatically, with no human step:"
aws elbv2 deregister-targets --target-group-arn "$tg_arn" --targets Id="$az_a_1" Id="$az_a_2"
after=$(aws elbv2 describe-target-health --target-group-arn "$tg_arn" --output json)
after_count=$(echo "$after" | grep -c '"State"')
surviving_az=$(aws ec2 describe-instances --instance-ids "$az_b_1" --query 'Reservations[0].Instances[0].Placement.AvailabilityZone' --output text)
echo "targets remaining after AZ-a failure: $after_count"
echo "the surviving targets are confirmed to be in: $surviving_az"
check "all four targets across both AZs were registered and healthy before the failure" "$before" "4"
check "exactly two targets survive after an entire AZ is removed" "$after_count" "2"
check "the surviving targets are confirmed to be in the OTHER Availability Zone" "$surviving_az" "us-east-1b"
check "the failed AZ's instances are genuinely gone from the pool, not just marked down" \
"$(echo "$after" | grep -c "$az_a_1\|$az_a_2")" "0"What it printed, when the build ran it
targets registered, split across two real AZs: 4 simulating an entire AZ-a failure -- deregistering exactly the instances that live there, the same outcome a real failed health check produces automatically, with no human step: targets remaining after AZ-a failure: 2 the surviving targets are confirmed to be in: us-east-1b ok all four targets across both AZs were registered and healthy before the failure ok exactly two targets survive after an entire AZ is removed ok the surviving targets are confirmed to be in the OTHER Availability Zone ok the failed AZ's instances are genuinely gone from the pool, not just marked down
Recovering the database: proving a snapshot restore actually works, before you need it
"We take backups" is not disaster recovery -- disaster recovery is having actually restored from one, at least once, before the day it matters. This is the exact real RDS mechanism lesson 24 introduced, run to completion: a real snapshot, and a real restore into a brand new, independent instance.
aws rds create-db-instance --db-instance-identifier checkout-db --db-instance-class db.t3.micro \
--engine postgres --master-username courseadmin --master-user-password coursepassword123 \
--allocated-storage 20 --output json > /dev/null
echo "the production database, running:"
aws rds describe-db-instances --db-instance-identifier checkout-db \
--query 'DBInstances[0].{Id:DBInstanceIdentifier,Status:DBInstanceStatus}' --output json
echo
echo "a real, manually-triggered snapshot -- the artefact recovery actually depends on:"
aws rds create-db-snapshot --db-instance-identifier checkout-db \
--db-snapshot-identifier checkout-db-manual-snap --output json > /dev/null
snapshots=$(aws rds describe-db-snapshots --db-instance-identifier checkout-db --output json)
snap_count=$(echo "$snapshots" | grep -c '"DBSnapshotIdentifier"')
echo "snapshots that exist for this database: $snap_count"
echo "(one is the automated snapshot RDS itself took on creation; one is the manual one just taken)"
echo
echo "the actual disaster recovery action -- restoring into a NEW, independent instance,"
echo "not overwriting the original, which is exactly how a real restore has to work:"
aws rds restore-db-instance-from-db-snapshot \
--db-instance-identifier checkout-db-recovered \
--db-snapshot-identifier checkout-db-manual-snap --output json > /dev/null
recovered_engine=$(aws rds describe-db-instances --db-instance-identifier checkout-db-recovered \
--query 'DBInstances[0].Engine' --output text)
recovered_storage=$(aws rds describe-db-instances --db-instance-identifier checkout-db-recovered \
--query 'DBInstances[0].AllocatedStorage' --output text)
both_exist=$(aws rds describe-db-instances --query 'length(DBInstances)' --output text)
echo "recovered instance's engine: $recovered_engine, storage: ${recovered_storage}GB"
echo "total real, independent database instances that now exist: $both_exist"
check "at least two snapshots exist -- the automated one and the manual one just taken" \
"$([ "$snap_count" -ge 2 ] && echo yes)" "yes"
check "the recovered instance genuinely restored the same engine" "$recovered_engine" "postgres"
check "the recovered instance genuinely restored the same storage size" "$recovered_storage" "20"
check "the original and the recovered instance both exist, independently, right now" "$both_exist" "2"What it printed, when the build ran it
the production database, running:
{
"Id": "checkout-db",
"Status": "available"
}
a real, manually-triggered snapshot -- the artefact recovery actually depends on:
snapshots that exist for this database: 2
(one is the automated snapshot RDS itself took on creation; one is the manual one just taken)
the actual disaster recovery action -- restoring into a NEW, independent instance,
not overwriting the original, which is exactly how a real restore has to work:
recovered instance's engine: postgres, storage: 20GB
total real, independent database instances that now exist: 2
ok at least two snapshots exist -- the automated one and the manual one just taken
ok the recovered instance genuinely restored the same engine
ok the recovered instance genuinely restored the same storage size
ok the original and the recovered instance both exist, independently, right now
Why the restored instance has a different identifier, on purpose. A real restore never overwrites the original -- doing so would destroy your only other copy if the restore itself went wrong. The correct sequence is: restore into a new instance, verify the data is actually correct against it, THEN switch the application's connection string (or a DNS record pointing at it) to the new one. Skipping the verification step is the most common way "we tested our backups" turns out to have meant "we confirmed the restore command exits zero," which is not the same claim.
Pricing the design that was just proven, not a different one
The exact hourly-rate arithmetic from lesson 45, applied to the fleet size this design actually needs -- 16 instances, not the naive 8 -- because a design's real cost is the cost of the version that survives the failure, not the version that only handles average load.
# A quoted heredoc ('PYEOF'), not python3 -c "...", because the f-strings below need
# a literal dollar sign and a quoted heredoc is never touched by bash's own expansion.
python3 << 'PYEOF'
HOURLY_RATE_SMALL = 0.0208 # t3.small, the same published rate lesson 45 used
HOURS_PER_MONTH = 730
naive_fleet = 8
az_aware_fleet = 16
naive_monthly = naive_fleet * HOURLY_RATE_SMALL * HOURS_PER_MONTH
az_aware_monthly = az_aware_fleet * HOURLY_RATE_SMALL * HOURS_PER_MONTH
extra_for_survival = az_aware_monthly - naive_monthly
print(f'naive fleet (8 instances, does not survive an AZ loss): ${naive_monthly:.2f}/month')
print(f'AZ-aware fleet (16 instances, survives losing one AZ): ${az_aware_monthly:.2f}/month')
print(f'the real cost of surviving the failure this design needs: ${extra_for_survival:.2f}/month')
print()
print('that number -- not a diagram -- is what you tell whoever approves the budget.')
PYEOF
check "the naive fleet costs about 121 dollars a month" \
"$(python3 -c "print(round(8*0.0208*730))")" "121"
check "the AZ-survivable fleet costs about 243 dollars a month" \
"$(python3 -c "print(round(16*0.0208*730))")" "243"What it printed, when the build ran it
naive fleet (8 instances, does not survive an AZ loss): $121.47/month AZ-aware fleet (16 instances, survives losing one AZ): $242.94/month the real cost of surviving the failure this design needs: $121.47/month that number -- not a diagram -- is what you tell whoever approves the budget. ok the naive fleet costs about 121 dollars a month ok the AZ-survivable fleet costs about 243 dollars a month
What the interviewer is actually listening for
- Did you ask what "survives" has to mean before sizing anything? Zero dropped requests and "back up in five minutes" are different designs with different costs, and assuming one without asking is the single most common way this question goes wrong.
- Did you compute the AZ-failure fleet size, not just the peak-load one? The panel above is the whole trick: each surviving AZ must cover full peak alone, which is a doubling, not a rounding error.
- Did you actually prove the failover, rather than assert it? "Multi-AZ" on a diagram is a claim. Deregistering the failed AZ's real targets and confirming the survivors are correctly in the other one is evidence.
- Did you name disaster recovery as a RESTORE that was tested, not a backup that exists? A snapshot nobody has ever restored from is an unverified assumption with a false sense of safety attached.
- Did you put a number on the design, not just a description of it? "It's more resilient" convinces nobody who has to approve a budget; "$243 a month, and here is exactly what that buys" does.
The through-line of this whole course, in one paragraph. Every lesson ran something real rather than describing it, because the job itself is exactly that: not knowing the name of a service, but knowing what happens when you actually run it, what it costs, and what specifically breaks when a piece of it fails. The five checks above are not a script for this one interview question -- they are the same five questions worth asking about any system before it goes anywhere near production, whether anyone is interviewing you about it or not.