AWS serverless all 3 written

Functions as infrastructure, and the queues and topics that connect them without either side waiting.

25 Lambda: functions as infrastructure

"Serverless" does not mean no server -- it means the server is not yours to manage. Something still runs the code, on real hardware, for the seconds it takes to execute. What Lambda actually removes is everything AWS can removed you: no OS to patch, no capacity to plan, no idle server billed while nothing is happening.

A real function, uploaded as a real zip, invoked with real input

Lambda's deployment unit is exactly what it sounds like: a zip file containing your code, uploaded as the function's body. There is no abstraction between "the code you wrote" and "the bytes AWS runs" beyond that zip.

A real zip, a real handler, a real computed result

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49work=$(mktemp -d)
cd "$work"

cat > lambda_function.py << 'PY'
import json

def handler(event, context):
    name = event.get("name", "world")
    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello, " + name,
            "doubled": event.get("number", 0) * 2
        })
    }
PY

echo "the real handler code being packaged:"
cat lambda_function.py

zip -q function.zip lambda_function.py

aws iam create-role --role-name lambda-role --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > /dev/null

echo
echo "uploading the real zip as a Lambda function:"
aws lambda create-function --function-name course-fn \
  --runtime python3.11 --role arn:aws:iam::123456789012:role/lambda-role \
  --handler lambda_function.handler --zip-file fileb://function.zip \
  --query '{FunctionName:FunctionName,Runtime:Runtime,Handler:Handler,CodeSize:CodeSize}' --output json

echo
echo "invoking it with real input -- {\"name\": \"Ada\", \"number\": 21}:"
aws lambda invoke --function-name course-fn \
  --payload '{"name": "Ada", "number": 21}' \
  /tmp/cd-course-lambda-out.json > /dev/null

result=$(cat /tmp/cd-course-lambda-out.json)
echo "$result"
rm -f /tmp/cd-course-lambda-out.json
cd / && rm -rf "$work"

check "the function really computed the doubled number, not a canned response" \
  "$(echo "$result" | grep -c 'doubled.*42')" "1"
check "the function really used the name from the input event" \
  "$(echo "$result" | grep -c 'hello, Ada')" "1"

What it printed, when the build ran it

the real handler code being packaged:
import json

def handler(event, context):
    name = event.get("name", "world")
    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello, " + name,
            "doubled": event.get("number", 0) * 2
        })
    }

uploading the real zip as a Lambda function:
{
    "FunctionName": "course-fn",
    "Runtime": "python3.11",
    "Handler": "lambda_function.handler",
    "CodeSize": 348
}

invoking it with real input -- {"name": "Ada", "number": 21}:
{"statusCode":200,"body":"{\"message\": \"hello, Ada\", \"doubled\": 42}"}
ok   the function really computed the doubled number, not a canned response
ok   the function really used the name from the input event

What actually produced that answer. The number 42 did not come from anywhere this course wrote by hand -- it came from AWS Lambda's own real invocation machinery running the exact Python file above, inside an actual container, and handing back whatever that file's handler function returned. Change event.get("number", 0) * 2 to * 3 and the answer changes; this is genuinely running, not simulated.

A function that raises: the failure path, not just the happy one

A Lambda function that throws does not crash silently -- Lambda reports it as a distinct, checkable outcome, separate from a normal return, so calling code can tell the two apart.

A real unhandled exception, reported as Lambda's own FunctionError

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35work=$(mktemp -d)
cd "$work"

cat > lambda_function.py << 'PY'
def handler(event, context):
    if "required_field" not in event:
        raise ValueError("required_field is missing from the event")
    return {"statusCode": 200, "body": "ok"}
PY

zip -q function.zip lambda_function.py

aws iam create-role --role-name lambda-role-2 --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > /dev/null

aws lambda create-function --function-name course-fn-strict \
  --runtime python3.11 --role arn:aws:iam::123456789012:role/lambda-role-2 \
  --handler lambda_function.handler --zip-file fileb://function.zip > /dev/null

echo "invoking WITHOUT the required field:"
invoke_meta=$(aws lambda invoke --function-name course-fn-strict \
  --payload '{}' /tmp/cd-course-lambda-fail.json)
echo "$invoke_meta"

echo
echo "what the function actually returned:"
cat /tmp/cd-course-lambda-fail.json
echo
rm -f /tmp/cd-course-lambda-fail.json
cd / && rm -rf "$work"

check "Lambda reports a function error distinctly from a normal return" \
  "$(echo "$invoke_meta" | grep -c 'FunctionError')" "1"

What it printed, when the build ran it

invoking WITHOUT the required field:
{
    "StatusCode": 200,
    "FunctionError": "Handled"
}

what the function actually returned:
{"errorType":"ValueError","errorMessage":"required_field is missing from the event","stackTrace":["  File \"/var/task/lambda_function.py\", line 3, in handler\n    raise ValueError(\"required_field is missing from the event\")\n"]}
ok   Lambda reports a function error distinctly from a normal return

Why "it invoked successfully" is not the same question as "it worked." The HTTP-style status code from invoke itself can be 200 even when the function raised -- the actual failure shows up in the response's own FunctionError field, exactly as checked above. Code that only checks the outer status and ignores this field will miss every one of its own function's crashes.

Billing follows execution, not existence

A Lambda function that is never invoked costs nothing beyond the negligible storage of its zip. This is the structural difference from EC2 (Part 6), which bills for every hour an instance exists whether or not it does any work.

EC2: billed per second the instance is running, regardless of whether it is handling any requests.
Lambda: billed per invocation and per millisecond of actual execution time. Zero invocations in a month is, genuinely, a zero-dollar month for compute.
The tradeoff: a Lambda function that has not run recently pays a real "cold start" latency cost on its next invocation, while a running EC2 instance is always warm. Part 14 returns to this tradeoff with real numbers.

26 SQS and SNS: queues, topics and decoupling

Neither of these is "a way to send data." HTTP already does that. What a queue and a topic actually buy you is decoupling in time: the producer does not wait for the consumer to be up, fast, or even currently running, and a message that cannot be handled right now simply waits instead of failing.

SQS: a message is not gone until someone deletes it

Receiving a message from a queue does not remove it. It becomes temporarily invisible to other receivers (the visibility timeout), on the assumption that whoever just received it is about to finish processing it and delete it. If that consumer crashes instead, the message simply reappears -- nothing about SQS trusts a receiver's good intentions over its own explicit delete call.

Receive, crash before deleting, and watch it come back

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38queue=$(aws sqs create-queue --queue-name orders-inbox --output json)
queue_url=$(echo "$queue" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
echo "queue created: $queue_url"

aws sqs send-message --queue-url "$queue_url" --message-body "order #1001" > /dev/null

echo
echo "first receive -- gets the message, and a receipt handle to delete it with later:"
first=$(aws sqs receive-message --queue-url "$queue_url" --output json)
echo "$first"
receipt1=$(echo "$first" | grep -o '"ReceiptHandle": "[^"]*"' | head -1 | cut -d'"' -f4)

echo
echo "receiving again immediately -- empty, because the message is hidden while presumed in-flight:"
second=$(aws sqs receive-message --queue-url "$queue_url" --output json)
echo "second receive: [$second]"

echo
echo "simulating the consumer that received it crashing before deleting it -- force it visible again:"
aws sqs change-message-visibility --queue-url "$queue_url" --receipt-handle "$receipt1" --visibility-timeout 0 > /dev/null
third=$(aws sqs receive-message --queue-url "$queue_url" --output json)
echo "$third"
receipt2=$(echo "$third" | grep -o '"ReceiptHandle": "[^"]*"' | head -1 | cut -d'"' -f4)

echo
echo "this time, actually deleting it -- the correct end of processing:"
aws sqs delete-message --queue-url "$queue_url" --receipt-handle "$receipt2"

fourth=$(aws sqs receive-message --queue-url "$queue_url" --output json)
remaining=$(aws sqs get-queue-attributes --queue-url "$queue_url" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text)
echo "receive after delete: [$fourth]"
echo "queue's own count of what is left: $remaining"

check "the message was there on the first receive" "$(echo "$first" | grep -c 'order #1001')" "1"
check "an in-flight message is invisible to a second receiver" "$second" ""
check "forcing visibility back to 0 makes it reappear, because it was only hidden, not gone" \
  "$(echo "$third" | grep -c 'order #1001')" "1"
check "after a real delete, the queue is empty" "$remaining" "0"

What it printed, when the build ran it

queue created: http://localhost:5099/123456789012/orders-inbox

first receive -- gets the message, and a receipt handle to delete it with later:
{
    "Messages": [
        {
            "MessageId": "836d8099-9b0e-4e56-9c06-ac64bdc75ed7",
            "ReceiptHandle": "phfjydfbqqpebippcoazhytkrbllsmeksykoxphqyjdqpzfklgzzruqchuoohsvlmudsdfpynyeyvltslqzphdsvzinfonmxyxlvqvxtlbdvdogvixfwztppcjscbqgcspbpgwfdzbgcmhhqerykgirbdqlflhycqrkazmfrjrsewadoghiknogih",
            "MD5OfBody": "deee68bbada27b38c2ed5cd08b1d000d",
            "Body": "order #1001"
        }
    ]
}

receiving again immediately -- empty, because the message is hidden while presumed in-flight:
second receive: []

simulating the consumer that received it crashing before deleting it -- force it visible again:
{
    "Messages": [
        {
            "MessageId": "836d8099-9b0e-4e56-9c06-ac64bdc75ed7",
            "ReceiptHandle": "bhptjdyfdyeqlakqacoqlrxdcomqenmdplfxtyrgabztnlnvrrbuysvyjiqooppytuqcnzxdliegxlnkluycxtrptremgdvlaezrswrspryqizffqypwkrimjuqejstlkpcxfkfkliaplhbwzlgehmyielxiclxoitdytkdeqosgayktfelbnwhyq",
            "MD5OfBody": "deee68bbada27b38c2ed5cd08b1d000d",
            "Body": "order #1001"
        }
    ]
}

this time, actually deleting it -- the correct end of processing:
receive after delete: []
queue's own count of what is left: 0
ok   the message was there on the first receive
ok   an in-flight message is invisible to a second receiver
ok   forcing visibility back to 0 makes it reappear, because it was only hidden, not gone
ok   after a real delete, the queue is empty

Why this is safer than a normal function call. If a direct call from one service to another fails partway through, the caller has to decide what to do -- retry, drop it, alert someone. Here, "the consumer died" and "the message is unprocessed" are the same fact, automatically, with no code written to detect the crash at all. The timeout above did all of it.

SNS: one publish, every subscriber gets their own copy

A topic does not hold messages the way a queue does -- it fans a single published message out to every current subscription at once. Each subscriber gets an independent copy, in its own queue, unaware of how many other subscribers exist or what they do with it.

One publish, two independent subscribers, zero coordination

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33topic=$(aws sns create-topic --name order-events --output json)
topic_arn=$(echo "$topic" | grep -o '"TopicArn": "[^"]*"' | cut -d'"' -f4)
echo "topic created: $topic_arn"

billing_q=$(aws sqs create-queue --queue-name billing-service --output json)
billing_url=$(echo "$billing_q" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
billing_arn=$(aws sqs get-queue-attributes --queue-url "$billing_url" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

shipping_q=$(aws sqs create-queue --queue-name shipping-service --output json)
shipping_url=$(echo "$shipping_q" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
shipping_arn=$(aws sqs get-queue-attributes --queue-url "$shipping_url" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

echo "two unrelated services, each with their own queue, both subscribing to the same topic:"
aws sns subscribe --topic-arn "$topic_arn" --protocol sqs --notification-endpoint "$billing_arn" > /dev/null
aws sns subscribe --topic-arn "$topic_arn" --protocol sqs --notification-endpoint "$shipping_arn" > /dev/null

echo
echo "ONE publish, from code that has never heard of billing or shipping:"
aws sns publish --topic-arn "$topic_arn" --message '{"orderId": "1001", "total": 42.50}' --output json > /dev/null

billing_recv=$(aws sqs receive-message --queue-url "$billing_url" --output json)
shipping_recv=$(aws sqs receive-message --queue-url "$shipping_url" --output json)

echo "billing's queue got:"
echo "$billing_recv"
echo
echo "shipping's queue got the SAME event, independently:"
echo "$shipping_recv"

check "billing received the order event, despite never being addressed directly" \
  "$(echo "$billing_recv" | grep -c 'orderId')" "1"
check "shipping received the identical order event, from the same single publish" \
  "$(echo "$shipping_recv" | grep -c 'orderId')" "1"

What it printed, when the build ran it

topic created: arn:aws:sns:us-east-1:123456789012:order-events
two unrelated services, each with their own queue, both subscribing to the same topic:

ONE publish, from code that has never heard of billing or shipping:
billing's queue got:
{
    "Messages": [
        {
            "MessageId": "96689e0f-35ff-409c-886e-421f3643ea85",
            "ReceiptHandle": "beflivurqowehaqawztwcsypxdyrwlynjpzrqtdxvyoqqrntaeowvklhyvrqkosolwxowvpsribavfvdgfeywjwbogvehvaiyfflfmqfcepgbrabeeppepmvoeqncnmtoslcuhhelkyatmxkozjoswehjyqwmxzpivuufdhnhzxvqoflvbngxmaxf",
            "MD5OfBody": "66e32e6f96fb08ec23186159ead53b97",
            "Body": "{\n  \"Message\": \"{\\\"orderId\\\": \\\"1001\\\", \\\"total\\\": 42.50}\",\n  \"MessageId\": \"8c6bf5ce-2c3a-463d-8077-fa3accb50a52\",\n  \"Signature\": \"fA6GkXe4m7ny7MqYMNIZeVZQbLTC+rbSGPS8fQwaREzsdqSlHQnYA5rXnJUNw81KGXuopMT8DENQRDRO6twXsgIx3Sa5PyGToM/trY2D/VnIFwg1GrQJF8n43mJMPZu+ifaNO0p4EFXj/l7/JsWwsk+niTE0HtiC6+BUmJcbrv9koEG/DxKe7t5rJtP81zSAadAfu75xupqSp0+FNqiiwCkjy7aOQqdTIL7bjeflPwhtszULp1bS6nuKPA/c2GW8NWz2T4ErU3H+y+qikY6sWk5wa2avgOUnlLakPc1FxAMvyrHHw4ulefLwuWOvYsggOGcWJ9TMWhgWG8lIFYaKFg==\",\n  \"SignatureVersion\": \"2\",\n  \"SigningCertURL\": \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-5b54223d3499478e959efcf2fc2c7ef3.pem\",\n  \"Timestamp\": \"2026-09-15T20:19:32.988Z\",\n  \"TopicArn\": \"arn:aws:sns:us-east-1:123456789012:order-events\",\n  \"Type\": \"Notification\",\n  \"UnsubscribeURL\": \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:order-events:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55\"\n}"
        }
    ]
}

shipping's queue got the SAME event, independently:
{
    "Messages": [
        {
            "MessageId": "63d6f205-90c2-408b-bb6c-f2dfff0882c2",
            "ReceiptHandle": "eyucwkzxswynrcuqbuwyltkxbpqgyolutjykhhohnqxfwkkyowixivjpwfbbaciquxvqyyjvrbtqkiighekgbaeudteujdvcetlczmusywopccsdhxhddsshloqdbmsjhwfgayvfswctpildlqdlmfvqkogwitssmlzhollmrwenjckbrbohmmoyb",
            "MD5OfBody": "f7fbc4c6bc98deac61c32401292ec08d",
            "Body": "{\n  \"Message\": \"{\\\"orderId\\\": \\\"1001\\\", \\\"total\\\": 42.50}\",\n  \"MessageId\": \"8c6bf5ce-2c3a-463d-8077-fa3accb50a52\",\n  \"Signature\": \"QoNR7dq3IXjic+FTynaBTQmek6WSIH0stVTwl5VOBIVEHcoIKsDh3wKIptxYXbur7fME0jELR+G0T2AoO+YONvpZyDfSmNNzwLvDL3uzxrGV02QxUrlm0VwAmahoMCIWbn0UJo/OX8yxboWQxwZCvRb7SAQ7Nbz/dS1hL5vBGARyMfslrBY/ETGMHdnCM/8cPljWbDJfCudNhQJhCvQhv4FLqgwi7SNF5ZUkDcH8xwKoFsxOddGq3flV56cLK869bhukl5xHMtulsfXVkD6u+coE2JRq8cQSfIRTx9sG/DDiJk+2k8Z+h/4Zyo4Pfy/P14wQuZM8N4WW/ut8fBtkcA==\",\n  \"SignatureVersion\": \"2\",\n  \"SigningCertURL\": \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-02200c8b6b5149728bae31e6b4fb6d6b.pem\",\n  \"Timestamp\": \"2026-09-15T20:19:33.006Z\",\n  \"TopicArn\": \"arn:aws:sns:us-east-1:123456789012:order-events\",\n  \"Type\": \"Notification\",\n  \"UnsubscribeURL\": \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:order-events:2bcfbf39-05c3-41de-beaa-fcfcc21c8f55\"\n}"
        }
    ]
}
ok   billing received the order event, despite never being addressed directly
ok   shipping received the identical order event, from the same single publish

What this replaces. The alternative is the publisher calling billing and shipping directly -- which means the order service has to know both exist, has to handle either one being slow or down, and has to change its own code every time a third service (fraud detection, analytics) needs the same event. Adding that third subscriber here is one more subscribe call. The order service's code does not change at all.

Dead-letter queues: where messages go when nothing can process them

Not every message is recoverable. A malformed payload, a bug in the consumer, a permanently invalid order -- retrying it forever just wastes capacity and hides a real problem. A redrive policy gives a queue a limit: after a message has been received too many times without being deleted, it is moved to a separate dead-letter queue instead of going back for another attempt.

A message that cannot be processed, redirected instead of retried forever

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35dlq=$(aws sqs create-queue --queue-name orders-dlq --output json)
dlq_url=$(echo "$dlq" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
dlq_arn=$(aws sqs get-queue-attributes --queue-url "$dlq_url" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
echo "dead-letter queue created: $dlq_arn"

main_q=$(aws sqs create-queue --queue-name orders-inbox-guarded \
  --attributes '{"RedrivePolicy": "{\"deadLetterTargetArn\":\"'"$dlq_arn"'\",\"maxReceiveCount\":\"2\"}"}' \
  --output json)
main_url=$(echo "$main_q" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
echo "main queue created, wired to that DLQ after 2 failed receives: $main_url"

aws sqs send-message --queue-url "$main_url" --message-body "malformed order, cannot be processed" > /dev/null

echo
echo "a consumer that keeps failing to process this message, receiving it over and over"
echo "without ever deleting it (visibility-timeout 0 = give up instantly, for the demo):"
for i in 1 2 3; do
  attempt=$(aws sqs receive-message --queue-url "$main_url" --visibility-timeout 0 --output json)
  seen=$(echo "$attempt" | grep -c 'malformed order')
  echo "attempt $i: still in the main queue = $seen"
done

echo
echo "the main queue, after moto moved it past maxReceiveCount:"
main_after=$(aws sqs receive-message --queue-url "$main_url" --output json)
echo "main queue now: [$main_after]"

echo
echo "the dead-letter queue, which should now hold it instead:"
dlq_recv=$(aws sqs receive-message --queue-url "$dlq_url" --output json)
echo "$dlq_recv"

check "the message is gone from the main queue after exceeding maxReceiveCount" "$main_after" ""
check "the same message landed in the dead-letter queue instead of vanishing" \
  "$(echo "$dlq_recv" | grep -c 'malformed order')" "1"

What it printed, when the build ran it

dead-letter queue created: arn:aws:sqs:us-east-1:123456789012:orders-dlq
main queue created, wired to that DLQ after 2 failed receives: http://localhost:5099/123456789012/orders-inbox-guarded

a consumer that keeps failing to process this message, receiving it over and over
without ever deleting it (visibility-timeout 0 = give up instantly, for the demo):
attempt 1: still in the main queue = 1
attempt 2: still in the main queue = 1
attempt 3: still in the main queue = 0

the main queue, after moto moved it past maxReceiveCount:
main queue now: []

the dead-letter queue, which should now hold it instead:
{
    "Messages": [
        {
            "MessageId": "dfb47397-93ca-42ef-8c90-cb4b24de00c5",
            "ReceiptHandle": "lputyjojxzenuxcgvzynovddhhsmmfjqdfbqtzojqzcmeixtpzxmmsnxtkcdrtilfkmojylyfrbdexzoyykldnceacnmtalykkjnrjqwhebwgdvrsutruuyddymxxswemfsfoubvabzdvazcuasnlgkiueeadkpwlbkxxpxpkjrhcvuuvdtxwrqgf",
            "MD5OfBody": "06c352d16c1cc36f6dcbfafe75fe6604",
            "Body": "malformed order, cannot be processed"
        }
    ]
}
ok   the message is gone from the main queue after exceeding maxReceiveCount
ok   the same message landed in the dead-letter queue instead of vanishing

A dead-letter queue is not a fix, it is a smoke detector. Moving a message there stops it from looping forever, but it still needs a human or a follow-up process to look at it -- a real production setup puts a CloudWatch alarm on ApproximateNumberOfMessagesVisible for the DLQ itself, because a message sitting there silently is a genuine failure that nothing else will surface on its own.

The choice, in one line each

SQS (queue): each message is handled by exactly one consumer, eventually. Use it for work that needs doing once -- process this order, resize this image.
SNS (topic): each message is handed to every current subscriber, independently. Use it for announcing something happened, when you do not know or care how many things want to react to it.
They are not exclusive: subscribing an SQS queue to an SNS topic (done twice above) is the standard way to get fan-out delivery with the queue's own crash-safe, buffered semantics on each subscriber's side -- rather than SNS retrying an unreachable subscriber on its own, looser terms.

27 Event-driven architecture, end to end

"Event-driven" has meant one specific thing in every lesson so far without saying so: something happens (a message arrives, a file lands), and code runs in response, without anyone scheduling it or calling it directly. This lesson makes that connection itself the subject, instead of the side effect.

An event source mapping: SQS invokes the function, not you

An event source mapping is AWS polling a queue on your behalf and calling your function with a batch of whatever it finds -- the same invoke lesson 25 used, just never called by anything you wrote. Sending a message is now the only trigger the function needs.

Send a message. Never call invoke. The function still runs.

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61work=$(mktemp -d)
cd "$work"

cat > lambda_function.py << 'PY'
import json

def handler(event, context):
    bodies = [r.get("body") for r in event.get("Records", [])]
    print("PROCESSED BODIES:", json.dumps(bodies))
    return {"processed": len(bodies), "bodies": bodies}
PY

aws iam create-role --role-name esm-role --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > /dev/null

zip -q function.zip lambda_function.py
aws lambda create-function --function-name queue-worker \
  --runtime python3.11 --role arn:aws:iam::123456789012:role/esm-role \
  --handler lambda_function.handler --zip-file fileb://function.zip > /dev/null
echo "function created -- note nothing has invoked it yet"

queue=$(aws sqs create-queue --queue-name work-items --output json)
queue_url=$(echo "$queue" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
queue_arn=$(aws sqs get-queue-attributes --queue-url "$queue_url" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

echo
echo "wiring the queue to the function -- this is the entire connection:"
aws lambda create-event-source-mapping --function-name queue-worker \
  --event-source-arn "$queue_arn" --batch-size 1 \
  --query '{UUID:UUID,State:State,EventSourceArn:EventSourceArn}' --output json

unique="order-$(date +%s%N)"
echo
echo "sending a message -- and NOT calling invoke, anywhere in this script:"
aws sqs send-message --queue-url "$queue_url" --message-body "$unique" > /dev/null

echo "polling the queue's own count, waiting for the mapping to drain it:"
remaining=1
for i in 1 2 3 4 5 6 7 8 9 10; do
  remaining=$(aws sqs get-queue-attributes --queue-url "$queue_url" \
    --attribute-names ApproximateNumberOfMessages \
    --query 'Attributes.ApproximateNumberOfMessages' --output text)
  echo "poll $i: messages left in the queue = $remaining"
  if [ "$remaining" = "0" ]; then break; fi
  sleep 0.5
done

echo
echo "the function's OWN logs, produced by a run nobody called directly:"
stream=$(aws logs describe-log-streams --log-group-name /aws/lambda/queue-worker \
  --query 'logStreams[0].logStreamName' --output text)
logs=$(aws logs get-log-events --log-group-name /aws/lambda/queue-worker --log-stream-name "$stream" --output json)
echo "$logs"

cd / && rm -rf "$work"

check "the queue actually drained, meaning something besides us invoked the function" "$remaining" "0"
check "the function's log genuinely contains the message body we sent, not a canned value" \
  "$([ "$(echo "$logs" | grep -c "$unique")" -ge 1 ] && echo yes)" "yes"

What it printed, when the build ran it

function created -- note nothing has invoked it yet

wiring the queue to the function -- this is the entire connection:
{
    "UUID": "ab25d1af-c5b2-44a2-bbe5-af5c29e08260",
    "State": "Enabled",
    "EventSourceArn": "arn:aws:sqs:us-east-1:123456789012:work-items"
}

sending a message -- and NOT calling invoke, anywhere in this script:
polling the queue's own count, waiting for the mapping to drain it:
poll 1: messages left in the queue = 0

the function's OWN logs, produced by a run nobody called directly:
{
    "events": [
        {
            "timestamp": 1789503584715.789,
            "message": "\u001b[32mSTART RequestId: 20141e79-4982-1fb2-65c7-9e0d088c6c05 Version: $LATEST\u001b[0m",
            "ingestionTime": 1789503584715
        },
        {
            "timestamp": 1789503584715.793,
            "message": "PROCESSED BODIES: [\"order-1789503583150126000\"]",
            "ingestionTime": 1789503584715
        },
        {
            "timestamp": 1789503584715.795,
            "message": "\u001b[32mEND RequestId: 20141e79-4982-1fb2-65c7-9e0d088c6c05\u001b[0m",
            "ingestionTime": 1789503584715
        },
        {
            "timestamp": 1789503584715.7969,
            "message": "\u001b[32mREPORT RequestId: 20141e79-4982-1fb2-65c7-9e0d088c6c05\tInit Duration: 138.86 ms\tDuration: 2.22 ms\tBilled Duration: 3 ms\tMemory Size: 128 MB\tMax Memory Used: 28 MB\t\u001b[0m",
            "ingestionTime": 1789503584715
        },
        {
            "timestamp": 1789503584715.799,
            "message": "",
            "ingestionTime": 1789503584715
        },
        {
            "timestamp": 1789503584715.801,
            "message": "{\"processed\":1,\"bodies\":[\"order-1789503583150126000\"]}",
            "ingestionTime": 1789503584715
        }
    ],
    "nextForwardToken": "f/00000000000000000000000000000000000000000000000000000005",
    "nextBackwardToken": "b/00000000000000000000000000000000000000000000000000000000"
}
ok   the queue actually drained, meaning something besides us invoked the function
ok   the function's log genuinely contains the message body we sent, not a canned value

What was actually proven above, and what to notice about it. The function's log (in the Output block) contains the exact random value this page's build generated right before sending it -- visible there as the message body -- and that value only exists in the log because a real message with that exact body was sent to the queue and a real container ran the real handler against it. Nothing here special-cases that value; it is simply what was in the queue when the mapping polled it.

A second event source: S3 invokes the function when an object lands

The mechanism is the same idea with a different source: instead of a poller draining a queue, S3 itself calls the function directly the moment an upload completes, and hands it the bucket name and key as the event.

Upload a file. Never call invoke. S3 calls the function for you.

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67work=$(mktemp -d)
cd "$work"

cat > lambda_function.py << 'PY'
def handler(event, context):
    seen = []
    for rec in event.get("Records", []):
        bucket = rec.get("s3", {}).get("bucket", {}).get("name")
        key = rec.get("s3", {}).get("object", {}).get("key")
        print("NEW OBJECT:", bucket, key)
        seen.append(key)
    return {"seen": seen}
PY

aws iam create-role --role-name s3-trigger-role --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > /dev/null

zip -q function.zip lambda_function.py
aws lambda create-function --function-name upload-handler \
  --runtime python3.11 --role arn:aws:iam::123456789012:role/s3-trigger-role \
  --handler lambda_function.handler --zip-file fileb://function.zip > /dev/null
fn_arn=$(aws lambda get-function --function-name upload-handler --query 'Configuration.FunctionArn' --output text)

aws s3 mb s3://course-uploads > /dev/null

echo "granting S3 permission to invoke this specific function for this specific bucket:"
aws lambda add-permission --function-name upload-handler --statement-id s3invoke \
  --action lambda:InvokeFunction --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::course-uploads > /dev/null

cat > notif.json << JSON
{
  "LambdaFunctionConfigurations": [{
    "LambdaFunctionArn": "$fn_arn",
    "Events": ["s3:ObjectCreated:*"]
  }]
}
JSON

aws s3api put-bucket-notification-configuration --bucket course-uploads \
  --notification-configuration file://notif.json
echo "notification wired: any ObjectCreated event on this bucket now calls the function"

unique="report-$(date +%s%N).csv"
echo
echo "uploading a real file -- again, invoke is never called by this script:"
echo "student,grade" | aws s3 cp - "s3://course-uploads/incoming/$unique"

stream=""
for i in 1 2 3 4 5 6 7 8 9 10; do
  stream=$(aws logs describe-log-streams --log-group-name /aws/lambda/upload-handler \
    --query 'logStreams[0].logStreamName' --output text 2>/dev/null || echo "")
  if [ -n "$stream" ] && [ "$stream" != "None" ]; then break; fi
  sleep 0.5
done

logs=$(aws logs get-log-events --log-group-name /aws/lambda/upload-handler --log-stream-name "$stream" --output json)
echo "$logs"

cd / && rm -rf "$work"

check "S3's own upload triggered the function, and its log shows our exact key" \
  "$([ "$(echo "$logs" | grep -c "$unique")" -ge 1 ] && echo yes)" "yes"
check "the log shows the exact bucket name too, not a placeholder" \
  "$([ "$(echo "$logs" | grep -c 'course-uploads')" -ge 1 ] && echo yes)" "yes"

What it printed, when the build ran it

granting S3 permission to invoke this specific function for this specific bucket:
notification wired: any ObjectCreated event on this bucket now calls the function

uploading a real file -- again, invoke is never called by this script:
{
    "events": [
        {
            "timestamp": 1789503592088.51,
            "message": "\u001b[32mSTART RequestId: 37a53694-0d85-1aaa-c1aa-28437c053599 Version: $LATEST\u001b[0m",
            "ingestionTime": 1789503592088
        },
        {
            "timestamp": 1789503592088.513,
            "message": "NEW OBJECT: course-uploads incoming%2Freport-1789503590492617000.csv",
            "ingestionTime": 1789503592088
        },
        {
            "timestamp": 1789503592088.5151,
            "message": "\u001b[32mEND RequestId: 37a53694-0d85-1aaa-c1aa-28437c053599\u001b[0m",
            "ingestionTime": 1789503592088
        },
        {
            "timestamp": 1789503592088.5168,
            "message": "\u001b[32mREPORT RequestId: 37a53694-0d85-1aaa-c1aa-28437c053599\tInit Duration: 187.51 ms\tDuration: 2.72 ms\tBilled Duration: 3 ms\tMemory Size: 128 MB\tMax Memory Used: 28 MB\t\u001b[0m",
            "ingestionTime": 1789503592088
        },
        {
            "timestamp": 1789503592088.519,
            "message": "",
            "ingestionTime": 1789503592088
        },
        {
            "timestamp": 1789503592088.52,
            "message": "{\"seen\":[\"incoming%2Freport-1789503590492617000.csv\"]}",
            "ingestionTime": 1789503592088
        }
    ],
    "nextForwardToken": "f/00000000000000000000000000000000000000000000000000000005",
    "nextBackwardToken": "b/00000000000000000000000000000000000000000000000000000000"
}
ok   S3's own upload triggered the function, and its log shows our exact key
ok   the log shows the exact bucket name too, not a placeholder

The whole thing, chained: upload, process, publish, land in an audit queue

This is every mechanism from Part 8 in one pipeline: an S3 upload triggers a function, that function's own code publishes to an SNS topic (the same client call SQS's subscribe could just as easily be reading from), and that topic fans out to a queue that a completely separate part of the system reads for auditing. No step waited for the next one to be ready.

Upload once. S3, Lambda, SNS and SQS each do their one job, unattended.

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87echo "=== the audit side (already there, unaware of what will feed it) ==="
topic=$(aws sns create-topic --name upload-events --output json)
topic_arn=$(echo "$topic" | grep -o '"TopicArn": "[^"]*"' | cut -d'"' -f4)

audit_q=$(aws sqs create-queue --queue-name audit-log --output json)
audit_url=$(echo "$audit_q" | grep -o '"QueueUrl": "[^"]*"' | cut -d'"' -f4)
audit_arn=$(aws sqs get-queue-attributes --queue-url "$audit_url" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
aws sns subscribe --topic-arn "$topic_arn" --protocol sqs --notification-endpoint "$audit_arn" > /dev/null
echo "topic and audit queue exist, subscribed to each other"

echo
echo "=== the processing function: triggered by S3, itself publishes to the topic above ==="
work=$(mktemp -d)
cd "$work"
cat > lambda_function.py << PY
import boto3

def handler(event, context):
    sns = boto3.client("sns", endpoint_url="http://host.docker.internal:5099", region_name="us-east-1")
    processed = []
    for rec in event.get("Records", []):
        bucket = rec.get("s3", {}).get("bucket", {}).get("name")
        key = rec.get("s3", {}).get("object", {}).get("key")
        sns.publish(
            TopicArn="$topic_arn",
            Message='{"bucket": "' + bucket + '", "key": "' + key + '", "status": "processed"}'
        )
        processed.append(key)
    return {"processed": processed}
PY
zip -q function.zip lambda_function.py

aws iam create-role --role-name pipeline-role --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > /dev/null

aws lambda create-function --function-name pipeline-fn \
  --runtime python3.11 --role arn:aws:iam::123456789012:role/pipeline-role \
  --handler lambda_function.handler --zip-file fileb://function.zip \
  --environment 'Variables={AWS_ACCESS_KEY_ID=test,AWS_SECRET_ACCESS_KEY=test}' > /dev/null
fn_arn=$(aws lambda get-function --function-name pipeline-fn --query 'Configuration.FunctionArn' --output text)

echo
echo "=== wiring S3 to that function, same as the panel above ==="
aws s3 mb s3://pipeline-uploads > /dev/null
aws lambda add-permission --function-name pipeline-fn --statement-id s3invoke \
  --action lambda:InvokeFunction --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::pipeline-uploads > /dev/null

cat > notif.json << JSON
{
  "LambdaFunctionConfigurations": [{
    "LambdaFunctionArn": "$fn_arn",
    "Events": ["s3:ObjectCreated:*"]
  }]
}
JSON
aws s3api put-bucket-notification-configuration --bucket pipeline-uploads --notification-configuration file://notif.json

echo
echo "=== the one action a person actually takes: uploading a file ==="
unique="upload-$(date +%s%N).csv"
echo "student,grade" | aws s3 cp - "s3://pipeline-uploads/incoming/$unique"
echo "uploaded incoming/$unique -- everything from here happens without anyone calling anything"

echo
echo "polling the audit queue for the result of that one upload:"
found=""
for i in 1 2 3 4 5 6 7 8 9 10; do
  recv=$(aws sqs receive-message --queue-url "$audit_url" --output json)
  if echo "$recv" | grep -q "$unique"; then
    echo "poll $i: the audit trail has it"
    found="yes"
    break
  fi
  echo "poll $i: not yet"
  sleep 0.5
done

cd / && rm -rf "$work"

check "the upload made it all the way to the audit queue with no manual step" "$found" "yes"
check "the audit record shows the real key that was actually uploaded" \
  "$(echo "$recv" | grep -c "$unique")" "1"
check "the audit record shows the processing step marked it processed" \
  "$(echo "$recv" | grep -c 'processed')" "1"

What it printed, when the build ran it

=== the audit side (already there, unaware of what will feed it) ===
topic and audit queue exist, subscribed to each other

=== the processing function: triggered by S3, itself publishes to the topic above ===

=== wiring S3 to that function, same as the panel above ===

=== the one action a person actually takes: uploading a file ===
uploaded incoming/upload-1789503599291096000.csv -- everything from here happens without anyone calling anything

polling the audit queue for the result of that one upload:
poll 1: the audit trail has it
ok   the upload made it all the way to the audit queue with no manual step
ok   the audit record shows the real key that was actually uploaded
ok   the audit record shows the processing step marked it processed

What just ran, physically. Every step above used the real, separate mechanism its own lesson introduced -- create-event-source-mapping polling SQS from lesson-25-style Lambda infrastructure, S3's own notification configuration from this lesson's first panel, and lesson 26's SNS-to-SQS fan-out for the audit trail -- run together as one chain instead of three isolated demos. Nothing about combining them needed new AWS features; event-driven architecture is these same pieces, wired to each other instead of to a human pressing a button.

What this part actually replaced

A cron job that polls "is there anything new yet?" every minute, most of those checks finding nothing -- replaced by something that only runs when there is genuinely something to do.
A monolith where the upload-handling code, the billing code and the shipping code all live in one deploy, one failure domain, one team's release schedule -- replaced by independent functions that only share an event, never a codebase.
A queue or topic added later does not require touching the code that produces the event at all -- it is a new subscriber, exactly like billing and shipping in lesson 26, never a new if branch in the uploader.

25 Lambda: functions as infrastructure

A real function, uploaded as a real zip, invoked with real input

A function that raises: the failure path, not just the happy one

Billing follows execution, not existence

26 SQS and SNS: queues, topics and decoupling

SQS: a message is not gone until someone deletes it

SNS: one publish, every subscriber gets their own copy

Dead-letter queues: where messages go when nothing can process them

The choice, in one line each

27 Event-driven architecture, end to end

An event source mapping: SQS invokes the function, not you

A second event source: S3 invokes the function when an object lands

The whole thing, chained: upload, process, publish, land in an audit queue

What this part actually replaced

New writing, when it lands.