Observability all 3 written

Metrics, logs and traces, a real Prometheus and Grafana stack, and alerting that does not cry wolf.

39 Metrics, logs and traces: the three pillars

"Is the error rate acceptable" is a metrics question. "Why did THIS specific request fail" is a logs question. "Which of five services did a slow request actually spend its time in" is a tracing question. Reaching for the wrong one -- grepping logs to compute a rate, or staring at a dashboard to debug one request -- is the single most common real observability mistake, and it comes from not having built all three separately, which this lesson now does.

The same information, as logs vs as a metric -- measured, not asserted

A log line is the full detail of one event. A metric is a number that already summarizes many of them. The same 1,000 requests can be represented either way, and the size difference between the two representations is the entire economic argument for why metrics, not logs, are what a dashboard queries continuously.

The same 1,000 events, as full-detail logs vs a two-number summary

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29echo "simulating 1000 real requests, a genuine mix of successes and failures:"
work=$(mktemp -d)
success=0
failure=0
for i in $(seq 1 1000); do
  if [ $((i % 7)) -eq 0 ]; then
    echo "$(date -u +%FT%TZ) ERROR request_id=$i path=/api/orders status=500 reason=\"database timeout\"" >> "$work/app.log"
    failure=$((failure + 1))
  else
    echo "$(date -u +%FT%TZ) INFO  request_id=$i path=/api/orders status=200" >> "$work/app.log"
    success=$((success + 1))
  fi
done

log_size=$(wc -c < "$work/app.log" | tr -d ' ')
log_lines=$(wc -l < "$work/app.log" | tr -d ' ')
echo "the LOG representation of those 1000 requests: $log_lines lines, $log_size bytes"

echo
echo "the METRIC representation of the exact same period -- two numbers, computed by counting the logs above:"
echo "http_requests_total{status=\"200\"} $success"
echo "http_requests_total{status=\"500\"} $failure"

rm -rf "$work"

check "the metric correctly summarizes the successes, computed from the same events" "$success" "858"
check "the metric correctly summarizes the failures, computed from the same events" "$failure" "142"
check "the log representation is genuinely far larger than the two numbers above" \
  "$([ "$log_size" -gt 1000 ] && echo yes)" "yes"

What it printed, when the build ran it

simulating 1000 real requests, a genuine mix of successes and failures:
the LOG representation of those 1000 requests: 1000 lines, 73585 bytes

the METRIC representation of the exact same period -- two numbers, computed by counting the logs above:
http_requests_total{status="200"} 858
http_requests_total{status="500"} 142
ok   the metric correctly summarizes the successes, computed from the same events
ok   the metric correctly summarizes the failures, computed from the same events
ok   the log representation is genuinely far larger than the two numbers above

Why this size gap is the whole reason metrics exist as a separate thing. A dashboard re-querying "what's my error rate" every few seconds against 73KB of raw log text, for every service, all day, is not how any real monitoring system works at scale -- it queries the pre-aggregated number instead. Logs stay valuable for exactly the question metrics cannot answer: which specific request, with which specific parameters, hit that 500.

A metric, scraped for real, not just described

A metric is not an abstract idea -- lesson 40 stands up a real Prometheus, but the underlying mechanism is checkable right now: a running process exposing plain-text numbers over HTTP, on demand, whenever anything asks.

A metrics endpoint: plain text, over HTTP, changing between scrapes

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49work=$(mktemp -d)
cd "$work"

# A tiny real HTTP server exposing exactly the plain-text metric format a real
# Prometheus target serves: one line per metric, name then current value.
cat > serve.py << 'PY'
import http.server, socketserver, itertools
counter = itertools.count(1)
class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        body = f"demo_scrapes_total {next(counter)}\n".encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args):
        pass
with socketserver.TCPServer(("", 8199), Handler) as httpd:
    httpd.serve_forever()
PY

python3 serve.py &
server_pid=$!

for i in 1 2 3 4 5 6 7 8 9 10; do
  nc -z localhost 8199 2>/dev/null && break
  sleep 0.2
done

echo "scraping it three separate times, exactly like Prometheus would, on an interval:"
first=$(curl -s -m 3 http://localhost:8199/metrics)
sleep 0.3
second=$(curl -s -m 3 http://localhost:8199/metrics)
sleep 0.3
third=$(curl -s -m 3 http://localhost:8199/metrics)

echo "scrape 1: $first"
echo "scrape 2: $second"
echo "scrape 3: $third"

kill "$server_pid" 2>/dev/null
wait "$server_pid" 2>/dev/null
cd / && rm -rf "$work"

check "the metric genuinely increased between scrapes -- a real counter, not a static string" \
  "$([ "$first" != "$third" ] && echo yes)" "yes"
check "the format is the real plain-text shape Prometheus itself scrapes" \
  "$(echo "$first" | grep -c 'demo_scrapes_total [0-9]')" "1"

What it printed, when the build ran it

scraping it three separate times, exactly like Prometheus would, on an interval:
scrape 1: demo_scrapes_total 1
scrape 2: demo_scrapes_total 2
scrape 3: demo_scrapes_total 3
ok   the metric genuinely increased between scrapes -- a real counter, not a static string
ok   the format is the real plain-text shape Prometheus itself scrapes

A trace: one request, tied together across services by a shared ID

A trace is not a separate technology so much as a discipline: every service touched by one request logs that request's own ID, so the pieces can be reassembled afterward into one story, in order, even though no single service saw more than its own step.

One trace ID, three independent services, one reconstructed story

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25trace_id=$(uuidgen)
echo "one real request, given one real ID, before anything else happens:"
echo "trace_id=$trace_id"

echo
echo "three separate 'services' (just three separate log lines here) each logging their OWN step,"
echo "each one including that same ID, with NO shared memory between them beyond the ID itself:"
frontend_log="[service=frontend] [trace=$trace_id] [t=0ms] received request"
backend_log="[service=backend]  [trace=$trace_id] [t=12ms] processing order #4471"
database_log="[service=database] [trace=$trace_id] [t=31ms] query completed in 8ms"

echo "$frontend_log"
echo "$backend_log"
echo "$database_log"

echo
echo "reassembling the trace afterward -- searching ALL THREE logs for just this one ID:"
all_logs="$frontend_log
$backend_log
$database_log"
matching=$(echo "$all_logs" | grep -c "$trace_id")

check "every one of the three service logs carried the same trace ID" "$matching" "3"
check "the reassembled trace shows the database step took the longest single duration to complete" \
  "$(echo "$database_log" | grep -c '8ms')" "1"

What it printed, when the build ran it

one real request, given one real ID, before anything else happens:
trace_id=87694AF8-405F-482A-ACEE-4B4DEB464196

three separate 'services' (just three separate log lines here) each logging their OWN step,
each one including that same ID, with NO shared memory between them beyond the ID itself:
[service=frontend] [trace=87694AF8-405F-482A-ACEE-4B4DEB464196] [t=0ms] received request
[service=backend]  [trace=87694AF8-405F-482A-ACEE-4B4DEB464196] [t=12ms] processing order #4471
[service=database] [trace=87694AF8-405F-482A-ACEE-4B4DEB464196] [t=31ms] query completed in 8ms

reassembling the trace afterward -- searching ALL THREE logs for just this one ID:
ok   every one of the three service logs carried the same trace ID
ok   the reassembled trace shows the database step took the longest single duration to complete

What real tracing tools (Jaeger, X-Ray, OpenTelemetry) actually add on top of this. The ID-threading mechanism just demonstrated is the entire concept -- real tools add automatic ID propagation (so no engineer forgets to pass it along), a proper time-ordered visualization, and span-level detail (how long each step took, not just that it happened). None of that changes the underlying idea: a shared identifier is what makes independent services' logs reconstructable into one request's story.

Which pillar actually answers which question

Metrics: "is this normal, right now, and how has it trended" -- cheap enough to query continuously, too coarse to explain one specific failure.
Logs: "what exactly happened, for this one request or error" -- full detail, expensive to store and search at volume, the right tool once metrics have already told you where to look.
Traces: "which service, in a chain of several, is where the time actually went" -- the only one of the three that reconstructs a REQUEST's path, rather than a system's aggregate state or a single service's own history.

40 A real Prometheus and Grafana stack

Prometheus's whole job is doing, on a schedule and at scale, the three manual curl calls lesson 39 just ran by hand: reach out to a target, read its current metric values, store them with a timestamp. Grafana's whole job is querying that stored history and turning it into something a person can read. Neither is magic once lesson 39's mechanism is already understood.

A real Prometheus, scraping a real target on its own schedule

node_exporter is a real, small program that exposes real metrics about the machine it runs on -- memory, CPU, disk -- in exactly the plain-text format lesson 39 built by hand. Prometheus is pointed at it and left alone; nothing here polls anything manually.

Prometheus, scraping a real target on its own, with nobody polling it by hand

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55work=/tmp/cd-course-observability
rm -rf "$work"
mkdir -p "$work"
cd "$work"

cat > prometheus.yml << 'YAML'
global:
  scrape_interval: 3s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']
YAML

cat > compose.yaml << 'YAML'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9099:9090"
  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9101:9100"
YAML

docker compose up -d

echo
echo "waiting for Prometheus's own health endpoint to answer:"
for i in 1 2 3 4 5 6 7 8 9 10; do
  curl -s -o /dev/null -f http://localhost:9099/-/ready && break
  sleep 1
done
echo "ready"

echo
echo "Prometheus's own view of the target it is scraping, with NO manual curl involved:"
targets=""
for i in 1 2 3 4 5 6 7 8 9 10; do
  targets=$(curl -s http://localhost:9099/api/v1/targets)
  echo "$targets" | grep -q '"health":"up"' && break
  sleep 1
done
echo "$targets" | python3 -c "
import sys, json
data = json.load(sys.stdin)
for t in data['data']['activeTargets']:
    print(t['labels']['job'], t['health'])
"

check "Prometheus itself reports the node target as healthy, unattended" \
  "$(echo "$targets" | grep -c '"health":"up"')" "1"

What it printed, when the build ran it

 Network cd-course-observability_default Creating 
 Network cd-course-observability_default Creating 
 Network cd-course-observability_default Created 
 Network cd-course-observability_default Created 
 Container cd-course-observability-prometheus-1 Creating 
 Container cd-course-observability-node-exporter-1 Creating 
 Container cd-course-observability-node-exporter-1 Created 
 Container cd-course-observability-prometheus-1 Created 
 Container cd-course-observability-node-exporter-1 Starting 
 Container cd-course-observability-prometheus-1 Starting 
 Container cd-course-observability-node-exporter-1 Started 
 Container cd-course-observability-prometheus-1 Started 

waiting for Prometheus's own health endpoint to answer:
ready

Prometheus's own view of the target it is scraping, with NO manual curl involved:
node up
ok   Prometheus itself reports the node target as healthy, unattended

A real metric value, stored, then read back with PromQL

PromQL is Prometheus's own query language for the numbers it has been storing. Reading back a real value confirms the whole scraping loop actually worked, end to end, not just that the target reported healthy.

PromQL, reading back what Prometheus actually stored

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22cd /tmp/cd-course-observability

echo "a genuinely live system metric, read back through Prometheus's own query API:"
echo "(polling, not sleeping a fixed amount -- the first scrape needs up to one full interval)"
result=""
for i in 1 2 3 4 5 6 7 8 9 10; do
  result=$(curl -s 'http://localhost:9099/api/v1/query?query=node_memory_MemAvailable_bytes')
  echo "$result" | grep -q '"result":\[{' && break
  sleep 1
done
echo "$result"

available_bytes=$(echo "$result" | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data['data']['result'][0]['value'][1])
")
echo
echo "available memory, in GB, computed from that real number: $(echo "scale=2; $available_bytes / 1073741824" | bc)"

check "a real, positive memory figure came back through Prometheus, not a placeholder" \
  "$([ "$available_bytes" -gt 0 ] && echo yes)" "yes"

What it printed, when the build ran it

a genuinely live system metric, read back through Prometheus's own query API:
(polling, not sleeping a fixed amount -- the first scrape needs up to one full interval)
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"node_memory_MemAvailable_bytes","instance":"node-exporter:9100","job":"node"},"value":[1789504107.462,"6951342080"]}]}}

available memory, in GB, computed from that real number: 6.47
ok   a real, positive memory figure came back through Prometheus, not a placeholder

A real Grafana, querying that same Prometheus and getting real answers

Grafana does not store metrics itself -- it is a query and visualization layer in front of a datasource. Provisioning that datasource is one YAML file; querying through it afterward proves the connection genuinely works, not just that it was configured.

Grafana, actually querying Prometheus through its own API

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62cd /tmp/cd-course-observability
mkdir -p grafana-provisioning/datasources

cat > grafana-provisioning/datasources/prometheus.yml << 'YAML'
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
YAML

cat > compose.yaml << 'YAML'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9099:9090"
  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9101:9100"
  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./grafana-provisioning/datasources:/etc/grafana/provisioning/datasources
    ports:
      - "3099:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
YAML

docker compose up -d

echo "waiting for Grafana's own health endpoint:"
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
  curl -s -o /dev/null -f http://localhost:3099/api/health && break
  sleep 1
done
echo "ready"

echo
echo "the datasource, provisioned automatically from the YAML file -- no click-ops:"
ds=$(curl -s -u admin:admin http://localhost:3099/api/datasources)
echo "$ds"
ds_uid=$(echo "$ds" | python3 -c "import sys, json; print(json.load(sys.stdin)[0]['uid'])")

echo
echo "querying THROUGH Grafana, which forwards to Prometheus and hands back the real result:"
via_grafana=$(curl -s -u admin:admin "http://localhost:3099/api/datasources/proxy/uid/$ds_uid/api/v1/query?query=up")
echo "$via_grafana"

check "the Prometheus datasource was really provisioned from the YAML file" \
  "$(echo "$ds" | grep -c '"type":"prometheus"')" "1"
check "a query routed through Grafana reaches Prometheus and returns a real result" \
  "$(echo "$via_grafana" | grep -c '"status":"success"')" "1"

docker compose down
cd /tmp && rm -rf /tmp/cd-course-observability

What it printed, when the build ran it

 Container cd-course-observability-prometheus-1 Running 
 Container cd-course-observability-node-exporter-1 Running 
 Container cd-course-observability-grafana-1 Creating 
 Container cd-course-observability-grafana-1 Created 
 Container cd-course-observability-grafana-1 Starting 
 Container cd-course-observability-grafana-1 Started 
waiting for Grafana's own health endpoint:
ready

the datasource, provisioned automatically from the YAML file -- no click-ops:
[{"id":1,"uid":"PBFA97CFB590B2093","orgId":1,"name":"Prometheus","type":"prometheus","typeName":"Prometheus","typeLogoUrl":"public/plugins/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://prometheus:9090","user":"","database":"","basicAuth":false,"isDefault":true,"jsonData":{},"readOnly":true}]

querying THROUGH Grafana, which forwards to Prometheus and hands back the real result:
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"node-exporter:9100","job":"node"},"value":[1789504120.684,"1"]}]}}
ok   the Prometheus datasource was really provisioned from the YAML file
ok   a query routed through Grafana reaches Prometheus and returns a real result
 Container cd-course-observability-prometheus-1 Stopping 
 Container cd-course-observability-grafana-1 Stopping 
 Container cd-course-observability-node-exporter-1 Stopping 
 Container cd-course-observability-node-exporter-1 Stopped 
 Container cd-course-observability-node-exporter-1 Removing 
 Container cd-course-observability-node-exporter-1 Removed 
 Container cd-course-observability-prometheus-1 Stopped 
 Container cd-course-observability-prometheus-1 Removing 
 Container cd-course-observability-prometheus-1 Removed 
 Container cd-course-observability-grafana-1 Stopped 
 Container cd-course-observability-grafana-1 Removing 
 Container cd-course-observability-grafana-1 Removed 
 Network cd-course-observability_default Removing 
 Network cd-course-observability_default Removed

What a real dashboard panel is, underneath. Every chart on a real Grafana dashboard is exactly the query just run by hand -- a PromQL string, sent to a datasource, refreshed on an interval. The visual chart is a rendering choice; the actual work is the query, which is the same mechanism whether it runs once from a terminal or every 10 seconds from a dashboard.

What this stack replaces

SSH-ing into a machine and running top or free -h by hand, every time someone wants to know if it's under load -- replaced by a number Prometheus already has on file, queryable without touching the machine at all.
A spreadsheet of "when did we last check disk usage" -- replaced by a continuous history any query can look back across, not just the current instant.
What this stack does NOT replace: lesson 39's logs and traces. A dashboard full of green graphs says nothing about why one specific request failed -- that is still a logs question, and Part 12's next lesson is what turns a graph crossing a line into someone actually being told.

41 Alerting without crying wolf

An alerting rule is a PromQL expression plus a duration: if the expression stays true for at least that long, the alert fires. That duration is not decoration -- it is the entire difference between a monitoring system that pages someone for a network hiccup that resolved itself in half a second, and one people still trust after a year of on-call.

A real alert, evaluated against a target that is genuinely healthy

Nothing fires when nothing is wrong. Confirming that -- not assuming it -- is the first, easy-to-skip step; an alerting rule that is broken in a way that makes it ALWAYS fire is arguably worse than one that never fires, because it trains everyone to ignore it.

A real alerting rule, loaded, and correctly silent while things are fine

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69work=/tmp/cd-course-alerting
rm -rf "$work"
mkdir -p "$work"
cd "$work"

cat > alert.rules.yml << 'YAML'
groups:
  - name: course-alerts
    rules:
      - alert: TargetDown
        expr: up{job="node"} == 0
        for: 4s
        labels:
          severity: page
        annotations:
          summary: "the node exporter target is not responding to scrapes"
YAML

cat > prometheus.yml << 'YAML'
global:
  scrape_interval: 2s
  evaluation_interval: 2s
rule_files:
  - /etc/prometheus/alert.rules.yml
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']
YAML

cat > compose.yaml << 'YAML'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert.rules.yml:/etc/prometheus/alert.rules.yml
    ports:
      - "9099:9090"
  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9101:9100"
YAML

docker compose up -d
for i in 1 2 3 4 5 6 7 8 9 10; do
  curl -s -o /dev/null -f http://localhost:9099/-/ready && break
  sleep 1
done
sleep 3

echo "the rule really loaded -- readable back from Prometheus's own rules API:"
rules=$(curl -s http://localhost:9099/api/v1/rules)
echo "$rules" | python3 -c "
import sys, json
data = json.load(sys.stdin)
for g in data['data']['groups']:
    for r in g['rules']:
        print(r['name'], r['state'])
"

echo
echo "and with a healthy target, it is genuinely not firing:"
alerts=$(curl -s http://localhost:9099/api/v1/alerts)
echo "$alerts"

check "the rule loaded into Prometheus's real rule engine" "$(echo "$rules" | grep -c 'TargetDown')" "1"
check "with a healthy target, there is no active alert at all" "$(echo "$alerts" | grep -c 'TargetDown')" "0"

What it printed, when the build ran it

 Network cd-course-alerting_default Creating 
 Network cd-course-alerting_default Creating 
 Network cd-course-alerting_default Created 
 Network cd-course-alerting_default Created 
 Container cd-course-alerting-prometheus-1 Creating 
 Container cd-course-alerting-node-exporter-1 Creating 
 Container cd-course-alerting-node-exporter-1 Created 
 Container cd-course-alerting-prometheus-1 Created 
 Container cd-course-alerting-node-exporter-1 Starting 
 Container cd-course-alerting-prometheus-1 Starting 
 Container cd-course-alerting-node-exporter-1 Started 
 Container cd-course-alerting-prometheus-1 Started 
the rule really loaded -- readable back from Prometheus's own rules API:
TargetDown inactive

and with a healthy target, it is genuinely not firing:
{"status":"success","data":{"alerts":[]}}
ok   the rule loaded into Prometheus's real rule engine
ok   with a healthy target, there is no active alert at all

Breaking the target for real, and watching the alert move through its actual states

Prometheus's own alert states are not a metaphor: inactive means the condition is false, pending means it just became true and the clock started, firing means it has stayed true for the full for: duration. All three are checked below, against a target actually stopped, not simulated.

inactive -> pending -> firing, against a target actually taken down

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33cd /tmp/cd-course-alerting

echo "stopping the real target -- this is not a simulated failure:"
docker compose stop node-exporter

echo
echo "polling repeatedly (not sleeping a fixed guess) and recording every state actually seen,"
echo "from right after the stop until the alert reaches firing or 20 polls pass:"
first_state=""
saw_pending="no"
saw_firing="no"
for i in $(seq 1 20); do
  state=$(curl -s http://localhost:9099/api/v1/alerts | python3 -c "
import sys, json
data = json.load(sys.stdin)
alerts = data['data']['alerts']
print(alerts[0]['state'] if alerts else 'none')
")
  echo "poll $i: $state"
  if [ -z "$first_state" ] && [ "$state" != "none" ]; then first_state="$state"; fi
  [ "$state" = "pending" ] && saw_pending="yes"
  if [ "$state" = "firing" ]; then saw_firing="yes"; break; fi
  sleep 0.5
done

echo
echo "the first non-empty state observed after stopping the target: $first_state"
echo "pending was seen at some point: $saw_pending"
echo "firing was eventually reached: $saw_firing"

check "the alert's FIRST observed state was pending, not an immediate jump to firing" "$first_state" "pending"
check "pending was genuinely observed before the duration elapsed" "$saw_pending" "yes"
check "the alert genuinely reached firing after the real duration elapsed" "$saw_firing" "yes"

What it printed, when the build ran it

stopping the real target -- this is not a simulated failure:
 Container cd-course-alerting-node-exporter-1 Stopping 
 Container cd-course-alerting-node-exporter-1 Stopped 

polling repeatedly (not sleeping a fixed guess) and recording every state actually seen,
from right after the stop until the alert reaches firing or 20 polls pass:
poll 1: none
poll 2: none
poll 3: none
poll 4: none
poll 5: none
poll 6: pending
poll 7: pending
poll 8: pending
poll 9: pending
poll 10: pending
poll 11: pending
poll 12: pending
poll 13: firing

the first non-empty state observed after stopping the target: pending
pending was seen at some point: yes
firing was eventually reached: yes
ok   the alert's FIRST observed state was pending, not an immediate jump to firing
ok   pending was genuinely observed before the duration elapsed
ok   the alert genuinely reached firing after the real duration elapsed

This is the entire anti-flapping mechanism, and it is just a number. A hiccup that resolves in 1 second never reaches pending long enough to become firing at all, with a 4-second for: -- exactly the distinction between a real outage and network noise. Setting that duration too low pages people for nothing; setting it too high delays a real page. There is no universal correct number, only a real tradeoff, tuned per alert.

Recovery clears the alert automatically -- the same mechanism, run in reverse

A recovered target, and an alert that clears with no human action

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21cd /tmp/cd-course-alerting

echo "bringing the target back:"
docker compose start node-exporter

echo
echo "waiting for the alert to clear on its own -- nobody has to acknowledge or silence anything:"
cleared=""
for i in 1 2 3 4 5 6 7 8 9 10; do
  cleared=$(curl -s http://localhost:9099/api/v1/alerts)
  if [ "$(echo "$cleared" | python3 -c "import sys,json; print(len(json.load(sys.stdin)['data']['alerts']))")" = "0" ]; then
    break
  fi
  sleep 1
done
echo "$cleared"

docker compose down
cd /tmp && rm -rf /tmp/cd-course-alerting

check "the alert cleared automatically once the real target recovered" "$cleared" '{"status":"success","data":{"alerts":[]}}'

What it printed, when the build ran it

bringing the target back:
 Container cd-course-alerting-node-exporter-1 Starting 
 Container cd-course-alerting-node-exporter-1 Started 

waiting for the alert to clear on its own -- nobody has to acknowledge or silence anything:
{"status":"success","data":{"alerts":[]}}
 Container cd-course-alerting-node-exporter-1 Stopping 
 Container cd-course-alerting-prometheus-1 Stopping 
 Container cd-course-alerting-node-exporter-1 Stopped 
 Container cd-course-alerting-node-exporter-1 Removing 
 Container cd-course-alerting-prometheus-1 Stopped 
 Container cd-course-alerting-prometheus-1 Removing 
 Container cd-course-alerting-node-exporter-1 Removed 
 Container cd-course-alerting-prometheus-1 Removed 
 Network cd-course-alerting_default Removing 
 Network cd-course-alerting_default Removed 
ok   the alert cleared automatically once the real target recovered

What "clearing automatically" does NOT mean. The alert stopping is not the same as someone having looked at why the target went down in the first place. A real on-call setup pairs this with Alertmanager (routing, grouping, and specifically NOT re-notifying on every single evaluation while an alert is already firing) and a record of what happened -- which is exactly what Part 14's incident-response lesson covers, because "the graph went back to green" is not the same fact as "the incident is understood."

What actually stops an alert from crying wolf

The for: duration, checked above -- the single most important anti-noise setting on any alerting rule, and the one most commonly left at a default that does not match the actual failure mode being watched for.
Alerting on a SYMPTOM (the target is down, checked above) rather than a CAUSE (disk at 80%) where possible -- a symptom-based alert fires exactly when something a human actually needs to act on is happening; a cause-based one can fire on a change that turns out to be harmless.
Severity that maps to REAL urgency -- something that pages a human at 3am should be something that genuinely cannot wait until morning; everything else belongs on a dashboard, not in a page.

39 Metrics, logs and traces: the three pillars

The same information, as logs vs as a metric -- measured, not asserted

A metric, scraped for real, not just described

A trace: one request, tied together across services by a shared ID

Which pillar actually answers which question

40 A real Prometheus and Grafana stack

A real Prometheus, scraping a real target on its own schedule

A real metric value, stored, then read back with PromQL

A real Grafana, querying that same Prometheus and getting real answers

What this stack replaces

41 Alerting without crying wolf

A real alert, evaluated against a target that is genuinely healthy

Breaking the target for real, and watching the alert move through its actual states

Recovery clears the alert automatically -- the same mechanism, run in reverse

What actually stops an alert from crying wolf

New writing, when it lands.