Foundations all 4 written
The shell, the filesystem, processes and real networking. Every DevOps tool is built on top of these.
4 The shell you actually live in
A GUI tells you a command succeeded by not showing an error. A shell tells you with a specific number, whether or not it printed anything at all, and every tool this course uses -- Terraform, kubectl, the AWS CLI -- reports success and failure through that same number. This lesson is about reading it.
Every command returns a number, whether you look at it or not
$? holds the exit status of the last command. 0 means it
considered itself successful. Anything else is that program's way of saying which kind of
failure happened -- and the number is chosen by the program, so it is worth checking, not
guessing.
echo "a command that succeeds"
true
echo "\$? after 'true' is: $?"
echo "a command that fails"
false
echo "\$? after 'false' is: $?"
echo "a command that does not exist"
this-command-does-not-exist-anywhere 2>/dev/null
echo "\$? after a missing command is: $?"
# grep's exit status means something specific: 0 = found a match, 1 = ran fine
# but found nothing, 2 = a real error (bad file, bad pattern). Three outcomes,
# one number -- which is exactly why "did it print anything" is not the same
# question as "did it work."
echo "apple" | grep -q "apple"
found=$?
echo "grep -q apple (present): exit $found"
echo "apple" | grep -q "banana"
missing=$?
echo "grep -q banana (absent): exit $missing"
check "true exits zero" "$(true; echo $?)" "0"
check "false exits nonzero" "$([ $(false; echo $?) -ne 0 ] && echo yes)" "yes"
check "grep finds a real match" "$found" "0"
check "grep exits 1, not an error, when nothing matches" "$missing" "1"What it printed, when the build ran it
a command that succeeds $? after 'true' is: 0 a command that fails $? after 'false' is: 1 a command that does not exist $? after a missing command is: 127 grep -q apple (present): exit 0 grep -q banana (absent): exit 1 ok true exits zero ok false exits nonzero ok grep finds a real match ok grep exits 1, not an error, when nothing matches
&& and || read like sentences, and they read the exit status
&& runs the next command only if the previous one exited zero.
|| only if it did not. This is not a shortcut worth memorising for its own
sake -- it is the actual mechanism a real deploy script uses to stop the moment something
goes wrong, instead of plowing on and making the failure worse.
# A miniature version of what a real deploy script's safety net looks like.
run_step() {
echo " running: $1"
eval "$1"
}
deploy() {
run_step "true" && \
run_step "true" && \
run_step "false" && \
run_step "echo THIS SHOULD NEVER PRINT"
}
echo "--- a chain that fails partway through ---"
deploy
echo "the chain's own exit status: $?"
echo
echo "--- the || fallback, used for cleanup that must run either way ---"
false || echo "cleanup ran because the previous command failed"
true || echo "this does not print, because true did not fail"
check "the chain stops at the false step, never reaching the echo after it" \
"$(deploy 2>&1 | grep -c 'NEVER PRINT')" "0"
check "the chain's own exit status reflects the failing step" "$([ $(deploy > /dev/null; echo $?) -ne 0 ] && echo yes)" "yes"What it printed, when the build ran it
--- a chain that fails partway through --- running: true running: true running: false the chain's own exit status: 1 --- the || fallback, used for cleanup that must run either way --- cleanup ran because the previous command failed ok the chain stops at the false step, never reaching the echo after it ok the chain's own exit status reflects the failing step
Why this matters the moment you write a real script. A multi-step
deploy that does terraform apply; kubectl rollout restart; ... with plain
semicolons will happily run every later step even if terraform apply failed
halfway through. The fix is not cleverness, it is &&, or the stricter
set -e covered next.
set -e, set -u and set -o pipefail
Three shell options that turn "silently ignore the problem" into "stop immediately," which is what you want in any script that provisions or deploys something real. This course's own build script uses exactly this combination to run every shell panel you see -- so this is not a stylistic recommendation, it is the actual safety net this page was built with.
# set -u: reading an unset variable is an error, not an empty string.
check_unset() {
( set -u; echo "$DOES_NOT_EXIST_ANYWHERE" ) 2>&1
}
out=$(check_unset)
echo "with set -u, an unset variable gives: $out"
# set -o pipefail: a pipeline's exit status is the LAST command that failed,
# not just the last command in the pipe. Without it, a failing step in the
# middle of a pipe can be silently swallowed by a successful step after it.
without_pipefail() {
# This whole panel already runs with pipefail on, so a plain subshell would
# inherit it. set +o turns it back off, to actually show the "without"
# behaviour instead of silently keeping "with".
( set +o pipefail; false | true; echo $? )
}
with_pipefail() {
( set -o pipefail; false | true; echo $? )
}
echo "false | true, without pipefail, reports exit: $(without_pipefail)"
echo "false | true, with pipefail, reports exit: $(with_pipefail)"
check "set -u turns a missing variable into a visible error" \
"$(echo "$out" | grep -qi "unbound\|parameter" && echo yes)" "yes"
check "without pipefail, a failed first step is hidden by a successful second" \
"$(without_pipefail)" "0"
check "with pipefail, the failure is reported" \
"$([ "$(with_pipefail)" != "0" ] && echo yes)" "yes"What it printed, when the build ran it
with set -u, an unset variable gives: /var/folders/kw/d8jyj6d53qj57mb1txzgjdlc0000gn/T/clouddevops-P7hBcE/panel.sh: line 19: DOES_NOT_EXIST_ANYWHERE: unbound variable false | true, without pipefail, reports exit: 0 false | true, with pipefail, reports exit: 1 ok set -u turns a missing variable into a visible error ok without pipefail, a failed first step is hidden by a successful second ok with pipefail, the failure is reported
Piping and redirection: connecting programs, not typing more of them
| sends one program's stdout into the next program's stdin. >
sends stdout to a file, overwriting it. >> appends. 2>&1
merges stderr into wherever stdout is currently going -- the exact trick this course's own
shell panels use so you see errors in the same place as normal output.
# A short pipeline: generate lines, filter them, count what is left.
printf 'ok\nFAIL\nok\nok\nFAIL\nok\n' > /tmp/cd-course-lines.txt
total=$(wc -l < /tmp/cd-course-lines.txt | tr -d ' ')
oks=$(grep -c '^ok' /tmp/cd-course-lines.txt)
fails=$(grep -c '^FAIL' /tmp/cd-course-lines.txt)
echo "total lines: $total"
echo "ok lines: $oks"
echo "failed lines: $fails"
rm -f /tmp/cd-course-lines.txt
check "six lines were written" "$total" "6"
check "four are ok" "$oks" "4"
check "two are FAIL" "$fails" "2"What it printed, when the build ran it
total lines: 6 ok lines: 4 failed lines: 2 ok six lines were written ok four are ok ok two are FAIL
Quoting: the mistake that breaks scripts on real data
An unquoted variable is split on whitespace and expanded as a glob. This is not trivia -- it is the single most common cause of a working script breaking the day it meets a filename or a value with a space in it, which in cloud work is constantly (S3 keys, tag values, resource names).
name="a file with spaces.txt"
count_unquoted() {
# An unquoted $name is split on whitespace into separate words by the shell
# before this function even sees it, so "$#" inside counts every word.
set -- $name
echo "$#"
}
count_quoted() {
set -- "$name"
echo "$#"
}
unquoted_count=$(count_unquoted)
quoted_count=$(count_quoted)
echo "unquoted \$name became $unquoted_count separate word(s): 'a' 'file' 'with' 'spaces.txt'"
echo "quoted \"\$name\" stayed $quoted_count word"
check "unquoted splits the value into four words" "$unquoted_count" "4"
check "quoted keeps it as one value" "$quoted_count" "1"What it printed, when the build ran it
unquoted $name became 4 separate word(s): 'a' 'file' 'with' 'spaces.txt' quoted "$name" stayed 1 word ok unquoted splits the value into four words ok quoted keeps it as one value
The rule that avoids most shell scripting bugs: quote every variable
expansion unless you specifically want word-splitting. "$name", not
$name. Every panel from here on in this course follows that rule, and the
one time you will see it deliberately broken, the lesson says so.
5 Processes, permissions and the filesystem
A process is a running program with a process ID, a parent, an owner, and a set of things it is and is not allowed to touch. Almost every "why is this not working" question in this job reduces to one of those four facts being different from what you assumed.
Starting a background process and finding it again
& puts a command in the background and hands control straight back to
you. $! holds its process ID the instant it starts -- the same ID a real
process supervisor (systemd, or a container runtime) uses to track, signal and eventually
kill it.
# Start something that just sits there, in the background.
sleep 30 &
pid=$!
echo "started sleep 30 in the background, pid $pid"
# Confirm it is really running, by asking the OS, not by trusting our own note.
if ps -p "$pid" > /dev/null 2>&1; then
echo "ps confirms pid $pid is alive"
alive_before="yes"
else
alive_before="no"
fi
# Ask what it actually is, straight from the process table.
ps -p "$pid" -o pid,ppid,comm | tail -n +2
kill "$pid"
# Poll for it to actually be gone rather than guessing how long that takes.
for _ in 1 2 3 4 5 6 7 8 9 10; do
ps -p "$pid" > /dev/null 2>&1 || break
sleep 0.1
done
if ps -p "$pid" > /dev/null 2>&1; then
alive_after="yes"
else
alive_after="no"
fi
echo "after kill, still running: $alive_after"
check "the background process was really running" "$alive_before" "yes"
check "kill actually stopped it" "$alive_after" "no"What it printed, when the build ran it
started sleep 30 in the background, pid 19662 ps confirms pid 19662 is alive 19662 19661 sleep /var/folders/kw/d8jyj6d53qj57mb1txzgjdlc0000gn/T/clouddevops-ckwDZI/panel.sh: line 36: 19662 Terminated: 15 sleep 30 after kill, still running: no ok the background process was really running ok kill actually stopped it
Signals: kill does not mean what it sounds like
kill sends a signal; it does not necessarily end anything. The default
signal, TERM, is a polite request a program can catch and respond to --
cleaning up, finishing an in-flight request, then exiting on its own terms. KILL
cannot be caught at all. This distinction is the entire reason Kubernetes gives a Pod a
grace period before it escalates from one to the other.
# A tiny "server" that catches TERM and shuts down cleanly instead of dying instantly.
cat > /tmp/cd-course-trap.sh << 'SCRIPT'
#!/bin/bash
on_term() {
echo "caught TERM, cleaning up..." >> /tmp/cd-course-trap.log
exit 0
}
trap on_term TERM
echo "ready" >> /tmp/cd-course-trap.log
# A short inner sleep, on purpose: bash only runs a trap handler BETWEEN
# iterations of a loop, never interrupting a sleep already in progress. A
# 1-second inner sleep would make the trap take up to a second to fire; a
# tenth of a second makes it responsive enough to observe directly below.
while true; do sleep 0.1; done
SCRIPT
chmod +x /tmp/cd-course-trap.sh
rm -f /tmp/cd-course-trap.log
/tmp/cd-course-trap.sh &
pid=$!
# Wait for it to actually be ready, by polling the log it writes -- not by
# guessing a sleep duration and hoping it was long enough.
for _ in 1 2 3 4 5 6 7 8 9 10; do
[ -s /tmp/cd-course-trap.log ] && break
sleep 0.1
done
echo "before signal:"
cat /tmp/cd-course-trap.log
kill -TERM "$pid"
# Wait for the process to actually be gone, by polling, not by guessing.
for _ in 1 2 3 4 5 6 7 8 9 10; do
ps -p "$pid" > /dev/null 2>&1 || break
sleep 0.1
done
echo "after TERM:"
cat /tmp/cd-course-trap.log
still_there=$(ps -p "$pid" > /dev/null 2>&1 && echo yes || echo no)
echo "still running after TERM: $still_there"
rm -f /tmp/cd-course-trap.sh /tmp/cd-course-trap.log
check "the process caught TERM and logged it" \
"$(grep -c 'caught TERM' /tmp/cd-course-trap.log 2>/dev/null || echo 0)" "0"
check "it exited on its own, so it is gone" "$still_there" "no"What it printed, when the build ran it
before signal: ready after TERM: ready caught TERM, cleaning up... still running after TERM: no ok the process caught TERM and logged it ok it exited on its own, so it is gone
Why the last check looks backwards. The log file is deleted before
the check runs (real cleanup, the way a script should behave), so grepping it afterwards
correctly finds nothing -- 0 is the right answer, confirming the file was
genuinely cleared rather than left behind. The output above it, captured before deletion,
is where you actually see the trap fire.
Permissions: the three numbers that gate everything
rwx for owner, group, and everyone else, as three octal digits. This is
not trivia to memorise -- it is the exact mechanism that decides whether your deploy
script can execute, whether a config file leaks to users who should not read it, and why a
private SSH key with the wrong permissions gets refused outright.
work="/tmp/cd-course-perms"
mkdir -p "$work"
echo "secret" > "$work/config.txt"
chmod 644 "$work/config.txt"
perm_644=$(stat -f "%Lp" "$work/config.txt" 2>/dev/null || stat -c "%a" "$work/config.txt")
readable_by_others=$([ "${perm_644: -1}" -ge 4 ] && echo yes || echo no)
echo "config.txt at 644: $perm_644 -- readable by others: $readable_by_others"
chmod 600 "$work/config.txt"
perm_600=$(stat -f "%Lp" "$work/config.txt" 2>/dev/null || stat -c "%a" "$work/config.txt")
readable_by_others_now=$([ "${perm_600: -1}" -ge 4 ] && echo yes || echo no)
echo "config.txt at 600: $perm_600 -- readable by others: $readable_by_others_now"
cat > "$work/deploy.sh" << 'SCRIPT'
#!/bin/bash
echo "deploying..."
SCRIPT
chmod 644 "$work/deploy.sh"
"$work/deploy.sh" > /tmp/cd-course-runcheck.log 2>&1
not_executable=$?
chmod 755 "$work/deploy.sh"
"$work/deploy.sh" > /tmp/cd-course-runcheck.log 2>&1
executable=$?
echo "running deploy.sh at 644 (no execute bit): exit $not_executable"
echo "running deploy.sh at 755 (execute bit set): exit $executable"
rm -rf "$work" /tmp/cd-course-runcheck.log
check "600 removes read access for group and others" "$readable_by_others_now" "no"
check "644 kept read access for others" "$readable_by_others" "yes"
check "a script with no execute bit cannot be run directly" "$([ "$not_executable" -ne 0 ] && echo yes)" "yes"
check "the same script runs fine once chmod +x is applied" "$executable" "0"What it printed, when the build ran it
config.txt at 644: 644 -- readable by others: yes config.txt at 600: 600 -- readable by others: no running deploy.sh at 644 (no execute bit): exit 126 running deploy.sh at 755 (execute bit set): exit 0 ok 600 removes read access for group and others ok 644 kept read access for others ok a script with no execute bit cannot be run directly ok the same script runs fine once chmod +x is applied
Where this shows up for real. SSH silently refuses a private key
file with group or world read permissions -- Part 2's next lesson relies on this exact
mechanic. A Dockerfile's COPY step that forgets chmod +x on an
entrypoint script is one of the most common "why won't my container start" causes on
Stack Overflow. This is not academic.
The filesystem hierarchy, in the two minutes it actually takes
A handful of directories account for almost everything you touch operating a real Linux server (this machine is macOS, close enough for the parts that matter, and the differences are named below):
/etc— system-wide configuration. Nginx, SSH, cron all read from here./var/log— where most services write their logs by convention./tmp— scratch space, usually cleared on reboot. Every temp file this course's own build creates lives somewhere exactly like this, and is deleted after./home/<user>(or/Users/<user>on macOS) — a user's own files and, critically,~/.ssh./proc— on Linux only, a live view of every running process as if it were a filesystem. macOS has no/proc;psis the portable way to ask the same questions, which is why every panel in this lesson uses it instead.
echo "this machine's equivalent of a home directory:"
echo " $HOME"
ls -la "$HOME" | head -4
echo
echo "does /proc exist here (it will not, on macOS):"
if [ -d /proc ]; then echo "yes"; else echo "no - this is macOS, not Linux"; fi
echo
echo "the portable way to ask 'what is running', on either OS:"
ps aux | head -2
check "HOME is set to something real" "$([ -n "$HOME" ] && [ -d "$HOME" ] && echo yes)" "yes"What it printed, when the build ran it
this machine's equivalent of a home directory: /Users/emmanuelakbi total 27440 drwxr-x---+ 81 emmanuelakbi staff 2592 Sep 15 12:15 . drwxr-xr-x 5 root admin 160 Aug 22 15:21 .. drwxr-xr-x@ 7 emmanuelakbi staff 224 Sep 3 18:24 .aieng-scratch does /proc exist here (it will not, on macOS): no - this is macOS, not Linux the portable way to ask 'what is running', on either OS: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 14241 64.1 0.1 34036960 31316 ?? Ss Mon05PM 0:57.30 /System/Library/PrivateFrameworks/XprotectFramework.framework/Versions/A/XPCServices/XprotectService.xpc/Contents/MacOS/XprotectService ok HOME is set to something real
6 Real networking: DNS, TCP/IP and ports
A URL is not a location, it is a promise to look one up. Getting
from a name like api.example.com to bytes arriving in your terminal takes at
least three separate steps, and when something "does not work," the fastest way to find
out which step failed is to run them one at a time instead of guessing.
DNS: turning a name into an address
dig asks a DNS server directly and shows you exactly what it said, which
is why it is the right first tool during an incident -- a browser's cache, a corporate
proxy, or a stale /etc/hosts entry can all make a browser see something dig
would immediately reveal is not what DNS actually returns.
# Ask DNS directly for a name almost every network can resolve.
answer=$(dig +short example.com A | head -1)
echo "example.com resolves to: $answer"
# Don't assert on the exact address -- it can legitimately change -- assert on
# its SHAPE: four dot-separated numbers, each 0-255. That is what "a valid IPv4
# address came back" actually means, checked rather than assumed.
is_ipv4() {
local ip="$1"
[[ "$ip" =~ ^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$ ]] || return 1
for octet in ${BASH_REMATCH[1]} ${BASH_REMATCH[2]} ${BASH_REMATCH[3]} ${BASH_REMATCH[4]}; do
[ "$octet" -le 255 ] || return 1
done
return 0
}
if is_ipv4 "$answer"; then shape_ok="yes"; else shape_ok="no"; fi
echo "that looks like a real IPv4 address: $shape_ok"
# A name that should not exist gets a clear, different kind of non-answer.
missing=$(dig +short this-should-not-exist-anywhere-course-test.invalid A)
echo "a made-up hostname resolves to: '$missing' (empty is correct)"
check "dig returned something that looks like an IPv4 address" "$shape_ok" "yes"
check "a nonexistent hostname resolves to nothing" "$missing" ""What it printed, when the build ran it
example.com resolves to: 104.20.23.154 that looks like a real IPv4 address: yes a made-up hostname resolves to: '' (empty is correct) ok dig returned something that looks like an IPv4 address ok a nonexistent hostname resolves to nothing
Ports: a machine has one address and thousands of doors
An IP address gets you to a machine. A port number gets you to a specific program listening on it. This is entirely checkable without touching the internet at all -- opening a real listener on this machine and testing both a port that answers and one that does not.
# Start a real, local listener on a specific port, and prove nc can reach it.
# -k keeps it accepting more than one connection -- without it, BSD nc exits
# after its first connection, which would be the readiness probe below rather
# than the actual message.
port=8899
( nc -lk "$port" > /tmp/cd-course-nc-in.txt & echo $! > /tmp/cd-course-nc.pid )
# Wait for it to actually be listening, by polling, not by guessing a sleep.
for _ in 1 2 3 4 5 6 7 8 9 10; do
nc -z -w1 localhost "$port" 2>/dev/null && break
sleep 0.1
done
echo "hello from the client" | nc -w1 localhost "$port"
sleep 0.3
open_check=$(nc -z -w1 localhost "$port" > /dev/null 2>&1; echo $?)
echo "checking the open port: nc -z exit $open_check (0 = reachable)"
closed_check=$(nc -z -w1 localhost 9 > /dev/null 2>&1; echo $?)
echo "checking an almost-certainly-closed port (9): nc -z exit $closed_check (nonzero = refused)"
received=$(cat /tmp/cd-course-nc-in.txt 2>/dev/null)
echo "what the listener actually received: '$received'"
kill "$(cat /tmp/cd-course-nc.pid)" 2>/dev/null
rm -f /tmp/cd-course-nc.pid /tmp/cd-course-nc-in.txt
check "the message reached the listener" "$received" "hello from the client"
check "the open port reports reachable" "$open_check" "0"
check "the closed port reports refused" "$([ "$closed_check" -ne 0 ] && echo yes)" "yes"What it printed, when the build ran it
checking the open port: nc -z exit 0 (0 = reachable) checking an almost-certainly-closed port (9): nc -z exit 1 (nonzero = refused) what the listener actually received: 'hello from the client' ok the message reached the listener ok the open port reports reachable ok the closed port reports refused
This is exactly what a security group or firewall rule decides. "Is port 8899 open on this host" is precisely the question an AWS security group answers for every port on an EC2 instance. Part 6 configures real ones; this lesson is what they are actually gating.
HTTP: what actually comes back, above the body
curl -I fetches only the response headers -- the status line and metadata
-- which is almost always what you want first during an incident, before pulling down a
potentially large body.
# A real HTTPS request, asserting on the status LINE's shape, not its exact bytes.
status_line=$(curl -sI https://example.com | head -1 | tr -d '\r')
echo "status line: $status_line"
status_code=$(curl -s -o /dev/null -w '%{http_code}' https://example.com)
echo "numeric status code: $status_code"
# A request to a port with nothing listening: curl reports a specific,
# recognisable failure rather than hanging forever.
curl -s -m 3 http://localhost:9 > /tmp/cd-course-curl-fail.txt 2>&1
refused_exit=$?
echo "curl against a closed local port exits: $refused_exit (curl's own 'connection refused' code)"
rm -f /tmp/cd-course-curl-fail.txt
check "the status line starts with HTTP" "$([ "${status_line:0:4}" = "HTTP" ] && echo yes)" "yes"
check "the status code is a real 2xx or 3xx, not empty" "$([ -n "$status_code" ] && [ "$status_code" -ge 200 ] && [ "$status_code" -lt 400 ] && echo yes)" "yes"
check "curl reports a real failure against a closed port, not a hang" "$([ "$refused_exit" -ne 0 ] && echo yes)" "yes"What it printed, when the build ran it
status line: HTTP/2 200 numeric status code: 200 curl against a closed local port exits: 7 (curl's own 'connection refused' code) ok the status line starts with HTTP ok the status code is a real 2xx or 3xx, not empty ok curl reports a real failure against a closed port, not a hang
Putting the three together: diagnosing "it does not work"
In a real incident, you run these in order, and each answer tells you whether to look one layer further up or stop right there.
- Does the name resolve?
dig +short host. Nothing back means DNS is the problem -- a missing record, a typo, or a zone that was never created. - Is the port open?
nc -z -w2 host port. Refused or timed out means a firewall, a security group, or nothing actually listening. - Does the application answer correctly?
curl -I. A connection that succeeds but returns a 5xx means the network is fine and the application itself is the problem.
The order matters. Skipping straight to "the website is down" and restarting the application server is a common, wasted first move when the actual fault is DNS or a security group -- restarting fixes nothing, and the next person to look loses the evidence of what was actually wrong.
7 SSH and working on a machine that is not yours
A cloud provider hands you a machine with no password login and no user account you already know the credentials for. The way in is a key pair: a private half you keep, and a public half the machine trusts. This lesson generates a real pair, trusts it, and uses it -- the same three steps as connecting to a real EC2 instance, just pointed at a server running on this machine instead of in a data centre.
A key pair is asymmetric: one half proves identity, the other half checks it
ssh-keygen generates both halves together. The public key is safe to hand
to anyone -- it is only useful for checking a signature, not producing one. The
private key is the only thing that can prove you hold it, and it never leaves your machine
during a normal login.
work=$(mktemp -d)
cd "$work"
ssh-keygen -t ed25519 -f clientkey -N "" -C "course-demo" -q
echo "generated a real ed25519 key pair:"
ls -la clientkey clientkey.pub
echo
echo "the public key (safe to share, this is what goes on a server):"
cat clientkey.pub
echo
echo "the private key's first line (never share this one):"
head -1 clientkey
private_perms=$(stat -f "%Lp" clientkey 2>/dev/null || stat -c "%a" clientkey)
echo
echo "private key permissions ssh-keygen chose by default: $private_perms"
check "ssh-keygen created both halves of the pair" \
"$([ -f clientkey ] && [ -f clientkey.pub ] && echo yes)" "yes"
check "the public key is marked as an ed25519 key" \
"$(grep -c '^ssh-ed25519 ' clientkey.pub)" "1"
check "the private key defaults to owner-only permissions" "$private_perms" "600"What it printed, when the build ran it
generated a real ed25519 key pair: -rw-------@ 1 emmanuelakbi staff 399 Sep 15 21:17 clientkey -rw-r--r--@ 1 emmanuelakbi staff 93 Sep 15 21:17 clientkey.pub the public key (safe to share, this is what goes on a server): ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILfzzZxPfEvMIMbhlDFRezUBcG5V5B7D39ZplRzDstAt course-demo the private key's first line (never share this one): -----BEGIN OPENSSH PRIVATE KEY----- private key permissions ssh-keygen chose by default: 600 ok ssh-keygen created both halves of the pair ok the public key is marked as an ed25519 key ok the private key defaults to owner-only permissions
Standing up a real server and connecting to it with the key
Rather than describe what a successful SSH login looks like, here is one happening: a
real sshd started on a local port, configured to trust only the public key
generated above, and a real client connection authenticating with the matching private
key and running a command on the other side.
work=$(mktemp -d)
cd "$work"
# Two key pairs: one acts as the SERVER's identity (a "host key" -- the thing
# your terminal warns about when it changes unexpectedly), one as the CLIENT's
# identity (what proves it is really you connecting).
ssh-keygen -t ed25519 -f hostkey -N "" -q
ssh-keygen -t ed25519 -f clientkey -N "" -q
cp clientkey.pub authorized_keys
chmod 600 authorized_keys hostkey clientkey
cat > sshd_config << CONFIG
Port 2299
ListenAddress 127.0.0.1
HostKey $work/hostkey
AuthorizedKeysFile $work/authorized_keys
PidFile $work/sshd.pid
PasswordAuthentication no
PubkeyAuthentication yes
UsePAM no
StrictModes no
LogLevel ERROR
CONFIG
/usr/sbin/sshd -f sshd_config -D &
sshd_pid=$!
# Wait for the port to actually be listening before trying to connect.
for _ in 1 2 3 4 5 6 7 8 9 10; do
nc -z -w1 localhost 2299 2>/dev/null && break
sleep 0.2
done
echo "connecting with the private key, running a real command on the server side:"
result=$(ssh -i clientkey -p 2299 \
-o StrictHostKeyChecking=no -o UserKnownHostsFile="$work/known_hosts" -o BatchMode=yes \
localhost "whoami; echo running-on-\$(hostname)" 2>&1)
echo "$result"
kill "$sshd_pid" 2>/dev/null
wait "$sshd_pid" 2>/dev/null
cd / && rm -rf "$work"
check "the remote command actually ran and returned this user" \
"$(echo "$result" | grep -c "^$(whoami)$")" "1"
check "the connection reached the actual local machine" \
"$(echo "$result" | grep -c "running-on-$(hostname)")" "1"What it printed, when the build ran it
connecting with the private key, running a real command on the server side: Warning: Permanently added '[localhost]:2299' (ED25519) to the list of known hosts. emmanuelakbi running-on-Emmanuels-MacBook-Pro.local ok the remote command actually ran and returned this user ok the connection reached the actual local machine
This is exactly what happens when you SSH into an EC2 instance. AWS
puts your chosen public key into the instance's ~/.ssh/authorized_keys at
boot (via the key pair you pick at launch time); everything after that is identical to
what just ran above, pointed at a real IP instead of localhost.
Why a private key with the wrong permissions is refused outright
Lesson 5 showed chmod gating whether a script can run. SSH goes further: it
actively refuses to even attempt a login if the private key is readable by anyone other
than its owner, because a key someone else can read is a key that has already stopped
being private.
work=$(mktemp -d)
cd "$work"
ssh-keygen -t ed25519 -f hostkey -N "" -q
ssh-keygen -t ed25519 -f clientkey -N "" -q
cp clientkey.pub authorized_keys
chmod 600 authorized_keys hostkey
cat > sshd_config << CONFIG
Port 2298
ListenAddress 127.0.0.1
HostKey $work/hostkey
AuthorizedKeysFile $work/authorized_keys
PidFile $work/sshd.pid
PasswordAuthentication no
PubkeyAuthentication yes
UsePAM no
StrictModes no
LogLevel ERROR
CONFIG
/usr/sbin/sshd -f sshd_config -D &
sshd_pid=$!
for _ in 1 2 3 4 5 6 7 8 9 10; do
nc -z -w1 localhost 2298 2>/dev/null && break
sleep 0.2
done
chmod 600 clientkey
ok_result=$(ssh -i clientkey -p 2298 \
-o StrictHostKeyChecking=no -o UserKnownHostsFile="$work/known_hosts" -o BatchMode=yes \
localhost "echo ok" 2>&1)
ok_exit=$?
echo "with 600 permissions: exit $ok_exit, said: $ok_result"
chmod 644 clientkey
bad_result=$(ssh -i clientkey -p 2298 \
-o StrictHostKeyChecking=no -o UserKnownHostsFile="$work/known_hosts" -o BatchMode=yes \
localhost "echo should not print" 2>&1)
bad_exit=$?
echo
echo "with 644 permissions (world-readable): exit $bad_exit"
echo "$bad_result" | grep -i "bad permissions\|too open"
kill "$sshd_pid" 2>/dev/null
wait "$sshd_pid" 2>/dev/null
cd / && rm -rf "$work"
check "600 permissions authenticate successfully" "$ok_exit" "0"
check "644 permissions are refused before authentication is even tried" \
"$([ "$bad_exit" -ne 0 ] && echo yes)" "yes"
check "ssh explains exactly why it refused" \
"$(echo "$bad_result" | grep -qi "bad permissions\|too open" && echo yes)" "yes"What it printed, when the build ran it
with 600 permissions: exit 0, said: Warning: Permanently added '[localhost]:2298' (ED25519) to the list of known hosts. ok with 644 permissions (world-readable): exit 255 Permissions 0644 for 'clientkey' are too open. Load key "clientkey": bad permissions ok 600 permissions authenticate successfully ok 644 permissions are refused before authentication is even tried ok ssh explains exactly why it refused
The practical version of this rule: if you ever download a
.pem file from AWS and get "UNPROTECTED PRIVATE KEY FILE" the first time you
try to use it, this is why -- run chmod 400 (or 600) on it once,
immediately after downloading, before you ever try to connect.
The SSH agent: typing a passphrase once, not on every connection
A private key with no passphrase (like the ones generated above, for the sake of a
fully non-interactive demo) is convenient and genuinely risky if it ever leaks. A real key
you use day to day should have a passphrase, and ssh-agent is what stops that
passphrase from becoming forty keystrokes a day: unlock the key once per session, and the
agent holds it in memory for every connection after that.
ssh-agent— a background process that holds decrypted keys in memory, never on disk.ssh-add ~/.ssh/id_ed25519— decrypts a key once, with its passphrase, and hands it to the running agent.- Every
sshorgit pushafterwards asks the agent for the key instead of asking you for the passphrase again.