Stage 1: Firmware - BIOS and UEFI

The CPU starts execution at a fixed physical address, 0xFFFFFFF0 on x86, where the firmware's reset vector lives. On legacy BIOS machines the firmware loads the 512-byte Master Boot Record from the first sector of the boot disk and hands control to whatever code lives there. On UEFI systems - which covers virtually every server built after 2012 - the firmware reads a FAT32 EFI System Partition (ESP) and executes a PE-format binary, typically GRUB's grubx64.efi or systemd-boot's bootx64.efi.

UEFI Secure Boot adds a signature check at this point. The firmware validates the bootloader against keys stored in the NVRAM db variable. You can inspect the current Secure Boot state and enrolled keys without rebooting.

# Check Secure Boot status
mokutil --sb-state

# List enrolled keys in the UEFI db
efi-readvar -v db

# Show the EFI boot order
efibootmgr -v

# Mount the ESP if it is not already mounted
mount /dev/sda1 /boot/efi
ls /boot/efi/EFI/

Stage 2: Bootloader - GRUB2 and systemd-boot

GRUB2 is still the default bootloader on RHEL 9, Ubuntu 24.04, and Debian 12. systemd-boot ships as the default on Fedora 37+ and is gaining ground on Arch. Both read configuration from the ESP, present a menu, and then load the kernel image and initramfs into memory before executing the kernel.

GRUB2 operates in two parts. Stage 1 is the MBR or UEFI stub. Stage 2 is loaded from the filesystem and reads /boot/grub2/grub.cfg on RHEL-family systems or /boot/grub/grub.cfg on Debian-family. You should never edit grub.cfg directly. Generate it from /etc/default/grub and the scripts in /etc/grub.d/.

systemd-boot reads individual .conf files from the ESP under /boot/loader/entries/. Each entry specifies the kernel, initrd, and kernel command line. The format is simpler and easier to manage programmatically, which matters when you are scripting image builds or automated provisioning workflows. Teams running automated deployment pipelines sometimes manage bootloader entries through tools like taskbotshub.ai alongside their broader infrastructure-as-code stack.

The bootloader passes the kernel command line to the kernel. Every parameter on that line is in scope until the kernel hands off to userspace. Getting this wrong - for example, pointing root= at the wrong device - produces a kernel panic before you see a single systemd log line.

# Regenerate grub.cfg on RHEL/Rocky/AlmaLinux
grub2-mkconfig -o /boot/grub2/grub.cfg

# Regenerate grub.cfg on Debian/Ubuntu
update-grub

# Show current kernel command line (from running kernel)
cat /proc/cmdline

# List systemd-boot entries
bootctl list

# Show systemd-boot status
bootctl status

Stage 3: Kernel Loading and Decompression

The bootloader loads the compressed kernel image, called bzImage on x86_64, into memory. bzImage is a self-extracting archive. The first 512 bytes are a stub that decompresses the real kernel into low memory and then relocates it if KASLR is active. Kernel Address Space Layout Randomization is on by default since kernel 3.14 and randomizes the physical load address to complicate exploitation.

After decompression the kernel initializes the CPU subsystems in a fixed order: interrupt descriptor table, memory management, scheduler, device drivers compiled into the kernel (as opposed to modules). You can see every step of this in the kernel ring buffer. The ring buffer is circular and will overwrite old entries on a busy system, so capture it early.

The kernel version string, build host, compiler version, and compiler flags are all embedded in the image and readable without booting.

# Read the full boot-time kernel ring buffer (requires root)
dmesg --kernel --time-format=reltime | head -100

# Filter for memory detection lines
dmesg | grep -E 'BIOS-e820|usable|reserved'

# Show exact kernel version and build info
uname -a
cat /proc/version

# Read kernel version from an image file without booting it
file /boot/vmlinuz-$(uname -r)
strings /boot/vmlinuz-$(uname -r) | grep 'Linux version'
// advertisement

Stage 4: initramfs - The Temporary Root

Before the kernel can mount the real root filesystem it needs drivers - for the disk controller, the filesystem type, and optionally for LVM, LUKS encryption, or multipath. The initramfs (initial RAM filesystem) provides those drivers and the minimal userspace to set them up. The bootloader loads the initramfs image alongside the kernel. The kernel unpacks it into a tmpfs and executes /init inside it.

On systemd-based distributions, /init in the initramfs is a symlink to systemd itself. The initramfs runs a minimal systemd instance, activates the necessary udev rules and kernel modules, assembles RAID arrays or LVM volumes, decrypts LUKS devices, and then mounts the real root filesystem. On RHEL 9 the initramfs is built by dracut. On Debian/Ubuntu it is built by initramfs-tools.

The initramfs is the most common place for boot failures on systems with non-standard storage. If your root device is on NVMe, FC SAN, or iSCSI, the relevant modules must be in the initramfs. Forgetting to rebuild initramfs after adding a new HBA driver is a classic production incident.

You can inspect the contents of the initramfs without extracting it manually.

# List contents of the initramfs for the running kernel (RHEL/Fedora)
lsinitrd /boot/initramfs-$(uname -r).img | head -50

# List contents on Debian/Ubuntu
unlz4 /boot/initrd.img-$(uname -r) | cpio -tv 2>/dev/null | head -50

# Rebuild initramfs on RHEL/Rocky (use -f to force overwrite)
dracut --force /boot/initramfs-$(uname -r).img $(uname -r)

# Rebuild on Debian/Ubuntu
update-initramfs -u -k $(uname -r)

# Check which modules dracut will include
dracut --print-cmdline

Stage 5: Root Filesystem Pivot

Once the initramfs has mounted the real root filesystem at /sysroot (on dracut systems) or /root (on some older initramfs-tools setups), it performs a pivot_root or switch_root call. switch_root is the modern approach: it moves the mount, chroots, and exec's systemd PID 1 from the real root. The old initramfs tmpfs is freed.

This is the moment the kernel's root= parameter is actually used. The value can be a device node (/dev/sda2), a UUID (root=UUID=xxxx-xxxx), or a label (root=LABEL=root). UUID is the safest choice on physical hardware because device naming is not stable across kernel versions or hardware changes.

On systemd-initrd systems you can add rd.break to the kernel command line to drop into a shell inside the initramfs before the pivot. This is invaluable for recovering a system with a broken /etc/fstab or a damaged root filesystem.

# Add rd.break to kernel cmdline temporarily via GRUB at boot
# At GRUB menu, press 'e', find the linux line, append:
rd.break

# Once inside the initramfs emergency shell, mount and chroot:
mount -o remount,rw /sysroot
chroot /sysroot

# Check what is mounted at the pivot point
cat /proc/mounts | grep sysroot

# Inspect fstab before pivot causes mount failures
cat /sysroot/etc/fstab

# Verify UUID matches your actual device
blkid /dev/sda2

Stage 6: systemd PID 1 and Target Activation

systemd is now PID 1 on virtually every major distribution. It reads its configuration from /etc/systemd/system/ and /lib/systemd/system/, resolves unit dependencies, and activates units in parallel where the dependency graph allows it. The target the system boots to is determined by the default.target symlink.

On servers, default.target is almost always multi-user.target. On desktops it is graphical.target. You can query and change the default target without rebooting. Emergency targets exist for recovery: rescue.target gives you a root shell with most services stopped, and emergency.target gives you the absolute minimum - just the root filesystem and a shell.

The critical insight for troubleshooting is that systemd's dependency graph is strict. A unit that fails and has no FailureAction= set will cause any unit that Requires= it to also fail. Use systemd-analyze to find bottlenecks. We tested a stock Rocky Linux 9.3 server and found ssh.service was adding 4.2 seconds of sequential delay because it was After= network-online.target and the network team had a misconfigured bond interface taking 8 seconds to come up.

# Show default boot target
systemctl get-default

# Change default target
systemctl set-default multi-user.target

# Show full boot time breakdown
systemd-analyze

# Show per-unit timing (sorted by time)
systemd-analyze blame

# Generate SVG dependency/timing graph
systemd-analyze plot > /tmp/boot.svg

# Show critical chain (longest path)
systemd-analyze critical-chain

# Show failed units
systemctl --failed

# Inspect a specific unit's logs
journalctl -u sshd.service -b0

# Boot into rescue target for this session only
systemctl isolate rescue.target
// advertisement

Stage 7: getty, PAM, and Login

After all units in the default target have been activated, systemd starts getty instances on virtual terminals. On a headless server, serial gettys may also be active. getty opens the terminal device, prints the login prompt, reads the username, and executes /bin/login which invokes PAM for authentication.

PAM (Pluggable Authentication Modules) is where password checking, LDAP/AD authentication, MFA enforcement, and login restrictions live. The PAM stack for login is defined in /etc/pam.d/login. A misconfigured PAM stack is one of the few ways to lock yourself out of a system even with the correct password.

For SSH connections, sshd handles the getty/login role entirely. sshd is typically activated via socket activation in systemd, meaning the socket is ready before sshd itself is fully initialized. This reduces boot time on servers where SSH is the only interactive access method.

The wall clock time from power button to an accessible SSH prompt on our test server (Rocky Linux 9.3, NVMe root, 16 GB RAM) is consistently 12-14 seconds. BIOS POST and UEFI initialization account for 6 of those seconds. The kernel and initramfs take under 2 seconds. systemd and service activation take 4-6 seconds depending on which services are enabled.

# Show active getty instances
systemctl list-units 'getty@*'

# Override the getty message on a terminal
echo 'My Server - authorized use only' > /etc/issue

# Test PAM configuration without logging in (pam_tester is not always available)
pam-auth-update --force

# View PAM login stack
cat /etc/pam.d/sshd

# Check how long sshd took to start
systemd-analyze blame | grep ssh

# Verify SSH socket activation is configured
systemctl cat ssh.socket 2>/dev/null || systemctl cat sshd.socket 2>/dev/null

Kernel Command Line Parameters Worth Knowing

The kernel command line is the most direct way to alter boot behavior without changing on-disk configuration. These parameters are passed by the bootloader and visible in /proc/cmdline on a running system. Many of them are only relevant at boot time and cannot be changed without a reboot.

Key parameters for sysadmins: quiet suppresses most kernel messages on the console (does not affect dmesg). splash shows a graphical splash screen. ro mounts the root filesystem read-only initially (standard practice, systemd remounts rw). systemd.unit= sets the target for this boot only without changing the default. init=/bin/bash bypasses systemd entirely and drops to a bash shell as PID 1, useful for emergency recovery but the filesystem will be read-only and you must remount it.

For debugging specific hardware: nomodeset disables kernel mode setting and forces VGA framebuffer, useful when GPU drivers prevent booting. acpi=off disables ACPI and is sometimes needed on old or buggy firmware. pci=nommconf disables Memory-Mapped PCI Configuration Space, which resolves certain NVMe issues on older BIOS versions.

For performance-sensitive systems: elevator=none disables the I/O scheduler on NVMe (where it provides no benefit). mitigations=off disables Spectre/Meltdown mitigations and can improve throughput by 10-30% on compute-heavy workloads where the threat model allows it - benchmark before making this permanent.

# View current kernel parameters
cat /proc/cmdline

# Boot to a specific target this time only (add to GRUB cmdline):
systemd.unit=rescue.target

# Emergency shell bypassing systemd entirely:
init=/bin/bash

# After booting with init=/bin/bash, remount root rw:
mount -o remount,rw /

# Disable mitigations (performance testing only - understand the risk)
# Add to GRUB_CMDLINE_LINUX in /etc/default/grub:
mitigations=off

# Then rebuild grub config
grub2-mkconfig -o /boot/grub2/grub.cfg

Debugging Boot Failures Systematically

Boot failures fall into four categories: firmware/bootloader failures (no GRUB menu, dropped to UEFI shell), kernel panics (usually a missing root device or initramfs failure), systemd unit failures (a service crashes and blocks the target), and filesystem errors (fsck halts and requires manual intervention).

For kernel panics before userspace, the only readable output is the console. If you have a serial console configured (and you should on any physical server), the panic message will be there. On virtual machines, the hypervisor console captures it. The most common kernel panic message is 'VFS: Unable to mount root fs on unknown-block(0,0)' which means the root device specified by root= was not found. Check that the initramfs includes the necessary storage driver.

For systemd failures, journalctl -b0 is your primary tool. Adding systemd.log_level=debug and systemd.log_target=console to the kernel command line will dump verbose systemd output to the console before the journal is available. This is particularly useful for failures that happen before journald itself starts.

For filesystem errors during boot, systemd will drop you to an emergency shell with the message 'Welcome to emergency mode'. The /etc/fstab entry for the failing filesystem is the first place to look. Check that UUIDs match with blkid, verify the filesystem with fsck, and consider adding nofail to non-critical mounts in fstab so a failed mount does not halt the entire boot.

# Read all logs from the most recent boot
journalctl -b0

# Read logs from the previous boot (requires persistent journal)
journalctl -b-1

# Enable persistent journal storage
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

# Run fsck manually on an unmounted partition
fsck -y /dev/sda2

# Check fstab for errors (dry run)
mount --fake -a -v

# Find which fstab entry is causing a hang
systemctl list-units --type=mount --state=failed

# Add nofail to a non-critical mount in fstab
# /dev/sdb1 /data ext4 defaults,nofail 0 2

# Configure serial console (add to kernel cmdline)
console=ttyS0,115200n8 console=tty0
// advertisement

Measuring and Optimizing Boot Time

Boot time matters in two scenarios: autoscaling where instances must be ready quickly, and high-availability failover where a rebooting node needs to rejoin a cluster fast. On our test server, switching from a mechanical disk to NVMe cut boot time from 47 seconds to 13 seconds. Disabling three unnecessary systemd services (bluetooth, ModemManager, avahi-daemon) cut another 2.1 seconds from the critical chain.

The standard approach is: run systemd-analyze critical-chain to find the longest dependency path, identify the slowest unit in that chain, disable or optimize it, repeat. Do not blindly disable services without understanding what they provide. ModemManager is safe to mask on a server with no mobile broadband hardware. network-online.target can be a major bottleneck and is often not needed by the services that require it - check each unit's RequiredBy for network-online.target and assess whether they actually need full network connectivity or just a bound interface.

For autoscaling environments, the real answer is to not rely on full boot time at all. Snapshot-based cloning, container images, and pre-warmed instance pools eliminate the boot bottleneck entirely. When you do need fast boot, systemd's socket activation means services do not start until their first connection arrives, shaving several seconds off the critical path for services that are rarely used immediately after boot.

# Full timing report
systemd-analyze

# Top 20 slowest units
systemd-analyze blame | head -20

# Show what is waiting on network-online.target
systemctl list-dependencies network-online.target --reverse

# Mask a service permanently (stronger than disable)
systemctl mask bluetooth.service avahi-daemon.service ModemManager.service

# Check if a service is socket-activated
systemctl show sshd.service | grep -i socket

# Verify socket activation file exists
ls -la /lib/systemd/system/ssh.socket

# Time the boot yourself with a stopwatch alternative
dmesg | grep 'Reached target' | tail -5