How Kernel Modules Work
A kernel module is a .ko file (kernel object) compiled against the headers of a specific kernel version. When you run modprobe or insmod, the kernel's module loader copies the object into kernel memory, resolves symbol references against the running kernel's symbol table (available at /proc/kallsyms), and calls the module's init function. On unload, the exit function is called and the memory is freed.
Modules are versioned with a VERMAGIC string that must match the running kernel exactly. If you try to load a module built against 6.6.20 on a 6.6.21 kernel, the load will fail unless CONFIG_MODVERSIONS was enabled during both builds and the CRCs match.
Dependencies between modules are encoded in modules.dep, generated by depmod. This is why modprobe handles dependency chains automatically while insmod does not. Always use modprobe in scripts and automation; insmod is for testing a single .ko file by explicit path.
# Show VERMAGIC and dependency info for a specific module
modinfo dm_crypt
# Rebuild the dependency database after installing a new module
depmod -a
# depmod for a specific kernel version (e.g., after cross-compiling)
depmod -a 6.6.21-mykernelListing Loaded Modules with lsmod
lsmod reads /proc/modules and formats the output as three columns: module name, size in bytes, and use count followed by the names of modules that depend on it.
The use count matters operationally. A module with a non-zero use count cannot be unloaded. The dependent module names tell you exactly what holds the reference. On a typical server you will see dm_crypt held by nothing (use count 0) if no encrypted volumes are open, and held by a non-zero count with no listed dependents if a dm-crypt device is active - the reference is held by the device mapper core, not another module.
For grep-friendly output and scripting, /proc/modules gives raw data. For detailed per-module state including load address and parameter values, use /sys/module/.
# Basic listing
lsmod
# Find all loaded modules related to networking
lsmod | grep -E '^(nf_|xt_|ip_|ipv6|bridge)'
# Check if a specific module is loaded (exit code 0 = found)
lsmod | grep -qw dm_crypt && echo loaded || echo not loaded
# Raw /proc/modules - module name, size, use count, deps, state, load address
cat /proc/modules | sort -k3 -rn | head -20
# List parameters of a loaded module via sysfs
ls /sys/module/dm_crypt/parameters/
cat /sys/module/dm_crypt/parameters/max_read_sizeInspecting Modules with modinfo
modinfo extracts metadata from a .ko file without loading it. The most useful fields are filename, description, author, license, version, srcversion, depends, and parm. The parm fields tell you every parameter the module accepts, along with the type.
License matters beyond attribution. A module with a non-GPL license (e.g., proprietary) cannot call kernel symbols exported with EXPORT_SYMBOL_GPL. If you're debugging why an out-of-tree driver fails to link, check the license field first.
The srcversion field is a hash of the source files. On systems where the kernel was built reproducibly, this lets you verify that the installed module matches the source tree you have on record.
# Full metadata for a module by name (searches /lib/modules/$(uname -r)/)
modinfo e1000e
# Show only parameters
modinfo -p e1000e
# Inspect a specific .ko file by path (useful for out-of-tree modules)
modinfo /lib/modules/6.6.21/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko
# Extract a single field
modinfo -F license nvidia
modinfo -F depends btrfs
# List all modules that depend on nothing (standalone, no deps)
for mod in $(lsmod | awk 'NR>1 {print $1}'); do
deps=$(modinfo -F depends $mod 2>/dev/null)
[ -z "$deps" ] && echo "$mod"
doneLoading Modules with modprobe
modprobe is the standard tool for loading modules in production. It reads /etc/modprobe.d/*.conf for aliases, options, blacklists, and install/remove hooks, then resolves the full dependency chain before loading.
Passing parameters at load time is straightforward. Parameters set via modprobe override defaults but do not persist across reboots unless you write them to /etc/modprobe.d/. To make a parameter permanent, create a conf file rather than adding it to a generic file - one file per module makes auditing clean and diffs readable.
The --dry-run flag is invaluable before touching production. It shows exactly what modprobe would do, including which dependencies would be loaded, without actually doing it.
# Load a module
modprobe dm_crypt
# Load with a parameter
modprobe dm_crypt max_read_size=524288
# Dry run - shows what would happen without loading
modprobe --dry-run --verbose dm_crypt
# Persistent parameter configuration
cat > /etc/modprobe.d/dm-crypt.conf <<'EOF'
options dm_crypt max_read_size=524288
EOF
# Verify the conf file is picked up
modprobe --showconfig | grep dm_crypt
# Load a module by alias (e.g., filesystem type)
modprobe -v fs-xfs
# Force load (bypass VERMAGIC check) - dangerous, for recovery only
modprobe --force-vermagic dm_cryptUnloading Modules with modprobe -r and rmmod
Unloading a module requires its use count to be zero and all modules that depend on it to be unloaded first. modprobe -r handles the dependency chain in reverse order. rmmod removes a single module and will fail if anything depends on it.
A common failure mode is trying to unload a module that has an open file descriptor or an active device node referencing it. In that case, the kernel returns EBUSY. You can check /sys/module/
The --wait flag (available since util-linux 2.36 and kernel 5.3) tells the kernel to mark the module for removal and wait until all references drop, rather than failing immediately. Use this with caution on production systems - it can block indefinitely if a process keeps the reference open.
# Unload a module and its unused dependencies
modprobe -r dm_crypt
# Remove a single module (no dependency handling)
rmmod dm_crypt
# Check what holds a reference (holders directory)
ls /sys/module/dm_crypt/holders/
# Find processes with open file descriptors into a module's devices
lsof | grep crypto
# Force removal (dangerous - can corrupt kernel state)
rmmod --force dm_crypt
# Wait-based removal (kernel marks module dying, waits for refcount to drop)
rmmod --wait dm_cryptBlacklisting and Aliases
Blacklisting prevents a module from loading automatically, but does not prevent manual loading with modprobe or insmod. This distinction catches people out when they blacklist nouveau but then find it still loads because another module or udev rule triggers it via an alias.
To completely prevent a module from loading under any circumstances, use the install directive in modprobe.d to replace the load action with /bin/false or /bin/true. The install directive runs instead of the normal module load sequence.
Aliases map generic names to specific module names. The kernel generates aliases for hardware (PCI IDs, USB IDs) during the build and stores them in modules.alias. You can add custom aliases in /etc/modprobe.d/ to override or supplement these.
# Blacklist a module (prevents automatic loading only)
cat > /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
blacklist nouveau
EOF
# Hard block: replace load with /bin/false (prevents all loading)
cat > /etc/modprobe.d/block-nouveau.conf <<'EOF'
install nouveau /bin/false
EOF
# Check what alias resolves to
modprobe --showconfig | grep 'alias.*nouveau'
# Look up hardware PCI alias manually
cat /lib/modules/$(uname -r)/modules.alias | grep '10de:' | head -5
# Add a custom alias
cat >> /etc/modprobe.d/custom-aliases.conf <<'EOF'
alias mynet e1000e
EOFPersistent Module Loading at Boot
There are two mechanisms for loading modules at boot: /etc/modules-load.d/ (processed by systemd-modules-load.service) and the legacy /etc/modules file on older Debian-based systems.
systemd-modules-load reads all *.conf files under /etc/modules-load.d/ and /usr/lib/modules-load.d/, loading each named module using modprobe. The service runs early in boot, before most userspace is available, so it's appropriate for modules that storage or network subsystems depend on.
For modules that should load based on hardware detection, rely on udev rules rather than static lists. Static lists are for modules that have no automatic trigger: specific tuning modules like tcp_bbr, crypto algorithm modules like crc32c_intel, or custom drivers for hardware that doesn't expose PCI/USB IDs properly.
After modifying modules-load.d, you can test without rebooting by running systemctl restart systemd-modules-load and checking the journal.
# Create a modules-load.d entry for tcp_bbr (BBR congestion control)
cat > /etc/modules-load.d/bbr.conf <<'EOF'
tcp_bbr
EOF
# Apply immediately without reboot
systemctl restart systemd-modules-load
journalctl -u systemd-modules-load --no-pager
# Verify it loaded
lsmod | grep tcp_bbr
# Enable BBR after the module is loaded
cat >> /etc/sysctl.d/99-bbr.conf <<'EOF'
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr
EOF
sysctl --system
# Confirm
sysctl net.ipv4.tcp_congestion_controlBuilding and Installing Out-of-Tree Modules with DKMS
DKMS (Dynamic Kernel Module Support) solves the most painful operational problem with out-of-tree modules: they break every time the kernel updates because VERMAGIC changes. DKMS stores the module source and rebuilds it automatically for every new kernel version.
The DKMS configuration file dkms.conf lives in /usr/src/
On systems managed at scale, we recommend using DKMS with a pre-built binary cache rather than rebuilding on every host. DKMS supports this via the --binaries-only flag for dkms install, which produces a tarball that can be distributed without headers on target hosts. This is standard practice on fleets where kernel headers are not installed on production nodes.
For teams building automated deployment pipelines, tools like taskbotshub.ai can handle DKMS build triggers as part of CI workflows, automatically rebuilding and distributing module packages when a new kernel version appears in your apt or dnf repository.
# Install DKMS
apt install dkms # Debian/Ubuntu
dnf install dkms # RHEL/Fedora
# Minimal dkms.conf example
cat > /usr/src/mydriver-1.0/dkms.conf <<'EOF'
PACKAGE_NAME="mydriver"
PACKAGE_VERSION="1.0"
BUILT_MODULE_NAME[0]="mydriver"
DEST_MODULE_LOCATION[0]="/kernel/drivers/misc"
AUTOINSTALL="yes"
EOF
# Register, build, and install the module
dkms add -m mydriver -v 1.0
dkms build -m mydriver -v 1.0
dkms install -m mydriver -v 1.0
# Check DKMS status across all kernels
dkms status
# Build a binary-only tarball (no source) for distribution
dkms install --binaries-only -m mydriver -v 1.0 -k $(uname -r)
tar czf mydriver-1.0-$(uname -r)-$(uname -m).tar.gz \
/var/lib/dkms/mydriver/1.0/$(uname -r)/
# Remove a DKMS module
dkms remove -m mydriver -v 1.0 --allDebugging Module Load Failures
Module load failures fall into four categories: VERMAGIC mismatch, unresolved symbols, missing firmware, and parameter errors. The error message in dmesg is always the first place to look, and it is usually specific enough to diagnose without guessing.
VERMAGIC mismatches produce: "disagrees about version of symbol module_layout". Unresolved symbols produce: "Unknown symbol
For production kernel debugging, /sys/kernel/debug/dynamic_debug/control lets you enable verbose pr_debug output from specific modules at runtime without recompiling. This is far less disruptive than adding printk calls and rebuilding.
When a module loads but behaves incorrectly, ftrace and perf are your tools. For module-specific tracing, the trace_printk infrastructure and kernel probes (kprobes) let you instrument arbitrary kernel functions without rebuilding.
# Check dmesg immediately after a failed modprobe
modprobe mydriver; dmesg | tail -20
# Check for unresolved symbols in a .ko before loading
nm mydriver.ko | grep ' U ' | awk '{print $2}'
# Cross-reference against the running kernel's symbol table
grep -Ff <(nm mydriver.ko | grep ' U ' | awk '{print $2}') /proc/kallsyms
# Enable dynamic debug for a module (verbose pr_debug output)
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
# Enable dynamic debug for a specific function
echo 'file drivers/misc/mydriver.c func mydriver_probe +p' \
> /sys/kernel/debug/dynamic_debug/control
# Trace all calls to a kernel function using ftrace
cd /sys/kernel/debug/tracing
echo mydriver_probe > set_ftrace_filter
echo function > current_tracer
echo 1 > tracing_on
# ... trigger the event ...
cat trace | head -50
echo 0 > tracing_on
# Find missing firmware
strace -e openat modprobe mydriver 2>&1 | grep firmwareSecurity Considerations: Module Signing and Lockdown
Since kernel 3.7, the kernel can enforce module signature verification. When CONFIG_MODULE_SIG_FORCE is enabled, only modules signed with a key trusted by the kernel's built-in keyring will load. This is the default on RHEL 9, Ubuntu 22.04 with Secure Boot enabled, and most modern enterprise distributions.
Checking whether signing is enforced: read /sys/module/module/parameters/sig_enforce. A value of Y means unsigned modules will be rejected with EKEYREJECTED.
To sign an out-of-tree module, you need the private key and certificate that were used during the kernel build. On Red Hat systems, the keys are enrolled in the Machine Owner Key (MOK) database and the signing tool is scripts/sign-file from the kernel source. On Ubuntu, mokutil handles MOK enrollment for Secure Boot.
Kernel lockdown mode (available since 5.4) goes further than module signing. In lockdown=integrity mode, the kernel also blocks /dev/mem access, kexec of unsigned images, and hibernation. In lockdown=confidentiality mode, it additionally blocks reading of kernel memory from userspace. Check the current lockdown level at /sys/kernel/security/lockdown.
# Check if module signature enforcement is active
cat /sys/module/module/parameters/sig_enforce
# Check current lockdown mode
cat /sys/kernel/security/lockdown
# Sign a module with the kernel's build key (requires kernel-devel)
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 \
/path/to/signing_key.pem \
/path/to/signing_cert.pem \
/lib/modules/$(uname -r)/extra/mydriver.ko
# Verify a module's signature
modinfo mydriver | grep signer
# Generate a new signing key pair (for custom kernels)
openssl req -new -x509 -newkey rsa:4096 \
-keyout signing_key.pem -out signing_cert.pem \
-days 36500 -subj '/CN=MyDriver Module Signing/' -nodes
# Enroll a certificate into MOK database (Ubuntu/Fedora with Secure Boot)
sudo mokutil --import signing_cert.pem
# Reboot, enter MOK manager, confirm enrollmentAutomating Module Management at Scale
Managing kernel modules across a fleet of hundreds or thousands of nodes requires treating module configuration as code. All /etc/modprobe.d/ files and /etc/modules-load.d/ entries should be in version control and deployed via configuration management - Ansible, Puppet, Salt, or similar.
Ansible's community.general.modprobe module handles idempotent loading, but it does not manage modprobe.d configuration files or DKMS registration. For a complete solution, you need the modprobe module for runtime state, the template module for /etc/modprobe.d/ files, and a custom role for DKMS lifecycle management.
For teams building this into CI/CD pipelines - especially when custom kernel builds or out-of-tree drivers are involved - platforms like taskbotshub.ai can automate the DKMS rebuild and signing workflow as a triggered job, dispatching rebuilds to a signing server and pushing signed .ko packages to an internal repository whenever a new kernel appears.
For inventory and audit purposes, collecting lsmod and modinfo output as structured data is straightforward with a simple script pushed via your configuration management tool. Store the output in your CMDB or log aggregation system - unexpected module changes on production hosts are a useful signal for both debugging and security monitoring.
# Ansible task: ensure a module is loaded with specific parameters
- name: Load dm_crypt with tuned read size
community.general.modprobe:
name: dm_crypt
state: present
params: 'max_read_size=524288'
# Ansible task: deploy modprobe.d config
- name: Configure dm_crypt parameters persistently
ansible.builtin.template:
src: dm-crypt.conf.j2
dest: /etc/modprobe.d/dm-crypt.conf
owner: root
group: root
mode: '0644'
# Collect loaded module inventory as JSON for CMDB
python3 -c "
import subprocess, json
result = subprocess.run(['lsmod'], capture_output=True, text=True)
modules = []
for line in result.stdout.strip().split('\n')[1:]:
parts = line.split()
modules.append({'name': parts[0], 'size': int(parts[1]), 'used_by': parts[3].split(',') if len(parts) > 3 else []})
print(json.dumps(modules, indent=2))
" > /var/log/modules-inventory-$(date +%Y%m%d).json