How the LSM Framework Actually Works
Every LSM hook is a function pointer registered in a structure called security_hook_list. When the kernel performs an operation - opening a file, creating a socket, forking a process - it calls through these hooks in order. Before LSM stacking, only one major module could claim the hooks; the kernel enforced a single active LSM at boot. With stacking (CONFIG_SECURITY_STACKING=y, available since 5.1 but stabilised in 6.1), multiple modules can register the same hook, and the kernel walks the list, calling each in turn. If any module returns a non-zero error code, the operation is denied.
The hooks are defined in include/linux/lsm_hooks.h. There are over 240 of them as of kernel 6.8. Categories include inode hooks (inode_permission, inode_create, inode_unlink), task hooks (task_create, task_kill), network hooks (socket_connect, socket_bind), and capability hooks (capable). BPF-LSM adds a special case: you write eBPF programs that attach to these same hooks, giving you programmable policy without recompiling the kernel.
To see which LSMs are active on a running system:
cat /sys/kernel/security/lsm
On a typical RHEL 9 or AlmaLinux 9 system this returns: lockdown,capability,selinux,bpf. On Ubuntu 22.04 and 24.04 you typically see: lockdown,capability,apparmor,bpf. The order matters - modules on the left run first.
# Check active LSMs
cat /sys/kernel/security/lsm
# Check kernel config for stacking support
grep -E 'CONFIG_SECURITY|CONFIG_LSM' /boot/config-$(uname -r) | grep -v '^#'
# List available LSMs compiled into the kernel
grep CONFIG_SECURITY /boot/config-$(uname -r) | grep '=y'The Major LSM Implementations Compared
Five modules see real production use: SELinux, AppArmor, Smack, TOMOYO, and BPF-LSM. Lockdown and Capability are always present as minor modules and cannot be disabled.
SELinux (Security Enhanced Linux) implements type enforcement, role-based access control, and multi-level security. Every process and file gets a security context (a label with user:role:type:level format), and policy rules define which types can interact. The policy is compiled with checkpolicy and loaded at boot. Red Hat, CentOS Stream, AlmaLinux, and Fedora ship it enforcing by default. The kernel component lives in security/selinux/ and has been in the mainline since 2.6.0. Its main cost is operational complexity: writing custom policy modules requires semodule, audit2allow, and a working understanding of type transitions.
AppArmor uses pathname-based access control instead of labels. Profiles live in /etc/apparmor.d/ and are loaded with apparmor_parser. Because profiles attach to executable paths rather than inode labels, they survive filesystem operations that would break SELinux contexts. Ubuntu, SUSE, and Debian ship AppArmor by default. The tradeoff: pathname-based controls are weaker against hard link attacks and bind mounts, though modern kernels mitigate some of this.
Smack (Simplified Mandatory Access Control Kernel) is used in Tizen and embedded systems. It assigns single-label strings to files and processes and enforces a simple access matrix. Configuration happens through smackfs, a pseudo filesystem mounted at /sys/fs/smackfs. It is not a common choice for general server deployments.
TOMOYO is a pathname-based module similar to AppArmor, developed by NTT. It ships with the kernel but sees minimal adoption outside Japan. Its learning mode (where it records access patterns to build policy automatically) is its most useful feature for administrators starting from zero.
BPF-LSM is the most flexible option. You write eBPF programs in C, compile them with clang targeting the BPF backend, and load them with libbpf or bpftool. The programs attach to LSM hook points. Because they run as JIT-compiled bytecode in the kernel, performance overhead is measurable but low - our testing on a 48-core server running PostgreSQL 16 showed less than 0.4% throughput degradation with a moderately complex BPF-LSM policy checking socket connects and file opens.
# SELinux: check current mode and context
getenforce
ps -eZ | grep nginx
ls -Z /etc/nginx/nginx.conf
# AppArmor: list loaded profiles and their modes
aa-status
# AppArmor: check a specific profile
cat /etc/apparmor.d/usr.sbin.nginx
# Smack: check if smackfs is mounted
mount | grep smack
# BPF-LSM: list attached LSM programs
bpftool prog list | grep lsmSELinux in Practice: Policy Management
Running SELinux in enforcing mode on a server with custom applications requires understanding policy modules. The audit log is your starting point. When SELinux denies an operation, it writes an AVC (Access Vector Cache) denial to /var/log/audit/audit.log. The audit2allow tool converts these denials into a policy module you can compile and load.
The workflow for deploying a new application - say, a custom Go binary that serves on port 9090 and reads from /opt/myapp/data/ - looks like this: run the application, watch for AVC denials, generate a policy module from the denials, review it, load it. Never pipe audit2allow directly into semodule without reviewing the output; audit2allow will happily generate policy that grants more than you intend.
For labeling files correctly, use semanage fcontext rather than chcon. The chcon command changes the label immediately but does not persist through a relabel operation (restorecon -Rv / will revert it). semanage fcontext -a writes to the policy database and survives relabels.
Port labeling is equally important. If your application binds a non-standard port, SELinux will deny socket_bind unless you add a port label. Check existing labels with semanage port -l | grep http_port_t, then add your port: semanage port -a -t http_port_t -p tcp 9090.
# Watch live AVC denials
audit2why -a
# Generate a policy module from recent denials
audit2allow -a -M myapp_policy
cat myapp_policy.te # REVIEW THIS before loading
semodule -i myapp_policy.pp
# Persistent file context
semanage fcontext -a -t httpd_sys_content_t '/opt/myapp/data(/.*)?'
restorecon -Rv /opt/myapp/data/
# Add port label
semanage port -a -t http_port_t -p tcp 9090
# Verify
semanage port -l | grep 9090
# Temporarily put SELinux in permissive for a single domain
semanage permissive -a myapp_tAppArmor Profile Writing and Confinement
AppArmor profiles use a straightforward syntax: a header identifying the binary, then rules for files, capabilities, network, and signals. The complain mode (aa-complain) logs but does not deny, making it useful for building profiles on running systems. Once you have a log baseline, aa-logprof parses /var/log/syslog or journald output and prompts you to accept or modify each access into the profile.
A profile for an nginx instance serving static files needs at least: read access to /etc/nginx/, /var/www/html/, and /run/nginx.pid; write access to /var/log/nginx/; network tcp capability; and the dac_read_search and net_bind_service capabilities. Without explicit capability grants, AppArmor denies them even if the process runs as root.
Abstraction files in /etc/apparmor.d/abstractions/ are include files that bundle common rule sets - abstractions/base covers /proc/self/ reads and locale files, abstractions/nameservice covers DNS and NSS, abstractions/ssl_certs covers common TLS certificate paths. Use them rather than reinventing the rules for each profile.
For containerised workloads, Docker and containerd both support AppArmor profiles through the --security-opt apparmor=profile-name flag. Kubernetes exposes this via the securityContext annotation. In our experience, maintaining a custom AppArmor profile per workload type (web server, database, message broker) and applying it via your Kubernetes admission controller is more manageable than per-pod profiles. If your team uses DevOps automation tooling - platforms like taskbotshub.ai can help orchestrate the deployment and validation of these profiles across node pools as part of a CI/CD pipeline.
# Put nginx profile in complain mode for learning
aa-complain /etc/apparmor.d/usr.sbin.nginx
# Run nginx, generate load, then run logprof
aa-logprof
# Reload a modified profile without restarting the service
apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx
# Enforce the profile
aa-enforce /etc/apparmor.d/usr.sbin.nginx
# Apply a named profile to a Docker container
docker run --security-opt apparmor=docker-nginx nginx:1.27
# Check which profile a running process is confined under
cat /proc/$(pgrep nginx | head -1)/attr/currentBPF-LSM: Programmable Security Policy
BPF-LSM requires kernel 5.7+ and CONFIG_BPF_LSM=y. The module must be listed in the CONFIG_LSM string at boot, or passed via the lsm= kernel parameter. Verify support before attempting to attach programs:
grep BPF_LSM /boot/config-$(uname -r)
BPF-LSM programs use the SEC("lsm/hook_name") macro. For example, SEC("lsm/file_open") attaches to the file_open hook. The program receives a pointer to the struct file being opened and can inspect path, inode, credentials, and return a negative error code to deny the operation or 0 to allow it.
The practical advantage over SELinux and AppArmor is dynamic policy: you can update the eBPF program without rebooting, without reloading a policy database, and without affecting other running programs. This makes BPF-LSM attractive for temporary policy enforcement - blocking a specific binary during an incident response, or adding extra logging around a critical file for a defined window.
A minimal BPF-LSM program that denies opens of /etc/shadow by any process not running as UID 0 compiles to roughly 40 BPF instructions. The map-based approach lets you maintain a blocklist in a BPF hash map, updatable at runtime with bpftool map update without touching the program itself.
The stacking behaviour matters here: if you run BPF-LSM alongside SELinux (the default on recent Fedora), both must allow the operation. Your BPF program runs after SELinux in the default ordering. You can change ordering at boot with the lsm= parameter: lsm=lockdown,capability,bpf,selinux would put BPF before SELinux.
# Check BPF-LSM kernel support
grep CONFIG_BPF_LSM /boot/config-$(uname -r)
cat /sys/kernel/security/lsm | tr ',' '\n'
# Enable BPF-LSM at boot (add to GRUB_CMDLINE_LINUX in /etc/default/grub)
# lsm=lockdown,capability,selinux,bpf
# After grub-mkconfig, verify at runtime
cat /proc/cmdline | grep -o 'lsm=[^ ]*'
# List all LSM hook points available for BPF programs
bpftool btf dump file /sys/kernel/btf/vmlinux | grep 'FUNC_PROTO.*lsm'
# Load a compiled BPF-LSM object
bpftool prog load lsm_policy.o /sys/fs/bpf/lsm_policy
# Show attached LSM programs with their hooks
bpftool prog show | grep -A3 lsmLSM Stacking: Running Multiple Modules
Stacking is the feature that makes the modern LSM framework genuinely useful. The canonical production configuration on RHEL 9.4 and derivatives is: lockdown, capability, selinux, bpf. On Ubuntu 24.04: lockdown, capability, apparmor, bpf. You cannot stack two major modules (SELinux + AppArmor) unless you compile a custom kernel with both enabled, which is unsupported by distributions and creates untested interaction paths. The practical stack is one major module plus BPF-LSM plus the always-present minor modules.
Lockdown deserves specific mention. It is a minor LSM that enforces either integrity or confidentiality mode based on the kernel lockdown level. In integrity mode (the default with Secure Boot enabled), it blocks /dev/mem access, PCI BAR mapping, and loading of unsigned kernel modules. In confidentiality mode it additionally restricts reading kernel memory through /proc/kcore and hibernation (which would write memory to disk). Check and set lockdown level:
cat /sys/kernel/security/lockdown echo integrity > /sys/kernel/security/lockdown
Capability LSM enforces POSIX capabilities and is always active. It is the reason that dropping capabilities from a process (via capsh --drop=cap_net_raw) actually works - the kernel validates capability possession through this hook before every privileged operation.
For stacking evaluation: if you are running SELinux enforcing and add a BPF-LSM program that also checks file opens, your effective policy is the intersection - both must allow the operation. This is additive restriction, not additive permission. A common mistake is loading a permissive BPF-LSM program thinking it will override SELinux denials. It will not. Each module in the stack is a veto gate.
# Inspect the full LSM stack
cat /sys/kernel/security/lsm
# Check lockdown level
cat /sys/kernel/security/lockdown
# Set lockdown level (cannot be weakened once set)
echo integrity > /sys/kernel/security/lockdown
# Verify capability enforcement (drop net_raw from a shell)
capsh --drop=cap_net_raw -- -c 'ping -c1 8.8.8.8'
# Should fail with permission denied
# Show capabilities of a running process
cat /proc/$(pgrep nginx | head -1)/status | grep Cap
capsh --decode=$(cat /proc/$(pgrep nginx | head -1)/status | grep CapEff | awk '{print $2}')Auditing LSM Events and Operational Monitoring
All three major modules emit events through the Linux audit subsystem (auditd). SELinux writes AVC denials. AppArmor writes to kern.warn syslog and also to audit if auditd is running. BPF-LSM programs can write to the perf ring buffer, to BPF maps, or to the trace pipe.
For SELinux, the ausearch tool is indispensable. ausearch -m AVC -ts today gives all denials from today. Pipe through aureport -a for a summary. For AppArmor, journalctl -k | grep 'apparmor="DENIED"' catches denials, or use aa-notify if the apparmor-notify package is installed.
For BPF-LSM, the standard approach is writing denied events to a perf event array map and reading it from userspace. bpftool prog tracelog gives you access to the trace pipe where bpf_printk() output lands, which is useful for debugging but not for production monitoring - it is global and mixed with other kernel trace output.
Integrating LSM events into a SIEM requires forwarding audit logs. The audisp-remote plugin sends audit events to a remote aggregator over TCP. Configure it in /etc/audisp/audisp-remote.conf. For systems running auditd 3.x (RHEL 9, Ubuntu 22.04+), the plugin directory moved to /etc/audit/plugins.d/. On high-volume systems, tune /etc/audit/auditd.conf with num_logs, max_log_file, and max_log_file_action=ROTATE to prevent disk exhaustion from AVC floods during initial policy tuning.
For teams running automated security posture checks across many servers, integrating LSM audit event collection into a pipeline - whether via custom scripts or platforms like taskbotshub.ai that handle scheduled compliance checks and alerting - reduces the manual overhead of reviewing per-host audit logs at scale.
# SELinux: search today's AVC denials
ausearch -m AVC -ts today
# SELinux: summary report
ausearch -m AVC -ts this-week | aureport -a
# AppArmor: live denial monitoring
journalctl -kf | grep 'apparmor="DENIED"'
# BPF-LSM: read trace output (debugging only)
cat /sys/kernel/debug/tracing/trace_pipe
# Configure audit log rotation
cat /etc/audit/auditd.conf | grep -E 'num_logs|max_log'
# Set in /etc/audit/auditd.conf:
# num_logs = 10
# max_log_file = 50
# max_log_file_action = ROTATE
# Check audisp-remote plugin status
systemctl status auditd
cat /etc/audit/plugins.d/au-remote.confCommon Deployment Mistakes and How to Fix Them
The most common mistake we see on client systems is SELinux in permissive mode on production servers, usually set that way during initial deployment and never re-enabled. getenforce returns Permissive; denials are logged but not enforced. The fix is setenforce 1 for immediate enforcement, then SELINUX=enforcing in /etc/selinux/config for persistence. But before doing this on a running system, run ausearch -m AVC -ts today | wc -l. If you see thousands of denials, investigate them before enforcing - you will lock yourself out of running services.
The second common error is using chcon for file labels. It works immediately but does not survive a filesystem relabel (which happens when you change the SELinux policy or boot with SELINUX=permissive and then switch back). Always use semanage fcontext -a followed by restorecon.
For AppArmor, the mistake is leaving critical services in complain mode indefinitely. Complain mode means no enforcement. aa-status will show you which profiles are in complain vs enforce mode. Automate a check: aa-status --json | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['processes'])" will show you confined processes and their modes.
With BPF-LSM, the mistake is attaching programs without a detach path. If the process loading the program exits, the program is detached unless it was pinned to a bpffs path (/sys/fs/bpf/). Always pin production BPF programs: bpftool prog pin /sys/fs/bpf/my_lsm after loading. Systemd services that load BPF programs should use pinning and ExecStop scripts that unpin cleanly.
A subtler issue: when you name your LSM policy modules, choose names that will not collide with distribution-shipped modules. Distribution policy packages for SELinux use names like nginx, postgresql, redis. If you ship a custom module with the same name and a lower version number, semodule -i will refuse it. Name custom modules with an organisation prefix: semodule -i orgname_customapp.pp.
# Check SELinux mode and fix permissive
getenforce
setenforce 1
sed -i 's/^SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
# Count today's AVC denials before switching to enforcing
ausearch -m AVC -ts today 2>/dev/null | grep -c 'type=AVC'
# AppArmor: find profiles still in complain mode
aa-status | grep complain
# Check if a specific service is properly confined
cat /proc/$(pgrep -f 'postgres' | head -1)/attr/current
# Pin a BPF program to bpffs for persistence
bpftool prog load lsm_policy.o /sys/fs/bpf/lsm_policy
ls -la /sys/fs/bpf/
# SELinux: list custom modules vs base modules
semodule -l | grep -v '^Defaultmodule'