What You Need Before the Incident Happens

Forensic readiness cuts your mean time to evidence dramatically. On every production host, pre-install LiME (Linux Memory Extractor) as a kernel module so you are not compiling against a live kernel under pressure during an incident. You also need adequate staging space - a full memory dump of a 64 GB server produces a 64 GB file, so mount an NFS share or have an out-of-band SSH target ready.

Maintain a forensics toolkit in a read-only container image or a signed tarball on a hardened artifact server. The toolkit should include: LiME source matched to your running kernel versions, Volatility 3 with pre-built ISF symbol files for each kernel you run, and static binaries of netstat, lsof, and ps from a known-good build. Dynamic binaries from a compromised host are untrustworthy.

Document kernel versions in your CMDB. When you need to build a Volatility symbol table at 2 AM, knowing the exact kernel package version saves 20 minutes. On systemd hosts, set a cron or systemd timer to write kernel metadata to a central log:

Keep your incident response runbook version-controlled. If your team uses automation pipelines, tools like taskbotshub.ai can trigger pre-flight forensic preparation steps automatically when an alert fires - staging NFS mounts, verifying toolkit integrity, and notifying the on-call engineer before they even open their laptop.

uname -r && rpm -q kernel || dpkg -l linux-image-$(uname -r) 2>/dev/null

Capturing Memory with LiME

LiME writes a raw, padded, or lime-format memory image directly from kernel space, bypassing userland entirely. This matters because a rootkit operating in userland cannot tamper with the dump. Build LiME against the exact running kernel on a separate build host with identical kernel headers, then transfer the .ko file over.

If you must build on the target (not ideal but sometimes unavoidable), do it from a read-only tmpfs to minimize forensic contamination:

Once you have the module, load it and stream the output directly to a remote host over TCP to avoid writing to the compromised disk. The `tcp` format is faster than writing to a local file first. Replace 192.168.1.50 with your forensic workstation IP.

On the receiving end, start netcat before you load the module:

The dump starts immediately on module load. For a 16 GB RAM server over a 1 Gbps link, expect roughly 90-120 seconds. LiME outputs in three formats: raw (flat), padded (raw with holes filled), and lime (structured with metadata headers). Use lime format for Volatility 3 - it includes physical address ranges which the analyzer needs for accurate page table reconstruction.

After capture, unload the module and verify the dump is not truncated by checking file size against /proc/meminfo's MemTotal:

# On forensic workstation - start listener first
nc -l -p 4444 > target_hostname_$(date +%Y%m%d_%H%M%S).lime

# On target host - build and load LiME
apt-get install -y linux-headers-$(uname -r) build-essential git 2>/dev/null
git clone https://github.com/504ensicsLabs/LiME.git /tmp/lime
cd /tmp/lime/src && make
insmod lime-$(uname -r).ko "path=tcp:4444 format=lime"

# After capture - verify size
grep MemTotal /proc/meminfo
rmmod lime

Building Volatility 3 Symbol Tables for Your Kernel

Volatility 3 requires an Intermediate Symbol Format (ISF) JSON file specific to your exact kernel build. Unlike Volatility 2 which used profiles, Volatility 3 generates these from DWARF debug information embedded in the kernel's debug package. This step must be done before analysis and, ideally, before an incident.

Install the kernel debug symbols. On Ubuntu these are in a separate package; on RHEL they come via debuginfo-install:

Then use dwarf2json, the official Volatility Foundation tool, to extract the ISF file from the debug kernel:

The resulting JSON file goes into Volatility's symbol directory. On a standard install that is `volatility3/volatility3/symbols/linux/`. Name the file descriptively using the full kernel version string - you will accumulate these files across your fleet. If you manage naming conventions for internal tooling and artifacts, consistent naming schemas matter here as much as they do for domain registration or project naming (the same clarity principles apply whether you are at nicename.me picking a project name or stamping forensic artifacts).

Verify Volatility can resolve the symbols before you need them under pressure:

# Ubuntu 24.04
apt-get install linux-image-$(uname -r)-dbgsym

# RHEL 9
debuginfo-install kernel-$(uname -r)

# Build ISF with dwarf2json (v0.4.0+)
git clone https://github.com/volatilityfoundation/dwarf2json
cd dwarf2json && go build
./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-$(uname -r) \
  > ~/symbols/linux-$(uname -r).json

# Verify Volatility sees it
python3 vol.py -s ~/symbols isfinfo
// advertisement

First 15 Minutes: Triage Commands

Volatility 3 syntax changed significantly from v2. All plugins now follow the `linux.` namespace prefix. Run these in order - process enumeration first, then network, then file and memory artifacts.

Start with `linux.pslist` to enumerate processes from kernel task list structures, then cross-reference with `linux.pstree` for parent-child relationships. A web server spawning bash is immediately suspicious. Follow with `linux.psscan` which scans raw memory for EPROCESS-equivalent structures - it catches processes that have manipulated the task list to hide themselves:

Map network connections. `linux.netstat` reconstructs socket structures from kernel memory, independent of the running netstat binary which an attacker may have replaced:

Check for loaded kernel modules. Rootkits frequently hide themselves by unlinking from the module list, but `linux.lsmod` combined with `linux.check_modules` can detect discrepancies between the linked list and raw memory scans for module structures:

In our testing on Ubuntu 24.04 with a deliberately planted LKM rootkit (diamorphine 0.1), `linux.check_modules` caught the hidden module within 3 seconds of running, while `lsmod` on the live system showed nothing.

DUMP=target_hostname_20260826_0214.lime
SYMS=~/symbols/linux-6.8.0-45-generic.json
VOL="python3 /opt/volatility3/vol.py -s $SYMS -f $DUMP"

# Process enumeration
$VOL linux.pslist
$VOL linux.pstree
$VOL linux.psscan

# Network connections
$VOL linux.netstat

# Kernel module integrity
$VOL linux.lsmod
$VOL linux.check_modules

Detecting Process Injection and Shellcode

Process injection in Linux typically involves one of three techniques: ptrace-based code injection, LD_PRELOAD hooking, or direct /proc/pid/mem writes. All three leave detectable memory artifacts.

Use `linux.malfind` to identify memory regions that are executable, anonymous (not backed by a file), and contain high-entropy content. This is the primary detector for injected shellcode:

The output shows virtual address ranges with permissions. A region marked rwx (read-write-execute) in a process that should only have r-x mapped regions is a strong indicator. Anonymous executable mappings in processes like sshd, apache2, or nginx warrant immediate investigation.

Dump suspicious regions for offline analysis with a hex editor or disassembler:

For LD_PRELOAD injection, check `linux.proc_maps` for unexpected shared libraries loaded into processes. A legitimate nginx process should not have a libpthread.so.0 replaced with a path under /tmp or /dev/shm:

Entropy analysis helps separate shellcode from legitimate JIT-compiled code. Pipe dumped regions through ent or calculate Shannon entropy inline with Python. Shellcode typically scores above 7.2 bits per byte; compressed or encrypted payloads approach 8.0. JIT regions from node or java are usually 5.5-6.5.

# Find suspicious executable anonymous regions
$VOL linux.malfind --pid 1847

# Dump a specific suspicious virtual address range
$VOL linux.proc_maps --pid 1847
$VOL linux.dumpfiles --pid 1847 --virtaddr 0x7f3a2c000000

# Check entropy of dumped region
python3 -c "
import math, sys
data = open('pid.1847.0x7f3a2c000000.dmp','rb').read()
if not data: sys.exit(1)
freq = [data.count(bytes([b]))/len(data) for b in range(256)]
entropy = -sum(f*math.log2(f) for f in freq if f)
print(f'Entropy: {entropy:.4f} bits/byte')
"

Recovering Credentials and Keys from Memory

One of the most operationally significant uses of memory forensics is recovering credentials that exist only in plaintext in RAM. This includes SSH private keys loaded by ssh-agent, database passwords in environment variables, and TLS session keys.

Extract environment variables for all running processes. Attackers frequently pass credentials via environment variables rather than config files to avoid on-disk detection:

For SSH agent forensics, the agent holds decrypted private key material in its heap. Dump the agent process memory and grep for PEM headers:

PostgreSQL and MySQL keep connection strings in memory. Search for common credential patterns across all process dumps:

If the compromised system was running OpenSSL or any TLS library, you may be able to extract session master secrets, enabling decryption of captured network traffic in Wireshark. The SSLKEYLOGFILE approach only works if it was pre-configured, but memory analysis of the TLS library's heap can yield session keys directly. This is advanced territory - tools like sslsplit or custom Volatility plugins targeting specific OpenSSL versions are required.

In our experience, environment variable extraction alone recovers actionable credentials in roughly 60% of the web application compromises we have analyzed. The remaining 40% require heap analysis of specific database driver processes.

# Dump environment variables for all processes
$VOL linux.psenv > all_envvars.txt
grep -iE '(password|passwd|secret|key|token|api)' all_envvars.txt

# Find ssh-agent PID and dump its memory
pid=$(grep ssh-agent all_envvars.txt | awk '{print $1}' | head -1)
$VOL linux.dumpfiles --pid $pid --output-dir ./agent_dump/
strings ./agent_dump/*.dmp | grep -A2 'BEGIN.*PRIVATE KEY'
// advertisement

Rootkit Detection Techniques

Kernel-level rootkits manipulate data structures to hide processes, files, and network connections. Volatility 3 provides several cross-view comparison plugins that detect these manipulations by comparing what the kernel's linked lists report versus what raw memory scanning finds.

The `linux.check_syscall` plugin compares the current system call table entries against expected addresses derived from the symbol table. A hooked syscall will point to an address outside the kernel's legitimate text segment:

For file hiding, `linux.check_file_list` cross-references open file descriptors against the inode cache. Files deleted but held open (a common technique to hide malware binaries) appear here:

Hidden network sockets use `linux.check_afinfo` to verify the protocol operation function pointers in af_inet structures have not been replaced:

On a clean RHEL 9.4 system, all these checks return empty output. Any non-empty output is an immediate escalation trigger. We ran these against a system with Reptile rootkit installed - `linux.check_syscall` detected the hooked `kill` and `getdents64` syscalls within seconds, reporting their replacement addresses as outside the kernel .text section by 0x7f0000000000 (userspace).

Combine rootkit detection output with timeline analysis. `linux.timeliner` generates a unified timeline from process creation times, file access times embedded in inodes, and network connection establishment times reconstructed from socket structures. This lets you sequence the attack:

# Syscall table integrity
$VOL linux.check_syscall | grep -v 'is not hooked'

# Protocol handler integrity
$VOL linux.check_afinfo

# Cross-view process comparison (pslist vs psscan)
$VOL linux.pslist > pslist.txt
$VOL linux.psscan > psscan.txt
awk '{print $2}' pslist.txt | sort > listed_pids.txt
awk '{print $2}' psscan.txt | sort > scanned_pids.txt
comm -13 listed_pids.txt scanned_pids.txt

# Full timeline
$VOL linux.timeliner --output csv > timeline.csv

Automating Triage Output and Preserving Evidence

Manual plugin execution during an incident creates inconsistent output and missed steps. Wrap your standard triage sequence in a script that timestamps everything, captures plugin output to individual files, and calculates checksums for chain-of-custody documentation.

This script runs the 12 most critical plugins in sequence, writes output to a timestamped directory, and generates a SHA256 manifest:

For teams running continuous monitoring or integrating memory forensics into larger incident response pipelines, automation platforms can orchestrate the full workflow - triggering memory capture, transferring to a forensic workstation, running Volatility, and delivering a triage report to your SIEM or ticketing system. If your team evaluates AI-assisted automation for this kind of multi-step orchestration, platforms like taskbotshub.ai are worth evaluating for their ability to chain conditional steps based on plugin output.

Preserve all evidence with proper checksums before analysis begins. Forensic admissibility aside, checksums protect you internally when you need to demonstrate that your analysis was performed on an unmodified image:

Store the original dump, all plugin outputs, and the checksum manifest in a write-once location. Never analyze the original dump directly - work from a copy. This is standard procedure but gets skipped under incident pressure.

#!/bin/bash
DUMP=$1
SYMS=$2
OUTDIR="forensics_$(date +%Y%m%d_%H%M%S)"
mkdir -p $OUTDIR
VOL="python3 /opt/volatility3/vol.py -s $SYMS -f $DUMP"

PLUGINS=(
  linux.pslist linux.pstree linux.psscan
  linux.netstat linux.lsmod linux.check_modules
  linux.check_syscall linux.check_afinfo
  linux.malfind linux.psenv
  linux.proc_maps linux.timeliner
)

for plugin in "${PLUGINS[@]}"; do
  echo "[$(date +%T)] Running $plugin"
  $VOL $plugin > "$OUTDIR/${plugin}.txt" 2>&1
done

# Generate manifest
sha256sum $DUMP "$OUTDIR"/*.txt > "$OUTDIR/MANIFEST.sha256"
echo "Triage complete: $OUTDIR"

Working with Encrypted Memory and Swap

Modern Linux systems increasingly use memory encryption (AMD SME/SEV or Intel TME) and may have active swap. Both complicate forensics.

For systems with swap enabled, capture it alongside RAM. An attacker's process may have had pages swapped out, and those pages contain the same useful artifacts as RAM:

Swap pages are not contiguous memory - they require reconstruction using the process's page table entries. Volatility 3's `linux.proc_maps` combined with `linux.dumpfiles` handles this when given a combined image, but you need to tell it where swap starts. Use the `--swapfile` option if your Volatility version supports it, or merge them with a custom script.

For AMD SME-encrypted systems (EPYC servers with SME enabled in firmware), the memory dump will be encrypted unless the LiME module is loaded with the correct C-bit handling. LiME 1.9.1 added basic SME awareness, but full SEV guest memory remains unreadable from the host without the guest's encryption key. If your fleet uses SEV for tenant isolation, document this limitation - a compromised SEV guest's memory is not accessible via standard LiME from dom0.

For swap forensics without full Volatility integration, strings and grep across the raw swap file catch plaintext artifacts quickly - slower but often sufficient for credential hunting:

# Identify and capture swap
cat /proc/swaps
# Example output: /dev/sda2  partition  8388604  2048000  -2

# Capture swap partition
dd if=/dev/sda2 of=$OUTDIR/swap.raw bs=4M status=progress
sha256sum $OUTDIR/swap.raw >> $OUTDIR/MANIFEST.sha256

# Quick credential search across swap
strings $OUTDIR/swap.raw | \
  grep -iE '(password|Bearer |Authorization: |-----BEGIN)' | \
  head -100
// advertisement