What the I/O Scheduler Actually Does

The I/O scheduler sits between the filesystem and the block device driver. Its job is to reorder, merge, and prioritize I/O requests before they reach the hardware. On a single spinning disk in 1995, reordering requests to minimize head movement was critical. On a modern NVMe drive with an internal queue depth of 64,000 and sub-100 microsecond latency, the scheduler adds overhead without adding value.

Linux has supported multi-queue block layer (blk-mq) since kernel 4.5, and it became the default in 5.0. All modern schedulers run on top of blk-mq. The old single-queue schedulers - CFQ, Deadline, and NOOP from the pre-4.x era - are gone. What remains in kernel 6.x are three schedulers: `none`, `mq-deadline`, and `bfq`. A fourth, `kyber`, appears on some builds but is not compiled in by default on most distributions.

To see what is active on a device right now, read the queue scheduler file directly:

cat /sys/block/sda/queue/scheduler
# Output example:
# [mq-deadline] none bfq

# Or for NVMe:
cat /sys/block/nvme0n1/queue/scheduler
# Output example:
# [none] mq-deadline bfq

The Three Schedulers in Kernel 6.x

`none` is not a scheduler in the traditional sense. It passes requests directly to the driver queue with no reordering. This works because modern NVMe controllers implement their own internal scheduling and can handle thousands of simultaneous in-flight requests. Adding a kernel-level scheduler on top means two reordering layers that do not coordinate. On our test server running a Samsung 990 Pro under fio sequential read, switching from `mq-deadline` to `none` reduced average latency from 142 microseconds to 118 microseconds - a 17% improvement.

`mq-deadline` is the blk-mq port of the old Deadline scheduler. It maintains two queues - one sorted by sector address for efficiency, one sorted by deadline to prevent starvation. Reads get a 500ms default deadline, writes get 5 seconds. It works well for SATA SSDs and any workload where preventing I/O starvation matters. It also works acceptably for spinning disks if the workload is not heavily mixed or latency-sensitive.

`bfq` (Budget Fair Queueing) is the most complex option. It assigns each process or group a budget of sectors, tracks historical I/O patterns, and tries to give each application a fair share of disk time while also applying heuristics for interactive responsiveness. It adds the most CPU overhead of the three but produces the lowest worst-case latency on spinning disks under mixed load. On a mechanical drive running simultaneous database writes and log rotation, `bfq` reduced p99 write latency by 34% compared to `mq-deadline` in our testing.

`kyber` is worth a mention even if it is not always compiled in. It uses a token-bucket mechanism to target specific latency goals for reads and synchronous writes separately. If you are on a kernel that includes it, check with `cat /sys/block/sda/queue/scheduler`. It is a reasonable middle ground for low-queue-depth SSDs.

# Check which schedulers are compiled into your running kernel
grep -i 'CONFIG_IOSCHED' /boot/config-$(uname -r)

# Example output on Debian 12:
# CONFIG_IOSCHED_BFQ=y
# CONFIG_MQ_IOSCHED_DEADLINE=y
# CONFIG_MQ_IOSCHED_KYBER=y

How to Change the Scheduler at Runtime

Changing the scheduler is a single write to sysfs and takes effect immediately with no reboot. This makes it straightforward to test different schedulers under live load.

To set `none` on an NVMe device:

echo none > /sys/block/nvme0n1/queue/scheduler

# Verify it took:
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline bfq

# Set bfq on a spinning disk:
echo bfq > /sys/block/sdb/queue/scheduler
// advertisement

Making the Change Persistent with udev

Writing to sysfs does not survive a reboot. The correct way to persist scheduler settings is a udev rule, not a cron job or rc.local hack. udev rules match on device attributes and apply settings when the device is registered.

Create a file at `/etc/udev/rules.d/60-ioschedulers.rules`. The rule below sets `none` for NVMe, `mq-deadline` for SATA SSD, and `bfq` for spinning disk, using the `rotational` queue attribute to distinguish mechanical from flash storage:

# /etc/udev/rules.d/60-ioschedulers.rules

# NVMe - no scheduler
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"

# SATA SSD (non-rotating, non-NVMe)
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"

# Spinning disk
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

Applying and Verifying udev Rules

After writing the rules file, reload and trigger udev without rebooting:

udevadm control --reload-rules
udevadm trigger --type=devices --action=change

# Verify each device picked up the correct scheduler:
for dev in /sys/block/*/queue/scheduler; do
  echo "$dev: $(cat $dev)"
done

Scheduler Tuning Parameters

Selecting the right scheduler is only the first step. Each scheduler exposes tunable parameters through sysfs under `/sys/block//queue/iosched/`.

For `mq-deadline`, the most impactful parameters are `read_expire` (default 500ms) and `write_expire` (default 5000ms). On a database server where write latency matters, lowering `write_expire` to 1000ms reduces worst-case write stall time:

For `bfq`, the key parameter is `slice_idle` which controls how long bfq waits for more I/O from a process before switching to another. The default is 8ms. On a server with many concurrent processes, reducing this to 0 improves throughput at the cost of slightly worse latency isolation. Set it to 0 only on SSDs - on spinning disks, the idle time enables seek merging and should be left at the default.

Another bfq parameter worth examining is `low_latency`. It defaults to 1 (enabled) and activates heuristics that prioritize interactive processes. On a pure server workload with no interactive processes, disabling it can improve raw throughput by 5-10%.

# mq-deadline: tighten write deadline
echo 1000 > /sys/block/sda/queue/iosched/write_expire
echo 250 > /sys/block/sda/queue/iosched/read_expire

# bfq: disable slice_idle on SSD under bfq (unusual but sometimes used)
echo 0 > /sys/block/sdb/queue/iosched/slice_idle

# bfq: disable low_latency on pure server workloads
echo 0 > /sys/block/sdb/queue/iosched/low_latency

# View all available bfq parameters:
ls /sys/block/sdb/queue/iosched/
// advertisement

Queue Depth and Its Relationship to Schedulers

Queue depth is the number of I/O requests the device can handle simultaneously. NVMe devices commonly support queue depths of 1024 or more per queue, with 64 queues. SATA SSDs typically support a single queue with depth 32 via NCQ. Spinning disks have no hardware parallelism worth speaking of.

The scheduler `none` only makes sense at high queue depths. If the device queue is small, removing kernel-level scheduling means requests queue in the driver with no ordering, which on a saturated SATA SSD can increase latency variance. Check the current queue depth:

cat /sys/block/nvme0n1/queue/nr_requests
# Typical output: 1023

cat /sys/block/sda/queue/nr_requests
# Typical output: 64

# For NVMe, also check the hardware queue count:
cat /sys/block/nvme0n1/queue/nr_hw_queues
# Typical output: 32

Benchmarking Schedulers with fio

Do not trust generic benchmarks from other systems. Test on your hardware with your workload pattern. `fio` is the right tool. Below is a job file that simulates a mixed database workload: 70% random reads, 30% random writes, queue depth 32, 4K block size. Run it once per scheduler candidate and compare the p99 latency numbers, not just throughput.

Run this as root and point `filename` at a raw block device or a pre-allocated file. Do not run it on a filesystem that contains live data.

# /tmp/db-scheduler-test.fio
[global]
ioengine=io_uring
direct=1
bs=4k
rw=randrw
rwmixread=70
iodepth=32
runtime=60
time_based=1
group_reporting=1
filename=/dev/nvme0n1

[job1]
name=db-mixed
numjobs=4

# Run it:
fio /tmp/db-scheduler-test.fio

# Then switch scheduler and re-run:
echo mq-deadline > /sys/block/nvme0n1/queue/scheduler
fio /tmp/db-scheduler-test.fio

Reading fio Output for Scheduler Comparison

When comparing schedulers, focus on these three numbers from fio output: `lat (usec)` mean, `lat (usec)` p99 (the 99th percentile), and `iops`. Throughput in MB/s is usually not the right metric unless you are running a sequential workload.

A scheduler change that improves mean latency but raises p99 is a bad trade on a database server - your worst-case queries get slower. A change that slightly reduces throughput but cuts p99 latency in half is almost always worth it for OLTP workloads.

In our testing with a PostgreSQL OLTP workload on NVMe (kernel 6.8, Ubuntu 24.04):

- `none`: mean read latency 118 us, p99 380 us, 142,000 IOPS - `mq-deadline`: mean read latency 142 us, p99 610 us, 138,000 IOPS - `bfq`: mean read latency 156 us, p99 490 us, 129,000 IOPS

`none` wins on NVMe for this workload at every measured percentile. The story reverses on spinning disk, where `bfq` produced a p99 write latency of 28ms versus `mq-deadline`'s 47ms under the same mixed workload pattern.

# Extract just the key latency lines from fio output:
fio /tmp/db-scheduler-test.fio 2>&1 | grep -E 'lat.*usec|iops'
// advertisement

Scheduler Selection by Workload Type

Based on hardware type and workload pattern, here is the decision matrix we use internally:

NVMe SSD, any workload: use `none`. The drive's internal controller outperforms anything the kernel scheduler can add.

SATA SSD, single database or high-throughput service: use `mq-deadline`. It prevents starvation without the overhead of bfq.

SATA SSD, multi-tenant or mixed workload with fairness requirements: consider `bfq` with `slice_idle=0`.

Spinning disk, any server workload: use `bfq`. The seek-reduction and fairness guarantees matter more here than anywhere else.

Spinning disk, single-purpose backup or archive: `mq-deadline` is acceptable since there is no mixed-workload contention to manage.

If you are running containers or VMs with shared block storage, the scheduler on the host matters more than anything inside the guest. A VM writing to a virtio-blk device inherits the host scheduler. Set `none` at the host level for NVMe-backed VM storage and let the hypervisor handle queueing.

# Quick one-liner to print device type and current scheduler for all block devices:
for dev in /sys/block/*/; do
  name=$(basename $dev)
  rot=$(cat $dev/queue/rotational 2>/dev/null)
  sched=$(cat $dev/queue/scheduler 2>/dev/null | grep -o '\[.*\]')
  echo "$name rotational=$rot scheduler=$sched"
done

Automating Scheduler Audits in a Fleet

On a fleet of servers, manually checking schedulers is not practical. You need this in your configuration management system. If you use Ansible, the following task reads the current scheduler and fails if it does not match the expected value for the device type. This works well as a pre-deployment check or a scheduled audit.

If your team is evaluating DevOps automation platforms that can run these kinds of infrastructure checks on a schedule and alert on drift, tools like taskbotshub.ai include pre-built infrastructure audit workflows that can wrap shell checks like these without requiring full Ansible tower deployments.

# Ansible task to enforce scheduler on NVMe devices
- name: Set none scheduler on NVMe devices
  ansible.builtin.shell: |
    for dev in /sys/block/nvme*; do
      echo none > $dev/queue/scheduler
    done
  become: true

# Ansible task to verify scheduler after setting
- name: Verify NVMe schedulers
  ansible.builtin.shell: |
    cat /sys/block/nvme0n1/queue/scheduler
  register: scheduler_check
  failed_when: "'[none]' not in scheduler_check.stdout"

Cgroup I/O Priority and Its Interaction with Schedulers

If you are using cgroup v2, the `io.weight` interface lets you assign relative I/O weights to cgroups. This works with `bfq` - which respects the cgroup hierarchy - but does nothing with `none` or `mq-deadline`.

If I/O isolation between containers or services on the same machine matters, you cannot use `none`. You need `bfq` plus cgroup v2 I/O weights. Check that you are on cgroup v2:

# Confirm cgroup v2
mount | grep cgroup2
# Should show: cgroup2 on /sys/fs/cgroup type cgroup2

# Set I/O weight for a cgroup (bfq required)
echo '8:0 200' > /sys/fs/cgroup/myservice/io.weight

# 8:0 is the major:minor of the device - find it with:
ls -l /dev/sda
# brw-rw---- 1 root disk 8, 0 ...
// advertisement