What BBR Actually Does Differently

CUBIC and most legacy congestion control algorithms are loss-based: they increase the congestion window until a packet is dropped, then back off. This works tolerably on low-latency LAN but performs poorly when buffers are large, paths are long, or both. The classic symptom is bufferbloat - queues fill up inside network equipment, RTT balloons to hundreds of milliseconds, and throughput oscillates.

BBR models the network path directly. It continuously estimates two values: BtlBw (bottleneck bandwidth) and RTprop (minimum round-trip propagation time). From those it calculates a pacing rate and sets the congestion window to BtlBw * RTprop, which is the theoretical bandwidth-delay product. It does not wait for loss. It periodically probes upward to detect bandwidth increases and drains the queue to refresh its RTprop estimate.

The practical consequence: on a 1 Gbps link with 50ms RTT, CUBIC running at steady state will have a queue of several hundred milliseconds in a deep-buffered switch. BBR keeps queuing delay near RTprop. Your application sees lower latency, iperf3 sees higher goodput, and you stop blaming the network team.

BBR is not magic on every topology. On shallow-buffered links where loss is genuine congestion signal rather than AQM artifact, the difference is smaller. On links shared with many CUBIC flows, early BBR versions were somewhat unfriendly to coexisting flows - BBR v2 and v3 address this with loss-signal integration.

Check Your Kernel and Available Algorithms

Before enabling BBR, confirm the kernel version and which algorithms are compiled in. BBR requires kernel 4.9+. BBR v2 landed in 5.13 as an out-of-tree patch before being upstreamed. BBR v3 is in 6.x kernels as of 2024.

Check available algorithms on your running system:

# Kernel version
uname -r

# Available congestion control algorithms (built-in and loaded modules)
sysctl net.ipv4.tcp_available_congestion_control

# Currently active algorithm
sysctl net.ipv4.tcp_congestion_control

# If bbr not listed, check if module exists
find /lib/modules/$(uname -r) -name 'tcp_bbr*'

# Load the module manually to test
modprobe tcp_bbr
lsmod | grep bbr

Enable BBR System-Wide in Two Steps

Enabling BBR requires setting the congestion control algorithm and pairing it with a suitable queue discipline. The fq (Fair Queue) scheduler is the recommended pairing - BBR's pacing depends on per-flow rate limiting that fq provides at the qdisc layer. Without fq, BBR still works but loses its pacing accuracy.

Apply immediately with sysctl -w for testing, then write to /etc/sysctl.d/ for persistence:

# Apply immediately (test first)
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq

# Verify it took
sysctl net.ipv4.tcp_congestion_control
sysctl net.core.default_qdisc

# Make persistent
cat >> /etc/sysctl.d/99-bbr.conf << 'EOF'
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr
EOF

# Reload
sysctl --system
// advertisement

Verify BBR Is Running on Active Connections

Setting sysctl does not retroactively change existing connections. New connections started after the sysctl change will use BBR. Use ss to inspect the congestion control on live sockets - the -i flag shows internal TCP information including which algorithm is active.

The cong field in ss output is what you want:

# Show all TCP sockets with congestion control info
ss -tni | grep -E '(ESTAB|cong)'

# More targeted: show established connections and their CC algorithm
ss -ti state established | grep bbr

# Count connections by CC algorithm
ss -ti | awk '/cong/{print $2}' | sort | uniq -c | sort -rn

# Check a specific port (e.g., HTTPS)
ss -ti '( dport = :443 or sport = :443 )' | grep cong

Tune the fq Queue Discipline

The default fq parameters work for most setups but there are two knobs worth examining: flow_limit and pacing_rate. flow_limit caps packets per flow in the fq bucket (default 100). On a very high-bandwidth server where a single flow dominates, increasing this prevents artificial throttling. pacing_rate sets a per-flow hard ceiling.

Check current fq settings on your primary interface:

# Show current qdisc settings on eth0 (replace with your interface)
tc qdisc show dev eth0

# Replace default fq with tuned settings
# flow_limit 200: allow deeper per-flow queue
# maxrate 1gbit: cap any single flow at 1Gbps (remove for 10G+ servers)
tc qdisc replace dev eth0 root fq flow_limit 200 maxrate 1gbit

# For 10GbE servers, remove per-flow cap and increase quantum
tc qdisc replace dev eth0 root fq flow_limit 500 quantum 4000

# Verify
tc -s qdisc show dev eth0

Additional sysctl Parameters That Matter With BBR

BBR works with standard socket buffers but you will leave performance on the table without adequate send and receive buffer sizing. The kernel's autotuning handles most of this but the maximums need to be large enough. On a 10 Gbps link with 100ms RTT, the bandwidth-delay product is 125 MB - your socket buffers must accommodate that.

We also recommend enabling tcp_no_metrics_save to prevent the kernel from caching stale RTT estimates from previous connections to the same host, which can interfere with BBR's path model:

cat >> /etc/sysctl.d/99-bbr.conf << 'EOF'
# Socket buffer sizing for high-BDP paths
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_rmem=4096 87380 134217728
net.ipv4.tcp_wmem=4096 65536 134217728

# Let kernel autotune within the above limits
net.ipv4.tcp_moderate_rcvbuf=1

# Do not cache per-host metrics (RTT, CWND) between connections
net.ipv4.tcp_no_metrics_save=1

# Timestamps help BBR's RTT measurement accuracy
net.ipv4.tcp_timestamps=1

# ECN: optional but improves BBR coexistence with AQM routers
net.ipv4.tcp_ecn=1
EOF

sysctl --system
// advertisement

BBR v1 vs v2 vs v3: Which Version Do You Have

The version matters because v1, v2, and v3 have meaningfully different behavior under competition with CUBIC flows and under packet loss.

BBR v1 (kernel 4.9-5.12): Tends to grab bandwidth aggressively, ignores loss signals, unfriendly to coexisting CUBIC flows on shared links. Fine for servers where you control both endpoints or on dedicated paths.

BBR v2 (kernel 5.13+ with out-of-tree patches, some distros): Adds loss-based pacing reduction. More conservative. Better coexistence. Google deployed this internally at scale.

BBR v3 (kernel 6.x, upstreamed 2024): Further refinements to the probe cycle timing, improved startup behavior, better handling of shallow buffers. This is what we ran our tests on (kernel 6.8 on Ubuntu 24.04 LTS).

Check which version your kernel ships:

# Look at kernel config for BBR version indicators
grep -i bbr /boot/config-$(uname -r)

# Or check module info
modinfo tcp_bbr 2>/dev/null | grep -E '(version|description|filename)'

# On systems with kernel 6.x, verify BBRv3 symbol presence
grep BBR /proc/net/protocols 2>/dev/null || true

# Check kernel changelog or package version
debian_kernel=$(dpkg -l | grep linux-image | grep $(uname -r | cut -d- -f1) | awk '{print $3}')
echo "Kernel package version: $debian_kernel"

# RHEL/Rocky/Alma
rpm -q kernel --qf '%{VERSION}-%{RELEASE}\n' | head -5

Benchmark BBR Against CUBIC With iperf3

Before committing to BBR in production, run controlled benchmarks. iperf3 with a simulated WAN delay using tc netem gives reproducible results on a single server pair. We used two VMs connected over a routed path with 80ms injected RTT and 0.1% random loss to simulate a realistic transcontinental path.

On our test setup (kernel 6.8, 1 Gbps virtual link, 80ms RTT, 0.1% loss), CUBIC averaged 412 Mbps, BBR v3 averaged 681 Mbps over 10 runs. Your numbers will vary but the methodology is repeatable:

# On the 'WAN' interface of the sending machine, inject delay and loss
tc qdisc add dev eth1 root netem delay 80ms loss 0.1%

# Start iperf3 server on remote machine
iperf3 -s -D -p 5201

# Test CUBIC first
sysctl -w net.ipv4.tcp_congestion_control=cubic
iperf3 -c  -p 5201 -t 30 -P 1

# Switch to BBR
sysctl -w net.ipv4.tcp_congestion_control=bbr
iperf3 -c  -p 5201 -t 30 -P 1

# Multi-stream test (more realistic)
iperf3 -c  -p 5201 -t 60 -P 4

# Clean up netem when done
tc qdisc del dev eth1 root

Monitoring BBR in Production

After deploying BBR system-wide you want ongoing visibility into whether it is behaving as expected. The key signals are RTT distribution, retransmit rate, and throughput per connection. netstat and ss provide per-connection data. For aggregate metrics, use /proc/net/netstat or export via node_exporter if you run Prometheus.

For teams using infrastructure automation, tools like taskbotshub.ai can automate rolling out sysctl changes across a fleet and collect baseline/post-change metrics from node_exporter, saving the manual before-and-after comparison work on dozens of nodes.

Watch for these specific BBR-related metrics:

# Per-connection RTT and retransmit stats
ss -ti state established | grep -E '(rtt|retrans|bbr)'

# System-wide TCP stats including retransmits
cat /proc/net/netstat | awk 'NR%2==1{header=$0} NR%2==0{for(i=1;i<=NF;i++) print header_arr[i]"="$i; split(header,header_arr)}'

# Simpler: watch retransmit counters
watch -n5 'netstat -s | grep -E "(retransmit|fast retransmit|timeout)"'

# If using nstat (iproute2)
nstat -az | grep -E '(Retrans|BBR|Congestion)'

# Prometheus node_exporter exposes these automatically
# Check: curl -s localhost:9100/metrics | grep node_netstat_Tcp_Retrans
// advertisement

Known Issues and When Not to Use BBR

BBR is not appropriate for every scenario. Three specific cases where you should think carefully before deploying it:

First: datacenter storage traffic. NVMe-oF and similar latency-sensitive protocols running over TCP (TCP/IP ROCEv2 setups) have tight latency SLAs. BBR's ProbeRTT phase periodically reduces the congestion window to 4 packets to get a clean RTT measurement. On a 100 Gbps storage fabric this is a deliberate throughput dip every 10 seconds per flow. CUBIC with a well-tuned buffer is often better here. BBR v3 reduced the ProbeRTT duration but did not eliminate it.

Second: heavily multiplexed outbound proxies. If your server is a forward proxy or SNAT gateway with tens of thousands of short flows, BBR's bandwidth probing adds overhead per flow that CUBIC avoids. Test with your actual flow size distribution.

Third: kernel versions below 5.4 on production workloads. BBR v1 has documented unfairness issues under competition with RENO and CUBIC. If you cannot upgrade the kernel, at minimum restrict BBR to services you control both endpoints of.

To set per-application congestion control without changing the global default (useful for testing or running mixed workloads), use the TCP_CONGESTION socket option. nginx and HAProxy expose this. In nginx:

# /etc/nginx/nginx.conf - set BBR only for this nginx instance
# (requires nginx built with --with-debug or on Linux 5.x+)
# nginx does not expose TCP_CONGESTION directly but you can use
# a wrapper script that sets the socket option before exec

# Alternative: use systemd socket activation with cc override
# Or set per-route with ip route
ip route change default via  congctl bbr

# Check it
ip route show default

# Revert to system default on a specific interface source address
ip route change default via   # omit congctl to revert

Automated Deployment and Config Management

For fleet deployments, encode the BBR configuration as an idempotent role. Below is a minimal Ansible task that handles the sysctl changes and persists them. This covers the common case of Ubuntu 22.04/24.04 and RHEL 9 derivatives running kernel 5.15+.

For teams building out their automation stack, taskbotshub.ai provides pre-built DevOps task templates including kernel parameter tuning playbooks that integrate with Prometheus alerting - worth evaluating if you are standardizing your automation toolchain.

# ansible/roles/tcp_bbr/tasks/main.yml
---
- name: Load tcp_bbr module
  community.general.modprobe:
    name: tcp_bbr
    state: present

- name: Persist tcp_bbr module load
  copy:
    dest: /etc/modules-load.d/tcp-bbr.conf
    content: "tcp_bbr\n"
    mode: '0644'

- name: Apply BBR sysctl settings
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/99-bbr.conf
    reload: yes
    state: present
  loop:
    - { key: 'net.core.default_qdisc', value: 'fq' }
    - { key: 'net.ipv4.tcp_congestion_control', value: 'bbr' }
    - { key: 'net.core.rmem_max', value: '134217728' }
    - { key: 'net.core.wmem_max', value: '134217728' }
    - { key: 'net.ipv4.tcp_no_metrics_save', value: '1' }
    - { key: 'net.ipv4.tcp_timestamps', value: '1' }

- name: Verify BBR is active
  command: sysctl -n net.ipv4.tcp_congestion_control
  register: cc_result
  changed_when: false
  failed_when: cc_result.stdout != 'bbr'

Quick Diagnostic Checklist

If BBR is enabled but you are not seeing throughput improvement, work through this list:

1. Confirm fq is the active qdisc, not pfifo_fast or anything else. Run tc qdisc show dev and look for 'fq'.

2. Check that new connections are actually using BBR, not connections established before the sysctl change. Kill and reconnect your test session.

3. Verify socket buffers are large enough for your BDP. If rmem_max is still at the default 212992, you are capped regardless of congestion control.

4. Confirm tcp_timestamps is enabled. BBR's RTT measurement degrades without timestamps.

5. Check if an intermediate device is stripping timestamps or doing aggressive buffering. Run mtr -r -c 100 and look for RTT variance exceeding 10x the minimum. That indicates bufferbloat at a hop you do not control - BBR helps but cannot fix a 2-second buffer at a carrier.

# Full diagnostic one-liner
echo "=== Congestion Control ==" && sysctl net.ipv4.tcp_congestion_control && \
echo "=== Default Qdisc ===" && sysctl net.core.default_qdisc && \
echo "=== Interface Qdisc ===" && tc qdisc show dev $(ip route | awk '/default/{print $5; exit}') && \
echo "=== Socket Buffers ===" && sysctl net.core.rmem_max net.core.wmem_max && \
echo "=== Timestamps ===" && sysctl net.ipv4.tcp_timestamps && \
echo "=== Active BBR connections ===" && ss -ti state established | grep -c bbr || echo 0
// advertisement