Salted. s

Thoughts on BBR congestion control in QUIC under high loss

posted in Network

BBR is a natural fit for QUIC, but it is not magic. It works well when packet loss is a bad congestion signal. It can work poorly when loss corrupts the measurements that BBR itself needs.

That distinction matters on high-loss networks: long-distance wireless, satellite, overloaded mobile backhaul, Wi-Fi with interference, tunnels over lossy UDP, and any path where packets disappear even when the bottleneck queue is not full.

The simple position is:

BBR should not treat every loss as congestion, but it also cannot ignore loss.

The robust design is not “BBR but more aggressive”. It is BBR with a better loss classifier, safer probing, stronger ACK-path handling, and an explicit policy for when random loss becomes too expensive.

Why QUIC changes the shape of the problem

QUIC gives the congestion controller cleaner signals than classic TCP:

  • packet numbers are not reused after retransmission
  • ACK frames can report multiple ACK ranges
  • RTT sampling can account for peer-reported ACK delay
  • loss detection is separated from retransmission bytes
  • the implementation can run in user space next to application scheduling
  • the sender can choose pacing and packetization policy per connection

That makes QUIC a good place to run a model-based controller like BBR. The controller can observe delivery rate, RTT, loss, ACK timing, and stream/application pressure with less ambiguity than TCP had historically.

The dangerous part is that QUIC also runs over UDP, so it often goes through paths with middleboxes, policers, tunnels, and wireless schedulers that do not behave like a clean FIFO bottleneck. If the ACK stream is noisy, delayed, compressed, or partially lost, BBR’s model can be wrong even if the algorithm is well-designed.

BBR’s useful mental model

BBR tries to estimate two path properties:

$$ B = \operatorname{BtlBw} $$$$ R = \operatorname{RTprop} $$

The bandwidth-delay product is:

$$ \operatorname{BDP} = B \cdot R $$

A simplified sender then chooses:

$$ \operatorname{pacingRate} = g_p \cdot B $$$$ \operatorname{inflightTarget} = g_c \cdot B \cdot R $$

where \(g_p\) is the pacing gain and \(g_c\) is the congestion-window gain. The exact BBR state machine is more complex, but this model is enough to reason about high loss.

Loss-based congestion control treats packet loss as the main signal that the sender overshot the path. BBR treats loss as one signal among several. That is why BBR can be much better on random-loss paths: if the path can deliver 100 Mbit/s with 1% non-congestion loss, the theoretical ceiling is not 1 Mbit/s. It is closer to:

$$ G_{\max} \approx C \cdot (1 - p) $$

where \(C\) is path capacity and \(p\) is the random packet loss probability. A controller that collapses its window on every random loss will never approach that ceiling.

The high-loss failure modes

High loss hurts BBR in five different ways.

First, random loss reduces delivered bytes, so the measured delivery rate can fall below the real bottleneck rate. If BBR updates its bandwidth model too eagerly, it can train itself to believe the path is slower than it is.

Second, burst loss can look like congestion even when the root cause is wireless corruption or link-layer scheduling. A single burst can erase a round of probes.

Third, ACK loss and ACK compression can corrupt the sender’s rate samples. BBR does not directly see the bottleneck; it sees ACK-clocked delivery observations.

Fourth, ProbeBW can be too sharp for a lossy queue. Pacing above the estimated bandwidth is necessary to discover more capacity, but a high-gain probe can cause loss bursts on shallow buffers.

Fifth, ProbeRTT can be expensive on high-BDP, high-loss paths. Draining inflight data to refresh the minimum RTT estimate may create idle gaps that are slow to recover from.

The controller has to decide whether loss is congestion, random corruption, ACK-path damage, or a sign that the sender’s own probes are too sharp.

Classify loss before reacting

The first improvement is to stop treating “loss rate” as one number.

Let:

$$ p_t = \frac{\operatorname{lostPackets}_t}{\operatorname{sentPackets}_t} $$

and:

$$ q_t = \operatorname{RTT}_t - \operatorname{RTprop} $$

where \(q_t\) is a rough queue-delay estimate. Smooth both:

$$ \hat p_t = \alpha p_t + (1-\alpha)\hat p_{t-1} $$$$ \hat q_t = \alpha q_t + (1-\alpha)\hat q_{t-1} $$

Now classify the round:

high loss + high queue delay      -> likely congestion
high loss + ECN-CE growth         -> likely congestion
high loss + low queue delay       -> likely random/link loss
loss burst after pacing probe     -> probe was too sharp
ACK gaps without forward loss     -> ACK path is damaged

This is not perfect, but it is better than one global reaction to all loss.

flowchart LR
  A["QUIC ACK ranges<br/>RTT samples<br/>ECN counters"] --> B["measurement filter"]
  B --> C["BBR path model<br/>BtlBw + RTprop + loss"]
  C --> D["pacing rate"]
  C --> E["inflight cap"]
  D --> F["QUIC packet scheduler"]
  E --> F
  F --> G["network path"]
  G --> A
  B --> H["loss classifier"]
  H --> C

The important engineering boundary is between loss detection and loss meaning. QUIC can tell you that a packet is lost. It cannot tell you why it was lost.

Make the bandwidth estimator loss-tolerant

BBR’s bandwidth estimate should not be a simple moving average of the last few delivery samples. On a lossy path, that average is biased downward.

Use a robust estimator:

$$ B_t = \operatorname{quantile}_{0.9}\{r_{t-k}, \ldots, r_t\} $$

or a max filter with aging:

$$ B_t = \max(\lambda B_{t-1}, r_t) $$

where \(0 < \lambda < 1\). The max filter keeps recent evidence that the path can deliver at a high rate, but slowly forgets old capacity.

Then avoid using samples from rounds that are obviously measurement-corrupted:

  • app-limited rounds
  • rounds with severe ACK compression
  • rounds dominated by retransmission repair
  • rounds immediately after path migration
  • rounds where receiver ACK frequency changed abruptly

The sender should still count loss against goodput, but it should not always let random loss lower the bottleneck estimate.

Bound inflight, but do not overreact

BBR must still protect the network. If it ignores loss completely, it can be unfair and unstable.

The right response is to bound inflight, not blindly halve it.

Define a loss pressure:

$$ L_t = w_p \hat p_t + w_q \frac{\hat q_t}{R} + w_e e_t $$

where \(e_t\) is ECN-CE fraction. Then choose an inflight bound:

$$ \operatorname{inflightMax}_t = \operatorname{BDP}_t \cdot \left(1 + \gamma\right) \cdot \frac{1}{1 + L_t} $$

This makes the controller stricter when loss correlates with queue growth or ECN marking. It is less strict when loss exists without queue growth.

For a high-random-loss path, a useful policy is:

if loss is random and queue delay is flat:
    preserve BtlBw estimate
    reduce probe gain
    cap inflight gently
else if loss is congestion-correlated:
    reduce inflight cap aggressively
    shorten probe bursts

This turns BBR into a controller that asks, “How much data can safely be in flight?” instead of, “Did any packet disappear?”

Make ProbeBW less sharp

ProbeBW is where BBR discovers extra capacity. It is also where lossy networks get hurt.

A sender can make probes safer with three changes:

  1. Lower the high-gain probe when loss is already elevated.
  2. Spread probe bytes over more pacing intervals.
  3. Stop the probe early when queue delay rises faster than delivery rate.

One simple rule:

$$ g_p = \begin{cases} 1.25, & \hat p_t < p_0 \\\\ 1.10, & p_0 \le \hat p_t < p_1 \\\\ 1.00, & \hat p_t \ge p_1 \end{cases} $$

That does not mean the sender gives up on probing forever. It means lossy paths get smaller, more frequent probes instead of large bursts.

The controller should also bound burst size independently of pacing rate:

$$ \operatorname{burstBytes} \le \min(4M, 0.05 \cdot \operatorname{BDP}) $$

where \(M\) is QUIC’s current max datagram size. On wireless and tunneled paths, burst size often matters more than average rate.

Treat ACK-path damage as a first-class problem

QUIC senders often talk about forward-path loss, but BBR depends on the reverse path too. If ACKs are lost, delayed, or compressed, delivery samples become lumpy.

A robust QUIC BBR implementation should track:

ACK inter-arrival variance
ACK range density
largest acknowledged gap
ack_delay changes
receiver max_ack_delay
ECN counter consistency

If ACK aggregation is detected, do not immediately treat one large ACK as evidence of a permanently higher bottleneck bandwidth. If ACK thinning is detected, do not immediately treat missing ACKs as evidence that the forward path slowed down.

The estimator should separate:

$$ \operatorname{deliveryRateSample} $$

from:

$$ \operatorname{deliveryRateBelief} $$

The first is raw observation. The second is what the controller should believe after filtering ACK-path artifacts.

When bit errors dominate, packet loss probability rises with packet size. If the bit error probability is \(p_b\) and the packet has \(8M\) bits:

$$ p_{\operatorname{packet}} = 1 - (1-p_b)^{8M} $$

For small \(p_b\), this is approximately:

$$ p_{\operatorname{packet}} \approx 8M p_b $$

So shrinking QUIC datagrams can reduce packet loss probability, but it increases header overhead and CPU cost. That gives a real tradeoff:

$$ G(M) \approx C \cdot (1-p_{\operatorname{packet}}(M)) \cdot \frac{M-H}{M} $$

where \(H\) is per-packet overhead. On a clean path, larger packets win. On a dirty path, a smaller datagram can win.

This should be adaptive. Do not globally force small packets. Use smaller packet sizes only when the path shows persistent random loss and the application benefits from smoother delivery.

Do not confuse reliability with congestion control

QUIC retransmits lost reliable data in new packets. That fixes correctness, not throughput. If the path loses 10% of packets, the sender can repair the stream, but repair traffic competes with new data.

The actual useful throughput is:

$$ G_{\operatorname{app}} = \frac{\operatorname{newBytesDelivered}}{\operatorname{time}} $$

not:

$$ G_{\operatorname{wire}} = \frac{\operatorname{allBytesSent}}{\operatorname{time}} $$

On high-loss paths, controllers can accidentally optimize wire rate while application goodput gets worse. The BBR policy should monitor application goodput and retransmission fraction:

$$ \rho = \frac{\operatorname{repairBytes}}{\operatorname{totalBytesSent}} $$

If \(\rho\) grows too high, the controller should reduce probe aggressiveness even if raw delivery samples still look good.

Practical implementation policy

For a QUIC stack, I would implement this in layers.

First, keep the RFC 9002 recovery machinery correct. Loss detection, PTO, persistent congestion handling, packet number spaces, and ACK processing must be right before congestion-control tuning matters.

Second, run BBR with explicit pacing. Without pacing, BBR degenerates into bursty window control, which is exactly what lossy queues dislike.

Third, use a BBRv3-style model that includes loss pressure and inflight bounds. BBRv1’s “loss is mostly not congestion” attitude is too optimistic for mixed Internet paths.

Fourth, classify loss per round:

round starts
  collect delivery samples
  collect loss samples
  collect ECN and RTT movement
  classify loss cause
  update BtlBw and RTprop
  update inflight bound
  update pacing gain
round ends

Fifth, add guardrails:

  • never let random loss erase the bandwidth model from one bad round
  • never let high ECN marking be ignored as random loss
  • never use huge GSO/UDP bursts on a lossy path
  • never optimize wire throughput without watching application goodput
  • never assume ACK timing is clean

When BBR is the wrong answer

There are paths where congestion control alone cannot fix the problem.

If loss is above several percent and bursty, reliable QUIC streams may spend too much capacity on repair. For media, gaming, telemetry, or replicated state, QUIC DATAGRAM plus application-level forward error correction may be better than forcing every byte through a reliable stream.

If the path is a tunnel, fix the tunnel first:

  • avoid fragmentation
  • clamp MTU correctly
  • avoid huge UDP segmentation bursts
  • pace before encryption when possible
  • keep buffers shallow with FQ-CoDel or CAKE

If the path is wireless, link-layer retry policy and airtime scheduling can matter more than transport tuning.

BBR can be made robust, but it cannot create capacity, remove interference, or repair broken queue management.

My preferred high-loss QUIC BBR recipe

For a production QUIC stack on high-loss networks, I would start with:

  1. BBRv3-style model and loss response.
  2. ECN support, but only when validation says the path reports it honestly.
  3. Quantile or aging-max delivery-rate filter.
  4. ProbeBW gain reduced by loss pressure.
  5. Independent burst cap below the pacing-rate calculation.
  6. ACK aggregation and ACK thinning detection.
  7. Packet-size adaptation for persistent random loss.
  8. Application-goodput feedback to avoid optimizing repair traffic.

The key idea is restraint. High-loss networks do not need a congestion controller that panics on every loss. They also do not need one that ignores loss until the path collapses. They need a controller that can say:

this loss is probably corruption
this loss is probably congestion
this loss is probably my own probe
this loss is probably measurement damage

BBR in QUIC has enough signal to make that distinction. The hard part is being conservative about when the distinction is trusted.

References