Split routing using netfilter

2020-01-11 posted in Network

Split routing, also called policy-based routing, means sending only some traffic through a different route while leaving the rest of the network alone. A common example is an OpenWrt router where normal traffic exits through the ISP WAN, but traffic for a few destination networks, a specific LAN host, or a specific service goes through a VPN tunnel.

On older OpenWrt releases this was usually done with iptables mangle rules. On current OpenWrt releases, the firewall is firewall4 (fw4), which uses nftables as its backend. The idea is the same, but the rule language and the way OpenWrt loads custom rules are different.

The basic design is:

Match the traffic you want to split.
Set a packet mark with nftables.
Add an ip rule that sends marked packets to a separate routing table.
Put a default route, or more specific routes, in that table.

Netfilter does not choose the route by itself. Netfilter classifies the packet and attaches a mark. Linux policy routing then reads that mark and chooses a routing table.

Example setup

Assume this OpenWrt router has:

LAN subnet: 192.168.1.0/24
Normal WAN: the default route in the main table
VPN interface: tun0
Split-routed destination: 203.0.113.0/24
Packet mark: 0x1
Custom routing table: 100

The goal is simple: traffic from LAN clients to 203.0.113.0/24 should use tun0; everything else should continue using the normal WAN route.

First, add a policy routing rule:

ip rule add fwmark 0x1 lookup 100 priority 100

Then add a route in table 100:

ip route add default dev tun0 table 100

If the VPN has a gateway address instead of a point-to-point tunnel interface, use that gateway:

ip route add default via 10.8.0.1 dev tun0 table 100

You can inspect the result with:

ip rule show
ip route show table 100

If your OpenWrt image does not support the needed ip rule or routing-table commands, install the fuller iproute2 package:

opkg update
opkg install ip-full

Marking traffic with nftables

With fw4, the cleanest way to add one rule to an existing generated chain is to use OpenWrt’s automatic include directories under /usr/share/nftables.d/. For forwarded LAN traffic, place the rule in the mangle_prerouting chain:

mkdir -p /usr/share/nftables.d/chain-pre/mangle_prerouting
vi /usr/share/nftables.d/chain-pre/mangle_prerouting/10-split-routing.nft

Add the rule itself, without wrapping it in a table or chain block:

ip saddr 192.168.1.0/24 ip daddr 203.0.113.0/24 meta mark set 0x1

Reload the firewall:

/etc/init.d/firewall reload

Then check that the rule was loaded:

nft list chain inet fw4 mangle_prerouting

For forwarded LAN traffic, prerouting is the important hook. The packet enters the router from LAN, nftables marks it before the routing decision, and then ip rule can select table 100.

For traffic generated by the router itself, use an output hook instead:

mkdir -p /usr/share/nftables.d/chain-pre/mangle_output
vi /usr/share/nftables.d/chain-pre/mangle_output/10-split-routing.nft

ip daddr 203.0.113.0/24 meta mark set 0x1

The difference matters. LAN client traffic passes through prerouting; local router traffic passes through output.

Matching by source host

You can also split-route a single LAN client instead of a destination network:

ip saddr 192.168.1.50 meta mark set 0x1

That sends all traffic from 192.168.1.50 to table 100.

If you only want that host’s HTTPS traffic to use the VPN:

ip saddr 192.168.1.50 tcp dport 443 meta mark set 0x1

Using a set for many destinations

One of the nicer differences between nftables and iptables is that nftables has first-class sets. Instead of creating many nearly identical rules, define a set and match against it.

Sets must be defined at table scope. Files in /etc/nftables.d/*.nft are included inside OpenWrt’s inet fw4 table context, so that is a good place for custom sets:

vi /etc/nftables.d/10-split-routing-sets.nft

set vpn_destinations {
    type ipv4_addr
    flags interval
    elements = {
        198.51.100.0/24,
        203.0.113.0/24
    }
}

Then reference that set from the chain include:

vi /usr/share/nftables.d/chain-pre/mangle_prerouting/10-split-routing.nft

ip saddr 192.168.1.0/24 ip daddr @vpn_destinations meta mark set 0x1

This is cleaner and easier to maintain than appending many separate mangle rules.

Making it persistent

The nftables rule is persistent if it lives under the fw4 include directories and fw4 includes it successfully. The routing policy also needs to survive reboot. There are several ways to do that, but the simple method is to put the ip rule and route commands in /etc/rc.local before exit 0:

ip rule add fwmark 0x1 lookup 100 priority 100 2>/dev/null
ip route replace default dev tun0 table 100

exit 0

Using ip route replace makes the route idempotent. The ip rule add command does not behave as cleanly if the rule already exists, so redirecting the duplicate-rule error is a simple workaround. A cleaner production setup would manage this through hotplug scripts so the route is added only after the VPN interface is up.

For a VPN interface, hotplug is usually better than rc.local because the interface may not exist when the router first boots. A minimal /etc/hotplug.d/iface/90-split-routing can look like this:

#!/bin/sh

[ "$ACTION" = "ifup" ] || exit 0
[ "$INTERFACE" = "vpn" ] || exit 0

ip rule add fwmark 0x1 lookup 100 priority 100 2>/dev/null
ip route replace default dev tun0 table 100

Use the OpenWrt interface name for $INTERFACE, not necessarily the Linux device name. For example, the OpenWrt interface might be named vpn while the device is tun0.

How this differs from iptables

The iptables version of the same idea usually looks like this:

iptables -t mangle -A PREROUTING \
    -s 192.168.1.0/24 -d 203.0.113.0/24 \
    -j MARK --set-mark 0x1

ip rule add fwmark 0x1 lookup 100 priority 100
ip route add default dev tun0 table 100

Conceptually, this is almost identical. The packet is matched, marked, and then policy routing handles the route decision.

The differences are mostly in the firewall layer:

iptables has separate tools and rule families such as iptables, ip6tables, ebtables, and arptables. nftables uses one nft command and a more unified ruleset model.
iptables has fixed tables such as filter, nat, and mangle. nftables lets you define tables and chains with explicit hooks and priorities.
iptables rules are commonly appended one at a time. nftables is designed around loading a ruleset, with better support for atomic updates.
nftables has native sets and maps, which makes large destination lists much cleaner.
OpenWrt 22.03 and later use fw4 with nftables by default. Older OpenWrt used fw3 with iptables. The UCI firewall config is still in /etc/config/firewall, but custom low-level rules must be written for the firewall backend you are actually using.

One important warning: do not blindly mix iptables and nftables rules. Some systems provide iptables compatibility commands backed by nftables, but OpenWrt’s generated firewall is still managed by fw4. If you add manual rules outside the expected fw4 include paths, they may disappear on firewall reload or behave differently than expected.

Debugging

Start with the generated ruleset:

fw4 print
nft list ruleset

Then check policy routing:

ip rule show
ip route show table 100

To test a routing decision, use:

ip route get 203.0.113.10 mark 0x1

If counters are enabled on your nftables rule, you can also watch whether packets hit it:

ip saddr 192.168.1.0/24 ip daddr 203.0.113.0/24 counter meta mark set 0x1

Then reload and inspect:

nft list chain inet fw4 mangle_prerouting

If the counter never increases, the packet is not matching the rule. If the counter increases but traffic still uses the wrong route, the problem is probably in ip rule, the custom route table, reverse-path filtering, or the VPN interface route.

Summary

Split routing on OpenWrt is still the same basic Linux policy-routing technique: mark selected packets, then route marked packets through a different table. The modern difference is that OpenWrt now uses fw4 and nftables, so the marking rule should be written as nftables rather than iptables.

For small setups, the nftables version is not much harder than iptables. For larger setups, it is usually better because sets make destination lists easier to read and update. The main thing to remember is the division of responsibility: nftables marks packets; ip rule and ip route decide where those marked packets go.

References: