| 1 |
# Multiple ISP |
| 2 |
|
| 3 |
Foomuuri supports multiple ISPs (aka multi-ISP aka multi-WAN aka multiple |
| 4 |
simultaneous uplink connections) with active-active (load balancing) or |
| 5 |
active-passive (failover) configuration. |
| 6 |
|
| 7 |
|
| 8 |
## Configuration with static ISP routes |
| 9 |
|
| 10 |
This example requires static ISP routes. It works with NetworkManager and |
| 11 |
systemd-networkd. Both active-active and active-passive configurations are |
| 12 |
supported. Both configuration types are very similar, with only one line |
| 13 |
changed. |
| 14 |
|
| 15 |
This example configuration assumes: |
| 16 |
|
| 17 |
* ISP #1 is network interface `enp1s0`, my own IP is 172.23.70.36/24, gateway |
| 18 |
is 172.23.70.254. Traffic will be marked with value 0x100 and uses route |
| 19 |
table 1001. |
| 20 |
* ISP #2 is network interface `enp2s0`, my own IP is 172.23.12.31/24, gateway |
| 21 |
is 172.23.12.254. Traffic will be marked with value 0x200 and uses route |
| 22 |
table 1002. |
| 23 |
* Zone `internal` network interface is `enp8s0`, with network 10.0.0.0/8. |
| 24 |
Outgoing traffic to `public` zone is masqueraded. |
| 25 |
|
| 26 |
Example `foomuuri.conf` file: |
| 27 |
|
| 28 |
``` |
| 29 |
# Basic configuration: |
| 30 |
|
| 31 |
zone { |
| 32 |
# Define zones with interfaces |
| 33 |
localhost |
| 34 |
public enp1s0 enp2s0 |
| 35 |
internal enp8s0 |
| 36 |
} |
| 37 |
|
| 38 |
foomuuri { |
| 39 |
# Reverse path filtering must be disabled for public interfaces |
| 40 |
rpfilter -enp1s0 -enp2s0 |
| 41 |
} |
| 42 |
|
| 43 |
snat { |
| 44 |
# Masquerade outgoing traffic from internal to public. Both ISPs must be |
| 45 |
# masqueraded separately. |
| 46 |
saddr 10.0.0.0/8 oifname enp1s0 masquerade |
| 47 |
saddr 10.0.0.0/8 oifname enp2s0 masquerade |
| 48 |
} |
| 49 |
|
| 50 |
# Multi-ISP magic is here, using marks to select which ISP to use. Order |
| 51 |
# of the rules is important. Specific rules should be first, generic last. |
| 52 |
|
| 53 |
prerouting { |
| 54 |
# Accept if mark is already set (not zero). Existing mark will be used. |
| 55 |
mark_match -0x0000/0xff00 |
| 56 |
|
| 57 |
# == Incoming traffic == |
| 58 |
|
| 59 |
# Mark traffic from enp1s0 as 0x100 (ISP1) and enp2s0 as 0x200 (ISP2). |
| 60 |
# This is needed for correctly routing reply packets. |
| 61 |
iifname enp1s0 mark_set 0x100/0xff00 |
| 62 |
iifname enp2s0 mark_set 0x200/0xff00 |
| 63 |
|
| 64 |
# == Outgoing traffic == |
| 65 |
|
| 66 |
# Specific rules should be added first. For example, uncomment next line to |
| 67 |
# route all SSH traffic from internal to public via ISP2. |
| 68 |
#iifname enp8s0 ssh mark_set 0x200/0xff00 |
| 69 |
|
| 70 |
# Similarly, some source IPs can always be routed via ISP1. |
| 71 |
#saddr 10.0.1.0/24 mark_set 0x100/0xff00 |
| 72 |
|
| 73 |
# For active-active configuration use following line. It uses random number |
| 74 |
# generator to mark traffic with 0x100 or 0x200. This routes 60% (0-5) |
| 75 |
# of outgoing traffic to ISP1 and 40% (6-9) to ISP2. |
| 76 |
nft "meta mark set numgen random mod 10 map { 0-5: 0x100, 6-9: 0x200 } ct mark set meta mark accept" |
| 77 |
|
| 78 |
# For active-passive configuration uncomment next line and add comment to |
| 79 |
# above nft-line. It simply assigns mark 0x100 (ISP1) to all traffic and |
| 80 |
# uses ISP2 only as fallback. |
| 81 |
#mark_set 0x100/0xff00 |
| 82 |
} |
| 83 |
|
| 84 |
# foomuuri-monitor config: |
| 85 |
|
| 86 |
target isp1 { |
| 87 |
# Monitor ISP1 connectivity by pinging 8.8.4.4. Ideally this would be |
| 88 |
# some ISP1's router's IP address. |
| 89 |
command fping --iface enp1s0 8.8.4.4 |
| 90 |
command_up /etc/foomuuri/multi-isp up 1 |
| 91 |
command_down /etc/foomuuri/multi-isp down 1 |
| 92 |
} |
| 93 |
|
| 94 |
target isp2 { |
| 95 |
# Monitor ISP2 connectivity by pinging their router 172.25.31.149. |
| 96 |
command fping --iface enp2s0 172.25.31.149 |
| 97 |
command_up /etc/foomuuri/multi-isp up 2 |
| 98 |
command_down /etc/foomuuri/multi-isp down 2 |
| 99 |
} |
| 100 |
|
| 101 |
# Normal zone-zone rules, copied from router firewall example configuration: |
| 102 |
|
| 103 |
public-localhost { |
| 104 |
ping saddr_rate "5/second burst 20" |
| 105 |
ssh saddr_rate "5/minute burst 5" |
| 106 |
drop log |
| 107 |
} |
| 108 |
|
| 109 |
internal-localhost { |
| 110 |
dhcp-server |
| 111 |
dhcpv6-server |
| 112 |
domain |
| 113 |
domain-s |
| 114 |
ntp |
| 115 |
ping |
| 116 |
ssh |
| 117 |
reject log |
| 118 |
} |
| 119 |
|
| 120 |
template outgoing_services { |
| 121 |
# Shared list of services for localhost-public and internal-public. |
| 122 |
domain |
| 123 |
domain-s |
| 124 |
http |
| 125 |
https |
| 126 |
ntp |
| 127 |
ping |
| 128 |
smtp |
| 129 |
ssh |
| 130 |
} |
| 131 |
|
| 132 |
localhost-public { |
| 133 |
template outgoing_services |
| 134 |
reject log |
| 135 |
} |
| 136 |
|
| 137 |
internal-public { |
| 138 |
template outgoing_services |
| 139 |
googlemeet |
| 140 |
imap |
| 141 |
reject log |
| 142 |
} |
| 143 |
|
| 144 |
public-internal { |
| 145 |
drop log |
| 146 |
} |
| 147 |
|
| 148 |
localhost-internal { |
| 149 |
dhcp-client |
| 150 |
dhcpv6-client |
| 151 |
ping |
| 152 |
ssh |
| 153 |
reject log |
| 154 |
} |
| 155 |
``` |
| 156 |
|
| 157 |
Example `/etc/foomuuri/multi-isp` script. Remember to save it as executable, |
| 158 |
`chmod 750 /etc/foomuuri/multi-isp`. |
| 159 |
|
| 160 |
``` bash |
| 161 |
#!/bin/sh |
| 162 |
|
| 163 |
# Path to "ip" command |
| 164 |
IP=/usr/sbin/ip |
| 165 |
|
| 166 |
case "${1}" in |
| 167 |
"start") |
| 168 |
# Started, run by foomuuri-multi-isp.service |
| 169 |
echo "Foomuuri multi-ISP start" |
| 170 |
|
| 171 |
# Create new route tables for both ISPs having default route |
| 172 |
${IP} route add table 1001 default dev enp1s0 via 172.23.70.254 |
| 173 |
${IP} route add table 1002 default dev enp2s0 via 172.23.12.254 |
| 174 |
|
| 175 |
# Delete default routes from main table |
| 176 |
${IP} route del table main default dev enp1s0 via 172.23.70.254 |
| 177 |
${IP} route del table main default dev enp2s0 via 172.23.12.254 |
| 178 |
|
| 179 |
# Start rules with main-table lookup |
| 180 |
${IP} rule add prio 1000 from all table main |
| 181 |
|
| 182 |
# Packets with specific source IP must go to correct ISP |
| 183 |
${IP} rule add prio 1001 from 172.23.70.36 table 1001 |
| 184 |
${IP} rule add prio 1002 from 172.23.12.31 table 1002 |
| 185 |
|
| 186 |
# Other ISP-specific rules will be added by monitor's up-event. |
| 187 |
;; |
| 188 |
|
| 189 |
"stop") |
| 190 |
# Stopped, run by foomuuri-multi-isp.service |
| 191 |
echo "Foomuuri multi-ISP stop" |
| 192 |
|
| 193 |
# Add default route back to main table |
| 194 |
${IP} route add table main default dev enp1s0 via 172.23.70.254 metric 100 |
| 195 |
${IP} route add table main default dev enp2s0 via 172.23.12.254 metric 101 |
| 196 |
|
| 197 |
# Delete added rules |
| 198 |
${IP} rule del prio 1000 |
| 199 |
${IP} rule del prio 1001 |
| 200 |
${IP} rule del prio 1002 |
| 201 |
${IP} rule del prio 1101 2> /dev/null |
| 202 |
${IP} rule del prio 1102 2> /dev/null |
| 203 |
${IP} rule del prio 1201 2> /dev/null |
| 204 |
${IP} rule del prio 1202 2> /dev/null |
| 205 |
|
| 206 |
# Delete added route tables |
| 207 |
${IP} route flush table 1001 |
| 208 |
${IP} route flush table 1002 |
| 209 |
;; |
| 210 |
|
| 211 |
"up") |
| 212 |
# ISP is up, add rules to route traffic to it |
| 213 |
echo "Foomuuri multi-ISP isp${2} up" |
| 214 |
# Use packet mark to select this ISP |
| 215 |
${IP} rule add prio 110${2} from all fwmark 0x${2}00/0xff00 table 100${2} |
| 216 |
# Fallback to this ISP if other is down |
| 217 |
${IP} rule add prio 120${2} from all table 100${2} |
| 218 |
;; |
| 219 |
|
| 220 |
"down") |
| 221 |
# ISP is down, delete its rules |
| 222 |
echo "Foomuuri multi-ISP isp${2} down" |
| 223 |
${IP} rule del prio 110${2} |
| 224 |
${IP} rule del prio 120${2} |
| 225 |
;; |
| 226 |
|
| 227 |
*) |
| 228 |
echo "syntax error" |
| 229 |
exit 1 |
| 230 |
esac |
| 231 |
``` |
| 232 |
|
| 233 |
Example `/etc/systemd/system/foomuuri-multi-isp.service` file. Remember to |
| 234 |
enable it with `systemctl enable foomuuri-multi-isp.service`. |
| 235 |
|
| 236 |
``` systemd |
| 237 |
[Unit] |
| 238 |
Description=Multizone bidirectional nftables firewall - Multi-ISP |
| 239 |
Documentation=https://foomuuri.foobar.fi/latest/ |
| 240 |
After=network-online.target |
| 241 |
Requires=network-online.target |
| 242 |
PartOf=foomuuri.service |
| 243 |
|
| 244 |
[Service] |
| 245 |
Type=oneshot |
| 246 |
RemainAfterExit=yes |
| 247 |
ExecStart=/etc/foomuuri/multi-isp start |
| 248 |
ExecStop=/etc/foomuuri/multi-isp stop |
| 249 |
|
| 250 |
[Install] |
| 251 |
WantedBy=multi-user.target |
| 252 |
``` |
| 253 |
|
| 254 |
|
| 255 |
## Configuration with routes dynamically allocated by ISPs |
| 256 |
|
| 257 |
This example configuration assumes: |
| 258 |
|
| 259 |
* ISP #1 is on the network interface `enp1s0` with routes and IP addresses allocated via DHCP/IPv6RA. Traffic will be marked with value 0x100. |
| 260 |
* ISP #2 is on the network interface `enp2s0` with routes and IP addresses allocated via DHCP/IPv6RA. Traffic will be marked with value 0x200. |
| 261 |
* Zone `internal` network interface is `enp8s0` with IP 10.0.0.1/24. Outgoing traffic to `public` zone is masqueraded. |
| 262 |
* At least the public interfaces and their routing tables will be managed by `systemd-networkd`. |
| 263 |
* This example demonstrates a purely failover setup from ISP #1 to ISP #2. For load balancing add a second target monitor for `enp2s0` in `foomuuri.conf` and adjust the `switch.sh` script to identify and replace the fwmark randomizer as documented in the static routes example above. |
| 264 |
|
| 265 |
Initiate the routing tables by creating the following two files. |
| 266 |
|
| 267 |
`/etc/systemd/networkd.conf.d/table-primary.conf`: |
| 268 |
|
| 269 |
``` |
| 270 |
[Network] |
| 271 |
RouteTable=primary:100 |
| 272 |
``` |
| 273 |
|
| 274 |
`/etc/systemd/networkd.conf.d/table-secondary.conf`: |
| 275 |
|
| 276 |
``` |
| 277 |
[Network] |
| 278 |
RouteTable=secondary:200 |
| 279 |
``` |
| 280 |
|
| 281 |
Configure `enp1s0` as followed in `/etc/systemd/network/enp1s0.network`: |
| 282 |
|
| 283 |
``` |
| 284 |
[Match] |
| 285 |
Name=enp1s0 |
| 286 |
|
| 287 |
[Network] |
| 288 |
DHCP=yes |
| 289 |
IPv6AcceptRA=yes |
| 290 |
|
| 291 |
[DHCPv4] |
| 292 |
RouteTable=primary |
| 293 |
|
| 294 |
[IPv6AcceptRA] |
| 295 |
RouteTable=primary |
| 296 |
|
| 297 |
[RoutingPolicyRule] |
| 298 |
FirewallMark=0x100/0xff00 |
| 299 |
Family=both |
| 300 |
Table=primary |
| 301 |
Priority=40100 |
| 302 |
|
| 303 |
[RoutingPolicyRule] |
| 304 |
Family=both |
| 305 |
Table=primary |
| 306 |
Priority=41000 |
| 307 |
``` |
| 308 |
|
| 309 |
Configure `enp2s0` as followed in `/etc/systemd/network/enp2s0.network`: |
| 310 |
|
| 311 |
``` |
| 312 |
[Match] |
| 313 |
Name=enp2s0 |
| 314 |
|
| 315 |
[Network] |
| 316 |
DHCP=yes |
| 317 |
IPv6AcceptRA=yes |
| 318 |
|
| 319 |
[DHCPv4] |
| 320 |
RouteTable=secondary |
| 321 |
|
| 322 |
[IPv6AcceptRA] |
| 323 |
RouteTable=secondary |
| 324 |
|
| 325 |
[RoutingPolicyRule] |
| 326 |
FirewallMark=0x200/0xff00 |
| 327 |
Family=both |
| 328 |
Table=secondary |
| 329 |
Priority=40200 |
| 330 |
|
| 331 |
[RoutingPolicyRule] |
| 332 |
Family=both |
| 333 |
Table=secondary |
| 334 |
Priority=42000 |
| 335 |
``` |
| 336 |
|
| 337 |
After a restart of `systemd-networkd` the `ip route show table main` command should show this output: |
| 338 |
|
| 339 |
`10.0.0.0/24 dev enp8s0 proto kernel scope link src 10.0.0.1` |
| 340 |
|
| 341 |
As you can see no default route is defined in the main routing table because `systemd-networkd` added them only to our separate primary and secondary tables as defined above. This can be confirmed by checking the outputs of `ip route show table 100` and `ip route show table 200`. You should see the routes added via DHCP (`ip -6 route show table 100` for ipv6 router advertisement). |
| 342 |
|
| 343 |
`ip rule` should output something like this: |
| 344 |
|
| 345 |
``` |
| 346 |
0: from all lookup local |
| 347 |
32766: from all lookup main |
| 348 |
32767: from all lookup default |
| 349 |
40100: from all fwmark 0x100/0xff00 lookup 100 proto static |
| 350 |
40200: from all fwmark 0x200/0xff00 lookup 200 proto static |
| 351 |
41000: from all lookup 100 proto static |
| 352 |
42000: from all lookup 200 proto static |
| 353 |
``` |
| 354 |
|
| 355 |
The catchall rules `41000` and `42000` are needed so `localhost` knows where to lookup the default routes. |
| 356 |
|
| 357 |
Now all that is missing is to configure Foomuuri to use the `fwmark` rules. |
| 358 |
|
| 359 |
Example `foomuuri.conf` file: |
| 360 |
|
| 361 |
``` |
| 362 |
zone { |
| 363 |
localhost |
| 364 |
internal enp8s0 |
| 365 |
public enp1s0 enp2s0 |
| 366 |
} |
| 367 |
|
| 368 |
foomuuri { |
| 369 |
rpfilter -enp1s0 -enp2s0 |
| 370 |
} |
| 371 |
|
| 372 |
snat { |
| 373 |
oifname enp1s0 masquerade |
| 374 |
oifname enp2s0 masquerade |
| 375 |
} |
| 376 |
|
| 377 |
prerouting { |
| 378 |
mark_match -0x0000/0xff00 |
| 379 |
iifname enp1s0 mark_set 0x100/0xff00 |
| 380 |
iifname enp2s0 mark_set 0x200/0xff00 |
| 381 |
} |
| 382 |
|
| 383 |
target main { |
| 384 |
command fping --iface enp1s0 8.8.8.8 |
| 385 |
command_up /etc/foomuuri/switch.sh up |
| 386 |
command_down /etc/foomuuri/switch.sh down |
| 387 |
} |
| 388 |
|
| 389 |
zone-zone rules... |
| 390 |
``` |
| 391 |
|
| 392 |
If the `fping` monitor recognizes no uplink on the main interface `enp1s0` the following `switch.sh` script (remember to `chmod 750`) will set the default fwmark to 0x200 from the backup interface `enp2s0` by directly manipulating the nft prerouting chain. |
| 393 |
|
| 394 |
``` bash |
| 395 |
#!/bin/bash |
| 396 |
|
| 397 |
case "$1" in |
| 398 |
up) |
| 399 |
echo "main is up" |
| 400 |
|
| 401 |
FWMARK="0x00000100" |
| 402 |
FWMASK="0xffff01ff" |
| 403 |
;; |
| 404 |
down) |
| 405 |
echo "main is down" |
| 406 |
|
| 407 |
FWMARK="0x00000200" |
| 408 |
FWMASK="0xffff02ff" |
| 409 |
;; |
| 410 |
*) |
| 411 |
echo "syntax error" |
| 412 |
exit 1 |
| 413 |
esac |
| 414 |
|
| 415 |
HANDLE="$( |
| 416 |
nft --handle list chain inet foomuuri filter_prerouting_mangle | |
| 417 |
grep --only-matching --perl-regexp '^\s+meta mark set meta mark.*handle\s\K([0-9]+)$' |
| 418 |
)" |
| 419 |
|
| 420 |
if [[ $HANDLE ]] |
| 421 |
then |
| 422 |
echo "catchall fwmark rule with handle $HANDLE found" |
| 423 |
|
| 424 |
nft replace rule inet foomuuri filter_prerouting_mangle handle "$HANDLE" \ |
| 425 |
meta mark set meta mark \& "$FWMASK" \| "$FWMARK" \ |
| 426 |
ct mark set meta mark accept |
| 427 |
|
| 428 |
echo "replaced handle $HANDLE with new fwmark ${FWMARK}/${FWMASK}" |
| 429 |
else |
| 430 |
echo "no existing catchall fwmark rule found" |
| 431 |
|
| 432 |
nft add rule inet foomuuri filter_prerouting_mangle \ |
| 433 |
meta mark set meta mark \& "$FWMASK" \| "$FWMARK" \ |
| 434 |
ct mark set meta mark accept |
| 435 |
|
| 436 |
echo "added catchall fwmark ${FWMARK}/${FWMASK}" |
| 437 |
fi |
| 438 |
``` |