Debian Codegraph @

docs/example/multiple-isp.md

raw
1 # Multiple ISP
2
3 Foomuuri supports multiple ISPs (aka multi-ISP aka multi-WAN aka multiple
4 simultaneous uplink connections) with active-active (load balancing) or
5 active-passive (failover) configuration.
6
7
8 ## Configuration with static ISP routes
9
10 This example requires static ISP routes. It works with NetworkManager and
11 systemd-networkd. Both active-active and active-passive configurations are
12 supported. Both configuration types are very similar, with only one line
13 changed.
14
15 This example configuration assumes:
16
17 * ISP #1 is network interface `enp1s0`, my own IP is 172.23.70.36/24, gateway
18 is 172.23.70.254. Traffic will be marked with value 0x100 and uses route
19 table 1001.
20 * ISP #2 is network interface `enp2s0`, my own IP is 172.23.12.31/24, gateway
21 is 172.23.12.254. Traffic will be marked with value 0x200 and uses route
22 table 1002.
23 * Zone `internal` network interface is `enp8s0`, with network 10.0.0.0/8.
24 Outgoing traffic to `public` zone is masqueraded.
25
26 Example `foomuuri.conf` file:
27
28 ```
29 # Basic configuration:
30
31 zone {
32 # Define zones with interfaces
33 localhost
34 public enp1s0 enp2s0
35 internal enp8s0
36 }
37
38 foomuuri {
39 # Reverse path filtering must be disabled for public interfaces
40 rpfilter -enp1s0 -enp2s0
41 }
42
43 snat {
44 # Masquerade outgoing traffic from internal to public. Both ISPs must be
45 # masqueraded separately.
46 saddr 10.0.0.0/8 oifname enp1s0 masquerade
47 saddr 10.0.0.0/8 oifname enp2s0 masquerade
48 }
49
50 # Multi-ISP magic is here, using marks to select which ISP to use. Order
51 # of the rules is important. Specific rules should be first, generic last.
52
53 prerouting {
54 # Accept if mark is already set (not zero). Existing mark will be used.
55 mark_match -0x0000/0xff00
56
57 # == Incoming traffic ==
58
59 # Mark traffic from enp1s0 as 0x100 (ISP1) and enp2s0 as 0x200 (ISP2).
60 # This is needed for correctly routing reply packets.
61 iifname enp1s0 mark_set 0x100/0xff00
62 iifname enp2s0 mark_set 0x200/0xff00
63
64 # == Outgoing traffic ==
65
66 # Specific rules should be added first. For example, uncomment next line to
67 # route all SSH traffic from internal to public via ISP2.
68 #iifname enp8s0 ssh mark_set 0x200/0xff00
69
70 # Similarly, some source IPs can always be routed via ISP1.
71 #saddr 10.0.1.0/24 mark_set 0x100/0xff00
72
73 # For active-active configuration use following line. It uses random number
74 # generator to mark traffic with 0x100 or 0x200. This routes 60% (0-5)
75 # of outgoing traffic to ISP1 and 40% (6-9) to ISP2.
76 nft "meta mark set numgen random mod 10 map { 0-5: 0x100, 6-9: 0x200 } ct mark set meta mark accept"
77
78 # For active-passive configuration uncomment next line and add comment to
79 # above nft-line. It simply assigns mark 0x100 (ISP1) to all traffic and
80 # uses ISP2 only as fallback.
81 #mark_set 0x100/0xff00
82 }
83
84 # foomuuri-monitor config:
85
86 target isp1 {
87 # Monitor ISP1 connectivity by pinging 8.8.4.4. Ideally this would be
88 # some ISP1's router's IP address.
89 command fping --iface enp1s0 8.8.4.4
90 command_up /etc/foomuuri/multi-isp up 1
91 command_down /etc/foomuuri/multi-isp down 1
92 }
93
94 target isp2 {
95 # Monitor ISP2 connectivity by pinging their router 172.25.31.149.
96 command fping --iface enp2s0 172.25.31.149
97 command_up /etc/foomuuri/multi-isp up 2
98 command_down /etc/foomuuri/multi-isp down 2
99 }
100
101 # Normal zone-zone rules, copied from router firewall example configuration:
102
103 public-localhost {
104 ping saddr_rate "5/second burst 20"
105 ssh saddr_rate "5/minute burst 5"
106 drop log
107 }
108
109 internal-localhost {
110 dhcp-server
111 dhcpv6-server
112 domain
113 domain-s
114 ntp
115 ping
116 ssh
117 reject log
118 }
119
120 template outgoing_services {
121 # Shared list of services for localhost-public and internal-public.
122 domain
123 domain-s
124 http
125 https
126 ntp
127 ping
128 smtp
129 ssh
130 }
131
132 localhost-public {
133 template outgoing_services
134 reject log
135 }
136
137 internal-public {
138 template outgoing_services
139 googlemeet
140 imap
141 reject log
142 }
143
144 public-internal {
145 drop log
146 }
147
148 localhost-internal {
149 dhcp-client
150 dhcpv6-client
151 ping
152 ssh
153 reject log
154 }
155 ```
156
157 Example `/etc/foomuuri/multi-isp` script. Remember to save it as executable,
158 `chmod 750 /etc/foomuuri/multi-isp`.
159
160 ``` bash
161 #!/bin/sh
162
163 # Path to "ip" command
164 IP=/usr/sbin/ip
165
166 case "${1}" in
167 "start")
168 # Started, run by foomuuri-multi-isp.service
169 echo "Foomuuri multi-ISP start"
170
171 # Create new route tables for both ISPs having default route
172 ${IP} route add table 1001 default dev enp1s0 via 172.23.70.254
173 ${IP} route add table 1002 default dev enp2s0 via 172.23.12.254
174
175 # Delete default routes from main table
176 ${IP} route del table main default dev enp1s0 via 172.23.70.254
177 ${IP} route del table main default dev enp2s0 via 172.23.12.254
178
179 # Start rules with main-table lookup
180 ${IP} rule add prio 1000 from all table main
181
182 # Packets with specific source IP must go to correct ISP
183 ${IP} rule add prio 1001 from 172.23.70.36 table 1001
184 ${IP} rule add prio 1002 from 172.23.12.31 table 1002
185
186 # Other ISP-specific rules will be added by monitor's up-event.
187 ;;
188
189 "stop")
190 # Stopped, run by foomuuri-multi-isp.service
191 echo "Foomuuri multi-ISP stop"
192
193 # Add default route back to main table
194 ${IP} route add table main default dev enp1s0 via 172.23.70.254 metric 100
195 ${IP} route add table main default dev enp2s0 via 172.23.12.254 metric 101
196
197 # Delete added rules
198 ${IP} rule del prio 1000
199 ${IP} rule del prio 1001
200 ${IP} rule del prio 1002
201 ${IP} rule del prio 1101 2> /dev/null
202 ${IP} rule del prio 1102 2> /dev/null
203 ${IP} rule del prio 1201 2> /dev/null
204 ${IP} rule del prio 1202 2> /dev/null
205
206 # Delete added route tables
207 ${IP} route flush table 1001
208 ${IP} route flush table 1002
209 ;;
210
211 "up")
212 # ISP is up, add rules to route traffic to it
213 echo "Foomuuri multi-ISP isp${2} up"
214 # Use packet mark to select this ISP
215 ${IP} rule add prio 110${2} from all fwmark 0x${2}00/0xff00 table 100${2}
216 # Fallback to this ISP if other is down
217 ${IP} rule add prio 120${2} from all table 100${2}
218 ;;
219
220 "down")
221 # ISP is down, delete its rules
222 echo "Foomuuri multi-ISP isp${2} down"
223 ${IP} rule del prio 110${2}
224 ${IP} rule del prio 120${2}
225 ;;
226
227 *)
228 echo "syntax error"
229 exit 1
230 esac
231 ```
232
233 Example `/etc/systemd/system/foomuuri-multi-isp.service` file. Remember to
234 enable it with `systemctl enable foomuuri-multi-isp.service`.
235
236 ``` systemd
237 [Unit]
238 Description=Multizone bidirectional nftables firewall - Multi-ISP
239 Documentation=https://foomuuri.foobar.fi/latest/
240 After=network-online.target
241 Requires=network-online.target
242 PartOf=foomuuri.service
243
244 [Service]
245 Type=oneshot
246 RemainAfterExit=yes
247 ExecStart=/etc/foomuuri/multi-isp start
248 ExecStop=/etc/foomuuri/multi-isp stop
249
250 [Install]
251 WantedBy=multi-user.target
252 ```
253
254
255 ## Configuration with routes dynamically allocated by ISPs
256
257 This example configuration assumes:
258
259 * ISP #1 is on the network interface `enp1s0` with routes and IP addresses allocated via DHCP/IPv6RA. Traffic will be marked with value 0x100.
260 * ISP #2 is on the network interface `enp2s0` with routes and IP addresses allocated via DHCP/IPv6RA. Traffic will be marked with value 0x200.
261 * Zone `internal` network interface is `enp8s0` with IP 10.0.0.1/24. Outgoing traffic to `public` zone is masqueraded.
262 * At least the public interfaces and their routing tables will be managed by `systemd-networkd`.
263 * This example demonstrates a purely failover setup from ISP #1 to ISP #2. For load balancing add a second target monitor for `enp2s0` in `foomuuri.conf` and adjust the `switch.sh` script to identify and replace the fwmark randomizer as documented in the static routes example above.
264
265 Initiate the routing tables by creating the following two files.
266
267 `/etc/systemd/networkd.conf.d/table-primary.conf`:
268
269 ```
270 [Network]
271 RouteTable=primary:100
272 ```
273
274 `/etc/systemd/networkd.conf.d/table-secondary.conf`:
275
276 ```
277 [Network]
278 RouteTable=secondary:200
279 ```
280
281 Configure `enp1s0` as followed in `/etc/systemd/network/enp1s0.network`:
282
283 ```
284 [Match]
285 Name=enp1s0
286
287 [Network]
288 DHCP=yes
289 IPv6AcceptRA=yes
290
291 [DHCPv4]
292 RouteTable=primary
293
294 [IPv6AcceptRA]
295 RouteTable=primary
296
297 [RoutingPolicyRule]
298 FirewallMark=0x100/0xff00
299 Family=both
300 Table=primary
301 Priority=40100
302
303 [RoutingPolicyRule]
304 Family=both
305 Table=primary
306 Priority=41000
307 ```
308
309 Configure `enp2s0` as followed in `/etc/systemd/network/enp2s0.network`:
310
311 ```
312 [Match]
313 Name=enp2s0
314
315 [Network]
316 DHCP=yes
317 IPv6AcceptRA=yes
318
319 [DHCPv4]
320 RouteTable=secondary
321
322 [IPv6AcceptRA]
323 RouteTable=secondary
324
325 [RoutingPolicyRule]
326 FirewallMark=0x200/0xff00
327 Family=both
328 Table=secondary
329 Priority=40200
330
331 [RoutingPolicyRule]
332 Family=both
333 Table=secondary
334 Priority=42000
335 ```
336
337 After a restart of `systemd-networkd` the `ip route show table main` command should show this output:
338
339 `10.0.0.0/24 dev enp8s0 proto kernel scope link src 10.0.0.1`
340
341 As you can see no default route is defined in the main routing table because `systemd-networkd` added them only to our separate primary and secondary tables as defined above. This can be confirmed by checking the outputs of `ip route show table 100` and `ip route show table 200`. You should see the routes added via DHCP (`ip -6 route show table 100` for ipv6 router advertisement).
342
343 `ip rule` should output something like this:
344
345 ```
346 0: from all lookup local
347 32766: from all lookup main
348 32767: from all lookup default
349 40100: from all fwmark 0x100/0xff00 lookup 100 proto static
350 40200: from all fwmark 0x200/0xff00 lookup 200 proto static
351 41000: from all lookup 100 proto static
352 42000: from all lookup 200 proto static
353 ```
354
355 The catchall rules `41000` and `42000` are needed so `localhost` knows where to lookup the default routes.
356
357 Now all that is missing is to configure Foomuuri to use the `fwmark` rules.
358
359 Example `foomuuri.conf` file:
360
361 ```
362 zone {
363 localhost
364 internal enp8s0
365 public enp1s0 enp2s0
366 }
367
368 foomuuri {
369 rpfilter -enp1s0 -enp2s0
370 }
371
372 snat {
373 oifname enp1s0 masquerade
374 oifname enp2s0 masquerade
375 }
376
377 prerouting {
378 mark_match -0x0000/0xff00
379 iifname enp1s0 mark_set 0x100/0xff00
380 iifname enp2s0 mark_set 0x200/0xff00
381 }
382
383 target main {
384 command fping --iface enp1s0 8.8.8.8
385 command_up /etc/foomuuri/switch.sh up
386 command_down /etc/foomuuri/switch.sh down
387 }
388
389 zone-zone rules...
390 ```
391
392 If the `fping` monitor recognizes no uplink on the main interface `enp1s0` the following `switch.sh` script (remember to `chmod 750`) will set the default fwmark to 0x200 from the backup interface `enp2s0` by directly manipulating the nft prerouting chain.
393
394 ``` bash
395 #!/bin/bash
396
397 case "$1" in
398 up)
399 echo "main is up"
400
401 FWMARK="0x00000100"
402 FWMASK="0xffff01ff"
403 ;;
404 down)
405 echo "main is down"
406
407 FWMARK="0x00000200"
408 FWMASK="0xffff02ff"
409 ;;
410 *)
411 echo "syntax error"
412 exit 1
413 esac
414
415 HANDLE="$(
416 nft --handle list chain inet foomuuri filter_prerouting_mangle |
417 grep --only-matching --perl-regexp '^\s+meta mark set meta mark.*handle\s\K([0-9]+)$'
418 )"
419
420 if [[ $HANDLE ]]
421 then
422 echo "catchall fwmark rule with handle $HANDLE found"
423
424 nft replace rule inet foomuuri filter_prerouting_mangle handle "$HANDLE" \
425 meta mark set meta mark \& "$FWMASK" \| "$FWMARK" \
426 ct mark set meta mark accept
427
428 echo "replaced handle $HANDLE with new fwmark ${FWMARK}/${FWMASK}"
429 else
430 echo "no existing catchall fwmark rule found"
431
432 nft add rule inet foomuuri filter_prerouting_mangle \
433 meta mark set meta mark \& "$FWMASK" \| "$FWMARK" \
434 ct mark set meta mark accept
435
436 echo "added catchall fwmark ${FWMARK}/${FWMASK}"
437 fi
438 ```