Round-robin load-balancing between two VPNs with iptables and policy-based routing
Posted on June 9, 2024 • 6 minutes • 1190 words
Table of contents
Experiment 1
We want to send http traffik generated by our computer through a VPN tunnel.
VPN tunnels
Create two VPN tunnels:
openvpn --config au.ovpn
openvpn --config at.ovpn
Two new interfaces will be created: tun0 and tun1.
Two configs should prevent setting of default gateways:
pull-filter ignore redirect-gateway
Routing
Create a new routing table:
echo "201 vpn1" >> /etc/iproute2/rt_tables
and add default routing to the table:
ip route add default dev tun1 table vpn1
Check:
ip route show table vpn1
Now, let’s add a rule:
ip rule add from all fwmark 1 lookup vpn1
The rule says that packages with mark 0x1 should be routed using table vpn1. Effectively, that means, the packages will go to tun1 interface.
Marking the packages
iptables -t mangle -A OUTPUT -p tcp --dport 80 -j MARK --set-mark 1
Testing
Let’s try to access some site on port 80:
curl ipinfo.io
The command times out. So what’s a problem? The packages are indeed marked with 0x1 and indeed go through tun1 but they have as source IP not tun1’s IP but IP of eth0 on which curl is bound and sends the request. So the packages with SYN flag go through tun1 interface but ACK packages from the destination are sent to eth0 IP.
Correction
We should somehow to change source IP of the packages to the IP of tun1 and when they are back at tun1 we need to send them back to eth0 on which curl communicates. But that’s exactly what NAT MASQUARADE exists for. So, we set tun1 into that mode:
iptables -t nat -A POSTROUTING -o tun1 -j MASQUERADE
Bingo!
curl ipinfo.io
works through tun1.
Experiment 2
Connections from port 4444 send through vpn1
Marking packets
Remove previous rules on mangle table:
iptables -t mangle -F OUTPUT
and set new:
iptables -t mangle -A OUTPUT -p tcp --sport 4444 -j MARK --set-mark 1
Because curl does not allow to specify source port in its --interface option, we’ll use nc:
echo -e "GET / HTTP/1.1\r\nHost: ipinfo.io\r\nConnection: close\r\n\r\n" | nc ipinfo.io 80 -p 4444
Ok, we get through vpn1. If we remove -p 4444 command line option, we go through our default interface, not through vpn1.
Note: nat rule on vpn1 is still there. It’s a must requirement.
Randomazing traffik between two VPNs
Routing
We create two routing tables:
cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
100 custom_table
200 vpn0
201 vpn1
We create default routes in the above tables:
ip route add default tun1 table vpn1
ip route add default tun0 table vpn0
Check:
ip route show table vpn1
default dev tun1 scope link
ip route show table vpn0
default dev tun0 scope link
We create two rules:
ip rule add from all fwmark 2 lookup vpn0
ip rule add from all fwmark 1 lookup vpn1
Check:
ip rule
0: from all lookup local
32760: from all fwmark 0x2 lookup vpn0
32761: from all fwmark 0x1 lookup vpn1
32766: from all lookup main
32767: from all lookup default
Masqurading on VPN interfaces
We should activate nat masquarading on both vpns:
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o tun1 -j MASQUERADE
Prevent killing VPN connections
We are going to put all traffik through VPNs, not only to port 80 or from port 4444. Therefore we need to prevent sending VPN client connections to VPN servers:
iptables -t mangle -A OUTPUT -d 146.70.116.194 -j RETURN
iptables -t mangle -A OUTPUT -d 103.108.231.98 -j RETURN
Marking packets randomly, only those which start connections
The next we want to randomly mark packets. If we just mark ALL packets randomly then we destroy the connection. Suppose, SYN packet was marked 1 and sent to tun1. The reponse with SYN-ACK was delivered to the sending interface ETH1 from VPN but now, ACK package from ETH1 is marked 2 and send to tun0. Thus connection is never established. It was not a case when we had only one VPN interface and all packets were sent to the same VPN interface.
Therefore, we mark only packets which start a connection:
iptables -t mangle -A OUTPUT -m conntrack --ctstate NEW -m statistic --mode random --probability 0.5 -j MARK --set-mark 2
The above marks 50% of connection starting packets. Then we match non-marked packets which start a connection and mark them with 1:
iptables -t mangle -A OUTPUT -m conntrack --ctstate NEW -m mark --mark 0 -j MARK --set-mark 1
Now we want to preserve the marks we gave for all the packets in the connection a marked packet started:
iptables -t mangle -A OUTPUT -m conntrack --ctstate NEW -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
The above rules save the mark of a new connection and restore that mark for all the packets in the established connection. The same mark in all packets of the connection guarantees all the packets arrive at the same VPN interface.
Summary
Masquarade:
iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
148 13070 MASQUERADE all -- * tun1 0.0.0.0/0 0.0.0.0/0
203 17720 MASQUERADE all -- * tun0 0.0.0.0/0 0.0.0.0/0
Mangle:
iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
7839 1001K RETURN tcp -- * * 0.0.0.0/0 146.70.116.194
55156 8718K RETURN tcp -- * * 0.0.0.0/0 103.108.231.98
377 32241 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW statistic mode random probability 0.50000000000 MARK set 0x2
274 30364 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW mark match 0x0 MARK set 0x1
493 48897 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW CONNMARK save
1658 224K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Routing tables:
cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
100 custom_table
200 vpn0
201 vpn1
Routing table routes:
ip route show table vpn1
default dev tun1 scope link
ip route show table vpn0
default dev tun0 scope link
Routing rules:
ip rule
0: from all lookup local
32760: from all fwmark 0x2 lookup vpn0
32761: from all fwmark 0x1 lookup vpn1
32766: from all lookup main
32767: from all lookup default
References
- https://datahacker.blog/industry/technology-menu/networking/routes-and-rules/iproute-and-routing-tables
- https://www.frozentux.net/iptables-tutorial/chunkyhtml/c962.html#TABLE.FORWARDEDPACKETS
- https://book.huihoo.com/iptables-tutorial/x9125.htm
Share
Tags
Counters
