In our last post we covered the basic setup and configuration of ExaBGP. While we were able to make use of ExaBGP for dynamic route advertisement, it wasn’t able to help us when it came to actually programming the servers routing table. In this post, I want to show you how you can leverage ExaBGP from a more programatic perspective. We’ll start by handling route advertisement to our peer and then tackle reading and processing received route updates. We’ll also start using another Python module (pyroute2) to program the routing table of the bgp_server host so that it begins acting more like a normal router. Enough talk – let’s dive in!
Im going to assume you’re starting off at the end of the last post. So the first thing we need to do is clean up a couple of items. We’re not going to rely on the static route we provisioned so to clean that up we can simply reapply the netplan network configuration using the command sudo netplan apply…
user@bgp_peer:~$ ip route
default via 192.168.127.100 dev ens3 proto static
10.10.10.0/30 dev ens7 proto kernel scope link src 10.10.10.1
10.10.10.4/30 dev ens6 proto kernel scope link src 10.10.10.5
10.10.10.12/30 via 10.10.10.6 dev ens6
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.5
user@bgp_peer:~$ sudo netplan apply
user@bgp_peer:~$ ip route
default via 192.168.127.100 dev ens3 proto static
10.10.10.0/30 dev ens7 proto kernel scope link src 10.10.10.1
10.10.10.4/30 dev ens6 proto kernel scope link src 10.10.10.5
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.5
user@bgp_peer:~$
Notice that our route to 10.10.10.12/30 is now gone. Great! So now let’s see what’s going on with our route advertisements…
user@bgp_peer:/opt/exabgp/bin$ ./exabgpcli show adj-rib out
neighbor 10.10.10.6 ipv4 unicast 10.10.10.0/30 next-hop 10.10.10.5
user@bgp_peer:/opt/exabgp/bin$
So we’re still advertising the route that we added manually. Let’s remove that so we can start with a clean slate…
user@bgp_peer:/opt/exabgp/bin$ ./exabgpcli neighbor 10.10.10.6 withdraw route 10.10.10.0/30
user@bgp_peer:/opt/exabgp/bin$ ./exabgpcli show adj-rib out
user@bgp_peer:/opt/exabgp/bin$
Cool! So now let’s talk about how we can modify the ExaBGP configuration to make things more dynamic. Before we get too deep into the weeds – let’s talk briefly about how you can programmatically interact with ExaBGP. ExaBGP has what’s called a command line API. What this boils down to is that you talk to the service through STDIN and STDOUT. So to talk to it – we will send commands to STDOUT. To read from it we’ll read from STDIN. This is covered in greater detail in the ExaBGP document aptly named READ ME FIRST.
We also need to understand ExaBGPs mode of operation. I like to think of ExaBGP as the sort of host program. That is – any code we write to send or receive route advertisements will actually be run by ExaBGP. Let me show you an example of how we might accomplish telling ExaBGP to advertise some prefixes for us. To do this, we need to modify our exabgp.conf file to look something like this…
process route-announce {
run python3 /opt/exabgp/run/exa_bgp_send.py;
encoder json;
}
neighbor 10.10.10.6 {
local-address 10.10.10.5;
local-as 65000;
peer-as 65000;
api send {
processes [route-announce];
}
}
Above we tell ExaBGP that we want it to run the Python script exa_bgp_send.py. The name route-announce is something I just came up with, it can be whatever you want it to be. So this seems pretty straight forward. Now let’s look at writing the script exa_bgp_send.py…
import sys
from time import sleep
final_prefixes = []
prefixes_to_advertise = [
{"prefix_to_advertise":"10.10.10.0/30",
"next_hop":"10.10.10.5",
}
]
for route in prefixes_to_advertise:
final_prefixes.append("announce route "+ route['prefix_to_advertise'] + " next-hop " + route['next_hop'])
# Iterate through routes
for route in final_prefixes:
sys.stderr.write(route + '\n')
sys.stderr.flush()
sys.stdout.write(route + '\n')
sys.stdout.flush()
sleep(1)
# Loop endlessly
while True:
sleep(1)
Alright – so this shouldn’t be too hard to interpret but let’s step through it just to make sure we’re all on the same page. After the imports we define an empty list called final_prefixes. Next we define another list that contains a dictionary of terms. Here we define the prefix we wish to advertise as well as the next hop a remote router would use to reach it. If you recall from our manual example in the last post, these are the two pieces of information we need in order to create the BGP route advertisement. Next – we loop through the routes that are defined (only one for now but you could have more (hence the loop)) and use the information in prefixes_to_advertise to build the actual commands that we’ll send into STDOUT for ExaBGP to use. Once this loop is completed we then loop through the route entries in the final_prefixes list and first send them to STDERR. This is just so we can see them in the log. Secondly, we send them to STDOUT so ExaBGP can see them. Lastly, we create a endless while loop so that ExaBGP doesn’t attempt to restart the script over and over again. So let’s try this out. Place the above script in /opt/exabgp/run/exa_bgp_send.py. Then all you have to do is restart the ExaBGP service with sudo systemctl restart exabgp. Now let’s see what we have going on by looking at vMX1…
[email protected]> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.10.10.5 65000 5 2 0 1 44 1/1/1/0 0/0/0/0
10.10.10.10 65001 84 84 0 0 36:02 0/1/1/0 0/0/0/0
[email protected]> show route receive-protocol bgp 10.10.10.5
inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 10.10.10.0/30 10.10.10.5 100 I
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
[email protected]>
Nice! Peering is back up and we are receiving the prefix we defined in our Python script. Let’s also take a look at our service logs on the bgp_peer server to see what’s going on there too…
user@bgp_peer:/opt/exabgp$ sudo journalctl -ru exabgp
-- Logs begin at Tue 2018-11-06 21:13:59 UTC, end at Fri 2019-02-08 21:04:04 UTC. --
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | reactor | connected to peer-1 with outgoing-1 10.10.10.5-10.10.10.6
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | api | route added to neighbor 10.10.10.6 local-ip 10.10.10.5 local-as 65000 peer-as 65000 router-id 10.10.10.5 family-allowed in-open : 10.10.10.0
Feb 08 21:01:42 bgp_peer exabgp[29448]: announce route 10.10.10.0/30 next-hop 10.10.10.5
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | reactor | loaded new configuration successfully
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | configuration | performing reload of exabgp 4.0.10-a8462350
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | cli control | to read responses /opt/exabgp/run/exabgp.out
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | cli control | to send commands /opt/exabgp/run/exabgp.in
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | cli control | named pipes for the cli are:
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | advice | generate it using "exabgp --fi > /opt/exabgp/etc/exabgp/exabgp.env"
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | advice | environment file missing
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | installation | /opt/exabgp
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | os | Linux bgp_peer 4.15.0-38-generic #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | interpreter | 3.6.6 (default, Sep 12 2018, 18:26:19) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | version | 4.0.10-a8462350
Feb 08 21:01:42 bgp_peer exabgp[29448]: 21:01:42 | 29448 | welcome | Thank you for using ExaBGP
Feb 08 21:01:42 bgp_peer systemd[1]: Started ExaBGP.
Now this is presented in reverse order so if you look at line 5 you’ll see our message we sent to STDERR and then on line 4 you’ll see ExaBGP sending the route advertisement as expected. Pretty simple right?
Now let’s move onto the slightly trickier part of processing the BGP updates that the server is receiving from vMX1. Much as we did before, we’ll start by modifying our exabgp.conf file to tell ExaBGP to run another script – this time for api receive…
process route-announce {
run python3 /opt/exabgp/run/exa_bgp_send.py;
encoder json;
}
process route-receive {
run python3 /opt/exabgp/run/exa_bgp_receive.py;
encoder json;
}
neighbor 10.10.10.6 {
local-address 10.10.10.5;
local-as 65000;
peer-as 65000;
api send {
processes [route-announce];
}
api receive {
processes [route-receive];
receive {
parsed;
update;
}
}
}
So you can see we’ve added the above highlighted lines. Same deal here – we’re just telling ExaBGP what script to run based on received routes. We’re also telling it to parse all of the BGP updates and that we want to see the BGP update messages. Now instead of jumping in all at once, let’s start our script exa_bgp_receive.py out small and then add on as we go. For now – let’s start with this…
import sys
while True:
line = sys.stdin.readline().strip()
sys.stderr.write(line + '\n')
sys.stderr.flush()
Save this in the file /opt/exabgp/run/exa_bgp_receive.py, restart ExaBGP, and check the logs…
user@bgp_peer:~$ sudo systemctl restart exabgp
user@bgp_peer:~$ sudo journalctl -ru exabgp
Feb 09 00:16:28 bgp_peer exabgp[29637]: { "exabgp": "4.0.1", "time": 1549671388.5721517, "host" : "bgp_peer", "pid" : 29637, "ppid" : 1, "counter": 1, "type": "update", "neighbor": { "address": { "local": "10.10.10.5", "peer": "10.10.10.6" }, "asn": { "local": 65000, "peer": 65000 } , "direction": "receive", "message": { "update": { "attribute": { "origin": "igp", "as-path": [ 65001 ], "confederation-path": [], "local-preference": 100 }, "announce": { "ipv4 unicast": { "10.10.10.10": [ { "nlri": "10.10.10.12/30" } ] } } } } } }
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | reactor | connected to peer-1 with outgoing-1 10.10.10.5-10.10.10.6
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | api | route added to neighbor 10.10.10.6 local-ip 10.10.10.5 local-as 65000 peer-as 65000 router-id 10.10.10.5 family-allowed in-open : 10.10.10.0/30 next-hop 10.10.10.5
Feb 09 00:16:28 bgp_peer exabgp[29637]: announce route 10.10.10.0/30 next-hop 10.10.10.5
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | reactor | loaded new configuration successfully
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | configuration | performing reload of exabgp 4.0.10-a8462350
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | cli control | to read responses /opt/exabgp/run/exabgp.out
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | cli control | to send commands /opt/exabgp/run/exabgp.in
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | cli control | named pipes for the cli are:
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | advice | generate it using "exabgp --fi > /opt/exabgp/etc/exabgp/exabgp.env"
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | advice | environment file missing
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | installation | /opt/exabgp
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | os | Linux bgp_peer 4.15.0-38-generic #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | interpreter | 3.6.6 (default, Sep 12 2018, 18:26:19) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | version | 4.0.10-a8462350
Feb 09 00:16:28 bgp_peer exabgp[29637]: 00:16:28 | 29637 | welcome | Thank you for using ExaBGP
Feb 09 00:16:27 bgp_peer systemd[1]: Started ExaBGP.
So what we’re interested in there is the highlighted line which is hard to read in this format so lets’ Json pretty print it…
{
"exabgp": "4.0.1",
"time": 1549671388.5721517,
"host": "bgp_peer",
"pid": 29637,
"ppid": 1,
"counter": 1,
"type": "update",
"neighbor": {
"address": {
"local": "10.10.10.5",
"peer": "10.10.10.6"
},
"asn": {
"local": 65000,
"peer": 65000
},
"direction": "receive",
"message": {
"update": {
"attribute": {
"origin": "igp",
"as-path": [
65001
],
"confederation-path": [],
"local-preference": 100
},
"announce": {
"ipv4 unicast": {
"10.10.10.10": [
{
"nlri": "10.10.10.12/30"
}
]
}
}
}
}
}
}
Ok – so as we can see – this is just a series of lists and dictionaries. There’s lots of data here – but we’re really after is the above highlighted lines. Line 16 tells us the peer we’re dealing with. Line 31 tells us the next hop we’re receiving for the route. Line 33 tells us the prefix being advertised to us. So let’s do some work in our script to pull these variables out. Let’s start with…
import sys
import json
while True:
line = sys.stdin.readline().strip()
json_line = json.loads(line)
neighbor_ip = json_line['neighbor']['address']['peer']
sys.stderr.write("Received a BGP update from "+neighbor_ip + '\n')
sys.stderr.flush()
Above we’re using the Python json module to convert the JSON data structures into something that Python can use – namely lists and dictionaries. If you were to check the type of json_line after you perform the json.lods() you would see it was a type dict. We can then access the values of the dictionary entries by referencing their keys. So in our case, to get the neighbor we have to dig into the neighbor, then address, then peer. The value for the key peer should be the BGP peer address we’re looking for. So save this as the script and restart ExaBGP again. If you check the logs afterwards you should see this in the log…
Feb 09 00:40:07 bgp_peer exabgp[29871]: Received a BGP update from 10.10.10.6
Great! So one value down two to go. Let’s next go after the next-hop address. Let’s try this…
import sys
import json
while True:
line = sys.stdin.readline().strip()
json_line = json.loads(line)
neighbor_ip = json_line['neighbor']['address']['peer']
for next_hop_address, nlri in json_line['neighbor']['message']['update']['announce']['ipv4 unicast'].items():
sys.stderr.write("Received a BGP update from " + neighbor_ip + '\n')
sys.stderr.write("With a next hop address of " + next_hop_address + '\n')
sys.stderr.flush()
Now this is getting a little tricky because instead of just getting a value based on a dictionary key, we need to actually get the key. To do this we create a loop and pull the key (next_hop_address) and value (nlri) at the same time. Let’s update our script again, restart ExaBGP, and check the log again…
Feb 09 00:48:48 bgp_peer exabgp[29983]: With a next hop address of 10.10.10.10
Feb 09 00:48:48 bgp_peer exabgp[29983]: Received a BGP update from 10.10.10.6
Great! Now all we need is the prefix. Let’s add a little more logic to get that…
import sys
import json
while True:
line = sys.stdin.readline().strip()
json_line = json.loads(line)
neighbor_ip = json_line['neighbor']['address']['peer']
for next_hop_address, nlri in json_line['neighbor']['message']['update']['announce']['ipv4 unicast'].items():
for prefix in nlri:
sys.stderr.write("Received a BGP update from " + neighbor_ip + '\n')
sys.stderr.write("For prefix " + prefix['nlri'] + '\n')
sys.stderr.write("With a next hop address of " + next_hop_address + '\n')
sys.stderr.flush()
So in the above case – we create another for loop to iterate through all the possible nlri. Then we simply reference the dict key we’re looking for (nlri). So if we modify our script as above, and then do our cycle of restarting the service and checking the logs we should now see output like this…
Feb 09 01:29:00 bgp_peer exabgp[30106]: With a next hop address of 10.10.10.10
Feb 09 01:29:00 bgp_peer exabgp[30106]: For prefix 10.10.10.12/30
Feb 09 01:29:00 bgp_peer exabgp[30106]: Received a BGP update from 10.10.10.6
Awesome! So now we have all the information we need to create a route. Well – actually we have more than we need (we don’t need to know the BGP neighbor IP but I pulled it out for a reason (you’ll see why soon)). So let’s talk about adding a route to the routing table of the server based on this information.
There’s a handy Python library called pyroute2 which we can use to modify the Linux routing table. The easiest way to install it will be with pip3 which you can install with the command sudo apt-get install python3-pip if you don’t already have it on your system. Once installed, we can install the module as follows…
user@bgp_peer:~$ sudo su
root@bgp_peer:/home/user# pip3 install pyroute2
Collecting pyroute2
Downloading https://files.pythonhosted.org/packages/19/1c/fc2efd9dfa5c1f7f713dbfee621365ad01f7b7ec0a6908e444f08bfd5873/pyroute2-0.5.3.tar.gz (687kB)
100% |████████████████████████████████| 696kB 1.8MB/s
Building wheels for collected packages: pyroute2
Running setup.py bdist_wheel for pyroute2 ... done
Stored in directory: /root/.cache/pip/wheels/3a/8d/19/a02a0af7742b48122d4ee7aa1f5ae9fa4833ec6043fc74052b
Successfully built pyroute2
Installing collected packages: pyroute2
Successfully installed pyroute2-0.5.3
root@bgp_peer:/home/user# exit
exit
user@bgp_peer:~$
Notice that I switched to the root user to install the package. This should make the module available for all users and is required since we’re running our Python code through a systemd service. So now that we have it installed, let’s use it in our code…
import sys
import json
from pyroute2 import IPRoute
ip = IPRoute()
while True:
line = sys.stdin.readline().strip()
json_line = json.loads(line)
neighbor_ip = json_line['neighbor']['address']['peer']
for next_hop_address, nlri in json_line['neighbor']['message']['update']['announce']['ipv4 unicast'].items():
for prefix in nlri:
sys.stderr.write("Received a BGP update from " + neighbor_ip + '\n')
sys.stderr.write("For prefix " + prefix['nlri'] + '\n')
sys.stderr.write("With a next hop address of " + next_hop_address + '\n')
sys.stderr.flush()
ip.route("add", dst=prefix['nlri'], gateway=next_hop_address)
Alright – so same deal – update your script, restart ExaBGP, and check the logs…
-- Logs begin at Tue 2018-11-06 21:13:59 UTC, end at Sat 2019-02-09 01:39:14 UTC. --
Feb 09 01:39:07 bgp_peer exabgp[4237]: pyroute2.netlink.exceptions.NetlinkError: (1, 'Operation not permitted')
Feb 09 01:39:07 bgp_peer exabgp[4237]: raise msg['header']['error']
Feb 09 01:39:07 bgp_peer exabgp[4237]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 679, in get
Feb 09 01:39:07 bgp_peer exabgp[4237]: return tuple(self._genlm_get(*argv, **kwarg))
Feb 09 01:39:07 bgp_peer exabgp[4237]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 355, in get
Feb 09 01:39:07 bgp_peer exabgp[4237]: callback=callback):
Feb 09 01:39:07 bgp_peer exabgp[4237]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 836, in nlm_request
Feb 09 01:39:07 bgp_peer exabgp[4237]: return tuple(self._genlm_request(*argv, **kwarg))
Feb 09 01:39:07 bgp_peer exabgp[4237]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 352, in nlm_request
Feb 09 01:39:07 bgp_peer exabgp[4237]: callback=callback)
Feb 09 01:39:07 bgp_peer exabgp[4237]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/iproute/linux.py", line 1561, in route
Feb 09 01:39:07 bgp_peer exabgp[4237]: ip.route("add", dst=prefix['nlri'], gateway=next_hop_address)
Feb 09 01:39:07 bgp_peer exabgp[4237]: File "/opt/exabgp/run/exa_bgp_receive.py", line 17, in <module>
Feb 09 01:39:07 bgp_peer exabgp[4237]: Traceback (most recent call last):
Feb 09 01:39:07 bgp_peer exabgp[4237]: With a next hop address of 10.10.10.10
Feb 09 01:39:07 bgp_peer exabgp[4237]: For prefix 10.10.10.12/30
Feb 09 01:39:07 bgp_peer exabgp[4237]: Received a BGP update from 10.10.10.6
Whelp – that didn’t work. If you look at the highlighted line above – you’ll see we’re having what looks like a permissions issue. Thinking about this some more – it seems likely that our Python script doesn’t have the permission it needs to modify the Linux routing table. While the “easy” fix here would be to just run this as the root user – I’d like to try a different approach. Let’s instead give the Python3 executable just the permissions it needs to work on the routing table. To do this, we can grant it a POSIX capability – specifically the CAP_NET_ADMIN capability. To do this – we simply run this command…
sudo setcap cap_net_raw,cap_net_admin=eip /usr/bin/python3.6
Note: If you look at which Python3 you’re using you’ll likely see that it doesn’t line up with the path I provide above. Look closer though – and it’s probably a symlink…
user@bgp_peer:~$ which python3
/usr/bin/python3
user@bgp_peer:~$ ls -al /usr/bin | grep python3
lrwxrwxrwx 1 root root 9 Oct 25 11:11 python3 -> python3.6
<some output excluded>
Alright – so let’s try that again. Restart the service and check the logs…
-- Logs begin at Tue 2018-11-06 21:13:59 UTC, end at Sat 2019-02-09 01:47:23 UTC. --
Feb 09 01:47:21 bgp_peer exabgp[17455]: pyroute2.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
Feb 09 01:47:21 bgp_peer exabgp[17455]: raise msg['header']['error']
Feb 09 01:47:21 bgp_peer exabgp[17455]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 679, in get
Feb 09 01:47:21 bgp_peer exabgp[17455]: return tuple(self._genlm_get(*argv, **kwarg))
Feb 09 01:47:21 bgp_peer exabgp[17455]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 355, in get
Feb 09 01:47:21 bgp_peer exabgp[17455]: callback=callback):
Feb 09 01:47:21 bgp_peer exabgp[17455]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 836, in nlm_request
Feb 09 01:47:21 bgp_peer exabgp[17455]: return tuple(self._genlm_request(*argv, **kwarg))
Feb 09 01:47:21 bgp_peer exabgp[17455]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 352, in nlm_request
Feb 09 01:47:21 bgp_peer exabgp[17455]: callback=callback)
Feb 09 01:47:21 bgp_peer exabgp[17455]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/iproute/linux.py", line 1561, in route
Feb 09 01:47:21 bgp_peer exabgp[17455]: ip.route("add", dst=prefix['nlri'], gateway=next_hop_address)
Feb 09 01:47:21 bgp_peer exabgp[17455]: File "/opt/exabgp/run/exa_bgp_receive.py", line 17, in <module>
Feb 09 01:47:21 bgp_peer exabgp[17455]: Traceback (most recent call last):
Feb 09 01:47:21 bgp_peer exabgp[17455]: With a next hop address of 10.10.10.10
Feb 09 01:47:21 bgp_peer exabgp[17455]: For prefix 10.10.10.12/30
Feb 09 01:47:21 bgp_peer exabgp[17455]: Received a BGP update from 10.10.10.6
GAHH. Now what? We’re getting the message Network is unreachable. So let’s think about this for a second. We learned a prefix (10.10.10.12/30), from a directly connected BGP peer (10.10.10.6), with a next hop of (10.10.10.10). Seems wrong right? Let us consult the diagram!
Yeah. So that ain’t right. The network engineers reading this likely picked up on the problem. BGP has that neat property where it can advertise routes with a next hop that’s not directly connected. Since the BGP advertisement is coming from vMX2 – the next hop is that of vMX2. Out server has no idea about that link so it’s kindly telling us that it has no idea. A general rule of thumb is for the AS boundary routers to set next-hops self on the prefix they advertise further into the AS. Let’s fix that on vMX1…
set policy-options policy-statement ibgp_route term nhs then next-hop self
set protocols bgp group internal export ibgp_route
Once added, we can restart ExaBGP again and see what we get…
Feb 09 01:53:37 bgp_peer exabgp[17455]: With a next hop address of 10.10.10.6
Feb 09 01:53:37 bgp_peer exabgp[17455]: For prefix 10.10.10.12/30
Feb 09 01:53:37 bgp_peer exabgp[17455]: Received a BGP update from 10.10.10.6
Logs look good! Let’s check the servers routing table…
user@bgp_peer:~$ ip route
default via 192.168.127.100 dev ens3 proto static
10.10.10.0/30 dev ens7 proto kernel scope link src 10.10.10.1
10.10.10.4/30 dev ens6 proto kernel scope link src 10.10.10.5
10.10.10.12/30 via 10.10.10.6 dev ens6 proto static
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.5
user@bgp_peer:~$
Yes! Success! Although – since we’re clearly working through this together – let me point out another problem we’re going to have. Restart the ExaBGP service one more time…
-- Logs begin at Tue 2018-11-06 21:13:59 UTC, end at Sat 2019-02-09 01:56:08 UTC. --
Feb 09 01:56:02 bgp_peer exabgp[29939]: pyroute2.netlink.exceptions.NetlinkError: (17, 'File exists')
Feb 09 01:56:02 bgp_peer exabgp[29939]: raise msg['header']['error']
Feb 09 01:56:02 bgp_peer exabgp[29939]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 679, in get
Feb 09 01:56:02 bgp_peer exabgp[29939]: return tuple(self._genlm_get(*argv, **kwarg))
Feb 09 01:56:02 bgp_peer exabgp[29939]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 355, in get
Feb 09 01:56:02 bgp_peer exabgp[29939]: callback=callback):
Feb 09 01:56:02 bgp_peer exabgp[29939]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 836, in nlm_request
Feb 09 01:56:02 bgp_peer exabgp[29939]: return tuple(self._genlm_request(*argv, **kwarg))
Feb 09 01:56:02 bgp_peer exabgp[29939]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/netlink/nlsocket.py", line 352, in nlm_request
Feb 09 01:56:02 bgp_peer exabgp[29939]: callback=callback)
Feb 09 01:56:02 bgp_peer exabgp[29939]: File "/usr/local/lib/python3.6/dist-packages/pyroute2/iproute/linux.py", line 1561, in route
Feb 09 01:56:02 bgp_peer exabgp[29939]: ip.route("add", dst=prefix['nlri'], gateway=next_hop_address)
Feb 09 01:56:02 bgp_peer exabgp[29939]: File "/opt/exabgp/run/exa_bgp_receive.py", line 17, in <module>
Feb 09 01:56:02 bgp_peer exabgp[29939]: Traceback (most recent call last):
Feb 09 01:56:02 bgp_peer exabgp[29939]: With a next hop address of 10.10.10.6
Feb 09 01:56:02 bgp_peer exabgp[29939]: For prefix 10.10.10.12/30
Feb 09 01:56:02 bgp_peer exabgp[29939]: Received a BGP update from 10.10.10.6
Sigh. Yeah – well we should have seen this coming. The server can’t add the same route twice. There are a few ways to deal with this – but since Im spending more time with pyroute2, I’m going to take this approach. First – go ahead and manually delete the route on the server…
sudo ip route del 10.10.10.12/30
Next – change your command to add the route to look like this…
ip.route("add", dst=prefix['nlri'], gateway=next_hop_address, priority=123)
Notice we added the priority parameter. The priority of 123 is just a made up number – you can pick anything you want. I’m going to use the priority as a means to filter the routes. Since I don’t actually care about the static route priority – seeing routes with a priority of 123 will indicate to me that they were added by this service. That also means they can be taken away by this service. So let’s change our script one more time…
import sys
import json
from pyroute2 import IPRoute
ip = IPRoute()
for route in ip.get_routes():
if route.get_attr('RTA_PRIORITY') == 123:
prefix = route.get_attr('RTA_DST')+"/"+str(route["dst_len"])
ip.route("delete", dst=prefix, gateway=route.get_attr('RTA_GATEWAY'), priority=route.get_attr('RTA_PRIORITY'))
while True:
line = sys.stdin.readline().strip()
json_line = json.loads(line)
neighbor_ip = json_line['neighbor']['address']['peer']
for next_hop_address, nlri in json_line['neighbor']['message']['update']['announce']['ipv4 unicast'].items():
for prefix in nlri:
sys.stderr.write("Received a BGP update from " + neighbor_ip + '\n')
sys.stderr.write("For prefix " + prefix['nlri'] + '\n')
sys.stderr.write("With a next hop address of " + next_hop_address + '\n')
sys.stderr.flush()
ip.route("add", dst=prefix['nlri'], gateway=next_hop_address, priority=123)
Notice the addition of the highlighted lines. These lines of code search for existing routes with a priority of 123 and delete them. The logic here is that when ExaBGP starts up we want a clean slate. So first go through and clean up any old routes that we learned. Then – enter our endless while loop listening for BGP updates and add them as you get them.
So once again – update your script, restart the service, and check your logs…
Feb 09 02:01:33 bgp_peer exabgp[1231]: With a next hop address of 10.10.10.6
Feb 09 02:01:33 bgp_peer exabgp[1231]: For prefix 10.10.10.12/30
Feb 09 02:01:33 bgp_peer exabgp[1231]: Received a BGP update from 10.10.10.6
Still looks good! Routing table checks out too…
user@bgp_peer:~$ ip route
default via 192.168.127.100 dev ens3 proto static
10.10.10.0/30 dev ens7 proto kernel scope link src 10.10.10.1
10.10.10.4/30 dev ens6 proto kernel scope link src 10.10.10.5
10.10.10.12/30 via 10.10.10.6 dev ens6 proto static metric 123
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.5
user@bgp_peer:~$
So at this point – ExaBGP should be in a position to add routes for any new prefixes it learns. Our ping from left_user to right_user should also be working as expected…
root@lab-box:~# ip netns exec left_user ping 10.10.10.14 -c 2
PING 10.10.10.14 (10.10.10.14) 56(84) bytes of data.
64 bytes from 10.10.10.14: icmp_seq=1 ttl=61 time=1.31 ms
64 bytes from 10.10.10.14: icmp_seq=2 ttl=61 time=1.18 ms
--- 10.10.10.14 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.185/1.251/1.317/0.066 ms
root@lab-box:~#
To test this out further – let’s add another IP address on right_user and advertise it through the network…
root@lab-box:~# ip netns exec right_user ip link add dummy0 type dummy
root@lab-box:~# ip netns exec right_user ip link set dev dummy0 up
root@lab-box:~# ip netns exec right_user ip addr add 9.9.9.9/32 dev dummy0
Let’s make sure that the left_user can’t reach it..
root@lab-box:~# ip netns exec left_user ping 9.9.9.9 -c 2
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
--- 9.9.9.9 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
Great! So now let’s add a static route on vMX2 pointing 9.9.9.9/32 at the interface of right_user and redistribute static routes into BGP…
set routing-options static route 9.9.9.9/32 next-hop 10.10.10.14
set policy-options policy-statement redistribute_connected term static from protocol static
set policy-options policy-statement redistribute_connected term static then accept
The instant we commit this – we should see the route pop into the routing table on the ExaBGP server…
user@bgp_peer:~$ ip route
default via 192.168.127.100 dev ens3 proto static
9.9.9.9 via 10.10.10.6 dev ens6 proto static metric 123
10.10.10.0/30 dev ens7 proto kernel scope link src 10.10.10.1
10.10.10.4/30 dev ens6 proto kernel scope link src 10.10.10.5
10.10.10.12/30 via 10.10.10.6 dev ens6 proto static metric 123
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.5
user@bgp_peer:~$
Nice! And our ping from left_user to 9.9.9.9 should now be working…
root@lab-box:~# ip netns exec left-user ping 9.9.9.9 -c 2
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=61 time=1.68 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=61 time=1.38 ms
--- 9.9.9.9 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.389/1.536/1.683/0.147 ms
root@lab-box:~#
Again – we’ve just scratched the surface of what we can do here programmatically. I also hope you’re thinking about the possible pitfalls of this approach and possible solutions for them. For instance – what happens if we receive a withdrawal? In the next post we’ll extend our code even further to deal with some more of these use cases. Stay tuned!
This is a very nice integration of exaBGP and Python based Linux routing tools. I employed exaBGP as the basis of a route server management portal, with Kubernetes orchestrated containerized application components, *for lab-only demonstration purposes* (http://cloudfulness.net). This was to help teach new cloud-native programmers how to program for cloud applications. Using a route server/route injection portal as the basis for a cloud based portal app seemed kind of silly at the time, but being a networking guy, networking topics were pretty familiar. But maybe not so silly. I’m updating the app so that the Python base program communicates with an exaBGP route server cluster indirectly via RabbitMQ fan-out messages and ExaBGP(s) communicates with quaggaBGP routers indirectly via Direct RabbitMQ messaging. In this fashion, cloud routing will be based on standard microservice communication. For small to medium sized cloud customers, this seems actually workable. Its likely already being applied in some similar fashion in AWS or Azure, but in a much more developed fashion than my simple lab setup.