One of my goals this year was to spend more time doing Python development so I thought I’d take a break from the MPLS deep dives (no worries – there are still lots more coming) and insert some development type things along the way. One of the opensource projects I’ve used in the past was ExaBGP by the folks over at Exa-Networks. Since then – they’ve released a new version (4) and I’ve been keen to play around with it some more.
The last time I played with ExaBGP was strictly from a testing perspective. This time – I want to focus on some use cases that are closer to real life. Things you might actually do – or at the very least – try. So to start things off – we’re going to once again start with a basic lab that looks like this…
So here we have our 2 friendly end users connected at the head to a small simple network. left_user is attached to an Ubuntu 18 server called ‘bgp_peer’ which is then connected to two other routers (Juniper vMX) and finally our friendly right_user. To start with – we’re going to configure the routers vMX1 and vMX2. The base configuration looks like this…
vMX1 Base Configuration….
system {
host-name vmx1.lab;
}
interfaces {
ge-0/0/0 {
unit 0 {
family inet {
address 10.10.10.6/30;
}
}
}
ge-0/0/1 {
unit 0 {
family inet {
address 10.10.10.9/30;
}
}
}
lo0 {
unit 0 {
family inet {
no-redirects;
address 1.1.1.1/32 {
primary;
}
}
}
}
}
routing-options {
static {
route 10.171.200.0/22 next-hop 192.168.127.100;
}
router-id 1.1.1.1;
autonomous-system 65000;
}
protocols {
bgp {
group external {
type external;
peer-as 65001;
neighbor 10.10.10.10;
}
group internal {
type internal;
peer-as 65000;
neighbor 10.10.10.5;
}
}
}
vMX2 Base Configuration…
system {
host-name vmx2.lab;
}
interfaces {
ge-0/0/0 {
unit 0 {
family inet {
address 10.10.10.10/30;
}
}
}
ge-0/0/1 {
unit 0 {
family inet {
address 10.10.10.13/30;
}
}
} lo0 {
unit 0 {
family inet {
no-redirects;
address 2.2.2.2/32 {
primary;
}
}
}
}
}
routing-options {
static {
route 10.171.200.0/22 next-hop 192.168.127.100;
}
router-id 2.2.2.2;
autonomous-system 65001;
}
protocols {
bgp {
group external {
export redistribute_connected;
peer-as 65000;
neighbor 10.10.10.9;
}
}
lldp {
port-id-subtype interface-name;
interface all;
}
lldp-med {
interface all;
}
}
policy-options {
prefix-list pfl_ge001 {
10.10.10.12/30;
}
policy-statement redistribute_connected {
term allow_ge001 {
from {
protocol direct;
prefix-list pfl_ge001;
}
then accept;
}
then reject;
}
}
Note: I’ve removed the obvious auth and other config pieces from the above configs so we can focus on just the routing pieces. I’ve also removed the fxp0 or management interface config so later on in the route outputs don’t be surprised if you see routes for 192.168.127.0/24
So there’s nothing terribly fancy here. vMX1 lives in AS 65000 and vMX2 lives in AS 65001. We then eBGP peer them together on their interface IP addresses. The only remotely fancy bit of config is the policy-statements/options to allow vMX2 to redistribute it’s network connected to right_user into BGP. So from a BGP perspective – you might say our diagram looks like this…
Before we get too crazy – let’s validate what routing looks like on each device. On the server bgp_peer, the Linux routing table looks like this…
user@bgp_peer:~$ ip route
default via 192.168.127.100 dev ens3 proto static
10.10.10.0/30 dev ens7 proto kernel scope link src 10.10.10.1
10.10.10.4/30 dev ens6 proto kernel scope link src 10.10.10.5
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.5
user@bgp_peer:~$
Note: Im using the network 192.168.127.0/24 as a management network to get to the devices. Ens3 on bgp_peer is connected to this network but not shown on the diagrams.
So just as we’d expect here. Beyond the management interface – the server sees it’s two directly connected interface networks but that’s about it. If we look at the inet.0
table on vMX1…
[email protected]> show route table inet.0
inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[Direct/0] 00:16:10
> via lo0.0
10.10.10.4/30 *[Direct/0] 00:13:51
> via ge-0/0/0.0
10.10.10.6/32 *[Local/0] 00:13:51
Local via ge-0/0/0.0
10.10.10.8/30 *[Direct/0] 00:13:51
> via ge-0/0/1.0
10.10.10.9/32 *[Local/0] 00:13:51
Local via ge-0/0/1.0
10.10.10.12/30 *[BGP/170] 00:09:24, localpref 100
AS path: 65001 I, validation-state: unverified
> to 10.10.10.10 via ge-0/0/1.0
10.171.200.0/22 *[Static/5] 00:16:10
> to 192.168.127.100 via fxp0.0
192.168.127.0/24 *[Direct/0] 00:16:10
> via fxp0.0
192.168.127.1/32 *[Local/0] 00:16:10
Local via fxp0.0
[email protected]>
Again – a bunch of directly connected interfaces and their networks. However – we do see the 10.10.10.12/30
network that we’re receiving from vMX2 through BGP as well. Looking at vMX2’s routing table we see…
[email protected]> show route table inet.0
inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2.2.2.2/32 *[Direct/0] 00:17:39
> via lo0.0
10.10.10.8/30 *[Direct/0] 00:14:07
> via ge-0/0/0.0
10.10.10.10/32 *[Local/0] 00:14:07
Local via ge-0/0/0.0
10.10.10.12/30 *[Direct/0] 00:14:07
> via ge-0/0/1.0
10.10.10.13/32 *[Local/0] 00:14:07
Local via ge-0/0/1.0
10.171.200.0/22 *[Static/5] 00:17:39
> to 192.168.127.100 via fxp0.0
192.168.127.0/24 *[Direct/0] 00:17:39
> via fxp0.0
192.168.127.2/32 *[Local/0] 00:17:39
Local via fxp0.0
[email protected]>
Just directly connected networks. Nothing exciting. So let’s get on with setting up ExaBGP!
The first thing we’re going to do is to use git
and download the code. We do that with this command…
sudo git clone https://github.com/Exa-Networks/exabgp.git /opt/exabgp
Once cloned – you’re /opt/exabgp
directory should look something like this…
user@bgp_peer:/opt/exabgp$ ls
CHANGELOG ISSUE_TEMPLATE.md PYPI.rst bin dev etc logo redhat run service
COPYRIGHT PEP8 README.md debian doc lib qa requirements.txt sbin setup.py
user@bgp_peer:/opt/exabgp$
Nice! Now when running ExaBGP – you have a few options. You can run it directly (in a number of ways) or you can have the system run it. In this case – Im going to go that route and have Ubuntu run it as a system service. Since Ubuntu 18 leverages systemd, we need to define a unit file for the service. ExaBGP provides a nice sample, but I found that I needed to extend it in order to get all the features I wanted to work. Mine ended up looking like this…
[Unit]
Description=ExaBGP
Documentation=man:exabgp(1)
Documentation=man:exabgp.conf(5)
Documentation=https://github.com/Exa-Networks/exabgp/wiki
After=network.target
ConditionPathExists=/etc/exabgp/exabgp.conf
[Service]
Environment=exabgp_daemon_daemonize=false
ExecStartPre=-/bin/mkdir /opt/exabgp/run
ExecStartPre=/bin/bash -c "/usr/bin/mkfifo -m 0666 /opt/exabgp/run/exabgp.{in,out}"
ExecStopPost=/bin/bash -c "/bin/rm -f /opt/exabgp/run/exabgp.{in,out}"
ExecStart=/opt/exabgp/sbin/exabgp /etc/exabgp/exabgp.conf
ExecReload=/bin/kill -USR1 $MAINPID
[Install]
WantedBy=multi-user.target
Let’s not dwell too much on this just yet – so for now – just define this as /lib/systemd/system/exabgp.service
. Once saved in that location, we can now define a simple ExaBGP configuration…
neighbor 10.10.10.6 {
router-id 9.9.9.9;
local-address 10.10.10.5;
local-as 65000;
peer-as 65000;
}
This is about as simple as it gets. Here we tell ExaBGP that we want to connect to a neighbor (10.10.10.6) using a local address of 10.10.10.5. We also imply this to be an iBGP peering by setting the peer and local AS to 65000. Let’s plop this in /etc/exabgp/exabgp.conf
.
Note: If that directory structure doesn’t exist just make it
And that’s it! Now all we need to do is to tell the server we updated a systemd unit file…
sudo systemctl daemon-reload
And start the ExaBGP service…
sudo systemctl start exabgp
Now let’s validate that it’s running…
user@bgp_peer:/opt/exabgp$ systemctl status exabgp
● exabgp.service - ExaBGP
Loaded: loaded (/lib/systemd/system/exabgp.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2019-02-07 02:48:58 UTC; 4s ago
Docs: man:exabgp(1)
man:exabgp.conf(5)
https://github.com/Exa-Networks/exabgp/wiki
Process: 28964 ExecStopPost=/bin/bash -c /bin/rm -f /opt/exabgp/run/exabgp.{in,out} (code=exited, status=0/SUCCESS)
Process: 28985 ExecStartPre=/bin/bash -c /usr/bin/mkfifo -m 0666 /opt/exabgp/run/exabgp.{in,out} (code=exited, status=0/SUCCESS)
Process: 28979 ExecStartPre=/bin/mkdir /opt/exabgp/run (code=exited, status=0/SUCCESS)
Main PID: 28988 (python3)
Tasks: 2 (limit: 4684)
CGroup: /system.slice/exabgp.service
├─28988 /usr/bin/python3 /opt/exabgp/lib/exabgp/application/bgp.py --root /opt/exabgp /etc/exabgp/exabgp.conf
└─29006 /usr/bin/python3 /opt/exabgp/lib/exabgp/application/bgp.py
user@bgp_peer:/opt/exabgp$
Success! And if you look at the BGP summary on vMX1 – you should see the peer up…
[email protected]> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.10.10.5 65000 4 3 0 2 42 0/0/0/0 0/0/0/0
10.10.10.10 65001 57 54 0 0 23:59 1/1/1/0 0/0/0/0
[email protected]>
So you’ll probably notice that vMX1 isn’t receiving any routes from ExaBGP yet. This makes sense since we haven’t told ExaBGP what to advertise yet. One of the neat things that came with ExaBGP 4 is exabgpcli
. So while the goal in later posts will be to programmatically send and receive BGP routes with ExaBGP – for now we can do it manually just to get our feet wet. You should be able to find exabgpcli
in the /opt/exabgp/bin
directory…
user@bgp_peer:/opt/exabgp/bin$ ls
exabgpcli exanetlink healthcheck
user@bgp_peer:/opt/exabgp/bin$
Let’s try checking the neighbor status…
user@bgp_peer:/opt/exabgp/bin$ ./exabgpcli show neighbor summary
Peer AS up/down state | #sent #recvd
10.10.10.6 65000 0:03:48 established 1 1
user@bgp_peer:/opt/exabgp/bin$
Pretty cool huh? So the next logical step here is to advertise the 10.10.10.0/30 network to vMX1 so he can tell vMX2 and so left_user can talk to right_user. But for those of you familiar with ExaBGP – you’ll know it isn’t that simple. ExaBGP only provides the BGP component – there is no integration with the Linux routing table. So that means we need to solve that problem ourselves (we will! (in a later post)). So for now – that means putting a static route on the bgp_peer server. Since left_user uses the bgp_peer server as it’s default gateway, and right_user uses vMX2 as it’s default gateway – we only really need to tell bgp_server about one route…
sudo ip route add 10.10.10.12/30 via 10.10.10.6
We also need to tell the server it’s OK to act like a router…
sudo sysctl -w net.ipv4.ip_forward=1
Cool – so at this point – left_user still can’t ping right_user…
root@lab-box:~# ip netns exec left_user ping 10.10.10.14
PING 10.10.10.14 (10.10.10.14) 56(84) bytes of data.
^C
--- 10.10.10.14 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
root@lab-box:~#
Note: Yes – Im using NetNS again to simulate the users.
And this is because, as we saw earlier, vMX1 and vMX2 have no idea about the 10.10.10.0/30
network. Let’s fix that by adverting that prefix to vMX1 with ExaBGP…
./exabgpcli neighbor 10.10.10.6 announce route 10.10.10.0/30 next-hop 10.10.10.5
Pretty simply syntax to understand. Now if we look on vMX1 – we should see a received route…
[email protected]> show route receive-protocol bgp 10.10.10.5
inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 10.10.10.0/30 10.10.10.5 100 I
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
[email protected]>
And our ping from left_user to right_user works!
root@lab-box:~# ip netns exec left_user ping 10.10.10.14 -c 4
PING 10.10.10.14 (10.10.10.14) 56(84) bytes of data.
64 bytes from 10.10.10.14: icmp_seq=1 ttl=61 time=1.30 ms
64 bytes from 10.10.10.14: icmp_seq=2 ttl=61 time=1.36 ms
64 bytes from 10.10.10.14: icmp_seq=3 ttl=61 time=1.41 ms
64 bytes from 10.10.10.14: icmp_seq=4 ttl=61 time=1.45 ms
--- 10.10.10.14 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.304/1.383/1.458/0.073 ms
root@lab-box:~# #
So we’ve barely scratched the surface here – but I hope this was enough to get you interested in ExaBGP. In the next post – we’ll cover more ExaBGP and talk about how we can interact with it more programmatically.
Thanks!
Thanks for the useful informations