In my last post I showed you how you can use plain old BGP to distribute labels and create LSPs. While sort of interesting to see – it wasn’t super handy all by itself. In this post, we’re going to try and layer on MPLS VPNs to the same setup and show you how that might work. So let’s dig right in. I’m assuming that you’ve read the last post and that we’re picking up where things left off there.
Note: There are no pretty diagrams here so if you’re looking for some context on the lab we’re using go back and checkout the last post.
The first thing we want to do is put our client subnets into VRFs or routing-instances in Juniper parlance. Let’s do that on each tail router….
vMX1
set routing-instances customer1 instance-type vrf
set routing-instances customer1 interface ge-0/0/0.0
set routing-instances customer1 route-distinguisher 1:1
set routing-instances customer1 vrf-target target:1:1
set routing-instances customer1 vrf-table-label
vMX7
set routing-instances customer1 instance-type vrf
set routing-instances customer1 interface ge-0/0/1.0
set routing-instances customer1 route-distinguisher 1:1
set routing-instances customer1 vrf-target target:1:1
set routing-instances customer1 vrf-table-label
So nothing fancy here – we’re just creating a routing instance, assigning an RT/RD, and mapping our client facing subnet into the VRF. At this point, we’ve effectively isolated the client subnets so our ping from left_client to right_client should have stopped working.
The next step we’d typically perform in a MPLS VPN scenario is peer the PE or tail routers together. When we do that peering, we’ll do so using the inet-vpn family so that we can exchange VPNv4 routes. Since we’re only advertising the loopbacks into BGP-LU we’ll need to perform the peering over those interfaces which means we need to specify a source address for the eBGP peering as well…
vMX1
set protocols bgp group vpn type external
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn peer-as 65007
set protocols bgp group vpn neighbor 7.7.7.7
set protocols bgp group vpn local-address 1.1.1.1
set protocols bgp group vpn multihop ttl 255
vMX7
set protocols bgp group vpn type external
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn peer-as 65001
set protocols bgp group vpn neighbor 1.1.1.1
set protocols bgp group vpn local-address 7.7.7.7
set protocols bgp group vpn multihop ttl 255
The only other interesting config here is the need to specify a multihop TTL since our eBGP peers are no directly connected. Once that config is in, let’s check and see what out BGP peer status is…
[email protected]> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
bgp.l3vpn.0
1 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
7.7.7.7 65007 6 5 0 0 1:10 Establ
bgp.l3vpn.0: 0/1/1/0
customer1.inet.0: 0/1/1/0
169.254.10.1 65002 16 14 0 0 5:06 Establ
inet.0: 1/1/1/0
[email protected]>
Awesome. Now let’s look and make sure that we’re getting the VPNv4 route we expect…
[email protected]> show route table bgp.l3vpn.0
bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both
1:1:10.10.10.0/24
*[Direct/0] 00:05:55
> via ge-0/0/0.0
1:1:10.10.10.1/32
*[Local/0] 00:05:55
Local via ge-0/0/0.0
[email protected]>
Huh. Well at least we have a clue here – see on line 3 there where it shows that we have a hidden route? Let’s take a look at that and see what’s up…
[email protected]> show route table bgp.l3vpn.0 hidden
bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both
1:1:192.168.10.0/24
[BGP/170] 00:03:28, localpref 100, from 7.7.7.7
AS path: 65007 I, validation-state: unverified
Unusable
[email protected]>
Hmmm. Well that’s not good – it looks like it thinks that route is unusable. Unfortunately, even looking at the extensive output of the route won’t tell us much more, but let’s look anyways…
[email protected]> show route table bgp.l3vpn.0 hidden extensive
bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
1:1:192.168.10.0/24 (1 entry, 0 announced)
BGP Preference: 170/-101
Route Distinguisher: 1:1
Next hop type: Unusable, Next hop index: 0
Address: 0xa0ef164
Next-hop reference count: 2
State: <Hidden Ext ProtectionPath ProtectionCand>
Local AS: 65001 Peer AS: 65007
Age: 3:26
Validation State: unverified
Task: BGP_65007.7.7.7.7+50474
AS path: 65007 I
Communities: target:1:1
Import Accepted
VPN Label: 16
Localpref: 100
Router ID: 7.7.7.7
Secondary Tables: customer1.inet.0
Indirect next hops: 1
Protocol next hop: 7.7.7.7
Label operation: Push 16
Label TTL action: prop-ttl
Load balance label: Label 16: None;
Indirect next hop: 0x0 - INH Session ID: 0x0
[email protected]>
So what’s going on? Well – if we forget about BGP LU for a moment and think back to a normal MPLS VPN scenario, what was the requirement for entering an LSP? Having an entry in the inet.3 table. So since we’re using an MPLS service – we still have the requirement of having to enter the LSP through inet.3. So how do we fix this?
One option for fixing this is to copy the route out of inet.0 table and into the inet.3 table. Let’s take a look at doing this to see how hard it is. If you’ve never worked with rib-groups this might seem overly complex but just bear with me – I think it will make sense once we talk through it.
The first thing we need to do is define a rib-group underneath the routing-options configuration. Let’s start the config on vMX1…
vMX1
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-remote-loop
The above defined a rib-group called from-inet0-to-inet3inet.0 so we need to put that one first. If you try to reverse them, you should get a commit error (I think). Lastly we specify a import-policy so let’s look at that…
set policy-options prefix-list remote-loop 7.7.7.7/32
set policy-options policy-statement copy-remote-loop term accept from prefix-list remote-loop
set policy-options policy-statement copy-remote-loop term accept then accept
set policy-options policy-statement copy-remote-loop then reject
Again – nothing too fancy here. We created a prefix list that is looking for the remote loopback of vMX1 and we’re accepting that. The last thing we need to do is to actually instantiate the rib-group. To do this, we configure it as part of the protocol that is receiving the prefix we wish to import. In our case, this is the BGP-LU BGP group…
set protocols bgp group external family inet labeled-unicast rib-group from-inet0-to-inet3
Once this configuration is in place and committed let’s look at our inet.3 table…
[email protected]> show route table inet.3
inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
7.7.7.7/32 *[BGP/170] 00:00:52, localpref 100
AS path: 65002 65003 65006 65007 I, validation-state: unverified
> to 169.254.10.1 via ge-0/0/1.0, Push 300928
[email protected]>
Nice! So that worked. Let’s put a similar configuration in vMX7…
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-remote-loop
set policy-options prefix-list remote-loop 1.1.1.1/32
set policy-options policy-statement copy-remote-loop term accept from prefix-list remote-loop
set policy-options policy-statement copy-remote-loop term accept then accept
set policy-options policy-statement copy-remote-loop then reject
set protocols bgp group external family inet labeled-unicast rib-group from-inet0-to-inet3
Once that’s committed – we should have the inet.3 entry on vMX7 for 1.1.1.1/32…
[email protected]> show route table inet.3
inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[BGP/170] 00:04:32, localpref 100
AS path: 65006 65003 65002 65001 I, validation-state: unverified
> to 169.254.10.10 via ge-0/0/0.0, Push 300992
[email protected]>
Nice! So at this point – our clients should once again be pinging. That wasn’t too bad to do – but let’s talk about some other ways to do this as well.
One of the options you have with BGP-LU is to tell it what RIB to use. As we’ve seen – by default it uses inet.0 but we can also configure it to use inet.3. Before we do that, let’s clear out some of the configuration we’ve done so that we’re not cluttering up the config (the worst)…
vMX1 and vMX7
delete policy-options policy-statement copy-remote-loop
delete routing-options rib-groups
delete protocols bgp group external family inet labeled-unicast rib-group
Alright – so back to a broken state. Let’s tell BGP-LU to use inet.3 now…
vMX1 and vMX7
set protocols bgp group external family inet labeled-unicast rib inet.3
So after we do that config let’s take a look and see what we’ve got…
[email protected]> show route table inet.3
[email protected]> show route receive-protocol bgp 169.254.10.1
inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
customer1.inet.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
mpls.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
customer1.inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
[email protected]>
Hmmmm. Not promising. But let’s think about what we just told BGP-LU to do. We asked it to use the inet.3 table. That means prefixes it receives will end up in inet.3 but it also means that prefixes need to be in inet.3 to be advertised to other peers. So to get the prefixes sent over the BGP-LU session they need to be in the inet.3 table to start with. So let’s try this out on vMX1…
vMX1
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-local-loop
set policy-options prefix-list local-loop 1.1.1.1/32
set policy-options policy-statement copy-local-loop term accept from prefix-list local-loop
set policy-options policy-statement copy-local-loop term accept then accept
set policy-options policy-statement copy-local-loop then reject
set routing-options interface-routes rib-group inet from-inet0-to-inet3
Be careful not to confuse the above configuration with what we did earlier. We’re still copying prefixes from inet.0 to inet.3 but we’re copying the local loopback instead of the remote loopback. Also notice that since this is already a local route that we’re doing the rib-group work under the routing-options and interface-routes config blocks. Once this configuration is in place on vMX1 we should see the route now in the inet.3 table…
[email protected]> show route table inet.3
inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[Direct/0] 00:00:04
> via lo0.0
[email protected]>
Cool – now we need to do a similar configuration on vMX7…
vMX7
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-local-loop
set policy-options prefix-list local-loop 7.7.7.7/32
set policy-options policy-statement copy-local-loop term accept from prefix-list local-loop
set policy-options policy-statement copy-local-loop term accept then accept
set policy-options policy-statement copy-local-loop then reject
set routing-options interface-routes rib-group inet from-inet0-to-inet3
With this configuration in place – let’s take a look at inet.3 on vMX7…
[email protected]> show route table inet.3
inet.3: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[BGP/170] 00:02:49, localpref 100
AS path: 65006 65003 65002 65001 I, validation-state: unverified
> to 169.254.10.10 via ge-0/0/0.0, Push 301008
7.7.7.7/32 *[Direct/0] 00:00:06
> via lo0.0
[email protected]>
Nice! So we have the local router we just imported as well as the one we’re learning through the BGP-LU session from vMX1. Everything seems to be in order! But it looks like our client ping is still failing. Let’s check our BGP sessions…
[email protected]> show bgp summary
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
0 0 0 0 0 0
bgp.l3vpn.0
0 0 0 0 0 0
inet.3
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
1.1.1.1 65001 0 0 0 2 8:17 Active
169.254.10.10 65006 23 22 0 0 9:13 Establ
inet.3: 1/1/1/0
[email protected]>
Hmmmm. So it looks like our VPN BGP peering is still down. Any one know why? Let’s look at the inet.0 routing table to see if we can reach that peer….
[email protected]> show route table inet.0 1.1.1.1/32
[email protected]>
So while we have a path to reach our remote peer 1.1.1.1 in inet.3 we don’t have one in inet.0 where we need it. The BGP peer has to be in inet.0 in order for BGP to use it. So what’s the fix for this? We need to copy the prefix from inet.3 into inet.0. MORE RIB-GROUP WORK! This is getting a little out of hand but let’s keep going to see if we can get this working…
vMX1 and vMX7
set routing-options rib-groups from-inet3-to-inet0 import-rib inet.3
set routing-options rib-groups from-inet3-to-inet0 import-rib inet.0
set routing-options rib-groups from-inet3-to-inet0 import-policy copy-remote-loop
set policy-options policy-statement copy-remote-loop term accept from prefix-list remote-loop
set policy-options policy-statement copy-remote-loop term accept then accept
set policy-options policy-statement copy-remote-loop then reject
set protocols bgp group external family inet labeled-unicast rib-group from-inet3-to-inet0
Note: We can use the same configuration in place on both since we already defined the prefix-list remote-loop earlier specifying the other tail router.
Now if we look we should see that we have the remote loopback back in inet.0…
[email protected]> show route table inet.0 1.1.1.1/32
inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[BGP/170] 00:02:37, localpref 100
AS path: 65006 65003 65002 65001 I, validation-state: unverified
> to 169.254.10.10 via ge-0/0/0.0, Push 301008
[email protected]>
And our BGP peering should be up…
[email protected]> show bgp summary
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
0 0 0 0 0 0
bgp.l3vpn.0
1 1 0 0 0 0
inet.3
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
1.1.1.1 65001 29 29 0 2 12:19 Establ
bgp.l3vpn.0: 1/1/1/0
customer1.inet.0: 1/1/1/0
169.254.10.10 65006 14915 14974 0 0 4d 16:17:32 Establ
inet.3: 1/1/1/0
[email protected]>
And if we check – we should see we’re getting the remote customer prefix in the customer routing instance…
[email protected]> show route table customer1.inet.0
customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.10.10.0/24 *[BGP/170] 00:02:56, localpref 100, from 1.1.1.1
AS path: 65001 I, validation-state: unverified
> to 169.254.10.10 via ge-0/0/0.0, Push 16, Push 301008(top)
192.168.10.0/24 *[Direct/0] 4d 16:27:41
> via ge-0/0/1.0
192.168.10.1/32 *[Local/0] 4d 16:27:41
Local via ge-0/0/1.0
[email protected]>
With all that now in place our left_client to right_client ping should now also be working…
root@left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=4.55 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=5.11 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.88 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=5.55 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=3.95 ms
--- 192.168.10.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 3.955/4.812/5.553/0.539 ms
root@left_client:~#
Alright – so that worked – but that was an awful lot of work. We had to copy the routes into the right table to advertise them (inet.0 -> inet.3), and then we had to move the remote prefix back to get the peering working (inet.3 -> inet.0). That seemed like a lot of work – so let me let you in on a little shortcut you can use when you’re using BGP-LU in the default model.
Let’s reset things again by clearing up some of the configuration…
vMX1 and vMX7
delete protocols bgp group external family inet labeled-unicast rib inet.3
delete routing-options rib-groups
delete policy-options policy-statement copy-local-loop
delete policy-options policy-statement copy-remote-loop
delete routing-options interface-routes
delete protocols bgp group external family inet labeled-unicast rib-group
With that cleared up – we’re back where we were when we started this post before we manually copied the remote loopback address from inet.0 to inet.3. Let’s look to make sure…
[email protected]> show route table inet.3
[email protected]>
What if I told you we could have the Juniper do this for us automatically?
vMX1 and vMX7
set protocols bgp group external family inet labeled-unicast resolve-vpn
If we add that to both vMX1 and vMX7 we should see that the remote loopback we’re learning through BGP-LU is now being automatically copied into the inet.3 table…
[email protected]> show route table inet.3
inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[BGP/170] 00:02:58, localpref 100
AS path: 65006 65003 65002 65001 I, validation-state: unverified
> to 169.254.10.10 via ge-0/0/0.0, Push 301040
[email protected]>
Nice! So what’s this command doing? Juniper doc says…
Allow labeled routes to be placed in the inet.3 routing table for route resolution. These routes are then resolved for PE router connections where the remote PE is located across another AS. For a PE router to install a route in the VRF, the next hop must resolve to a route stored within the inet.3 table.
So we know that we need an entry in inet.3 for MPLS VPNs to work. That’s pretty much what this is saying. So the command is telling the router to copy any BGP-LU prefixes from inet.0 (their default location) to inet.3 so long as they’re being used a BGP next-hop. In this case, we can see that the VPNv4 route we’re receiving lists the remote loopback as the next hop address…
[email protected]> show route receive-protocol bgp 1.1.1.1 table bgp.l3vpn.0
bgp.l3vpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
1:1:10.10.10.0/24
* 1.1.1.1 65001 I
[email protected]>
So that’s why this works. I know I saved the easiest option for the last, but we got some great experience with rib-groups along the way. In the next post we’re going to dig into some other use case for BGP-LU and how it can help solve some MPLS scaling problems for us.
hi Jon,
I have some questions regarding the two options in this article:
as you mentioned
option 2,”set protocols bgp group external family inet labeled-unicast resolve-vpn” will fill the inet.3 table, will it also fill inet.0 table ?
because for option1 you have both 1.1.1.1 and 7.7.7.7 in inet.0 and inet.3 tables and you mentioned u need inet.0 to bring up bgp-vpn.
actually I do not understand why we need inet.0 for bgp connection?
in option1 we can resolve bgp next hop via inet.3 and we have connectivity between 1.1.1.1 and 7.7.7.7, so why do we need to copy inet.3 into inet.0
Regards
Michael
Great walkthrough! Thank you for “going the long way around” with the rib-groups solutions – this really cleared up VPN resolution for me