BGP-LU and MPLS VPNs

      No Comments on BGP-LU and MPLS VPNs

In my last post I showed you how you can use plain old BGP to distribute labels and create LSPs. While sort of interesting to see – it wasn’t super handy all by itself. In this post, we’re going to try and layer on MPLS VPNs to the same setup and show you how that might work. So let’s dig right in. I’m assuming that you’ve read the last post and that we’re picking up where things left off there.

Note: There are no pretty diagrams here so if you’re looking for some context on the lab we’re using go back and checkout the last post.

The first thing we want to do is put our client subnets into VRFs or routing-instances in Juniper parlance. Let’s do that on each tail router….

vMX1

set routing-instances customer1 instance-type vrf
set routing-instances customer1 interface ge-0/0/0.0
set routing-instances customer1 route-distinguisher 1:1
set routing-instances customer1 vrf-target target:1:1
set routing-instances customer1 vrf-table-label 

vMX7

set routing-instances customer1 instance-type vrf
set routing-instances customer1 interface ge-0/0/1.0
set routing-instances customer1 route-distinguisher 1:1
set routing-instances customer1 vrf-target target:1:1
set routing-instances customer1 vrf-table-label 

So nothing fancy here – we’re just creating a routing instance, assigning an RT/RD, and mapping our client facing subnet into the VRF. At this point, we’ve effectively isolated the client subnets so our ping from left_client to right_client should have stopped working.

The next step we’d typically perform in a MPLS VPN scenario is peer the PE or tail routers together. When we do that peering, we’ll do so using the inet-vpn family so that we can exchange VPNv4 routes. Since we’re only advertising the loopbacks into BGP-LU we’ll need to perform the peering over those interfaces which means we need to specify a source address for the eBGP peering as well…

vMX1

set protocols bgp group vpn type external
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn peer-as 65007
set protocols bgp group vpn neighbor 7.7.7.7
set protocols bgp group vpn local-address 1.1.1.1
set protocols bgp group vpn multihop ttl 255 

vMX7

set protocols bgp group vpn type external
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn peer-as 65001
set protocols bgp group vpn neighbor 1.1.1.1
set protocols bgp group vpn local-address 7.7.7.7
set protocols bgp group vpn multihop ttl 255 

The only other interesting config here is the need to specify a multihop TTL since our eBGP peers are no directly connected. Once that config is in, let’s check and see what out BGP peer status is…

[email protected]> show bgp summary 
Groups: 2 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       1          1          0          0          0          0
bgp.l3vpn.0          
                       1          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
7.7.7.7               65007          6          5       0       0        1:10 Establ
  bgp.l3vpn.0: 0/1/1/0
  customer1.inet.0: 0/1/1/0
169.254.10.1          65002         16         14       0       0        5:06 Establ
  inet.0: 1/1/1/0

[email protected]> 

Awesome. Now let’s look and make sure that we’re getting the VPNv4 route we expect…

[email protected]> show route table bgp.l3vpn.0 

bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

1:1:10.10.10.0/24                
                   *[Direct/0] 00:05:55
                    > via ge-0/0/0.0
1:1:10.10.10.1/32                
                   *[Local/0] 00:05:55
                      Local via ge-0/0/0.0

[email protected]> 

Huh. Well at least we have a clue here – see on line 3 there where it shows that we have a hidden route? Let’s take a look at that and see what’s up…

[email protected]> show route table bgp.l3vpn.0 hidden              

bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

1:1:192.168.10.0/24                
                    [BGP/170] 00:03:28, localpref 100, from 7.7.7.7
                      AS path: 65007 I, validation-state: unverified
                      Unusable

[email protected]> 

Hmmm. Well that’s not good – it looks like it thinks that route is unusable. Unfortunately, even looking at the extensive output of the route won’t tell us much more, but let’s look anyways…

[email protected]> show route table bgp.l3vpn.0 hidden extensive 

bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)
1:1:192.168.10.0/24 (1 entry, 0 announced)
         BGP    Preference: 170/-101
                Route Distinguisher: 1:1
                Next hop type: Unusable, Next hop index: 0
                Address: 0xa0ef164
                Next-hop reference count: 2
                State: <Hidden Ext ProtectionPath ProtectionCand>
                Local AS: 65001 Peer AS: 65007
                Age: 3:26 
                Validation State: unverified 
                Task: BGP_65007.7.7.7.7+50474
                AS path: 65007 I 
                Communities: target:1:1
                Import Accepted
                VPN Label: 16
                Localpref: 100
                Router ID: 7.7.7.7
                Secondary Tables: customer1.inet.0
                Indirect next hops: 1
                        Protocol next hop: 7.7.7.7
                        Label operation: Push 16
                        Label TTL action: prop-ttl
                        Load balance label: Label 16: None; 
                        Indirect next hop: 0x0 - INH Session ID: 0x0

[email protected]> 

So what’s going on? Well – if we forget about BGP LU for a moment and think back to a normal MPLS VPN scenario, what was the requirement for entering an LSP? Having an entry in the inet.3 table. So since we’re using an MPLS service – we still have the requirement of having to enter the LSP through inet.3. So how do we fix this?

One option for fixing this is to copy the route out of inet.0 table and into the inet.3 table. Let’s take a look at doing this to see how hard it is. If you’ve never worked with rib-groups this might seem overly complex but just bear with me – I think it will make sense once we talk through it.

The first thing we need to do is define a rib-group underneath the routing-options configuration. Let’s start the config on vMX1…

vMX1

set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-remote-loop

The above defined a rib-group called from-inet0-to-inet3 and specifies two tables. The order in which you define the RIBs matter, but not in the way you think it might. The first RIB should be the one where the prefix normally lives or should live. In this case, that’s inet.0 so we need to put that one first. If you try to reverse them, you should get a commit error (I think). Lastly we specify a import-policy so let’s look at that…

set policy-options prefix-list remote-loop 7.7.7.7/32
set policy-options policy-statement copy-remote-loop term accept from prefix-list remote-loop
set policy-options policy-statement copy-remote-loop term accept then accept
set policy-options policy-statement copy-remote-loop then reject

Again – nothing too fancy here. We created a prefix list that is looking for the remote loopback of vMX1 and we’re accepting that. The last thing we need to do is to actually instantiate the rib-group. To do this, we configure it as part of the protocol that is receiving the prefix we wish to import. In our case, this is the BGP-LU BGP group…

set protocols bgp group external family inet labeled-unicast rib-group from-inet0-to-inet3

Once this configuration is in place and committed let’s look at our inet.3 table…

[email protected]> show route table inet.3 

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

7.7.7.7/32         *[BGP/170] 00:00:52, localpref 100
                      AS path: 65002 65003 65006 65007 I, validation-state: unverified
                    > to 169.254.10.1 via ge-0/0/1.0, Push 300928

[email protected]> 

Nice! So that worked. Let’s put a similar configuration in vMX7…

set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-remote-loop
set policy-options prefix-list remote-loop 1.1.1.1/32
set policy-options policy-statement copy-remote-loop term accept from prefix-list remote-loop
set policy-options policy-statement copy-remote-loop term accept then accept
set policy-options policy-statement copy-remote-loop then reject
set protocols bgp group external family inet labeled-unicast rib-group from-inet0-to-inet3

Once that’s committed – we should have the inet.3 entry on vMX7 for 1.1.1.1/32

[email protected]> show route table inet.3 

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 00:04:32, localpref 100
                      AS path: 65006 65003 65002 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 300992

[email protected]> 

Nice! So at this point – our clients should once again be pinging. That wasn’t too bad to do – but let’s talk about some other ways to do this as well.

One of the options you have with BGP-LU is to tell it what RIB to use. As we’ve seen – by default it uses inet.0 but we can also configure it to use inet.3. Before we do that, let’s clear out some of the configuration we’ve done so that we’re not cluttering up the config (the worst)…

vMX1 and vMX7

delete policy-options policy-statement copy-remote-loop
delete routing-options rib-groups 
delete protocols bgp group external family inet labeled-unicast rib-group 

Alright – so back to a broken state. Let’s tell BGP-LU to use inet.3 now…

vMX1 and vMX7

set protocols bgp group external family inet labeled-unicast rib inet.3

So after we do that config let’s take a look and see what we’ve got…

[email protected]> show route table inet.3 

[email protected]> show route receive-protocol bgp 169.254.10.1 

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

customer1.inet.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)

mpls.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.l3vpn.0: 3 destinations, 3 routes (2 active, 0 holddown, 1 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

customer1.inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

[email protected]> 

Hmmmm. Not promising. But let’s think about what we just told BGP-LU to do. We asked it to use the inet.3 table. That means prefixes it receives will end up in inet.3 but it also means that prefixes need to be in inet.3 to be advertised to other peers. So to get the prefixes sent over the BGP-LU session they need to be in the inet.3 table to start with. So let’s try this out on vMX1…

vMX1

set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-local-loop

set policy-options prefix-list local-loop 1.1.1.1/32
set policy-options policy-statement copy-local-loop term accept from prefix-list local-loop
set policy-options policy-statement copy-local-loop term accept then accept
set policy-options policy-statement copy-local-loop then reject

set routing-options interface-routes rib-group inet from-inet0-to-inet3

Be careful not to confuse the above configuration with what we did earlier. We’re still copying prefixes from inet.0 to inet.3 but we’re copying the local loopback instead of the remote loopback. Also notice that since this is already a local route that we’re doing the rib-group work under the routing-options and interface-routes config blocks. Once this configuration is in place on vMX1 we should see the route now in the inet.3 table…

[email protected]> show route table inet.3 

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[Direct/0] 00:00:04
                    > via lo0.0

[email protected]> 

Cool – now we need to do a similar configuration on vMX7…

vMX7

set routing-options rib-groups from-inet0-to-inet3 import-rib inet.0
set routing-options rib-groups from-inet0-to-inet3 import-rib inet.3
set routing-options rib-groups from-inet0-to-inet3 import-policy copy-local-loop

set policy-options prefix-list local-loop 7.7.7.7/32
set policy-options policy-statement copy-local-loop term accept from prefix-list local-loop
set policy-options policy-statement copy-local-loop term accept then accept
set policy-options policy-statement copy-local-loop then reject

set routing-options interface-routes rib-group inet from-inet0-to-inet3

With this configuration in place – let’s take a look at inet.3 on vMX7…

[email protected]> show route table inet.3 

inet.3: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 00:02:49, localpref 100
                      AS path: 65006 65003 65002 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 301008
7.7.7.7/32         *[Direct/0] 00:00:06
                    > via lo0.0

[email protected]> 

Nice! So we have the local router we just imported as well as the one we’re learning through the BGP-LU session from vMX1. Everything seems to be in order! But it looks like our client ping is still failing. Let’s check our BGP sessions…

[email protected]> show bgp summary 
Groups: 2 Peers: 2 Down peers: 1
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
bgp.l3vpn.0          
                       0          0          0          0          0          0
inet.3               
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
1.1.1.1               65001          0          0       0       2        8:17 Active
169.254.10.10         65006         23         22       0       0        9:13 Establ
  inet.3: 1/1/1/0

[email protected]> 

Hmmmm. So it looks like our VPN BGP peering is still down. Any one know why? Let’s look at the inet.0 routing table to see if we can reach that peer….

[email protected]> show route table inet.0 1.1.1.1/32 


[email protected]> 

So while we have a path to reach our remote peer 1.1.1.1 in inet.3 we don’t have one in inet.0 where we need it. The BGP peer has to be in inet.0 in order for BGP to use it. So what’s the fix for this? We need to copy the prefix from inet.3 into inet.0. MORE RIB-GROUP WORK! This is getting a little out of hand but let’s keep going to see if we can get this working…

vMX1 and vMX7

set routing-options rib-groups from-inet3-to-inet0 import-rib inet.3
set routing-options rib-groups from-inet3-to-inet0 import-rib inet.0
set routing-options rib-groups from-inet3-to-inet0 import-policy copy-remote-loop
set policy-options policy-statement copy-remote-loop term accept from prefix-list remote-loop
set policy-options policy-statement copy-remote-loop term accept then accept
set policy-options policy-statement copy-remote-loop then reject
set protocols bgp group external family inet labeled-unicast rib-group from-inet3-to-inet0

Note: We can use the same configuration in place on both since we already defined the prefix-list remote-loop earlier specifying the other tail router.

Now if we look we should see that we have the remote loopback back in inet.0

[email protected]> show route table inet.0 1.1.1.1/32 

inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 00:02:37, localpref 100
                      AS path: 65006 65003 65002 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 301008

[email protected]> 

And our BGP peering should be up…

[email protected]> show bgp summary 
Groups: 2 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
bgp.l3vpn.0          
                       1          1          0          0          0          0
inet.3               
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
1.1.1.1               65001         29         29       0       2       12:19 Establ
  bgp.l3vpn.0: 1/1/1/0
  customer1.inet.0: 1/1/1/0
169.254.10.10         65006      14915      14974       0       0 4d 16:17:32 Establ
  inet.3: 1/1/1/0

[email protected]> 

And if we check – we should see we’re getting the remote customer prefix in the customer routing instance…

[email protected]> show route table customer1.inet.0 

customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[BGP/170] 00:02:56, localpref 100, from 1.1.1.1
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 16, Push 301008(top)
192.168.10.0/24    *[Direct/0] 4d 16:27:41
                    > via ge-0/0/1.0
192.168.10.1/32    *[Local/0] 4d 16:27:41
                      Local via ge-0/0/1.0

[email protected]> 

With all that now in place our left_client to right_client ping should now also be working…

[email protected]_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=4.55 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=5.11 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.88 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=5.55 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=3.95 ms

--- 192.168.10.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 3.955/4.812/5.553/0.539 ms
[email protected]_client:~# 

Alright – so that worked – but that was an awful lot of work. We had to copy the routes into the right table to advertise them (inet.0 -> inet.3), and then we had to move the remote prefix back to get the peering working (inet.3 -> inet.0). That seemed like a lot of work – so let me let you in on a little shortcut you can use when you’re using BGP-LU in the default model.

Let’s reset things again by clearing up some of the configuration…

vMX1 and vMX7

delete protocols bgp group external family inet labeled-unicast rib inet.3
delete routing-options rib-groups
delete policy-options policy-statement copy-local-loop
delete policy-options policy-statement copy-remote-loop
delete routing-options interface-routes
delete protocols bgp group external family inet labeled-unicast rib-group

With that cleared up – we’re back where we were when we started this post before we manually copied the remote loopback address from inet.0 to inet.3. Let’s look to make sure…

[email protected]> show route table inet.3 

[email protected]>

What if I told you we could have the Juniper do this for us automatically?

vMX1 and vMX7

set protocols bgp group external family inet labeled-unicast resolve-vpn 

If we add that to both vMX1 and vMX7 we should see that the remote loopback we’re learning through BGP-LU is now being automatically copied into the inet.3 table…

[email protected]> show route table inet.3    

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 00:02:58, localpref 100
                      AS path: 65006 65003 65002 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 301040

[email protected]> 

Nice! So what’s this command doing? Juniper doc says…

Allow labeled routes to be placed in the inet.3 routing table for route resolution. These routes are then resolved for PE router connections where the remote PE is located across another AS. For a PE router to install a route in the VRF, the next hop must resolve to a route stored within the inet.3 table.

So we know that we need an entry in inet.3 for MPLS VPNs to work. That’s pretty much what this is saying. So the command is telling the router to copy any BGP-LU prefixes from inet.0 (their default location) to inet.3 so long as they’re being used a BGP next-hop. In this case, we can see that the VPNv4 route we’re receiving lists the remote loopback as the next hop address…

[email protected]> show route receive-protocol bgp 1.1.1.1 table bgp.l3vpn.0 

bgp.l3vpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
  1:1:10.10.10.0/24                    
*                         1.1.1.1                                 65001 I

[email protected]> 

So that’s why this works. I know I saved the easiest option for the last, but we got some great experience with rib-groups along the way. In the next post we’re going to dig into some other use case for BGP-LU and how it can help solve some MPLS scaling problems for us.

Leave a Reply

Your email address will not be published. Required fields are marked *