Carrier supporting Carrier with BGP-LU

      No Comments on Carrier supporting Carrier with BGP-LU

In our last post we talked about the less used method of deploying CsC where we ran OSPF and LDP inside the CSC-PE routing-instance.

Note: I can’t help myself apparently so be aware that Carrier of Carriers (CoC) is the same as Carrier supporting Carrier (CsC)

This required some changes to be made to our default LDP export policy as well as how we moved routes between the inet.3 and inet.0 tables. That being said, if you’re a single org it might make good sense to run things that way. I liked how you were able to see all of the remote LDP domain loopbacks in your local inet.3 table which in my mind made it easier to imagine the LSP paths.

That being said, it is clearly not the preferred deployment methodology. Most examples you’ll find leverage BGP (BGP-LU specifically) for the CSC-CE to CSC-PE connections as well as within the local label domains. So in this example, we’ll do just that. Larges chunks of the base configuration will be the same as they were in the previous post but for the sake of clarity I’ll post our starting post the starting configurations and diagrams here once again. Here is our base lab topology…

And once again we’ll split the lab into the orange carrier, who has a network on the left and the right, and the green carrier, who has the top network…

As before, our goal is to make the orange carrier whole by leveraging connectivity through the green carrier so that the orange carrier can support a VPN service for its customer CE routers. The interesting bit about this is that the connectivity we leverage in the green carrier is itself a VPN hence the terminology of Carrier supporting Carrier. So let’s take a look at our base configuration for each router once again…

Note: Im not going to talk through all the specifics of the base configuration in this post. If you’re curious about it – take a look at the walk through of the base configuration in the last post.

vMX1

set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.1/24
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.0/31
set routing-options autonomous-system 65001
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.1 peer-as 65002
set policy-options prefix-list customer_routes 10.10.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject

vMX2

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.1/31
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.2/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 2.2.2.2/32 primary
set routing-options router-id 2.2.2.2
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/1.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/0.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65002
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.0 peer-as 65001

vMX3

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.3/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.4/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 3.3.3.3/32 primary
set routing-options router-id 3.3.3.3
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX4

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.5/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.6/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 4.4.4.4/32 primary
set routing-options router-id 4.4.4.4
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX5

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.7/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.8/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 5.5.5.5/32 primary
set routing-options router-id 5.5.5.5
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX6

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.9/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.10/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 6.6.1.1/32
set routing-options router-id 6.6.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols mpls label-switched-path to-vmx9 to 9.9.1.1
set protocols mpls interface ge-0/0/1.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX7

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.11/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.12/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 7.7.1.1/32 primary
set routing-options router-id 7.7.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX8

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.13/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.14/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 8.8.1.1/32 primary
set routing-options router-id 8.8.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX9

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.15/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.16/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 9.9.1.1/32
set routing-options router-id 9.9.1.1
set routing-options autonomous-system 65000
set protocols rsvp interface ge-0/0/0.0
set protocols mpls label-switched-path to-vmx6 to 6.6.1.1
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p

vMX10

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.17/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.18/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 10.10.10.10/32 primary
set routing-options router-id 10.10.10.10
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX11

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.19/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.20/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 11.11.11.11/32 primary
set routing-options router-id 11.11.11.11
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX12

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.21/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.22/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces fxp0 unit 0 family inet address 192.168.127.12/24
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 12.12.12.12/32 primary
set routing-options static route 10.171.200.0/22 next-hop 192.168.127.100
set routing-options router-id 12.12.12.12
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX13

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.23/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.24/31
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 13.13.13.13/32 primary
set routing-options router-id 13.13.13.13
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/1.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65004
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.25 peer-as 65003

vMX14

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.25/31
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.1/24
set routing-options autonomous-system 65003
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.24 peer-as 65004
set policy-options prefix-list customer_routes 192.168.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject

Right – so if you recall from the last post. This sort of got us what I call our “base label domain” configuration. That is – each “block” should have a means to for an MPLS LSP through it. This is an exact copy and past of the base configuration from the previous post. There should be nothing terribly new or exciting to you about this configuration.

As with the last post, the next step is to configure the CSC-PE routers routing-instance for the orange carrier. This configuration differs slightly from our last configuration…

vMX6

set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/0.0
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 6.6.6.6
set routing-instances orange_carrier routing-options autonomous-system 65200
set routing-instances orange_carrier protocols bgp group to_orange_carrier type external
set routing-instances orange_carrier protocols bgp group to_orange_carrier family inet labeled-unicast
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.8 peer-as 65100
set routing-instances orange_carrier protocols mpls interface ge-0/0/0.0

Notice the lack of OSPF and LDP configuration. This is now replaced by a BGP peering configuration that will be used to peer to the CSC-PE routers. vMX9 looks largely the same…

vMX9

set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/1.0
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 9.9.9.9
set routing-instances orange_carrier routing-options autonomous-system 65200
set routing-instances orange_carrier protocols bgp group to_orange_carrier type external
set routing-instances orange_carrier protocols bgp group to_orange_carrier family inet labeled-unicast
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.17 peer-as 65100
set routing-instances orange_carrier protocols mpls interface ge-0/0/1.0

Notice that we’re using the same AS number for the peerings to the CSC-CE routers. As you might already be guessing, we’ll have to make some adjustments later on for that to work correctly but for now let’s table that.

The next thing to do is to configure BGP within the orange carrier blocks. To do this, we’ll peer the PE router to the CSC-CE node iBGP. Then we’ll also configure a peering between the CSC-CE and the CSC-PE nodes with eBGP. Let’s start on the left side with vMX2…

vMX2

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 2.2.2.2
set protocols bgp group internal family inet labeled-unicast resolve-vpn
set protocols bgp group internal neighbor 5.5.5.5 peer-as 65100

Nothing too fancy here but do note that we added the flag resolve-vpn to the BGP configuration. If you recall from our previous posts, this allows the router to add prefixes it receives through BGP-LU to the inet.3 table which is required for VPN route resolution. With that in place, we can configure vMX5 for that same peering as well as another BGP group so it can peer to the CSC-PE node (vMX6 in this case)…

vMX5

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 5.5.5.5
set protocols bgp group internal family inet labeled-unicast
set protocols bgp group internal neighbor 2.2.2.2 peer-as 65100
set protocols bgp group to_green_carrier type external
set protocols bgp group to_green_carrier family inet labeled-unicast
set protocols bgp group to_green_carrier export to_green_carrier
set protocols bgp group to_green_carrier neighbor 169.254.10.9 peer-as 65200
set policy-options policy-statement to_green_carrier term local_loops from prefix-list local_loops
set policy-options policy-statement to_green_carrier term local_loops then accept
set policy-options policy-statement to_green_carrier then reject
set policy-options prefix-list local_loops 2.2.2.2/32
set policy-options prefix-list local_loops 3.3.3.3/32
set policy-options prefix-list local_loops 4.4.4.4/32
set policy-options prefix-list local_loops 5.5.5.5/32

Notice that we’re doing the loopback export on the CSC-CE node. The reason for doing that here and not on vMX2 is that we also know about that route from OSPF. That is – vMX5 prefers the route from OSPF so it won’t advertise a BGP route it learned about through BGP to another BGP peer. Note that we’re advertising all of the loopbacks from the left orange block through the BGP-LU session to vMX6. Now let’s do the same configuration on the right side orange block…

vMX13

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 13.13.13.13
set protocols bgp group internal family inet labeled-unicast resolve-vpn
set protocols bgp group internal neighbor 10.10.10.10 peer-as 65100

vMX10

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 10.10.10.10
set protocols bgp group internal family inet labeled-unicast
set protocols bgp group internal neighbor 13.13.13.13 peer-as 65100
set protocols bgp group to_green_carrier type external
set protocols bgp group to_green_carrier family inet labeled-unicast
set protocols bgp group to_green_carrier export to_green_carrier
set protocols bgp group to_green_carrier neighbor 169.254.10.16 peer-as 65200
set policy-options policy-statement to_green_carrier term local_loops from prefix-list local_loops
set policy-options policy-statement to_green_carrier term local_loops then accept
set policy-options policy-statement to_green_carrier then reject
set policy-options prefix-list local_loops 10.10.10.10/32
set policy-options prefix-list local_loops 11.11.11.11/32
set policy-options prefix-list local_loops 12.12.12.12/32
set policy-options prefix-list local_loops 13.13.13.13/32

And last but not least, we need to configure the VPN peering within the green carrier…

vMX6

set routing-options autonomous-system 65200
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 local-address 6.6.1.1
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 peer-as 65200

vMX9

set routing-options autonomous-system 65200
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 local-address 9.9.1.1
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 peer-as 65200

Now what we’ve done here is slightly changed out BGP configuration to make things easier to read. The green carrier in this case is wholly inside of AS 65200 while the orange provider (both blocks) are in AS 65100. Now, we still have the sort of one off eBGP peering between the CE and PE routers in each orange block where we use AS 65001 to 65002 and 65003 to 65004. But to be honest, that really doesn’t matter too much for us at this point. As in the last post, that’s really just a means to get the customer prefixes to the orange PE routers. It could very easily have been anything else including static routes or an IGP etc. So our BGP config at this point looks like this…

The one BGP peering we’re clearly missing here is the iBGP VPNv4 peering between vMX2 and vMX13 but before we configure that let’s make sure we have good reachability…

[email protected]> show route table inet.3 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[LDP/9] 00:14:14, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0
4.4.4.4/32         *[LDP/9] 00:14:08, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0
5.5.5.5/32         *[LDP/9] 00:13:59, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0, Push 299920

[email protected]> 

Well that’s not what I had expected. I had hoped that I’d be getting some BGP-LU routes for the routers in the right orange block that be populated into the inet.3 table. So what’s going on? Well – I sort of mentioned this before, but since each CSC-CE is using the same AS number, we’ve got an AS loop. And Juniper is smart enough to know not even to try and advertise routes back to an AS that’s already in a routes AS path list. We can validate that here with just one of the prefixes from the right orange block…

[email protected]> show route table orange_carrier.inet.0 11.11.11.11/32              

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

11.11.11.11/32     *[BGP/170] 00:14:22, MED 1, localpref 100, from 9.9.1.1
                      AS path: 65100 I, validation-state: unverified
                    > to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9

[email protected]> show route advertising-protocol bgp 169.254.10.8 11.11.11.11/32    

[email protected]> 

Notice that the AS path includes 65100 which was added by vMX10 as it advertised the loopbacks to vMX9 over the eBGP peering. Also notice that despite the routes we want being in the table we need them to be in, they aren’t being sent to vMX5. To fix this, we can tell the green provider to do an AS override…

vMX6

set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.8 as-override

vMX9

set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.17 as-override 

AS-override is a commonly misunderstood tool. It lets a router replace all instances of a peer’s AS number in the AS-path list with it’s own. It’s important to know that it works on egress. That is, the AS path will be modified as the prefix leaves the green carrier and is advertised toward the orange carrier. We can see that now that this configuration is in place…

[email protected]> show route receive-protocol bgp 9.9.1.1 11.11.11.11/32   

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 11.11.11.11/32          9.9.1.1              1       100        65100 I

bgp.l3vpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
  1:1:11.11.11.11/32                    
*                         9.9.1.1              1       100        65100 I

[email protected]> show route advertising-protocol bgp 169.254.10.8 11.11.11.11/32    

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 11.11.11.11/32          Self                                    65200 I

[email protected]> 

We can see that when we get the route from vMX9 the AS path is unchanged. But when we advertise it to vMX5 we’ve replaced the 65100 AS with 65200. Now you might be looking at this and be wondering why that looks wrong. Indeed, it looks like we’re missing an AS path entry here. It’s each BGP routers job to add it’s local AS number to the route as it is advertised to an eBGP peer. In this case, it appears that all we’ve done is done the replacement. Where is the AS entry for the route coming from 65200 in the first place?

[email protected]> show route receive-protocol bgp 169.254.10.9 11.11.11.11/32 

inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 11.11.11.11/32          169.254.10.9                            65200 65200 I

[email protected]> 

As it turns out it’s there. We just don’t get to see it with the advertising-protocol command for some reason. A minor annoyance, but one that can be particularly irritating as you attempt to do troubleshooting.

Now that we have our AS loop sorted, let’s look back on vMX2 to see what we have in the inet.3 table…

[email protected]> show route table inet.3                        

inet.3: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.3/32         *[LDP/9] 00:25:07, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32         *[LDP/9] 00:24:59, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299904
5.5.5.5/32         *[LDP/9] 00:24:49, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299920
10.10.10.10/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300608, Push 299920(top)
11.11.11.11/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
12.12.12.12/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
13.13.13.13/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
169.254.10.16/31   *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300576, Push 299920(top)

[email protected]x2.lab> 

Success! So at this point, we should be able to configure out iBGP VPNv4 peering for the orange carrier…

vMX2

set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 13.13.13.13 local-address 2.2.2.2
set protocols bgp group vpn neighbor 13.13.13.13 peer-as 65100

vMX13

set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 2.2.2.2 local-address 13.13.13.13
set protocols bgp group vpn neighbor 2.2.2.2 peer-as 65100

And with that complete, our BGP peerings are now complete and should look like this…

But alas – our pings still aren’t working…

left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
  
--- 192.168.10.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
  
left_client:~# 

So what could be going on now? Well if I were a betting man, I’d guess that as long as the VPNv4 routes are being sent and accepted between vMX2 and vMX13 that we have a problem with LSP stitching…

[email protected]> show route table customer.inet.0 192.168.10.0/24   

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:03:55, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300592, Push 299920(top)

[email protected]> 

And that looks good. So let’s look at our stitching points…

[email protected]> show route table inet.0 13.13.13.13/32    

inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

13.13.13.13/32     *[OSPF/10] 00:33:21, metric 3
                    > to 169.254.10.19 via ge-0/0/1.0

[email protected]> 

You see – the CSC-CE routers think that their best route to get to 13.13.13.13/32 is through OSPF. Which is totally fair, but also means that we won’t be hopping into an LSP on vMX10 to reach vMX13. And since vMX13 is the endpoint that vMX2 wants to reach…

[email protected]> show route table customer.inet.0 192.168.10.0/24    

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:07:36, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300592, Push 299920(top)

[email protected]> 

We now have a problem. We’ve covered the fix(es) for this before and in this case, we should be able to get away with the safest option which is to use mpls-forwarding

vMX5 and vMX10

set protocols mpls traffic-engineering mpls-forwarding

With that in place, our preferred route on vMX10 should now become the LSP route from LDP…

[email protected]> show route table inet.0 13.13.13.13/32    

inet.0: 20 destinations, 23 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

13.13.13.13/32     @[OSPF/10] 00:00:48, metric 3
                    > to 169.254.10.19 via ge-0/0/1.0
                   #[LDP/9] 00:00:48, metric 1
                    > to 169.254.10.19 via ge-0/0/1.0, Push 299904

[email protected]> 

And our pings should now be working as expected…

[email protected]_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
[email protected]_client:~# 

Awesome. So let’s walk through one direction of this flow with labels just so you can see what’s going on. We’ll start with vMX2…

[email protected]> show route table customer.inet.0 192.168.10.0/24 

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:02:19, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300624, Push 299920(top)

[email protected]> 

vMX2 is pushing a whopping 3 labels onto the stack. Let’s see what vMX3 does with that top label…

[email protected]> show route table mpls.0 label 299920 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299920             *[LDP/9] 00:39:30, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0, Swap 299920

[email protected]> 

It will swap it for the same label. Now onto vMX4…

[email protected]> show route table mpls.0 label 299920 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299920             *[LDP/9] 00:40:00, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, Pop      
299920(S=0)        *[LDP/9] 00:40:00, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, Pop      

[email protected]> 

vMX4 is the PHP router for the LDP LSP ending at vMX5 so it will pop the label. This means that vMX5 will get a top label of 300624 which was imposed at vMX2…

[email protected]> show route table mpls.0 label 300624 

mpls.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300624             *[VPN/170] 00:04:22
                    > to 169.254.10.9 via ge-0/0/1.0, Swap 300832

[email protected]> 

vMX5 will swap that label for 300832

[email protected]> show route table orange_carrier.mpls.0 label 300832 

orange_carrier.mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300832             *[VPN/170] 00:05:07, metric2 3, from 9.9.1.1
                    > to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9

[email protected]>    

vMX6 will use that label to get the traffic into the RSVP LSP to vMX9. Let’s look at the actual label operation…

[email protected]> show route forwarding-table vpn orange_carrier label 300832 
Routing table: orange_carrier.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
300832             user     0                    indr  1048577     2
                              169.254.10.11     Swap 300848, Push 299904(top)      610     2 ge-0/0/1.0

[email protected]> 

Wow – so vMX6 is going to swap the label of 300832 for 300848 and then push 299904 onto the label. So our label stack at this point looks like 299904 300848 16 (assuming the topmost label is on the left). It’s worthwhile here to know where these labels come from. We can assume that 299904 is an RSVP label since it’s the topmost label and we are no in an RSVP domain…

[email protected]> show rsvp session    
Ingress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
9.9.1.1         6.6.1.1         Up       0  1 FF       -   299904 to-vmx9
Total 1 displayed, Up 1, Down 0

Egress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
6.6.1.1         9.9.1.1         Up       0  1 FF       3        - to-vmx6
Total 1 displayed, Up 1, Down 0

Transit RSVP: 0 sessions
Total 0 displayed, Up 0, Down 0

[email protected]> 

Jackpot – now what about 300848?

[email protected]> show route receive-protocol bgp 9.9.1.1 13.13.13.13/32 extensive 

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 13.13.13.13/32 (1 entry, 1 announced)
     Import Accepted
     Route Distinguisher: 1:1
     VPN Label: 300848
     Nexthop: 9.9.1.1
     MED: 3
     Localpref: 100
     AS path: 65100 I 
     Communities: target:1:1

bgp.l3vpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)

* 1:1:13.13.13.13/32 (1 entry, 0 announced)
     Import Accepted
     Route Distinguisher: 1:1
     VPN Label: 300848
     Nexthop: 9.9.1.1
     MED: 3
     Localpref: 100
     AS path: 65100 I 
     Communities: target:1:1

[email protected]b> 

It’s our BGP LU label used to reach the final LSP endpoint! Ok, let’s keep moving along…

[email protected]> show route forwarding-table label 299904 
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
299904             user     0 169.254.10.13     Swap 299904      584     2 ge-0/0/1.0

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      540     1

[email protected]> 

vMX7 will swap the top label with 299904 and send it on it’s way to vMX8…

[email protected]> show route forwarding-table label 299904 
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
299904             user     0 169.254.10.15     Pop        584     2 ge-0/0/1.0
299904(S=0)        user     0 169.254.10.15     Pop        585     2 ge-0/0/1.0

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      553     1

[email protected]> 

vMX8 is the PHP router in the RSVP domain so it will pop the label and send the remaining label stack to vMX9 (which is 300848 16 (assuming the topmost label is on the left))…

[email protected]> show route table mpls.0 label 300848 

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300848             *[VPN/170] 00:11:32
                    > to 169.254.10.17 via ge-0/0/1.0, Swap 300800

[email protected]> 

vMX9 will swap the label for 300800. Now this is crucial here. This is the BGP LU route label that vMX10 allocated and sent to vMX9 to tell it how to reach vMX13…

[email protected]> show route advertising-protocol bgp 169.254.10.16 13.13.13.13/32 extensive   

inet.0: 20 destinations, 23 routes (20 active, 0 holddown, 0 hidden)
@ 13.13.13.13/32 (2 entries, 2 announced)
 BGP group to_green_carrier type External
     Route Label: 300800
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     AS path: [65100] I 
     Entropy label capable

[email protected]> 

And if we look – vMX10 will use that label to stitch the traffic into the LDP LSP toward vMX13…

[email protected]> show route table mpls.0 label 300800 

mpls.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300800             *[VPN/170] 00:13:19
                    > to 169.254.10.19 via ge-0/0/1.0, Swap 299904

[email protected]> show ldp database                
Input label database, 10.10.10.10:0--11.11.11.11:0
Labels received: 4
  Label     Prefix
 299872      10.10.10.10/32
      3      11.11.11.11/32
 299888      12.12.12.12/32
 299904      13.13.13.13/32

Output label database, 10.10.10.10:0--11.11.11.11:0
Labels advertised: 4
  Label     Prefix
      3      10.10.10.10/32
 300720      11.11.11.11/32
 300736      12.12.12.12/32
 300752      13.13.13.13/32

[email protected]> 

vMX11 will then get a label of 299904

[email protected]> show route table mpls.0 label 299904 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:14:35, metric 1
                    > to 169.254.10.21 via ge-0/0/1.0, Swap 299904

[email protected]> 

Which it will then swap for 299904 before sending it to vMX12…

[email protected]> show route table mpls.0 label 299904 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:50:57, metric 1
                    > to 169.254.10.23 via ge-0/0/1.0, Pop      
299904(S=0)        *[LDP/9] 00:50:57, metric 1
                    > to 169.254.10.23 via ge-0/0/1.0, Pop      

[email protected]> 

vMX12 as the PHP router in this LDP domain will pop the label leaving only the initial VPN label imposed by vMX2 on the frame which it will send to vMX13…

[email protected]> show route table mpls.0 label 16 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

16                 *[VPN/0] 00:51:44
                    > via lsi.2 (customer), Pop      

[email protected]> 

Which vMX13 will pop off and dump into the customer routing-instance. If we put that all together in a diagram it would look like this…

Alright. So there you have it. As you can see, the configuration was just about as involved as using LDP and OSPF but strongly mimicked what we had seen earlier when using BGP-LU for LSP stitching. Clearly design requirements would win in terms of which way you do CsC but I can see the argument for having more routing policy flexibility when using BGP-LU.

Leave a Reply

Your email address will not be published. Required fields are marked *