In our last post we talked about the less used method of deploying CsC where we ran OSPF and LDP inside the CSC-PE routing-instance.
Note: I can’t help myself apparently so be aware that Carrier of Carriers (CoC) is the same as Carrier supporting Carrier (CsC)
This required some changes to be made to our default LDP export policy as well as how we moved routes between the inet.3 and inet.0 tables. That being said, if you’re a single org it might make good sense to run things that way. I liked how you were able to see all of the remote LDP domain loopbacks in your local inet.3 table which in my mind made it easier to imagine the LSP paths.
That being said, it is clearly not the preferred deployment methodology. Most examples you’ll find leverage BGP (BGP-LU specifically) for the CSC-CE to CSC-PE connections as well as within the local label domains. So in this example, we’ll do just that. Larges chunks of the base configuration will be the same as they were in the previous post but for the sake of clarity I’ll post our starting post the starting configurations and diagrams here once again. Here is our base lab topology…
And once again we’ll split the lab into the orange carrier, who has a network on the left and the right, and the green carrier, who has the top network…
As before, our goal is to make the orange carrier whole by leveraging connectivity through the green carrier so that the orange carrier can support a VPN service for its customer CE routers. The interesting bit about this is that the connectivity we leverage in the green carrier is itself a VPN hence the terminology of Carrier supporting Carrier. So let’s take a look at our base configuration for each router once again…
Note: Im not going to talk through all the specifics of the base configuration in this post. If you’re curious about it – take a look at the walk through of the base configuration in the last post.
vMX1
set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.1/24
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.0/31
set routing-options autonomous-system 65001
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.1 peer-as 65002
set policy-options prefix-list customer_routes 10.10.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject
vMX2
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.1/31
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.2/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 2.2.2.2/32 primary
set routing-options router-id 2.2.2.2
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/1.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/0.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65002
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.0 peer-as 65001
vMX3
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.3/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.4/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 3.3.3.3/32 primary
set routing-options router-id 3.3.3.3
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
vMX4
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.5/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.6/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 4.4.4.4/32 primary
set routing-options router-id 4.4.4.4
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
vMX5
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.7/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.8/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 5.5.5.5/32 primary
set routing-options router-id 5.5.5.5
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
vMX6
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.9/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.10/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 6.6.1.1/32
set routing-options router-id 6.6.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols mpls label-switched-path to-vmx9 to 9.9.1.1
set protocols mpls interface ge-0/0/1.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
vMX7
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.11/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.12/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 7.7.1.1/32 primary
set routing-options router-id 7.7.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
vMX8
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.13/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.14/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 8.8.1.1/32 primary
set routing-options router-id 8.8.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
vMX9
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.15/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.16/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 9.9.1.1/32
set routing-options router-id 9.9.1.1
set routing-options autonomous-system 65000
set protocols rsvp interface ge-0/0/0.0
set protocols mpls label-switched-path to-vmx6 to 6.6.1.1
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
vMX10
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.17/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.18/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 10.10.10.10/32 primary
set routing-options router-id 10.10.10.10
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
vMX11
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.19/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.20/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 11.11.11.11/32 primary
set routing-options router-id 11.11.11.11
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
vMX12
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.21/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.22/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces fxp0 unit 0 family inet address 192.168.127.12/24
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 12.12.12.12/32 primary
set routing-options static route 10.171.200.0/22 next-hop 192.168.127.100
set routing-options router-id 12.12.12.12
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
vMX13
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.23/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.24/31
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 13.13.13.13/32 primary
set routing-options router-id 13.13.13.13
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/1.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65004
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.25 peer-as 65003
vMX14
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.25/31
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.1/24
set routing-options autonomous-system 65003
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.24 peer-as 65004
set policy-options prefix-list customer_routes 192.168.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject
Right – so if you recall from the last post. This sort of got us what I call our “base label domain” configuration. That is – each “block” should have a means to for an MPLS LSP through it. This is an exact copy and past of the base configuration from the previous post. There should be nothing terribly new or exciting to you about this configuration.
As with the last post, the next step is to configure the CSC-PE routers routing-instance for the orange carrier. This configuration differs slightly from our last configuration…
vMX6
set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/0.0
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 6.6.6.6
set routing-instances orange_carrier routing-options autonomous-system 65200
set routing-instances orange_carrier protocols bgp group to_orange_carrier type external
set routing-instances orange_carrier protocols bgp group to_orange_carrier family inet labeled-unicast
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.8 peer-as 65100
set routing-instances orange_carrier protocols mpls interface ge-0/0/0.0
Notice the lack of OSPF and LDP configuration. This is now replaced by a BGP peering configuration that will be used to peer to the CSC-PE routers. vMX9 looks largely the same…
vMX9
set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/1.0
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 9.9.9.9
set routing-instances orange_carrier routing-options autonomous-system 65200
set routing-instances orange_carrier protocols bgp group to_orange_carrier type external
set routing-instances orange_carrier protocols bgp group to_orange_carrier family inet labeled-unicast
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.17 peer-as 65100
set routing-instances orange_carrier protocols mpls interface ge-0/0/1.0
Notice that we’re using the same AS number for the peerings to the CSC-CE routers. As you might already be guessing, we’ll have to make some adjustments later on for that to work correctly but for now let’s table that.
The next thing to do is to configure BGP within the orange carrier blocks. To do this, we’ll peer the PE router to the CSC-CE node iBGP. Then we’ll also configure a peering between the CSC-CE and the CSC-PE nodes with eBGP. Let’s start on the left side with vMX2…
vMX2
set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 2.2.2.2
set protocols bgp group internal family inet labeled-unicast resolve-vpn
set protocols bgp group internal neighbor 5.5.5.5 peer-as 65100
Nothing too fancy here but do note that we added the flag resolve-vpn to the BGP configuration. If you recall from our previous posts, this allows the router to add prefixes it receives through BGP-LU to the inet.3 table which is required for VPN route resolution. With that in place, we can configure vMX5 for that same peering as well as another BGP group so it can peer to the CSC-PE node (vMX6 in this case)…
vMX5
set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 5.5.5.5
set protocols bgp group internal family inet labeled-unicast
set protocols bgp group internal neighbor 2.2.2.2 peer-as 65100
set protocols bgp group to_green_carrier type external
set protocols bgp group to_green_carrier family inet labeled-unicast
set protocols bgp group to_green_carrier export to_green_carrier
set protocols bgp group to_green_carrier neighbor 169.254.10.9 peer-as 65200
set policy-options policy-statement to_green_carrier term local_loops from prefix-list local_loops
set policy-options policy-statement to_green_carrier term local_loops then accept
set policy-options policy-statement to_green_carrier then reject
set policy-options prefix-list local_loops 2.2.2.2/32
set policy-options prefix-list local_loops 3.3.3.3/32
set policy-options prefix-list local_loops 4.4.4.4/32
set policy-options prefix-list local_loops 5.5.5.5/32
Notice that we’re doing the loopback export on the CSC-CE node. The reason for doing that here and not on vMX2 is that we also know about that route from OSPF. That is – vMX5 prefers the route from OSPF so it won’t advertise a BGP route it learned about through BGP to another BGP peer. Note that we’re advertising all of the loopbacks from the left orange block through the BGP-LU session to vMX6. Now let’s do the same configuration on the right side orange block…
vMX13
set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 13.13.13.13
set protocols bgp group internal family inet labeled-unicast resolve-vpn
set protocols bgp group internal neighbor 10.10.10.10 peer-as 65100
vMX10
set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 10.10.10.10
set protocols bgp group internal family inet labeled-unicast
set protocols bgp group internal neighbor 13.13.13.13 peer-as 65100
set protocols bgp group to_green_carrier type external
set protocols bgp group to_green_carrier family inet labeled-unicast
set protocols bgp group to_green_carrier export to_green_carrier
set protocols bgp group to_green_carrier neighbor 169.254.10.16 peer-as 65200
set policy-options policy-statement to_green_carrier term local_loops from prefix-list local_loops
set policy-options policy-statement to_green_carrier term local_loops then accept
set policy-options policy-statement to_green_carrier then reject
set policy-options prefix-list local_loops 10.10.10.10/32
set policy-options prefix-list local_loops 11.11.11.11/32
set policy-options prefix-list local_loops 12.12.12.12/32
set policy-options prefix-list local_loops 13.13.13.13/32
And last but not least, we need to configure the VPN peering within the green carrier…
vMX6
set routing-options autonomous-system 65200
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 local-address 6.6.1.1
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 peer-as 65200
vMX9
set routing-options autonomous-system 65200
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 local-address 9.9.1.1
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 peer-as 65200
Now what we’ve done here is slightly changed out BGP configuration to make things easier to read. The green carrier in this case is wholly inside of AS 65200 while the orange provider (both blocks) are in AS 65100. Now, we still have the sort of one off eBGP peering between the CE and PE routers in each orange block where we use AS 65001 to 65002 and 65003 to 65004. But to be honest, that really doesn’t matter too much for us at this point. As in the last post, that’s really just a means to get the customer prefixes to the orange PE routers. It could very easily have been anything else including static routes or an IGP etc. So our BGP config at this point looks like this…
The one BGP peering we’re clearly missing here is the iBGP VPNv4 peering between vMX2 and vMX13 but before we configure that let’s make sure we have good reachability…
[email protected]> show route table inet.3
inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2.2.2.2/32 *[LDP/9] 00:14:14, metric 1
> to 169.254.10.2 via ge-0/0/0.0
4.4.4.4/32 *[LDP/9] 00:14:08, metric 1
> to 169.254.10.5 via ge-0/0/1.0
5.5.5.5/32 *[LDP/9] 00:13:59, metric 1
> to 169.254.10.5 via ge-0/0/1.0, Push 299920
[email protected]>
Well that’s not what I had expected. I had hoped that I’d be getting some BGP-LU routes for the routers in the right orange block that be populated into the inet.3 table. So what’s going on? Well – I sort of mentioned this before, but since each CSC-CE is using the same AS number, we’ve got an AS loop. And Juniper is smart enough to know not even to try and advertise routes back to an AS that’s already in a routes AS path list. We can validate that here with just one of the prefixes from the right orange block…
[email protected]> show route table orange_carrier.inet.0 11.11.11.11/32
orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
11.11.11.11/32 *[BGP/170] 00:14:22, MED 1, localpref 100, from 9.9.1.1
AS path: 65100 I, validation-state: unverified
> to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9
[email protected]> show route advertising-protocol bgp 169.254.10.8 11.11.11.11/32
[email protected]>
Notice that the AS path includes 65100 which was added by vMX10 as it advertised the loopbacks to vMX9 over the eBGP peering. Also notice that despite the routes we want being in the table we need them to be in, they aren’t being sent to vMX5. To fix this, we can tell the green provider to do an AS override…
vMX6
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.8 as-override
vMX9
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.17 as-override
AS-override is a commonly misunderstood tool. It lets a router replace all instances of a peer’s AS number in the AS-path list with it’s own. It’s important to know that it works on egress. That is, the AS path will be modified as the prefix leaves the green carrier and is advertised toward the orange carrier. We can see that now that this configuration is in place…
[email protected]> show route receive-protocol bgp 9.9.1.1 11.11.11.11/32
orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 11.11.11.11/32 9.9.1.1 1 100 65100 I
bgp.l3vpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
1:1:11.11.11.11/32
* 9.9.1.1 1 100 65100 I
[email protected]> show route advertising-protocol bgp 169.254.10.8 11.11.11.11/32
orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 11.11.11.11/32 Self 65200 I
[email protected]>
We can see that when we get the route from vMX9 the AS path is unchanged. But when we advertise it to vMX5 we’ve replaced the 65100 AS with 65200. Now you might be looking at this and be wondering why that looks wrong. Indeed, it looks like we’re missing an AS path entry here. It’s each BGP routers job to add it’s local AS number to the route as it is advertised to an eBGP peer. In this case, it appears that all we’ve done is done the replacement. Where is the AS entry for the route coming from 65200 in the first place?
[email protected]> show route receive-protocol bgp 169.254.10.9 11.11.11.11/32
inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 11.11.11.11/32 169.254.10.9 65200 65200 I
[email protected]>
As it turns out it’s there. We just don’t get to see it with the advertising-protocol command for some reason. A minor annoyance, but one that can be particularly irritating as you attempt to do troubleshooting.
Now that we have our AS loop sorted, let’s look back on vMX2 to see what we have in the inet.3 table…
[email protected]> show route table inet.3
inet.3: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
3.3.3.3/32 *[LDP/9] 00:25:07, metric 1
> to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32 *[LDP/9] 00:24:59, metric 1
> to 169.254.10.3 via ge-0/0/1.0, Push 299904
5.5.5.5/32 *[LDP/9] 00:24:49, metric 1
> to 169.254.10.3 via ge-0/0/1.0, Push 299920
10.10.10.10/32 *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
AS path: 65200 65200 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 300608, Push 299920(top)
11.11.11.11/32 *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
AS path: 65200 65200 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
12.12.12.12/32 *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
AS path: 65200 65200 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
13.13.13.13/32 *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
AS path: 65200 65200 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
169.254.10.16/31 *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
AS path: 65200 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 300576, Push 299920(top)
[email protected]>
Success! So at this point, we should be able to configure out iBGP VPNv4 peering for the orange carrier…
vMX2
set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 13.13.13.13 local-address 2.2.2.2
set protocols bgp group vpn neighbor 13.13.13.13 peer-as 65100
vMX13
set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 2.2.2.2 local-address 13.13.13.13
set protocols bgp group vpn neighbor 2.2.2.2 peer-as 65100
And with that complete, our BGP peerings are now complete and should look like this…
But alas – our pings still aren’t working…
left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
--- 192.168.10.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
left_client:~#
So what could be going on now? Well if I were a betting man, I’d guess that as long as the VPNv4 routes are being sent and accepted between vMX2 and vMX13 that we have a problem with LSP stitching…
[email protected]> show route table customer.inet.0 192.168.10.0/24
customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
192.168.10.0/24 *[BGP/170] 00:03:55, localpref 100, from 13.13.13.13
AS path: 65004 65003 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300592, Push 299920(top)
[email protected]>
And that looks good. So let’s look at our stitching points…
[email protected]> show route table inet.0 13.13.13.13/32
inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
13.13.13.13/32 *[OSPF/10] 00:33:21, metric 3
> to 169.254.10.19 via ge-0/0/1.0
[email protected]>
You see – the CSC-CE routers think that their best route to get to 13.13.13.13/32 is through OSPF. Which is totally fair, but also means that we won’t be hopping into an LSP on vMX10 to reach vMX13. And since vMX13 is the endpoint that vMX2 wants to reach…
[email protected]> show route table customer.inet.0 192.168.10.0/24
customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
192.168.10.0/24 *[BGP/170] 00:07:36, localpref 100, from 13.13.13.13
AS path: 65004 65003 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300592, Push 299920(top)
[email protected]>
We now have a problem. We’ve covered the fix(es) for this before and in this case, we should be able to get away with the safest option which is to use mpls-forwarding…
vMX5 and vMX10
set protocols mpls traffic-engineering mpls-forwarding
With that in place, our preferred route on vMX10 should now become the LSP route from LDP…
[email protected]> show route table inet.0 13.13.13.13/32
inet.0: 20 destinations, 23 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
13.13.13.13/32 @[OSPF/10] 00:00:48, metric 3
> to 169.254.10.19 via ge-0/0/1.0
#[LDP/9] 00:00:48, metric 1
> to 169.254.10.19 via ge-0/0/1.0, Push 299904
[email protected]>
And our pings should now be working as expected…
root@left_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
root@left_client:~#
Awesome. So let’s walk through one direction of this flow with labels just so you can see what’s going on. We’ll start with vMX2…
[email protected]> show route table customer.inet.0 192.168.10.0/24
customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
192.168.10.0/24 *[BGP/170] 00:02:19, localpref 100, from 13.13.13.13
AS path: 65004 65003 I, validation-state: unverified
> to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300624, Push 299920(top)
[email protected]>
vMX2 is pushing a whopping 3 labels onto the stack. Let’s see what vMX3 does with that top label…
[email protected]> show route table mpls.0 label 299920
mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
299920 *[LDP/9] 00:39:30, metric 1
> to 169.254.10.5 via ge-0/0/1.0, Swap 299920
[email protected]>
It will swap it for the same label. Now onto vMX4…
[email protected]> show route table mpls.0 label 299920
mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
299920 *[LDP/9] 00:40:00, metric 1
> to 169.254.10.7 via ge-0/0/1.0, Pop
299920(S=0) *[LDP/9] 00:40:00, metric 1
> to 169.254.10.7 via ge-0/0/1.0, Pop
[email protected]>
vMX4 is the PHP router for the LDP LSP ending at vMX5 so it will pop the label. This means that vMX5 will get a top label of 300624 which was imposed at vMX2…
[email protected]> show route table mpls.0 label 300624
mpls.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
300624 *[VPN/170] 00:04:22
> to 169.254.10.9 via ge-0/0/1.0, Swap 300832
[email protected]>
vMX5 will swap that label for 300832…
[email protected]> show route table orange_carrier.mpls.0 label 300832
orange_carrier.mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
300832 *[VPN/170] 00:05:07, metric2 3, from 9.9.1.1
> to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9
[email protected]>
vMX6 will use that label to get the traffic into the RSVP LSP to vMX9. Let’s look at the actual label operation…
[email protected]> show route forwarding-table vpn orange_carrier label 300832
Routing table: orange_carrier.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
300832 user 0 indr 1048577 2
169.254.10.11 Swap 300848, Push 299904(top) 610 2 ge-0/0/1.0
[email protected]>
Wow – so vMX6 is going to swap the label of 300832 for 300848 and then push 299904 onto the label. So our label stack at this point looks like 299904 300848 16 (assuming the topmost label is on the left). It’s worthwhile here to know where these labels come from. We can assume that 299904 is an RSVP label since it’s the topmost label and we are no in an RSVP domain…
[email protected]> show rsvp session
Ingress RSVP: 1 sessions
To From State Rt Style Labelin Labelout LSPname
9.9.1.1 6.6.1.1 Up 0 1 FF - 299904 to-vmx9
Total 1 displayed, Up 1, Down 0
Egress RSVP: 1 sessions
To From State Rt Style Labelin Labelout LSPname
6.6.1.1 9.9.1.1 Up 0 1 FF 3 - to-vmx6
Total 1 displayed, Up 1, Down 0
Transit RSVP: 0 sessions
Total 0 displayed, Up 0, Down 0
[email protected]>
Jackpot – now what about 300848?
[email protected]> show route receive-protocol bgp 9.9.1.1 13.13.13.13/32 extensive
orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 13.13.13.13/32 (1 entry, 1 announced)
Import Accepted
Route Distinguisher: 1:1
VPN Label: 300848
Nexthop: 9.9.1.1
MED: 3
Localpref: 100
AS path: 65100 I
Communities: target:1:1
bgp.l3vpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
* 1:1:13.13.13.13/32 (1 entry, 0 announced)
Import Accepted
Route Distinguisher: 1:1
VPN Label: 300848
Nexthop: 9.9.1.1
MED: 3
Localpref: 100
AS path: 65100 I
Communities: target:1:1
[email protected]>
It’s our BGP LU label used to reach the final LSP endpoint! Ok, let’s keep moving along…
[email protected]> show route forwarding-table label 299904
Routing table: default.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
299904 user 0 169.254.10.13 Swap 299904 584 2 ge-0/0/1.0
Routing table: __mpls-oam__.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 540 1
[email protected]>
vMX7 will swap the top label with 299904 and send it on it’s way to vMX8…
[email protected]> show route forwarding-table label 299904
Routing table: default.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
299904 user 0 169.254.10.15 Pop 584 2 ge-0/0/1.0
299904(S=0) user 0 169.254.10.15 Pop 585 2 ge-0/0/1.0
Routing table: __mpls-oam__.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 553 1
[email protected]>
vMX8 is the PHP router in the RSVP domain so it will pop the label and send the remaining label stack to vMX9 (which is 300848 16 (assuming the topmost label is on the left))…
[email protected]> show route table mpls.0 label 300848
mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
300848 *[VPN/170] 00:11:32
> to 169.254.10.17 via ge-0/0/1.0, Swap 300800
[email protected]>
vMX9 will swap the label for 300800. Now this is crucial here. This is the BGP LU route label that vMX10 allocated and sent to vMX9 to tell it how to reach vMX13…
[email protected]> show route advertising-protocol bgp 169.254.10.16 13.13.13.13/32 extensive
inet.0: 20 destinations, 23 routes (20 active, 0 holddown, 0 hidden)
@ 13.13.13.13/32 (2 entries, 2 announced)
BGP group to_green_carrier type External
Route Label: 300800
Nexthop: Self
Flags: Nexthop Change
MED: 3
AS path: [65100] I
Entropy label capable
[email protected]>
And if we look – vMX10 will use that label to stitch the traffic into the LDP LSP toward vMX13…
[email protected]> show route table mpls.0 label 300800
mpls.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
300800 *[VPN/170] 00:13:19
> to 169.254.10.19 via ge-0/0/1.0, Swap 299904
[email protected]> show ldp database
Input label database, 10.10.10.10:0--11.11.11.11:0
Labels received: 4
Label Prefix
299872 10.10.10.10/32
3 11.11.11.11/32
299888 12.12.12.12/32
299904 13.13.13.13/32
Output label database, 10.10.10.10:0--11.11.11.11:0
Labels advertised: 4
Label Prefix
3 10.10.10.10/32
300720 11.11.11.11/32
300736 12.12.12.12/32
300752 13.13.13.13/32
[email protected]>
vMX11 will then get a label of 299904…
[email protected]> show route table mpls.0 label 299904
mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
299904 *[LDP/9] 00:14:35, metric 1
> to 169.254.10.21 via ge-0/0/1.0, Swap 299904
[email protected]>
Which it will then swap for 299904 before sending it to vMX12…
[email protected]> show route table mpls.0 label 299904
mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
299904 *[LDP/9] 00:50:57, metric 1
> to 169.254.10.23 via ge-0/0/1.0, Pop
299904(S=0) *[LDP/9] 00:50:57, metric 1
> to 169.254.10.23 via ge-0/0/1.0, Pop
[email protected]>
vMX12 as the PHP router in this LDP domain will pop the label leaving only the initial VPN label imposed by vMX2 on the frame which it will send to vMX13…
[email protected]> show route table mpls.0 label 16
mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
16 *[VPN/0] 00:51:44
> via lsi.2 (customer), Pop
[email protected]>
Which vMX13 will pop off and dump into the customer routing-instance. If we put that all together in a diagram it would look like this…
Alright. So there you have it. As you can see, the configuration was just about as involved as using LDP and OSPF but strongly mimicked what we had seen earlier when using BGP-LU for LSP stitching. Clearly design requirements would win in terms of which way you do CsC but I can see the argument for having more routing policy flexibility when using BGP-LU.