MPLS 101 – The Basics

In this series of posts, I want to spend some time reviewing MPLS fundamentals.  This has been covered in many places many times before – but I’ve noticed lately that often times the basics are missed or skipped when looking at MPLS.  How many “Introduction to MPLS” articles have you read where the first step is “Enable LDP and MPLS on the interface” and they dont actually explain whats happening?  I disagree with that being a valid starting point so in this post I’d like to start with the basics.  Subsequent posts will build from here as we get more and more advanced with the configuration.

Warning: In order to get up and running with even a basic configuration we’ll need to introduce ourselves to some MPLS terminology and concepts in a very brief fashion.  The descriptions of these terms and concepts is being kept brief intentionally in this post and will be covered in much great depth in a future post.

Enough rambling from me, let’s get right into it…

So what is MPLS?  MPLS stands for Multi-Protocol Label Switching and it provides a means to forward multiple different protocols across a network.  To see what it’s capable of, let’s dive into a real working example of MPLS.

Note: I encourage you to follow along with the examples by using virtual routing instances.  I’ll be using the vMX from Juniper which is free to try and full featured.  You can check it out here.

Above we have a very simple network topology.  Four routers connected together in a chain with two clients (apparently people pointing their fingers in the air) at either end of the chain.  At this point, the routers have a simple configuration that includes their interface IP addressing and that’s about it.  The configurations look like this…

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.2.2.0/31;
            }
        }
    }
    ge-0/0/1 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.0/31;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.1/32;
            }
        }
    }
}

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.1/31;
            }
        }
    }
    ge-0/0/1 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.2/31;
            }
        }
    }
}

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.3/31;
            }
        }
    }
    ge-0/0/1 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.4/31;
            }
        }
    }
}

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.5/31;
            }
        }
    }
    ge-0/0/1 {
        enable;
        unit 0 {
            family inet {
                address 10.2.2.2/31;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 4.4.4.4/32;
            }
        }
    }
}

So nothing exciting here – what we currently have is a fairly broken IP network. None of the routers or clients can communicate to anything besides their directly connected interfaces (perhaps the reason the clients are raising their hands). To be fair, the clients can at this point talk to their directly connected router’s interfaces since they are using them as their default gateway. But they can’t talk past there.

So how do we fix this? Well typically we’d just configure something like OSPF on all of the routers, let them advertise their connected networks to each other, and things would magically work. That’s all well and good, but that also means we need to share a lot of state with all of the routers. Each router has to learn about all of the prefixes in the network. Certainly in this case, that’s not a concern, but what if this network was 100 million times bigger? If it was – it might be worth while to look at other options to keep the amount of state being shared between the devices as small as possible especially in the case of routers 2 and 3 which aren’t even directly connected to a client. However – without knowledge of all possible networks, we’d need to find another means to forward traffic instead of relying on standard IP forwarding. Enter MPLS.

MPLS offers a new means of forwarding traffic that relies on labels rather than IP addresses to perform forwarding actions.  So rather than relying on doing IP lookups at each hop, the router will perform a label lookup.  MPLS tags are inserted in between the layer 2 header and the IP packet…

Given the tag placement, MPLS is often said to use a ‘shim’ header.  While the MPLS header includes more than just a label (tag) let’s just focus on the label for now.  We’ll revisit this later on, but the fact that this is a shim header is important to understand.  Many people incorrectly believe that MPLS represents a totally different means to transport data across a network.  That’s not the case at all.  Since the L2 header is still the outer header that means that we’re still using L2 forwarding semantics to get the packet from point A to point B.  The MPLS header is just additional information that can be acted on once a device discards the L2 header.  This is the beginning of a longer theme about how tightly coupled MPLS is to the underlay network it rides on top of.

So now that we have a label we need to do something with it.  When an MPLS enabled router receives a packet – it can perform three basic actions.  It can push a label, swap a label, or pop a label.  Pushing a label would occur when you wish to have traffic enter the MPLS network.  Swapping happens inside of the MPLS network and represents the basic forwarding action MPLS relies on.  Popping a tag occurs as traffic leaves the MPLS network and egresses back onto a normal IP network.

Note: These are certainly not the only times these actions are used but for the sake of keeping things simple in this intro post let’s assume they are.  We’ll cover more advanced use cases in later posts.

So now that we know some of the basics – lets get some of the configuration done and then come back and explain it.

The first thing we have to do is configure the routers interfaces for MPLS. To do so on a Juniper router you configure both the physical interface for MPLS as well as specify the interface under the MPLS protocol configuration. Our router 1 configuration would now look like this…

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.2.2.0/31;
            }
        }
    }
    ge-0/0/1 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.0/31;
            }
            family mpls;
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.1/32;
            }
        }
    }
}
protocols {
    mpls {
        interface ge-0/0/1.0;
    }
}

Notice the MPLS configuration under both the interface as well as under the protocol mpls stanza. It’s important to enable MPLS in both places in order for MPLS transport to work. Let’s make the same changes to the rest of the routers to enable their transit interfaces for MPLS…

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.1/31;
            }
            family mpls;
        }
    }
    ge-0/0/1 {
        unit 0 {
            family inet {
                address 10.1.1.2/31;
            }
            family mpls;
        }
    }
}
protocols {
    mpls {
        interface ge-0/0/0.0;
        interface ge-0/0/1.0;
    }
}

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.3/31;
            }
            family mpls;
        }
    }
    ge-0/0/1 {
        unit 0 {
            family inet {
                address 10.1.1.4/31;
            }
            family mpls;
        }
    }
}
protocols {
    mpls {
        interface ge-0/0/0.0;
        interface ge-0/0/1.0;
    }
}

interfaces {
    ge-0/0/0 {
        enable;
        unit 0 {
            family inet {
                address 10.1.1.5/31;
            }
            family mpls;
        }
    }
    ge-0/0/1 {
        enable;
        unit 0 {
            family inet {
                address 10.2.2.2/31;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 4.4.4.4/32;
            }
        }
    }
}
protocols {
    mpls {
        interface ge-0/0/0.0;
    }
}

Once again – nothing exciting about this. All we’ve done at this point is enabled the interfaces for MPLS transport.

Note: This is where things diverge drastically between vendors, specifically Cisco and Juniper. Since I’ll be using the vMX for this series of posts we’ll be talking about how this is handled with Juniper. While many things are similar, and they certainly interoperate, just keep in mind that the default behavior for many things is different between the two vendors.

So let’s take a step back here and talk about how this works with IP. If the top client is using router 1 as it’s default gateway, when it sends traffic destined to the bottom client, router 1 has to sort out what to do with it. In the case of normal IP forwarding the router would look at the incoming packets destination IP, consult it’s routing table, hopefully find a prefix that matches the destination IP of the packet, and forward the packet on to the next hop the route specifies. If we wish to forward the top clients packet using MPLS transport, we need a means to tell the router to change it’s normal forwarding behavior. In other words – the router needs some mechanism to tell it what traffic should be sent using MPLS tags. That mechanism comes in the form of a LSP – or a label switched path. A LSP defines a path through an MPLS network. For the sake of keeping things simple as we begin our MPLS labs we’ll rely on statically defined LSPs. Static LSPs are defined under the MPLS protocol section of the configuration as follows…

protocols {
    mpls {
        static-label-switched-path router1->router4 {
            ingress {
                next-hop 10.1.1.1;
                to 4.4.4.4;
                push 1001001;
            }
        }
        interface ge-0/0/1.0;
    }
}

Above we have a new LSP called router1->router4. The LSP defines…

• A method of ingress meaning that this will be the entry point of traffic entering the LSP.
• A destination using the to definition. In this case the LSP is headed to or terminates at 4.4.4.4 which is the loopback IP address of router 4.
• A action of push meaning this router will push a label onto the packet
• A next-hop of 10.1.1.1.

Most of these items should make at least some sense to you.  We know we want to use MPLS to forward the traffic so the ingress method makes sense.  We know that we want to get the traffic all the way to router 4 since that’s what the bottom client is directly connected to so saying that we’re heading to 4.4.4.4 also seems to add up.  We also talked about how when we enter a MPLS network we need to push a label so I’m also OK with making that conclusion.  What’s curious is the next-hop of 10.1.1.1.  Why do we need a next hop IP if we aren’t relying on IP forwarding to move the traffic?  The next-hop is required so we know what interface to send the traffic out of.  While slightly deceiving, the IP address you specify for the next hop is used to resolve what interface should be used to send the labelled traffic out of.  The router will consult the IP routing table, determine which interface is used to reach that IP, and then use that as the egress interface when sending traffic down that LSP.  Remember, MPLS isn’t using the IP header to make any forwarding decisions.  The fact that you specify an IP for the next-hop is simply a matter of convenience and is then resolved to a next-hop interface for the local router.

So let’s install this static LSP on router 1.  If you want to copy and paste the configuration load merge terminal is your friend here.  So now that we have defined the ingress to our LSP, we need to define the rest of the path.  We just told router 1 to push a label of 1001001 onto the packet as it enters the LSP.  We also told it to send the newly labeled packet over to router 2.  When router 2 receives the labeled packet it needs to perform a MPLS forwarding operation.  In this case, it will be a swap.  Let’s tell router 2 to swap the label 1001001 with 1001002…

protocols {
    mpls {
        static-label-switched-path router1->router4 {
            transit 1001001 {
                next-hop 10.1.1.3;
                swap 1001002;
            }
        }
    }
}

The above label operation will be part of the same LSP we started on router 1 and defines…

• A method of transit meaning that this router will not be starting or terminating an LSP. Rather we’ll be transiting traffic across the router
• A action of swap meaning this router will examine the incoming label and if it matches the label defined in the method (1001001) it will swap the label for 1001002
• A next-hop of 10.1.1.3. As we saw above this means that it will send the relabeled packet out of its interface towards router 3.

Now our MPLS packet is making it’s way to router 3.  Router 3 will need to deal with the MPLS packet as well, however, its not going to perform a swap operation.  Rather, it’s going to perform a pop operation.  This might seem strange at first since there is another router (router 4) in path to reach the bottom client, however we want to try and minimize the operations each router has to do.  If we sent a packet with an MPLS header to router 4 it would not only need to pop the MPLS tag, it would also need to do an IP lookup.  So rather than making router 4 do all that work, we simply tell router 3 to pop the label off and send the packet on it’s way to router 4 who can then just act on the IP packet alone.  This function is called PHP or penultimate hop pop and is very common in MPLS networks.  So let’s configure this on router 3…

protocols {
    mpls {
        static-label-switched-path router1->router4 {
            transit 1001002 {
                next-hop 10.1.1.5;
                pop;
            }
        }
    }
}

Once again – the above label operation will be part of the same LSP we started on router 1 and 2 and defines…

• A method of transit meaning that this router will not be starting or terminating an LSP. Rather we’ll be transiting traffic across the router
• A action of pop meaning this router will examine the incoming label and if it matches the label defined in the method (1001002) it pop off the label before forwarding
• A next-hop of 10.1.1.5 to send the packet toward router 4.

Now that we’ve defined the LSP we need to tell the router to use it to get to the bottom client.  Let’s look at our routing table now on router 1 and see what it looks like…

root@vmx1> show route table inet

inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[Direct/0] 3d 19:49:30
                    > via lo0.0
10.1.1.0/31        *[Direct/0] 3d 20:31:11
                    > via ge-0/0/1.0
10.1.1.0/32        *[Local/0] 3d 20:31:11
                      Local via ge-0/0/1.0
10.2.2.0/31        *[Direct/0] 3d 13:21:20
                    > via ge-0/0/0.0
10.2.2.0/32        *[Local/0] 3d 13:21:20
                      Local via ge-0/0/0.0
10.20.30.0/24      *[Direct/0] 3d 21:13:17
                    > via fxp0.0
10.20.30.132/32    *[Local/0] 3d 21:13:17
                      Local via fxp0.0

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

4.4.4.4/32         *[MPLS/6/1] 3d 01:02:13, metric 0
                    > to 10.1.1.1 via ge-0/0/1.0, Push 1001001

Notice that in addition to the normal inet.0 routing table we now also have a inet.3 routing table.  Where did this come from?  When we defined the ingress LSP ending at 4.4.4.4 the router created an entry for 4.4.4.4 in the inet.3 table.  The table is often called by other names, but it’s a special table in Junos that is used to lookup BGP next hops.  Recall that BGP is unique in the fact that it can put a next hop in the routing table that is not directly connected.  To understand how this is used with MPLS we need to talk briefly about some more MPLS terminology that will be discussed in more detail in later posts.  The inet.3 table is sometimes also called the FEC mapping table.  So what is a FEC?  FEC stands for Forwarding Equivalence Class and defines a group of packets that are sent to the same next-hop, out the same interface, using the same behavior (think QOS etc).  FECs aren’t unique to MPLS.  In standard IP routing FECs are defined the same way, the difference is that they are defined on a hop by hop basis.  In MPLS, since we build LSPs across an MPLS cloud (virtual circuits) the FEC is the same end to end.  So how is this different than an LSP?  An LSP is a generic path through an MPLS cloud.  Many FECs could use the same LSP.  That is a FEC with a lower priority could use the same LSP as one with a higher priority.  So LSPs define the path of the virtual circuit while FECs define more granular policy on a more specific set of classified flows.

So why is the inet.3 table also called the FEC mapping table?  Because its used to determine the FEC for a given packet flow.  If we look at the routing table above we can see that the inet.3 table shows an entry for 4.4.4.4/32.  That’s the loopback of router 4 and the endpoint for the static LSP we defined.  It lists the other information we need to get to that endpoint.  Namely, the next hop (which resolves to the egress interface), and the MPLS label operation that we need to use to get into that FEC or LSP.  The problem we have now is that while we know how to get there, we dont know how to get traffic into the LSP.  Note that the inet.0 table still lacks an entry for the subnet of the bottom client (10.2.2.2/31).  So how do we fix that?  Well typically BGP would advertise the remote prefix for us (we’ll see this in a later post) and BGP would lookup the next-hop in the inet.3 table which would then put our entry into the inet.0 table.  Since we are doing everything manually, we can still make this happen with a static route.  For instance we can put this config in router 1…

routing-options {
    static {
        route 10.2.2.2/31 {
            next-hop 4.4.4.4;
            resolve;
        }
    }
}

Much like BGP would do for us, we’re telling router 1 here that the route to reach 10.2.2.2/31 is reachable through 4.4.4.4. Since 4.4.4.4 is not directly connected we need to tell the router to `resolve` 4.4.4.4 into a usable next hop.

Note: Another difference between IOS and JunOS is that in JunOS you need to tell the router to recurse (resolve) a route.  IOS does that automatically.  

When the router resolves the route, it consults the inet.3 table just like BGP would. When it does that, it finds an entry for 4.4.4.4 because of our static LSP. Now if we look at the inet.0 table we’ll see a usable entry to reach the bottom client…

root@vmx1> show route table inet

inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[Direct/0] 3d 20:08:57
                    > via lo0.0
10.1.1.0/31        *[Direct/0] 3d 20:50:38
                    > via ge-0/0/1.0
10.1.1.0/32        *[Local/0] 3d 20:50:38
                      Local via ge-0/0/1.0
10.2.2.0/31        *[Direct/0] 3d 13:40:47
                    > via ge-0/0/0.0
10.2.2.0/32        *[Local/0] 3d 13:40:47
                      Local via ge-0/0/0.0
10.2.2.2/31        *[Static/5] 00:02:25, metric2 0
                    > to 10.1.1.1 via ge-0/0/1.0, Push 1001001
10.20.30.0/24      *[Direct/0] 3d 21:32:44
                    > via fxp0.0
10.20.30.132/32    *[Local/0] 3d 21:32:44
                      Local via fxp0.0

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

4.4.4.4/32         *[MPLS/6/1] 3d 01:21:40, metric 0
                    > to 10.1.1.1 via ge-0/0/1.0, Push 1001001

Success! At this point, we should be able to see some of this in action. Let’s start a ping on the top client toward the bottom client and take a couple of packet captures alone the way.

Here is the ICMP request on the wire before it gets to router 1. Looks like a normal frame and packet…

Now below is the same frame and packet as it traverses the link between router 1 and router 2. Notice that addition of the MPLS header with it’s label of 1001001.  Based on the ingress LSP we configured on router 1 this makes total sense.  Also notice that in the ethernet (L2) header the type has changed from 0x0800 (IPv4) to 0x8847 (MPLS).  This is how a receiving router knows to process this frame as an MPLS datagram…

If we capture the frame again on the link between routers 2 and 3 we’ll see the MPLS header has changed to reflect the new label.  Also notice that the MPLS TTL has been decremented.  Since we’re dealing with a new forwarding paradigm here (not IP) we still need a means to keep track of TTL and MPLS acts in much the same way that IP does for TTL – decrementing it at each hop…

The capture between routers 3 and 4 is more interesting because, as you can see, we no longer have an MPLS header.  This is the PHP action I mentioned earlier.  Router 3 pops the MPLS header off before sending it to router 4 so that all router 4 has to do is perform an IP lookup…

And lastly we can see the normal frame and packet making it all the way to the bottom client as we’d expect using a normal L3 forwarding mechanism…

What the capture on the link between the bottom client and router 4 will also show is router 4 generating ICMP unreachable packets back to the bottom client (not pictured).  When the bottom client attempts to return the traffic to the top client it sends its traffic to router 4.  Router 4 has no means to reach the top client at 10.2.2.1 since it’s forwarding table has no entry for it…

root@vmx4> show route table inet

inet.0: 8 destinations, 8 routes (7 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

4.4.4.4/32         *[Direct/0] 3d 01:07:40
                    > via lo0.0
10.1.1.4/31        *[Direct/0] 3d 01:07:40
                    > via ge-0/0/0.0
10.1.1.5/32        *[Local/0] 3d 01:07:40
                      Local via ge-0/0/0.0
10.2.2.2/31        *[Direct/0] 3d 01:07:40
                    > via ge-0/0/1.0
10.2.2.2/32        *[Local/0] 3d 01:07:40
                      Local via ge-0/0/1.0
10.20.30.0/24      *[Direct/0] 3d 01:50:06
                    > via fxp0.0
10.20.30.135/32    *[Local/0] 3d 01:50:06
                      Local via fxp0.0

So this is strange. Why cant the return traffic from the bottom client use the same LSP that the top client used? It’s because LSPs are unidirectional. At this point our MPLS LSP looks like this…

When we defined the LSP on router 1 we defined an endpoint for it to use as router 4. To get return traffic to work we’ll need to define an LSP in the other direction as well.  So our LSPs will look like this…

Above we now show bidirectional LSPs.  With this configuration, the top and bottom client should be able to communicate normally.  The additional configuration for the second LSP on each router is shown below (again, load merge terminal is your friend here)…

protocols {
    mpls {
        static-label-switched-path router4->router1 {
            transit 1001004 {
                next-hop 10.1.1.0;
                pop;
            }
        }
    }
}

protocols {
    mpls {
        static-label-switched-path router4->router1 {
            transit 1001003 {
                next-hop 10.1.1.2;
                swap 1001004;
            }
        }
    }
}

protocols {
    mpls {
        static-label-switched-path router4->router1 {
            ingress {
                next-hop 10.1.1.4;
                to 1.1.1.1;
                push 1001003;
            }
        }
        interface ge-0/0/0.0;
    }
}
routing-options {
    static {
        route 10.2.2.0/31 {
            next-hop 1.1.1.1;
            resolve;
        }
    }
}

Notice how the router 4 configuration also includes the static route to get the traffic into the LSP.  At this point, the top and bottom client should be able to communicate to one another successfully.  And if we look at the routing table of router 2 or 3, we’ll see that they still have no clue about either clients subnet…

root@vmx3> show route table inet

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.1.2/31        *[Direct/0] 3d 23:15:15
                    > via ge-0/0/0.0
10.1.1.3/32        *[Local/0] 3d 23:15:15
                      Local via ge-0/0/0.0
10.1.1.4/31        *[Direct/0] 3d 04:04:58
                    > via ge-0/0/1.0
10.1.1.4/32        *[Local/0] 3d 04:04:58
                      Local via ge-0/0/1.0
10.20.30.0/24      *[Direct/0] 4d 00:02:49
                    > via fxp0.0
10.20.30.134/32    *[Local/0] 4d 00:02:49
                      Local via fxp0.0

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

ff02::2/128        *[INET6/0] 4d 00:02:50
                      MultiRecv

root@vmx3>

Now we’ve just scratched the surface of MPLS and if you’re new to MPLS Im sure you’ll still have many many questions. Hang in there – in the next post we’ll talk about LDP and BGP and how they work in conjunction with BGP. Comments and questions welcome!