eBGP Multihop vs TTL-Security

It’s a well known fact that eBGP peers need to be (by default) directly connected.  That is, the BGP packets generated by a BGP speaker have a TTL of one.  When a BGP peer receives the packet, it decrements the TTL on ingress and process the packet normally.  If the BGP peer is more than one layer 3 hop away, the first router to receive the packet from the BGP speaker will decrement the TTL of the packet.  So in the case of a BGP peer multiple hops away, the eBGP packet sent by the BGP speaker will have a TTL of one.  The next router in the path to the peer will decrement the TTL to 0, realize it can’t forward the packet, drop it, and generate an ICMP TTL expired in transit back to router1.  Let’s examine this behavior by using this lab topology…

Let’s assume here that we want to setup a eBGP peering between router1 and router4.  To make matters a little more interesting, let’s also use the loopback interfaces on each router as the update-source for the BGP session.  I’ll be using OSPF as the IGP to get reachability between all of the networks.  The BGP config on router1 and router4 will then look like this…

Router1

Router4

Taking a look at the BGP summary, we see that both routers are in the IDLE mode…

Let’s take a quick look at the BGP packets being sent from router1 and examine their TTL…

A capture on the router1’s interface facing router2 curiously doesn’t show any BGP traffic at all.  This is rather strange.  I would have expected to see BGP packets being sent towards 172.64.32.2 with the default TTL of 1.  However, it appears that this isn’t occurring.  A quick look at the ‘show ip bgp neighbors’ commands yields this output…

Interesting.  So the router knows that the neighbor isn’t directly connected.  Apparently this means that the router knows it’s hop count is 1 and that since it isn’t directly connected it doesn’t even try to establish a connection.  We can tell the router to still attempt the the connection by issuing this command under the BGP configuration…

Now if we take a look at our capture, we should see some BGP packets…

If you look at the above output closely we can see that router1 is attempting to establish a TCP connection on port 179 with router4.  We can also see that router2 (10.0.0.13) is replying back to router1 informing it that the TTL has expired in transit.  An up close examination of the TCP packet shows the TTL of the packet to be 1 which is what we’d expect to see…

Let’s go ahead and re-enable the directly connected neighbor check on router1 and instead change the eBGP multihop setting…

A second look at the capture shows the TCP packets being generated with a TTL of a 2 and the ICMP TTL expired in transit packets being generated by router3 (10.0.0.10)…

Let’s take a look at the diagram again to make sure we understand where we’re at…

We are now telling router1 to attempt it’s connection to router4 so long as it is less than or equal to 2 hops away.  We saw the TCP SYN leave router1 destined for router4 with a TTL of 2.  Router2 receives the packet, processes it, decrements the TTL by 1, and sends it on its way to router3 with a TTL of 1.  Router 3 process the packet and realizes that it cant send a packet with a TTL of 0 (after it decrements it), and replies with a ICMP TTL expired in transit. 

Side Note:  This is probably a good time to discuss how Cisco routers handle TTL in general.  From my experience, Cisco network devices process TTL in this manner.  If the packet is destined for the router (control plane etc.) a packet coming across the wire to the device with a TTL of 1 is fine.  The router will decrement the TTL of the packet on interface ingress, and then pass the packet up to the control plane.  This implies that a TTL of 0 is fine on a local device EVEN if you are headed to a loopback interface on the router.  However, if the packet is not destined for the router the router will decrement the TTL, do a forwarding lookup, realize the packet is not destined for itself, and will then drop the packet and generate a ICMP TTL expired message back to the sender.

Now let’s change the eBGP multihop setting to 3 on router1 and see what happens…

We can see that router4 is sending a TCP reset back to router1…

Much like how router1 didn’t want to play ball when it knew it’s neighbor was directly connected, router4 is in the same boat.  It’s completely refusing the TCP session.  If we were to enable the ‘disable-connected-check’ on router4 we’d see that router4’s replies to router1 would be dropped on router3 since they only have a TTL of 1.  Let’s confirm that theory…

Just as we thought, lets look at the packets in more detail…

Here’s router1 reaching router4.  By this time the TTL of the packet is 1…

Here’s the reply from router4 back to router1.  Notice that the TTL of this reply is also 1…

And finally, here’s router3 telling router4 that the TTL expired in transit…

So now let’s turn the eBGP multihop on router4 to three and see what happens…

And the session is now up!  Now let’s look at some quick verification commands to determine how eBGP multihop is configured…

The allowed hop count is recorded in the ‘show ip bgp neighbor’ command.  Note that when you are using the default hop count of 1, this command won’t display anything.  The full output of the ‘show ip bgp neighbor’ command will show whether or not the neighbor is directly connected.

So now that we beat the snot out of eBGP multihop, let’s talk about the other option which is TTL-Security.  When I first read about TTL-Security, I was rather confused.  Not by the concept of it, but why I would want to use it rather than eBGP multihop.  It should be noted here that the two features are mutually exclusive.  AKA, you can only use one of them at a time.  So let’s start with the definition of TTL-Security from Cisco…

This feature protects the eBGP peering session by comparing the value in the TTL field of received IP packets against a hop count that is configured locally for each eBGP peering session. If the value in the TTL field of the incoming IP packet is greater than or equal to the locally configured value, the IP packet
is accepted and processed normally. If the TTL value in the IP packet is less than the locally configured value, the packet is silently discarded and no ICMP message is generated. This is designed behavior; a response to a forged packet is unnecessary

Sounds quite a bit like eBGP multihop in some ways doesn’t it?  Well, the key to understanding TTL-Security (at least for me) was these two sentences…

eBGP multihop configures the maximum number of hops in which a eBGP speaker can use to reach a eBGP peer.  TTL-Security assumes the default TTL of 255 is being used and ensures that the TTL of the received packet is greater than or equal to the minimum TLL (255 minus configured hop count).

Confused?  Let’s run through a quick example so you can see what I mean.  I’m going to remove the eBGP multihop command from router1 and router4 and reconfigure them with the TTL-Security command…

So let’s check out what we see on the wire…

Here we can see the packet from router4 headed to router1.  Check out the TTL of the packet…

Now check out the TTL of the packet coming from router1 headed to router4 as it is seen on the wire between router3 and router4…

As you can see, the initial packets being generated by each router are starting at 255.  The packet sent by router1 reaches a TTL of 253 by the time it hits the wire between router3 and router4.  So what’s going on here?  The TTL is more than sufficient for the TCP packets to arrive at each router.  Why isn’t the session coming up?

TTL-Security takes the number you supply in the ‘ttl-security hop’ command and uses it to determine a minimum incoming TTL.  The maximum TTL is assumed on Cisco devices to be 255.  That being said, specifying a TTL-security value of 2 means my TTL minimum is 253 and the TTL maximum is 255.  When router 1 sends the packet to router 4 we saw the packet leaving router3 headed to router4 with a TTL of 253. 

So if the value of the incoming packet was 253 why doesn’t the connection work when the minimum TTL is 253?  The bottom line is that Cisco router decrement the TTL of a packet on ingress to an interface (see side note above).  The instant that packet is received by the router, the TTL is decremented before it can reach the control plane. 

Making sense?  Let’s have some picture to clarify the point…

As we can see above, the TTL starts off with 255 and then decrements.  The red lines show the TTL ‘on the wire’.  That is, the TTL generated by the router that pushes the frame onto that segment.  So while the TTL of the packet between router3 and router4 is 253, router4 still has to do it’s job and decrement the TTL to 252.  The general concept is the same between eBGP multihop and TTL-Security, but the purpose of each is rather different.  Take these two examples for instance…

Consider we have a rogue router that’s many hops away.  The rogue router and router1 both appear to be within the correct distance for router4 to accept the session.  Granted, router4’s reply to the rogue router won’t work since he likely has en eBGP multihop setting of 3 (like router1 in this example) but router4 will still accept the initial TCP SYN for the BGP session.  If we change this to TTL-Security we get something that looks like this…

Considering we had TTL-Security configured for 3 hops on both router1 and router4, the rogue router wouldn’t be able to create a session to router4 because it’s arriving TTL is not in between the minimum(252) and maximum(255) range.  However, the connection from router1 would work just fine since it’s arriving TTL at router4 is 253 (252 once router4 processes it).  Once configured we’d see something like this on router4…

So as you can see, there is a distinct use case for TTL-Security.  I hope this helps clear things up a bit.