eBGP Multihop vs TTL-Security

      16 Comments on eBGP Multihop vs TTL-Security

It’s a well known fact that eBGP peers need to be (by default) directly connected.  That is, the BGP packets generated by a BGP speaker have a TTL of one.  When a BGP peer receives the packet, it decrements the TTL on ingress and process the packet normally.  If the BGP peer is more than one layer 3 hop away, the first router to receive the packet from the BGP speaker will decrement the TTL of the packet.  So in the case of a BGP peer multiple hops away, the eBGP packet sent by the BGP speaker will have a TTL of one.  The next router in the path to the peer will decrement the TTL to 0, realize it can’t forward the packet, drop it, and generate an ICMP TTL expired in transit back to router1.  Let’s examine this behavior by using this lab topology…

image

Let’s assume here that we want to setup a eBGP peering between router1 and router4.  To make matters a little more interesting, let’s also use the loopback interfaces on each router as the update-source for the BGP session.  I’ll be using OSPF as the IGP to get reachability between all of the networks.  The BGP config on router1 and router4 will then look like this…

Router1
image

Router4
image

Taking a look at the BGP summary, we see that both routers are in the IDLE mode…

image

image

Let’s take a quick look at the BGP packets being sent from router1 and examine their TTL…

image

A capture on the router1’s interface facing router2 curiously doesn’t show any BGP traffic at all.  This is rather strange.  I would have expected to see BGP packets being sent towards 172.64.32.2 with the default TTL of 1.  However, it appears that this isn’t occurring.  A quick look at the ‘show ip bgp neighbors’ commands yields this output…

image

Interesting.  So the router knows that the neighbor isn’t directly connected.  Apparently this means that the router knows it’s hop count is 1 and that since it isn’t directly connected it doesn’t even try to establish a connection.  We can tell the router to still attempt the the connection by issuing this command under the BGP configuration…

image

Now if we take a look at our capture, we should see some BGP packets…

image

If you look at the above output closely we can see that router1 is attempting to establish a TCP connection on port 179 with router4.  We can also see that router2 (10.0.0.13) is replying back to router1 informing it that the TTL has expired in transit.  An up close examination of the TCP packet shows the TTL of the packet to be 1 which is what we’d expect to see…

image

Let’s go ahead and re-enable the directly connected neighbor check on router1 and instead change the eBGP multihop setting…

image

A second look at the capture shows the TCP packets being generated with a TTL of a 2 and the ICMP TTL expired in transit packets being generated by router3 (10.0.0.10)…

image

image

Let’s take a look at the diagram again to make sure we understand where we’re at…

image

We are now telling router1 to attempt it’s connection to router4 so long as it is less than or equal to 2 hops away.  We saw the TCP SYN leave router1 destined for router4 with a TTL of 2.  Router2 receives the packet, processes it, decrements the TTL by 1, and sends it on its way to router3 with a TTL of 1.  Router 3 process the packet and realizes that it cant send a packet with a TTL of 0 (after it decrements it), and replies with a ICMP TTL expired in transit. 

Side Note:  This is probably a good time to discuss how Cisco routers handle TTL in general.  From my experience, Cisco network devices process TTL in this manner.  If the packet is destined for the router (control plane etc.) a packet coming across the wire to the device with a TTL of 1 is fine.  The router will decrement the TTL of the packet on interface ingress, and then pass the packet up to the control plane.  This implies that a TTL of 0 is fine on a local device EVEN if you are headed to a loopback interface on the router.  However, if the packet is not destined for the router the router will decrement the TTL, do a forwarding lookup, realize the packet is not destined for itself, and will then drop the packet and generate a ICMP TTL expired message back to the sender.

Now let’s change the eBGP multihop setting to 3 on router1 and see what happens…

image 

We can see that router4 is sending a TCP reset back to router1…

image

Much like how router1 didn’t want to play ball when it knew it’s neighbor was directly connected, router4 is in the same boat.  It’s completely refusing the TCP session.  If we were to enable the ‘disable-connected-check’ on router4 we’d see that router4’s replies to router1 would be dropped on router3 since they only have a TTL of 1.  Let’s confirm that theory…

image

Just as we thought, lets look at the packets in more detail…

Here’s router1 reaching router4.  By this time the TTL of the packet is 1…
image

Here’s the reply from router4 back to router1.  Notice that the TTL of this reply is also 1…
image

And finally, here’s router3 telling router4 that the TTL expired in transit…
image

So now let’s turn the eBGP multihop on router4 to three and see what happens…

image

image

And the session is now up!  Now let’s look at some quick verification commands to determine how eBGP multihop is configured…

image

The allowed hop count is recorded in the ‘show ip bgp neighbor’ command.  Note that when you are using the default hop count of 1, this command won’t display anything.  The full output of the ‘show ip bgp neighbor’ command will show whether or not the neighbor is directly connected.

So now that we beat the snot out of eBGP multihop, let’s talk about the other option which is TTL-Security.  When I first read about TTL-Security, I was rather confused.  Not by the concept of it, but why I would want to use it rather than eBGP multihop.  It should be noted here that the two features are mutually exclusive.  AKA, you can only use one of them at a time.  So let’s start with the definition of TTL-Security from Cisco…

This feature protects the eBGP peering session by comparing the value in the TTL field of received IP packets against a hop count that is configured locally for each eBGP peering session. If the value in the TTL field of the incoming IP packet is greater than or equal to the locally configured value, the IP packet
is accepted and processed normally. If the TTL value in the IP packet is less than the locally configured value, the packet is silently discarded and no ICMP message is generated. This is designed behavior; a response to a forged packet is unnecessary

Sounds quite a bit like eBGP multihop in some ways doesn’t it?  Well, the key to understanding TTL-Security (at least for me) was these two sentences…

eBGP multihop configures the maximum number of hops in which a eBGP speaker can use to reach a eBGP peer.  TTL-Security assumes the default TTL of 255 is being used and ensures that the TTL of the received packet is greater than or equal to the minimum TLL (255 minus configured hop count).

Confused?  Let’s run through a quick example so you can see what I mean.  I’m going to remove the eBGP multihop command from router1 and router4 and reconfigure them with the TTL-Security command…

image

image

So let’s check out what we see on the wire…

Here we can see the packet from router4 headed to router1.  Check out the TTL of the packet…

image

Now check out the TTL of the packet coming from router1 headed to router4 as it is seen on the wire between router3 and router4…

image

As you can see, the initial packets being generated by each router are starting at 255.  The packet sent by router1 reaches a TTL of 253 by the time it hits the wire between router3 and router4.  So what’s going on here?  The TTL is more than sufficient for the TCP packets to arrive at each router.  Why isn’t the session coming up?

TTL-Security takes the number you supply in the ‘ttl-security hop’ command and uses it to determine a minimum incoming TTL.  The maximum TTL is assumed on Cisco devices to be 255.  That being said, specifying a TTL-security value of 2 means my TTL minimum is 253 and the TTL maximum is 255.  When router 1 sends the packet to router 4 we saw the packet leaving router3 headed to router4 with a TTL of 253. 

So if the value of the incoming packet was 253 why doesn’t the connection work when the minimum TTL is 253?  The bottom line is that Cisco router decrement the TTL of a packet on ingress to an interface (see side note above).  The instant that packet is received by the router, the TTL is decremented before it can reach the control plane. 

Making sense?  Let’s have some picture to clarify the point…

image

As we can see above, the TTL starts off with 255 and then decrements.  The red lines show the TTL ‘on the wire’.  That is, the TTL generated by the router that pushes the frame onto that segment.  So while the TTL of the packet between router3 and router4 is 253, router4 still has to do it’s job and decrement the TTL to 252.  The general concept is the same between eBGP multihop and TTL-Security, but the purpose of each is rather different.  Take these two examples for instance…

image

Consider we have a rogue router that’s many hops away.  The rogue router and router1 both appear to be within the correct distance for router4 to accept the session.  Granted, router4’s reply to the rogue router won’t work since he likely has en eBGP multihop setting of 3 (like router1 in this example) but router4 will still accept the initial TCP SYN for the BGP session.  If we change this to TTL-Security we get something that looks like this…

image

Considering we had TTL-Security configured for 3 hops on both router1 and router4, the rogue router wouldn’t be able to create a session to router4 because it’s arriving TTL is not in between the minimum(252) and maximum(255) range.  However, the connection from router1 would work just fine since it’s arriving TTL at router4 is 253 (252 once router4 processes it).  Once configured we’d see something like this on router4…

image

So as you can see, there is a distinct use case for TTL-Security.  I hope this helps clear things up a bit. 

16 thoughts on “eBGP Multihop vs TTL-Security

  1. jdsilva

    Great post Jon.

    I was playing around with this a bit and I noticed one thing that is only implied by the command reference that should be noted. The neighbor ttl-security “feature is not supported for internal BGP (iBGP) peers or iBGP peer groups”. That makes sense since the TTL restriction isn’t in place for iBGP peers… Unless we’re talking about Confederations.

    I tested this with eBGP Confed peers, which do enforce the directly connected rule, and this does not work. In that case, you have only the option of setting the ebg-multihop value.

    I see the logic here since with confederations you *should* have administrative control over the peers, and the purpose of ttl-security *should* be moot, but seeing as how ttl-security and ebgp-multihop are mutually exclusive, and can be used to solve the same problem, it’s a natural assumption that you can also use it to connect confed eBGP peers that are, say, using loopbacks to peer.

    Anyway, thanks for getting me to look a bit deeper at this feature. This is one of those things that I had read over, but never really labbed up. I figured this was one of those features I could wing it with the docs on Lab day if I needed to.

    Reply
      1. Jon Langemak Post author

        Ha! Yeah, I had thought that BGP was one of strong topics. It’s funny that the more I study the more I realize I didn’t know/understand. Thanks for reading!

        Reply
  2. anish

    Team, thanks for the post. As from my observation, Loopback is always 1 hope away from the same router. so i guess minimum 4 hope we have to put here from R1 to R4. Also, we cannot increase the hope with random figure, coz, it will keep on roam here and there. will increase the efficiency and CPU head. Suggest and correct me if i’m wrong, thanks.

    Reply
  3. JL

    Shouldn’t be the ebgp-multihop set to 4 instead of 3? R4 is 3 hops away but the target is the loopback and it is considered another hop isn’t it? I tried to establish an eBGP session between 2 directly connected routers between their loopbacks and I had to set the eBGP multihop value to 2.

    Reply
    1. Jon Langemak Post author

      Good question! Actually, the loopback isnt technically another hop. If you have two routers that are directly connected but you are eBGP peering with the loopbacks you have 2 options. Either change the multihop to 2 as you suggest, or disable the directly connected check. You see, the router knows if a peer is directly connected or not. Disabling the connected check bypasses this check and the router sends the BGP packet on its way with a TTL of 1. The receiving router decrements the TTL on ingress, but since the packet is headed for the control plane (the loopback) its processed. If the packet were destined out another interface, the router would just drop it. Thats what my testing showed.

      Does that make sense? Thanks for reading!

      Reply
      1. kovi

        Hi Jon

        What about ttl-security in this case ? , two routers connected directly but session created based on loopbacks.
        For me it make sense to set ttl-security hop 1 , because router is sending SYN with TTL 255, neighboring router decrements TTL to 254, and it still has to be accepted, and forwarded to control plan (loopback),
        But in lab it doesn’t work , when set ttl-security to 1 , they will not create session.

        Reply
  4. sidd

    Thanks for the great explanation Jon- I have a question,

    In your last picture what happens if the rogue router is connected to Router3 and 2 hops away- then by the time the rogue packet reaches router4 its TTL will be 253 same as router1 packet and will be accepted by router4.

    What is the advantage of ttl security over ebgp-multihop—seems like they both have same issue when it comes to the rogue advertisements- am I missing something?

    Reply
  5. Paulo

    Do ttl-security and ebgp-multihop are interoperable? I mean, they cannot be configured on the same router, but what if I configure a router with ebgp-multihop and the other with ttl-security??

    Paulo

    Reply
  6. Pingback: eBGP Multihop vs TTL-Security – skminhaj

  7. Larry

    Thanks for the excellent descriptions!

    One minor suggestion to increase the clarity of your text:

    it’s = it is
    its = possessive, for example, “its TTL is set to 255.”

    It’s not really a big deal, but I do a fair amount of technical writing and found this error to be distracting.

    Keep up the good work!

    Reply
  8. Ankit

    what if Rouge Router is one hop away from R2 then still R4 will receive packet with with TTL value 253 both from R1 and Rouge routers so in this scenario how R4 determine is that incoming BGP packets are not spoofed.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *