Working with VMware NSX – Logical to Physical connectivity

In our last post, we talked about how to deploy what I referred to as logical networking.  I classify logical networking as any type of switching or routing that occurs solely on the ESXi hosts.  It should be noted that with logical networking, the physical network is still used, but only for IP transport of overlay encapsulated packets. 

That being said, in this post I’d like to talk about how to connect our one of our tenants to the outside world.  In order for the logical tenant network to talk to the outside world, we need to find a means to connect the logical networks out to the physical network.  In VMware NSX, this is done with the edge gateway.  The edge gateway is similar to the DLR (distributed local router) we deployed in the last post, however there is one significant difference.  The edge gateway is in the data plane, that is, it’s actually in the forwarding path for the network traffic. 

Note – I will sometimes refer to the edge services gateway as the edge gateway or simply edge.  Despite both the edge services gateway and the DLR both being considered ‘NSX edges’ I will not refer to the DLR as an edge for the sake of clarity.

At the end of our last post, the lab logically looked like this…

image

The web and the app VMs were able to route to each other through the DLR.  In order to extend that connectivity to the physical network, we need to deploy the edge gateway and connect it to the physical MLS.  By the end of this post, our logical network should look like this…

image

So let’s jump right in!  Before we start deploying the edge, we need to do a little additional configuration on the DLR.  Since we want the logical segments connected to the DLR to talk to the physical network through the edge, we need the traffic to flow from the DLR to the edge gateway.  That being said, we need a network segment to connect the DLR and the edge.  Let’s add another logical switch called ‘Tenant1_DLR_Uplink’ for this purpose…

image

Now, let’s update the DLR to include an IP interface on this uplink which it can use to talk to the edge.  To do this, browse to your NSX edges, and double click on the edge you created in your last post (Yes, I know, there’s no edit button, I don’t know why…).  Then navigate to settings and interfaces and lets create an internal interface to be used to uplink to the DLR…

image

Now that we have the IP interface in place, let’s add a static default route to the DLR so it knows where to route traffic.  Navigate over to the Routing tab, click ‘Static Routes’, then the plus button to add one…

image

We’ll point the default route to the interface of the edge we’re going to deploy later on.  Once that’s done, we can deploy our edge appliance.  Navigate back to the NSX menu, click on the NSX edges tab, and the the plus sign to deploy a new edge…

image

We’ve seen this before, just make sure you’re deploying the ‘Edge Service Gateway’ and not the DLR this time.  On the next screen give it a username and password and enable SSH access if you wish…

Note – I find it slightly humorous that they enforce password complexity restrictions like this…
image 
But they don’t have you confirm the password.  Make sure you type it correctly!

image

On the next screen you’ll be asked to size the edge.  Despite really wanting to have a ‘Quad Large’ edge in my lab, I’m going to stick with compact.  I’m also going to leave the checkbox in place for the ‘auto rule generation’.  We’ll look at that further later on.  Next pick the cluster, host, and data store you wish to use for deployment…

image

The next task is to configure the IP interfaces for the edge.  In our case, we want to connect the DLR to the edge, and the edge to the physical network so we’ll need two IP interfaces.  Let’s first configure the matching interface to the one we just created on the DLR…

image

Next we’ll add the uplink interface that we’ll use to talk to the physical network.  Note that I’m connecting that interface to a distributed port group which is trunking the VLAN (120) that I’ll use for external connectivity.  AKA – There’s a VLAN 120 IP interface which is the other end of this /30 (10.20.20.50) on the physical switch…

image

Let’s leave the default gateway settings as is and move on…

image

On the next screen, lets change the default traffic policy to allow and configure the management IPs for the HA configuration.  Here again I just picked two random non-routable IPs…

image

The click ‘Finish’ to kick off the configuration!

image

After hitting finish, I got this error…

image

Weird – Last time I checked my subnet math those two IPs were in the exact same /30.  After trying multiple times to get HA working and receiving the same error I rebuilt the edge from scratch without HA and it seemed to work.  So no HA for now…

Once it’s deployed, you should see the edge listed under your NSX edge tab…

image

The next thing we have to do is configure some routes on the edge so it knows how to get places.  Double click on the edge to edit it.  We want to add 3 routes.  A default pointing out to the physical network, a route pointing in for the app layer, and another pointing for the web layer…

image

The last step is to tell my physical network to route to the edge to get to the app and web layers.  We do this by adding two static routes on the physical MLS…

image 
Now I should have full connectivity between the logical network VMs and the physical network VMs.  I can test this with ping the VMs from my desktop which is not inside of the vCenter environment.…

image

So now this is where the traffic flow get’s really interesting.  For instance, let’s take a look at what the traffic flow is for my desktop pinging the Tenant 1 App server…

image

1 – My traffic sourced from the IP of 10.20.30.41 hits the MLS which does a route lookup and determines that to get to the App server (10.20.20.74) I need to route to the edge gateway (10.20.20.49).  My traffic enters VLAN 120 through the VLAN interface on the switch.

2 – The traffic destined for the edge traverses the physical trunk to Thumper tagged on VLAN 120.  The host sends that traffic towards the edge gateway VM

3 – The edge gateway receives the traffic, does a route lookup, and sees that it has a route for 10.20.20.72 /29 pointing at the DLR interface.  Recall that the DLR is local to each host so the edge must now send the traffic back out to the physical network to get to the correct host.

4 – The traffic first encapsulated in VXLAN, and then in the dot1q header for VLAN 118 leaves the host and returns to the switch destined for the VTEP of Thumper3.

5 – The traffic routes between VLANs 118 and 119

6 – The traffic leaves the switch encapsulated in a dot1q header for VLAN 119 destined for the VTEP of Thumper3.

7 – The traffic can now be striped of its VXLAN header and delivered to the app guest VM

The cool part, is we can see pieces of this with a packet capture on the wire…

Step 1/2 – My initial ping heading towards the host, note the VLAN 120 dot1q tag

image

Step 4/5 – The VXLAN encapsulated traffic destined for the VTEP of Thumper3.  Note that the outer MAC address is destined for the Cisco switch VLAN interface and the layer2 header has a tag of 118 indicating it’s coming into the switch off of the Thumper VTEP interface

image

Step 6 – The VXLAN encapsulated packet leaves the switch interface destined for the VTEP of Thumper3.  Note that external layer2 header lists a destination of the Thumper3 (VMware) VTEP interface and we can verify this by seeing it’s leaving the switch tagged dot1q on VLAN 119.

image

Pretty cool huh?  A similar traffic flow would be seen in reverse as the VM replies to my pings with ICMP echo replies.  So there you have it.   Over the last 3 posts we’ve successfully deployed NSX, created logical networks, and routed them out to the physical network.  The primary goal of all of this for me was to see how NSX was using VXLAN and it seems to make good sense now.  Hopefully later I’ll have more time to dig into some of the other features of NSX.

1 thought on “Working with VMware NSX – Logical to Physical connectivity

Leave a Reply

Your email address will not be published. Required fields are marked *