So now that we have a good handle on the standard switches, lets look at distributed switches.  A distributed switch is one that’s managed at the vCenter level and propagates across all hosts.  This gives you a centralized management point for the vSwitch.  In the rest of this post, I’m going to refer to the distributed vSwitch as the DVS and the standard vSwitch as the SVS. 

So let’s say you’ve got an environment up and running on SVS and you’d like to migrate it over to DVS.  It’s actually pretty easy to do ,and as we’ll see, your VMs wont take a hit when you migrate over (if you do it correctly). 

Step 1 – Free up a uplink port from the SVS
So the first step is to free up a physical uplink from each SVS to use for our new DVS.  If you read the last post, you should know how to do this.  Just go into your SVS config, and remove vmnic1 from the list of network adapters associated with the SVS.   When I completed that, my SVS now looks like this…

 
Make sure you free up a NIC on any host you want to become a member of the new DVS. 

Step 2 – Create the new DVS
To create a new distributed switch in vCenter, go to home, inventory, and then networking.  Right click on your data center and select ‘New vSphere Distributed Switch’.

 
This loads the DVS wizard.  On the first page, leave the default option of version 5 selected and click next…

 
On the next page, you can name your DVS as well as determine how many uplink ports each server can have.  Mine only have two each , but I’ll leave the default at 4.  Then click next…

 
On the next screen, we get the option to add hosts and uplinks(vmnics) to the DVS.  You can do this now, or chose to do it later.  Since we already free’d up the uplinks we might as well do it now.  Select the hosts you want to add and select the uplink you free’d up from the SVS earlier.  Then click next…

 
On the next screen, I unchecked the box that said ‘Automatically create a default port group’.  I like my port groups to be for a specific purpose so I don’t want a default one.  Then click finish…

 
Step 3 – Create new distributed port-groups
Once the task completes, let’s add some port groups to the DVS that we can migrate hosts to.  In the network inventory view, right click the new DVS and select the ‘New Port group’ option…

 
In the wizard that comes up, enter your port-group name, and change the VLAN to whatever VLAN you had in the port-group on the SVS.  We are basically recreating all of the SVS port-groups on the DVS in this step.

 
Do the same for any other port-groups you need to add. 

 
Step 4 – Move your VMkernel ports over
So at this point, you have a brand new DVS that should have all of the connectivity it needs to act just like the SVS.  The first thing I like to do is move over the VMkernel ports.  But first, let’s talk about how that works on the DVS.  I’m not sure if you noticed, but when I was adding DVS port-groups, I added one for management.  Now, on the SVS, I didn’t have that.  I just had a VMkernel port tagged on VLAN 10.  While I cant find this documented anywhere, it’s apparent that you need a distributed port-group to match and specific VMkernel interface that you have.  That is, a VMkernel interface resides within a port-group on a DVS.  So now that we have that out of the way, let’s give it a try.

Go to hosts and clusters, select your host, go to the configuration tab, select networking, and then make sure you are viewing the DVS by click it at the top of the window.  Then click ‘Manage Virtual Adapters’…

 
At the top of the next screen, click ‘Add’.  On the wizard that pops up, chose the migration option…

 
On the next screen, select the appropriate port-groups for each VMkernel interface, then click Next…

 
Review the changes and click Finish…

 
Once the task completes, all of your VMkernel interfaces should be moved over. 

Step 5 – Move your VM’s over
This is done by simply editing the the VM settings, and choosing the new DVS port-group…

 
If all goes well, your VMs should move over to the new DVS without dropping a packet (or at least one that you notice).  Complete this change for all of your VMs.

Step 6 – Cleanup
So now that everything is up and running on the DVS, let’s do some final cleanup.  Remove the rest of the vmnics from the SVS, then remove the SVS entirely by click remove…

 
Now we can add the left over uplink to the DVS on each host.  Go into the DVS on each host and click the ‘Manage Physical Adapters’ link…

 
Click the ‘Click to add NIC’ link and then select the available NIC to add…

 
When you are all done, it should look like this…

 
Complete the same steps on your other hosts and you’re all done!

I’m not going to cover the basics in this post so I’m going to assume you know about port-groups, vmnics (why they are called vmnics is something that bothers me to this day.  vmnics = physical NICs = uplinks), and dot1q trunking.

Ok, I lied, let’s talk a little bit about the basics.  The important thing to remember about VMware networking is that an ESXi box is an end host.  What does that mean?  It mean’s that it’s not a router.  While an ESXi box can have multiple vmnics (or even VMs on it that may perform some routing) the box itself can’t route.  It’s just like any other PC or server out in the world.  The ESXi boxes IP identity is based on it’s VMkernel IP address.  A ESXi box can have multiple VMkernel ports each with different IP addresses.  However, only one of the VMkernel ports can have a default gateway.  The VMkernel port with the default gateway is generally the management VMkernel port.  Take a look at my management VMkernel port below…

 
Note how the VMkernel default gateway is an available IP on that subnet.  Now, take a look at my storage VMkernel port…

See how it’s the same gateway as the management interface?  This is (in my network engineering opinion) an important fact to remember.  The ESXi box itself doesn’t do layer 3 routing.  So I can create a ton of VMkernel ports on different VLANs, with different IPs, and trunk them all together.  But I can’t route to any of the other VLANs or VMkernel ports by going through the management interface.  So for instance, I needed a storage VMkernel port since my IP storage was on a different VLAN (15 in my case).  So the iSCSI host was 10.10.10.6/24.  If I want my ESXi host to talk to 10.10.10.6/24, I need a VMkernel port with an IP in that subnet. 

Bottom line – If you want the ESXi host to talk to something, it needs a VMkernel interface on that subnet.  One of those VMkernel interfaces can have a default gateway.

There are 4 different services that can be enabled on a VMkernel port.  They are…

Management
IP Storage (iSCSI port binding)
vMotion
FT logging

It’s generally recommended that you use a different VMkernel port for each of these functions.  Ideally, you have enough physical NICs to do so.  If not, you can always trunk them over the same physical links. 

In my lab configuration, I have one storage VMkernel port (note that it doesn’t have the ‘IP Storage’ function enabled on it, that’s only for iSCSI multipathing (another post on that coming soon)) and another management interface that has the management and vMotion features enabled on it.  Not best practice for production, but it works.  The last feature FT logging will be covered in an upcoming blog. 

Ok, so that sort of beat the snot out of VMkernel ports.  I’m not going to talk about the basics of VM port groups though since those are really just like ‘switchport access’ virtual ports.  Let’s talk about some of the other configuration features on VM port groups and VMkernel ports on the standard vSwitch. 

So here’s the configuration I have on one of the hosts…

So let’s pop into the properties of vSwitch0 to see what we are dealing with…

 
The first item in the list is the vSwitch itself.   There isn’t a ton of ‘interesting’ pieces to the switch but there are a couple of things to note.  First off, even though it’s a virtual switch, it has a hard limit on the number of ports it can have.  You can increase this all the way up to 4088 but any change requires a host reboot in order to take affect.  The MTU is also a configuration item of the vSwitch itself.  You’ll notice that the vSwitch has some duplicate configuration items in relation to VMkernel ports and port groups.  You can set ‘global’ configuration items on the vSwitch that the VMkernel and port groups then inherit.  These global settings can be overridden by simply configuring them on the port group or VMkernel interface. 

The Network Adapters tab shows the physical NICs that are associated with that vSwitch.  You can add and remove them from the vSwitch as well as change their speed and duplex in this view…

The Security Tab
In the security tab, we can configure promiscuous mode, MAC address changes, and forged transmits. 

 
Promiscuous mode
If you want to see all of the traffic traversing the vSwitch, it is possible to configure a port on the switch to be fully promiscuous.  I would recommend doing this on the port-group level rather than on the vSwitch itself.  Basically, you’d create a new port-group called ‘Promiscuous’ (or whatever you want to call it) and set the vlan tag to 4095 (All VLANs).  Then edit the port group, and change the promiscuous mode to accept by overriding the default from the vSwitch itself.  Default is to reject.

MAC Address Changes
This one’s pretty straight forward.  It the vSwitch sees a host trying to change it’s MAC address, it will stop sending it frames.  The only real situation I can think of where you’d want to allow MAC spoofing is if you were doing some type of clustering or had to change your MAC to comply with some type of licensing.  Default is to accept.

Forged Transmits
Another straight forward configuration item.  If the virtual host attempts to send frames with a source other than that of it’s own vNIC, the frames will get dropped.  The default is to accept. 

Traffic Shaping
In the traffic shaping tab, we have a few settings we can tweak in regards to traffic shaping VMs bandwidth.  It’s important to note that the traffic shaping settings only refer to outbound traffic with standard switches.  Distributed switches can shape inbound and outbound. 

 
Status
Not going to explain that setting….

Average Bandwidth
This is the amount of traffic you want to allow over the link averaged over time.

Peak Bandwidth

The maximum amount of traffic you want to allow in a burst.  This parameter can never be smaller than the average bandwidth.

Burst Size
Specifies how large the burst can be in kilobytes. Unlike the other two settings, this a kilobytes setting.  The other are kilobytes per second. 

NIC Teaming
In the NIC teaming tab, we have quite a few options.  We can determine the NIC load balancing method, failover/back, and set what NICs are active, standby, and not in use. 

Load Balancing
There are 4 different load balancing options. 

Route based on the originating virtual port ID – This is the simplest mechanism.  In this method, the virtual machines outbound traffic is mapped to a particular physical NIC.  The VMkernel chooses the NIC based off the ID of the virtual port that the VM is connected to.  In this method, one VM can never use more bandwidth than is provided by a single physical adapter. 

Route based on IP hash – This is the source/destination IP hashing mechanism we are used to seeing on port-channels.  When you use this method, the switch port’s on the MLS side need to be configured in an etherchannel. 

Route based on Source MAC hash – Same as the virtual port ID method but it uses the virtual NICs MAC address.

Explicit Failover – The use of active and standby adapters. 

Network Failover detection
This can be set to link status or beacon.  Beacon sends out broadcasts which should be forwarded to the other uplinks on that vSwitch.  This is really only useful if there are three or more physical links.  With two links, if one had issues, neither port would receive the broadcasts from the other port. 

Notify Switches
This is sort of an interesting one.  Notify switches is the mechanism in which the ESXi host notifies the physical switch that an IP address has moved.  It uses a gratuitous ARP which causes other devices to update their ARP tables.  For instance, when a vMotion occurs, we need some mechanism in which to tell the switches that the MAC address has moved to a new interface.  If you disable this setting, then the ESXi hosts will not send the gratuitous ARP and you have to rely on MAC aging on the switch or the normal MAC learning process to learn of the hosts new location.  I’d recommend you leave this one on. 

Failback
Pretty straight forward.  If the primary NIC goes down the secondary NIC will take over.  If the primary comes back, this setting determines if the host should fail back to the primary NIC. 

Failover Order
In this box you can determine the failover order as well as determine that some NICs are unused.  This setting comes into play in an upcoming post about iSCSI multipathing.

So that’s about it for the standard vSwitch.  In the next post we’ll talk about the distributed switch as well as the migration process from standard to distributed.  I’m hoping to also squeeze in the iSCSI multipathing post in the next day or two.