MDS

You are currently browsing articles tagged MDS.

So now that we have our base config running we can take a look at configuring the native FC interfaces, building the connection to the CNAs, and connecting the 5Ks to the IP network (I’ll refer to this as data network going forward). Before we dive into config, let’s talk about design considerations.

Design Considerations
Most deployments will (should) have redundant 5k’s as part of the setup.  So let’s take a look at a pretty common setup that you might see out in the wild from a logical perspective.  Drawing1

So this is how I like to conceptualize the basic setup (Yes I know I’m missing pieces, this is the ‘big picture’).  Both the native FC storage and the data network terminate into the 5Ks.  At that point, the FC VSAN gets mapped to a VLAN.  The VLAN is what’s passed to the 2K and on to the server.  So in my illustration, the green links are storage and the orange are data.  Pretty straightforward right?  I’m hoping you caught the fact that the link between the 5Ks only shows the data network.  Have you figured out why?  Even though we map the VSAN to a VLAN its still for the most part ‘its own thing’.  That being said, standard FC design concepts come into play.  One of those being having dual fabrics to each server.  If we allowed that VLAN across the trunk between the two 5Ks we’d be breaking that rule.  You’ll also need to take things like MPIO into account as well since you’ll have redundant active paths to the same storage.  The data network is still trunked between the 5Ks to facilitate a VPC configuration.  Now, on to the config…

Native Fiber Channel
At this point I’m assuming you have a native fiber channel card in your 5k and you’ve got some trunks going back to your MDS.  That being said, I’ll also assume you have more than one pair of fiber and you’d like to make the two pair that you do have into some type of port-channel.  Let’s get to it…

(Note: I’m only showing ‘one side’ of the config.  You’d obviously have to duplicate this on the other 5K as well.  Keep in mind that things like the Storage VLAN will be different between the two 5Ks)

Configure the ports to the MDS
Nexus5k(config)# interface san-port-channel <SAN Port Channel Number>
Nexus5k(config-if)# channel mode active
Nexus5k(config-if)# switchport trunk mode on
Nexus5k(config-if)# switchport mode NP [E]
(Note: Pick either NP or E depending on your deployment.  NP would be for NPV mode and E would be for Fabric Switch mode (or whatever they are calling it now))
Nexus5k(config-if)# switchport trunk allowed vsan 1
Nexus5k(config-if)# switchport trunk allowed vsan add <Assigned VSAN>
Nexus5k(config-if)# exit
Nexus5k(config)# int <Member 1>, <Member 2>, etc…
Nexus5k(config-if)# channel-group <San Port Channel Number> force
(Note: We are using force, make sure that your config for port-channel is the same on both ends.  A lot of it depends on the exact setup and what features (LACP, etc..) the gear supports)
Nexus5k(config-if)#
no shutdown

As an obvious side note here, make sure you configure the MDS side to match the settings on this end.

Map a VSAN to a VLAN
Nexus5k(config)# vlan <VSAN VLAN Number>
Nexus5k(config-vlan)# fcoe vsan <Assigned VSAN>

Building the CNA Connections
There are two components to building the connectivity to the CNAs.  One being the data component and two being the FCOE component.  The data component is easy (for now, we’ll add VPC later) so let’s tackle that first followed by the VFC interfaces for the FCOE piece.

Configure the Port and Port-channel going to the CNA Card 
Nexus5k(config)# interface Ethernet <Number>
Nexus5k(config-if)# switchport mode trunk
Nexus5k(config-if)# switchport trunk allowed vlan <Native VLAN>, <Data VLAN>, <VSAN VLAN>
Nexus5k(config-if)# spanning-tree port type edge trunk
Nexus5k(config-if)# speed 10000
Nexus5k(config-if)# channel-group <CNA Port Channel Number> mode active
Nexus5k(config-if)# exit
Nexus5k(config)# interface port-channel <CNA Port Channel Number>
Nexus5k(config-if)# spanning-tree port type edge trunk

Configure the VFC Interface
Nexus5k(config)# interface vfc <VFC Number>
Nexus5k(config-if)# bind interface port-channel<CNA Port Channel Number>
Nexus5k(config-if)# no shutdown

Create the VSAN and map it to the appropriate interfaces
(Note: In this step we are just telling the VSAN database what ports should be part of what VSAN.  This would include the VFC interface facing the server and the port-channel facing the SAN.)
Nexus5k(config)#
vsan database
Nexus5k(config-vsan-db)# vsan <Assigned VSAN>
Nexus5k(config-vsan-db)# vsan <Assigned VSAN> interface vfc<VFC Number>
Nexus5k(config-vsan-db)# vsan <Assigned VSAN> interface san-port-channel <San Port Channel Number>

Configure the Data Network pieces
In this section we’ll cover the trunk back to the data network (65K or 7K in most cases), the trunk between the 5Ks, and the VPC config.

Configure the connection back to the data network
I’m just going to list this as a config item.  There isn’t anything special about configuring a standard data trunk.  You can use a single port on each, make a port-channel, however you want to handle it.  You just need to get your data network VLANs onto the 5K.

Configure the trunk between the 5k’s
Nexus5k(config)# int <Member 1>, <Member 2>, etc…
Nexus5k(config-if)# switchport mode trunk
Nexus5k(config-if)# switchport trunk allowed vlan <Native VLAN>, <Data VLAN>
(Note: We are only allowing the Data VLANs across the trunk, not the storage VLAN)
Nexus5k(config-if)# spanning-tree port type network
Nexus5k(config-if)# channel-group <5K Port Channel Number> mode active

Configure VPC
(Note: This is a very basic VPC config.  I don’t have a dedicated keep-alive link in this example so the management VRF is being used.  Probably not a good idea for production)
Nexus5k(config)# vpc domain <VPC Domain Number>
Nexus5k(config-vpc-domain)# role priority 5000
Nexus5k(config-vpc-domain)# system-priority 4000
Nexus5k(config-vpc-domain)# peer-keepalive destination <Management IP of other Nexus>
Nexus5k(config-vpc-domain)# exit
Nexus5k(config)# interface port-channel <CNA Port Channel Number>
Nexus5k(config-if)# vpc <VPC Domain Number>
Nexus5k(config-if)# exit
Nexus5k(config)# interface port-channel<5K Port Channel Number>
Nexus5k(config-if)# vpc peer-link

Conclusion
I hate to say it, but there really isn’t much to it when you look at it as a whole.  The tricky part is troubleshooting.  There are many different pieces of new technology (VPC, FEX, FCoE, etc..) that it can be hard at times to see the issues.  At any rate, I’m anxious to hear if others have had any luck with their setups or what they have running currently.

Tags: , ,

As I laid out in my last post, these are the basic steps for the SAN configuration of a Cisco MDS series switch.  They are….

1.  Create the VSAN number you wish to use (1 is the default, not recommended to use the default for production SAN traffic)
2.  Add interfaces to your VSAN (just like you do with a VLAN)
3. Do any interface configuration needed on the FC interfaces (Just turn them on in most cases)
4. Verify the cabling and ensure that you have SAN connectivity
5. Create Aliases for WWNs (makes life easier)
6. Create required zones
7. Add members to your zones (I recommend using PWWNs)
8. Create a zoneset (I think Brocade calls this a ‘config’)
9. Add your zones to your zoneset
10. Activate the zoneset on the fabric

So lets walk through them one at a time and show the associated configuration.  I’ll use my same old color coding conventions.
-Insert your relevant information between <>
-Console prompts are shown in green

1.Create the VSAN number you wish to use (1 is the default, not recommended to use the default for production SAN traffic)
MDS(config)# vsan database
MDS(config-vsan-db)# vsan <new VSAN number>
MDS(config-vsan-db)# vsan <new VSAN number> name <VSAN name>
MDS(config-vsan-db)# exit

-Add interfaces to your VSAN (just like you do with a VLAN)
MDS(config)# vsan database
MDS(config-vsan-db)# vsan <VSAN number>
MDS(config-vsan-db)# vsan <VSAN number> interface <FC Interface>
MDS(config-vsan-db)# exit

-Do any interface configuration needed on the FC interfaces
MDS(config)# interface <FC Interface>
MDS(config-if)# no shutdown
MDS(config-if)# switchport mode <Either Auto, or a set type (E,F, etc.)>
MDS(config-if)# exit

-Verify the cabling and ensure that you have SAN connectivity
Your on your own here.  As with anything fiber, if you don’t have link, make sure you have the pair flipped correctly from SFP to SFP.

-Create Aliases for WWNs (makes life easier)
MDS(config)# fcalias name <name of the Alias> vsan <VSAN number>
MDS(config-fcalias)# member pwwn <WWN>
MDS(config-fcalias)# exit

-Create required zones and add members
MDS(config)# zone name <name of zone> vsan <VSAN number>
MDS(config-zone)# member fcalias <alias name 1>
MDS(config-zone)# member fcalias <alias name 2>
MDS(config-zone)# exit
(note: if you didn’t want to make aliases, you could use the keyword ‘pwwn’ rather than ‘fcalias’ and directly input the WWN)

-Create a zoneset (I think Brocade calls this a ‘config’) and add the zones to the zoneset.
MDS(config)# zoneset name <name of zoneset> vsan <VSAN number>
MDS(config-zoneset)# member <zone 1>
MDS(config-zoneset)# member <zone 2>
MDS(config-zoneset)# member <zone 3>
MDS(config-zoneset)# exit

-Activate the zoneset on the fabric
MDS(config)# zoneset activate name <zoneset name> vsan <VSAN number>

Conclusion
And that’s it.  It goes without saying, but save your config when you are done.

Tags: ,

It occurred to me while writing the NPV and NPIV article (coming up shortly) that there was a lot of the fiber channel protocol which I hadn’t covered.  That being said, I’m going to talk a little bit more about it here.  Specifically we’ll discuss FCID, WWNs, zoning, and VSANs

So if you look at the terms we listed in the last post.  You might be scratching your head and wondering what the difference is between WWNs and FCID/NPIDs.  This was something I had trouble understanding initially as well.  For starters, let’s agree that we’ll simplify and call NPIDS/FCIDS just FCIDs going forward.  The terms are used interchangeably, but for the sake of clarity, I’ll just call them FCIDs going forward.

That being said, let’s talk about about the FCID.  The FCID is a 24 bit (3 byte) field used to route frames through a FC network.  Let’s walk through each byte of the FCID….

1st byte – Domain ID.  The Domain ID is given to each FC switch in a fabric and must be unique for each switch.

2nd byte – Area ID.  The Area ID is very similar to the 3rd byte.  In fact, in Cisco documentation, you’ll find it called the exact same thing as the 3rd byte.  The Area ID is used to identify an N type port that is connected to a switch.

3rd byte – Port ID.  The Port ID is used to identify a single FC object on the FC fabric.

In a SAN each device that is going to talk on the fabric has a WWN and a FCID.  The WWN is very much like a MAC address in Ethernet speak.  Each WWN is globally unique with some exceptions.  There are instances where devices allow an administrator to manually configure a WWN but we won’t be discussing those instances.  For basic understanding, let’s say that each WWN is globally unique.  So an HBA that had multiple ports would have a WWN for itself, and then PWWN (port WWNs) for each one of its ports.  Additionally, each port on a fabric switch has a FWWN (fabric WWNs) which are sometimes also called FPWNNs (fabric port WWNs).

So, a device talking on the fabric has multiple identifiers.  Still with me?

The FCID is used in routing through the fabric.  So in a FC frame there is S_ID (Source ID) and a D_ID (Destination ID) field.  These are populated with the appropriate FCID to facilitate traffic through the fabric and to the correct node.

So what is the WWN used for?  Enter zones…  Put basically, for two devices to communicate, they MUST be in the same FC zone.  Zones are primarily a security practice.  In reality, if everything was in the same zone, and in the same fabric, everything could talk to everything.  That isn’t to say that all the LUNs would show up on all the servers, its just implying that it would be possible.  Zoning is a fabric configuration, not a disk array to host mapping configuration.  So let’s go over a quick example so that we are all on the same page.  Take the diagram below as an example. (Side note: I’ve had this wonderfully large whiteboard in my home office for almost a year and have hardly ever used it.  I will waste resources no longer, look forward to more whiteboard pictures with my hard to read hand writing.)

photo

So as we can see here (in my glorious white board depiction) we have two servers.  A backup server that needs access to the disk array for its local disk, and access to the tape library to run backups.  We also have a file server that just has access to the disk array.  It doesn’t need access to the tape library since the backup server is accessing the file server’s files over the network, and then putting them to tape itself.  That’s a pretty simple example of zoning.

To be clear, you don’t always have to use WWNs to create zones.  You can also create zones based on…
-PWWN or FWWN
-FC Alias (Basically, a name associated with a WWN)
-FCID
-Physical interface

Cisco recommends using a PWWN to do zoning.  This ties an HBA to a zone rather than a physical port to a zone.

Another concept of logical SAN separation is VSANs.  While its usually viewed as a Cisco fabric feature I believe its recently been approved as a global standard.  VSANs are much like, you guessed it, VLANs.  A VSAN allows you to have logically separate fabrics without actually having to have different physical fabrics.

photo5

(Additional side note: I’m going to get colored markers to make this more straight forward)
As you can see in the drawing above, all four devices hang off of the same physical fabric, but since they are in separate VSANs, they are unable to communicate.

So let’s take a step back and talk about the entire Cisco SAN configuration.  We spoke about zones and VSANs, now let’s talk about how to configure them.  This is the basic list I use when talking about SAN config on an MDS. 
1.  Create the VSAN number you wish to use (1 is the default, not recommended to use the default for production SAN traffic)
2.  Add interfaces to your VSAN (just like you do with a VLAN)
3. Do any interface configuration needed on the FC interfaces (Just turn them on in most cases)
4. Verify the cabling and ensure that you have SAN connectivity
5. Create Aliases for WWNs (makes life easier)
6. Create required zones
7. Add members to your zones (I recommend using PWWNs)
8. Create a zoneset (I think Brocade calls this a ‘config’)
9. Add your zones to your zoneset
10. Activate the zoneset on the fabric
(Don’t worry I’ll show you how to do each one of these steps in the next article)

You’ll notice that we introduced the term ‘zoneset’ in the steps.  We haven’t talked about them yet, so let’s take a brief moment to discuss them.  There are some basic rules I like to think of when talking about zones/zonesets that do a pretty decent job defining the term.  They are…..

-Zonesets are a container for one or more zones in the fabric
-Zones need to be a member of a zoneset in order to be used.  That is to say, zones once configured aren’t actually ‘on’ until they are applied to a zoneset and the zoneset is activated in a particular VSAN.
-Zones can be in more than one zoneset ‘container’.  That is, a zone can belong to multiple zonesets.
-Devices (WWNs) can be members of multiple zones
-Only one zoneset can be active at one time in any given VSAN
-If an alias or WWN is not assigned to a particular zone, it is a member of the default zone (1). 
-If zoning isn’t active at all, then all devices are default to being a member of the default VSAN.

So if we looked at the big picture, the layout would look like this….

photo7

I probably shouldn’t have used VSAN 1 in the diagram since thats the default, but you get the idea.

Please keep in mind that we are talking about a very basic configuration here.  There are far more advanced topics/concepts in regards to all of these functional items.  As you get more comfortable with the configuration, you can start playing with the additional functionality.

As I reread these posts, I’m starting to see they are a bit scatterbrained in regards to flow.  I apologize for that; I’ll do my best to keep these flowing as I’m sure that makes them easier to read.  Once we get all the basic FC understanding out of the way, I think the actual config posts will flow together nicely.

This is my outline for the next few posts…
-CLI config of an MDS (how to do what we’ve been talking about)
-NPV and NPIV
-Nexus 5k/2k config (just the base config)
-Nexus 5k/2k config (Connecting to the MDS and the IP network)
-Nexus 5k/2k config (Connecting to the hosts (CNAs))
-Follow up posts to answer any questions I see coming through in the comments.

Tags: ,