2vcps and a Truck

Equallogic PS Series Design Considerations

VMware vSphere introduces support for multipathing for iSCSI. Equallogic released a recommended configuration for using MPIO with iSCSI.   I have a few observations after working with MPIO and iSCSI. The main lesson is know the capabilities of the storage before you go trying to see how man paths you can have with active IO.

1. EqualLogic defines a host connection as 1 iSCSI path to a volume. At VMware Partner Exchange 2010 I was told by a Dell guy, “Yeah, gotta read those release notes!”
2. EqualLogic limits the number of hosts in the to 128 per pool or 256 per group connections in the 4000 series (see table 1 for full breakdown) and to 512/2048 per pool/group connections in the 6000 series arrays.
3. The EqualLogic MPIO recommendation mentioned above can consume many connections with just a few vSphere hosts.

I was under the false impression that by “hosts” we were talking about physical connections to the array. Especially since the datasheet says “Hosts Accessing PS series Group”. It actually means iSCSI connections to a volume. Therefore if you have 1 host with 128 volumes singly connected via 1 iSCSI path each, you are already at your limit (on the PS4000).

An example of how fast vSphere iSCSI MPIO (Round Robin) can consume available connections can be seen this this scenario. Five vSphere hosts with 2 network cards each on the iSCSI network. If we follow the whitepaper above we will create 4 vmkernel ports per host. Each vmkernel creates an additional connection per volume. Therefore if we have 10 300 GB volumes for datastores we already have 200 iSCSI connections to our Equallogic array. Really no problem for the 6000 series but the 4000 will start to drop connections. I have not even added the connections created by the vStorage API/VCB capable backup server. So here is a formula*:

N – number of hosts

V – number of vmkernel ports

T – number of targeted volumes

B – number of connections from the backup server

C – number of connections

(N * V * T) + B = C

Use multiple pools within the group in order to avoid dropped iSCSI connections and provide scalability. This reduces the number of spindles you are hitting with your IO. Using care to know the capacity of the array will help avoid big problems down the road.

*I have seen the connections actually be higher and I can only figure this is because the way EqualLogic does iSCSI redirection.

I was in my personal ESX about to upgrade to update 1. I was distracted by trying to setup iSCSI from the command line. Right before I looked to the vSphere Client to get the iqn I said, “There is surely a way for me to find this from the command line.”

Searching around I found the command vmkiscsi-tool. Really good stuff, I can complete the rest of my setup without the GUI. One thing though to list out the iqn for iscsi after you enable it you must know the device name (ie vmhba??).

Using this command:
vmkiscsi-tool -I -l

I usually guess the iscsi hba is vmhba33 or 32 but how do I know for sure?
Try:
esxcfg-scsidevs -a

Ok great, now we know it is vmhba33

[root@esxhost01 sbin]# vmkiscsi-tool -I -l vmhba33
iSCSI Node Name: iqn.1998-01.com.vmware:esxhost01-35151883
[root@esxhost01 sbin]#

Now with a few more vmkiscsi-tool commands I can finish configuring my iSCSI.
Add the Ip of the SAN:
[root@esxhost01 sbin]# vmkiscsi-tool -D -a 172.16.23.251 vmhba33
Now rescan:
[root@esxhost01 sbin]# esxcfg-rescan -a vmhba33

OK, SRM and View 4 are out. Go ahead and start planning those upgrades from 3.x to 4. I mean really, vSphere is out now for almost 6 months. Get Enterprise Plus or the Acceleration kit, just get to vSphere. Here are a few of my reason’s why.

1. Round Robin Storage IO. Those without can giant Fiber Channel SAN infrastructure can start to stack the Software iSCSI ports and get performance above and beyond what was possible before with iSCSI. Equalogic, Lefthand and other iSCSI SAN manufacturers have to be throwing huge parties about this. While talking about iSCSI you don’t need a Service Console just for every iSCSI VMkernel port. This always seemed like extra setup in 3.x.

2 Thin Provisioning, I am not technical enough with storage to know if SAN based thin provisioning is better for some reason. It is great to be able to save space with template and other large footprint VM’s.

3. dVSwitch, VMsafe and vShield zones. New hooks for security will eventually give us insight into areas of the VI we could not see before. VMsafe will let vendors tie into the kernel (at least that is how I understand it). Additionally the new dVSwitch (Distributed Virtual Switch, sometimes it is called something else) will give control and sight into the network stack in ways that was impossible before.

This is stuff many may have read on the release date in May, but now that I have seen vSphere in action and some of the biggest hurdles (SRM and View) have been overcome, it is now time to upgrade, already.

Note: I rarely do posts like this. I would rather explain an admin problem and solution. I hope this doesn’t scare too many away. I am in a rant mood.

Day 2 was a great day at VMworld. The key-note today combined with the announcement after the keynote sparked a couple thoughts. I would bet I am not the only one that noticed. VMware basically put Microsoft, Google and Amazon on notice. VMware now has the tools to make a challenge to these previously unchecked organizations.

Microsoft and Google
First the giant world domination bent company based in Redmond and its information hording rival from Nor Cal. The significance of the Springsource merger/purchase and vCloud API’s is telling everyone use your existing apps in the private/public cloud, also your custom developed applications will soon fly into the cloud. Nothing we didn’t already know but now they are all supplied by VMware. Virtualization in general can make Microsoft mortal, and who would use Google apps if the apps they actually know and like could be highly available in a per month charge model.

Amazon
The vCloud Express being made available means VMware can provide virtual services on demand to anyone with a credit card. All at the same time letting the hosting companies front the major expenses for datacenter buildout. The software gets sold no matter how successful everything is. How will Amazon be able to turn a profit when competing against the most proven Enterprise platform for providing virtual servers? Pretty hard to do in my opinion.

Now before being called a VMware fanboy. I think VMware is starting a game they better win. There may be a glimmer of hope that competition will benefit the consumer. The bullying by software vendors of their customers should turn into innovation to set themselves apart. The first company to become complacent will lose. At this point who would know what is going to happen.

First, I have lived in the South too long, because I said “Big ole disk” and couldn’t think of a more appropriate phrase. Now someone rescue me if I start to tell you to “mash” the power button on your server or SAN. I kid.

I am sure everyone out there has used this before but I like to document these things just case someone else needs help.

A coworker and I were installing a vSphere environment last week to support some new software for a customer. The software vendor required approximately 30 x 146GB drives in a Raid 5 to store images. Never would guess the software vendor happens to sell SANs too! I exaggerate it actually called for 3TB of usable space.

So my thought was to get over 2TB limit of VMFS we would need to use the MS iSCSI initiator inside the VM. Then my coworker thought we could enable MPIO using two virtual Nics with vmxnet3. We tied each vmxnet3 nic to a separate port group and assigned one of the 2 physical NICs to each port group. Additionally vmxnet3 lets you enable jumbo frames and the physical nics were already set to mtu 9000 because this was on the software iscsi vswitch. So we were able to get multiple paths from the VM to the network and have jumbo frames all the way through.

Next we presented the iSCSI volume of 3TB to the Windows machines. Of course at first it sees it as a couple of smaller volumes. Convert the disk to GPT and align to 64k, then format with NTFS. Just like that a 3TB disk inside a Virtual Machine.

Now we saw IOMETER push better sequential IO than an RDM that was set up for Round Robin, but not quite as good in the Random IO department as a RDM.

The main gain here is to get a file bigger than 2TB minus 512B. Useful for the scan/image servers that store a tons of files for a long time.

To sum up and make it clear.

1. Use the Microsoft iSCSI initiator and MPIO inside the VM when you need more than 2TB on  a disk.

2. Use 2 port groups and bind them to separate physical nics to let the MPIO actually work over 2 nics.

3. With vSphere use the VMXNET3 driver for network to use jumbo frames, the E1000 driver does not support this.

During the VMware Communities Roundtable today. I learned that Lab Manager 4 only works with vCenter Server 4. Ah! So be sure to plan ahead because even though ESX 3.5 and 4 hosts are supported your vCenter 2.5 is not. So once you upgrade to vCenter 4 you will need to upgrade to Lab Manger 4.

This is no fun because it means after hours work when the users of Lab Manager 3 are at home and happy. Which I started to wonder at what kind of pitfalls will there be upgrading. What do back up to be sure your linked clones stay linked? I guess it is time to read some docs.

Good Luck.

Well, before I can get to the VCDX Enterprise Administration Exam it becomes more urgent to get the new VCP for vSphere (aka VCP4).

VCP 4 Blueprint

Simon Long has a practice test on his Slog.

Keep check the practice test site, he has added to it a couple times. I really like practice tests. Helps me know if I know what I am doing. Many times scheduling the exam for me comes down to confidence, I hate to waste $175- $225.

Now to remember how much Memory is required for the vCenter Server…

So my first experience trying to deploy the new vShield Zones security product included in VMware’s vSphere.

First vShield Zones is different than VMsafe. The way I understand it is the vShield Zones is like your border security but inside of the vSphere. It divides and segregates networks and virtual machines. The VMsafe is end point protection built into the kernel. Reflex has the first VMsafe certified appliance but I have not had a chance to try it yet. (Need more hardware hint hint)

The User Guide talks about downloading an appliance but you actually download an ISO then run an installer that unzips a folder with the 2 appliances. One is the vShield Zones Manager and the other is the actual firewall. The extra step of using the ISO image was annoying buy I guess I am just a whiner. On a super basic level, (I am not here to re-write the user guide) Import the appliance for the manager then import the firewall. Convert the firewall into a template. The Manager appliance takes care of the rest. Note: Internet Explorer 8 and the Manager Web UI don’t work. I used IE 7 just fine.

1. You won’t get this far in IE8
2. Deploying the vShield is straight forward. It creates new vSwitches and port groups and the Manager UI indicates which network is protected and unprotected. This is not in Virtual Center still in the Web Interface.
3. 
4. As you deploy the vShield enjoy watching the tasks in vCenter.
   

All things considered it is a good product I don’t have enough throughput on my little lab machine to really test any impact using vShields would have on performance. If you are a Service Provider I think it would be a great add on to ensure some separation of virtuals.