2vcps and a Truck

*Disclaimer – I work for a Xsigo and VMware partner.

I was in the VMware View Design and Best practices class a couple weeks ago. Much of the class is built on the VMware View Reference Architecture. The picture below is from that PDF.

It really struck me how many IO connections (Network or Storage) it would take to run this POD. Minimum (in my opinion) would be 6 cables per host with ten 8 host clusters that is 480 cables! Let’s say that 160 of those are 4 gb Fiberchannel and the other 320 are 1 gb ethernet. The is 640 gb for storage and 320 for network.

Xsigo currently uses 20 gb infiniband and best practice would be to use 2 cards per server. The same 80 servers in the above cluster would have 3200 gb of bandwidth available. Add in the flexibility and ease of management you get using virtual IO. The cost savings in the number director class fiber switches and datacenter switches you no longer need and the ROI I would think the pays for the Xsigo Directors. I don’t deal with pricing so this is pure contemplation. So I will stick with the technical benefits. Being in the datacenter I like any solution that makes provisioning servers easier, takes less cabling, and gives me unbelievable bandwidth.

So just in the way VMware changed the way we think about the datacenter. Virtual IO will once again change how we deal with our deployments.

I just saw this on twitter and thought it was so cool I would post it.

The knowledge base team added some video KB entries.

Here is one: Restarting the Management Agents on ESX

Not everyone can just read a KB article and know right what to do. So this is a cool move by the KB team.

esxcfg-nas

Standard use of this command is to add or list your NFS mounts.

List: esxcfg-nas –l

Add: esxcfg-nas –a –o <host> -s <share> <name>

Not much more I can say. A little more detail here:

http://b2v.co.uk/b2vguide2vmware3.htm

A thread in the communities about a problem someone had where the nfsclient wasn’t loaded:
http://communities.vmware.com/message/864559

So go out and add some NFS datastores!

It has been almost 1 year since I started looking at the esxcfg-* commands. It initially came as a look at the first part of the Enterprise Administration Exam’s Blueprint very first bullet point. In that post I talked about using esxcfg-mpath to identify which luns are fiber, iSCSI, NFS or local.

Today lets look a little bit deeper at the command and how it can be used to help you day to day.

We can always use esxcfg-mpath -l to list all of the luns and their paths. A good thing to check here is that you have the same number of paths to each datastore that comes from the SAN. You may have a zoning issue if a certain lun can only be seen from 1 path rather than all of them. In general each hba will see the lun through each controller in an active active type fiber channel SAN. So hba A should see the lun from Controller A and B. Likewise, hba B should see Controller A and B for a total of 4 paths. If you are using fixed or MRU as the pathing policy only one will be active but esxcfg-mpath -l will show four paths.

Of course if you have more hbas and controllers you will have more paths.

You can follow the examples given by the esxcfg-mpath -h to get help. One useful tool is to create a crude script using esxcfg-mpath –policy with the –lun tag to change the policy from say MRU to Fixed. I am not a perl scripter and I sure someone already has a real shell or perl script to set the policy but I do like to prepare multiple command in notepad then paste them into the cli.

esxcfg-mpath –policy=fixed –lun=vmhba0:0:1
esxcfg-mpath –policy=fixed –lun=vmhba0:0:2
and so on…

Then try to feel really smart by alternating the paths so every fourth lun will use different paths.

esxcfg-mpath –path=vmhba1:0:1 –lun=vmhba0:0:1 –state=on
esxcfg-mpath –path=vmhba2:0:1 –lun=vmhba0:0:2 –state=on
esxcfg-mpath –preferred –path=vmhba1:0:1 –lun=vmhba0:0:1
esxcfg-mpath –preferred –path=vmhba2:0:1 –lun=vmhba0:0:2

The first two lines set the path for the respective luns to different hbas. The last two lines set the preferred path to that same port on the hba. So when there is failover the path will fail back to your set config when all is well.

Well I passed the VCP 4 and my CCNA expired (can’t get around to renewing it). At work I did the VTSP as required by the partner program. Since I am in a test taking grove I think I need to push to passing the Enterprise Administration Exam. There was a series I started a while back on command line management of ESX. So like several others I will set my next goal at the Enterprise Administration Exam.

Here are some good sites I have recently seen on studying for the Exam.

VCDX Study Guides
Simon Long’s Site
follow @vcdx001 on twitter he is giving hints about the design defense, great info, but don’t want to get too far ahead of myself.

I think for this first test you just have to know what you are doing with VI 3. My big problem is having worked with vSphere for a couple months now I hope that doesn’t hurt me.

First, I have lived in the South too long, because I said “Big ole disk” and couldn’t think of a more appropriate phrase. Now someone rescue me if I start to tell you to “mash” the power button on your server or SAN. I kid.

I am sure everyone out there has used this before but I like to document these things just case someone else needs help.

A coworker and I were installing a vSphere environment last week to support some new software for a customer. The software vendor required approximately 30 x 146GB drives in a Raid 5 to store images. Never would guess the software vendor happens to sell SANs too! I exaggerate it actually called for 3TB of usable space.

So my thought was to get over 2TB limit of VMFS we would need to use the MS iSCSI initiator inside the VM. Then my coworker thought we could enable MPIO using two virtual Nics with vmxnet3. We tied each vmxnet3 nic to a separate port group and assigned one of the 2 physical NICs to each port group. Additionally vmxnet3 lets you enable jumbo frames and the physical nics were already set to mtu 9000 because this was on the software iscsi vswitch. So we were able to get multiple paths from the VM to the network and have jumbo frames all the way through.

Next we presented the iSCSI volume of 3TB to the Windows machines. Of course at first it sees it as a couple of smaller volumes. Convert the disk to GPT and align to 64k, then format with NTFS. Just like that a 3TB disk inside a Virtual Machine.

Now we saw IOMETER push better sequential IO than an RDM that was set up for Round Robin, but not quite as good in the Random IO department as a RDM.

The main gain here is to get a file bigger than 2TB minus 512B. Useful for the scan/image servers that store a tons of files for a long time.

To sum up and make it clear.

1. Use the Microsoft iSCSI initiator and MPIO inside the VM when you need more than 2TB on  a disk.

2. Use 2 port groups and bind them to separate physical nics to let the MPIO actually work over 2 nics.

3. With vSphere use the VMXNET3 driver for network to use jumbo frames, the E1000 driver does not support this.

So I don’t have any real content today. So I will make up things you should not tweet from a customer site.

1. @duncanYB How do I add a host to vCenter again?

2. Just overwrote dudes LUNS, hope there wasn’t anything on them.

3. Licenses? Everyone can use my file.

4. Just P2V’d dudes server in the middle of the day. Hope no one noticed the reboot.

5. iSCSI, Service Console, VM Network, DMZ, VMotion. All on one NIC Sweet!
(don’t ruin the effect by commenting about 10GigE)

6. Exchange and SQL on SATA? No better use for a single 1 TB drive!

7. Just finished my first ESX install on a real server. Faster than it was in Workstation.

8. @Texiwill is there  a way to blank the root password for the Service Console?

9. Failover is for dummies that don’t do it right the first time.

10. Can’t wait to leave here so I can finish building my life size Yoda robot.

Add your suggestions in the comments or tweet them to me @2vcps

This is the kind of stuff I love to find. Good stuff all in one place. The Storage customer service team identified several of the top KB entries that could help in a pinch. Check them out on the VMware Knowledge Base Blog.

I have a personal experience with:

1002511 Recreating a missing VMDK (header) file

It would relate back to this post.

So thanks a bunch Storage Heavy Hitters!

Two new products and a new release for an older product will happen today. Applications aimed for those trying to scale their VMware environment and make it more strategic in their organization. These will be released in the vCenter family but may not actually run on you vCenter Server.

First up is VMware vCenter AppSpeed. This software will alow drilling into the performance of the application level in a VM. One feature I find compelling is the Assured Migration. Take a look at an apps performance before and after migration to be sure the same level of service post virtual migration. This is very cool and needed for VI admins trying to prove that it is not VMware causing the problem. At least AppSpeed helps pinpoint performance issues faster, which is a huge help to anyone with a VMware environment.

Second is vCenter Chargeback. You can guess it, Chargeback provides the ability to apply accounting to your resources and if not charge for those resources at least get a look back to determine the actual costs of a initiative.

Last is the Update to an existing product, vCenter LabManager 4. Big news here is all Stage Manager functionality has been rolled into Lab Manager. Stage Manager will go away supposedly. The pricing for Lab Manager will change. If you have a valid SnS for Lab Manager or Stage Manager you will supposedly get Lab Manager 4. Cool.

So that this doesn’t just look like a copy of a VMware press release. I wanted to include my own thoughts. The appearance of more “manage and automate” applications like these three show that the VMware install base is demanding more and more from their environment. Customers investing in this type of software shows to me they are not looking to bolt to another product any time soon. Wait this is way too positive. Lets say something I don’t like, Lab Manger price will go up but still be less than Lab Manager and Stage Manager combined. As a customer I would not like this move because maybe I just wanted Lab Manager now you say the price has increased due to functionality being added that I didn’t ask for. It could be like Microsoft including SQL 2008 with every 2008 Server license and increasing the cost of the server license by the cost of SQL. Then not letting you buy them separate. The problem with my complaint is I guess many customers came to VMware and said, “Hey, I sure wish Lab Manager and Stage Manager were all one product.”

So I was updating some of my blog posts on the esxcfg-* commands with any changes in ESX 4. I wrote earlier I did not know much about the esxcfg-advcfg command. Since writing that post at the end of 2008, I found Duncan Epping used esxcfg-advcfg in 3.5 to set the option rescan all the Hba’s. I thought this was a great shortcut and decided to try it out in vSphere but:

[root@esx4 ~]# esxcfg-advcfg -s 1 /Scsi/ScsiRescanAllHbas
Exception occured: Unable to find option ScsiRescanAllHbas

So I looked through vCenter 4 and did not find the option under Scsi I looked around some in the other Advanced Options and it is no where to be found.

Has this been removed or moved somewhere else? If you know hit me up on twitter @2vcps