2vcps and a Truck

So I often have epiphany teasers while driving long distances or stuck in traffic. I call them teasers because they are never fully developed ideas and often disappear into thoughts about passing cars, or yelling at the person on their cell phone going 15 MPH taking up 2 lanes.

Here is some I was able to save today (VMware related):

1. What if I DID want an HA cluster to be split in two different locations, Why?
2. Why must we over-subscribe iSCSI vmkernel ports to make the best use of the 1gbe phyical nics. Is it a just the software iSCSI in vSphere? Is just something that happens with IP storage? I should test that sometime…
3. If I had 10 GB nics I wouldn’t use them on Service Console or Vmotion that would be a waste. No wait, VMotion ports could use it to speed up  your VMotions.
4. Why do people use VLAN 1 for their production servers? Didnt’ their Momma teach em?
5.  People shouldn’t fear using extents, they are not that bad. No, maybe they are. Nah, I bet they are fine, how often does just 1 lun go down. What are the chances of it being the first lun in your extent? Ok maybe it happens a bunch. I am too scared to try it today.

*Disclaimer – I work for a Xsigo and VMware partner.

I was in the VMware View Design and Best practices class a couple weeks ago. Much of the class is built on the VMware View Reference Architecture. The picture below is from that PDF.

It really struck me how many IO connections (Network or Storage) it would take to run this POD. Minimum (in my opinion) would be 6 cables per host with ten 8 host clusters that is 480 cables! Let’s say that 160 of those are 4 gb Fiberchannel and the other 320 are 1 gb ethernet. The is 640 gb for storage and 320 for network.

Xsigo currently uses 20 gb infiniband and best practice would be to use 2 cards per server. The same 80 servers in the above cluster would have 3200 gb of bandwidth available. Add in the flexibility and ease of management you get using virtual IO. The cost savings in the number director class fiber switches and datacenter switches you no longer need and the ROI I would think the pays for the Xsigo Directors. I don’t deal with pricing so this is pure contemplation. So I will stick with the technical benefits. Being in the datacenter I like any solution that makes provisioning servers easier, takes less cabling, and gives me unbelievable bandwidth.

So just in the way VMware changed the way we think about the datacenter. Virtual IO will once again change how we deal with our deployments.

Equallogic PS Series Design Considerations

VMware vSphere introduces support for multipathing for iSCSI. Equallogic released a recommended configuration for using MPIO with iSCSI.   I have a few observations after working with MPIO and iSCSI. The main lesson is know the capabilities of the storage before you go trying to see how man paths you can have with active IO.

1. EqualLogic defines a host connection as 1 iSCSI path to a volume. At VMware Partner Exchange 2010 I was told by a Dell guy, “Yeah, gotta read those release notes!”
2. EqualLogic limits the number of hosts in the to 128 per pool or 256 per group connections in the 4000 series (see table 1 for full breakdown) and to 512/2048 per pool/group connections in the 6000 series arrays.
3. The EqualLogic MPIO recommendation mentioned above can consume many connections with just a few vSphere hosts.

I was under the false impression that by “hosts” we were talking about physical connections to the array. Especially since the datasheet says “Hosts Accessing PS series Group”. It actually means iSCSI connections to a volume. Therefore if you have 1 host with 128 volumes singly connected via 1 iSCSI path each, you are already at your limit (on the PS4000).

An example of how fast vSphere iSCSI MPIO (Round Robin) can consume available connections can be seen this this scenario. Five vSphere hosts with 2 network cards each on the iSCSI network. If we follow the whitepaper above we will create 4 vmkernel ports per host. Each vmkernel creates an additional connection per volume. Therefore if we have 10 300 GB volumes for datastores we already have 200 iSCSI connections to our Equallogic array. Really no problem for the 6000 series but the 4000 will start to drop connections. I have not even added the connections created by the vStorage API/VCB capable backup server. So here is a formula*:

N – number of hosts

V – number of vmkernel ports

T – number of targeted volumes

B – number of connections from the backup server

C – number of connections

(N * V * T) + B = C

Use multiple pools within the group in order to avoid dropped iSCSI connections and provide scalability. This reduces the number of spindles you are hitting with your IO. Using care to know the capacity of the array will help avoid big problems down the road.

*I have seen the connections actually be higher and I can only figure this is because the way EqualLogic does iSCSI redirection.

Tonight I was sitting with my Mac Book Pro, wife and kids all in bed, perfect blogging time. Except no idea what I wanted to write. After a burst of twitter encouragement from @rickvanover here we go:

Five things you should know about VMware Certs:

1. If you work for a VMware partner, start with the VSP/VTSP Certifications. They require a lot less initial investment and give you the start into VMware products. Partners only.

2. You can take the “Install and Configure” or “Design Secure and Analyze” or the “Fast Track” (which is some kind of combo of the previous 2) then study and take the VCP. The VMware Certified Professional is the standard in Virtualization industry and has personally opened many doors for me professionally. VMware Partners NEED them, but it is quite an investment. The class is far from free and the test isn’t cheap. Something to know though, if you go to VMworld the tests are discounted. So everyone at VMworld take advantage! Save a few bucks.

3. As good as the VCP is, there is still a demand for VMware knowledge more advanced than the base cert. I lost my job last year, while job hunting I saw listings asking for VCDX certified people. At the time no VCDX’s were publically known. It just shows the demand for proof of advanced knowledge in VMware and a possible dilution of the VCP due to brain dumping.

4. Testing experience is fairly standard technical testing. Although I wish it was all lab practical rather than memorization. Multiple choice type questions are the majority of the questions. My best test taking tip is to actually know how to Install and Manage Virtual Infrastructure. You can run it all in VMware Workstation 7, so there is no excuse to just memorize answers. I say that but you will have to memorize max and minimums and other facts in order to pass the VCP.

5. Check what is on the blue print on the VMware Certification website, this website is the final word on what is on the test.

Bonus: Use the communities/blogs/twitter to find information and answers. Don’t just post “Hey what is on the exam?”. Ask questions about technical topics, then make sure you could articulate the concept to someone else. I find if I understand enough to be able to teach it to someone else then I really am starting to learn things.

Bonus #2: Be willing to be always learning. If you already know everything you probably don’t need the certification.

*No clue who the original artist is of that picture. It is awesome, so if you know who to credit let me know.

Due to the performance hit while zeroing mentioned in the Thin Provisioning Performance white paper this article in the VMware knowledge base could be of some good use.

I would suggest using eagerzeroedthick for any high IO tier 1 type of Virtual Machine. This can be done when creating the VMDK from the GUI by selecting the “Support Clustering Features such as Fault Tolerance” check box.

So go out and check your VMDK’s.

Well time to get on over to Eric Siebert’s vSphere-land and vote for the top 25 VMware Blogs. The goal this year is to have someone besides my Mom and myself vote for me. So if you happen to like the content of this blog please vote for me. Now some recap of recent posts to remind you why a vote for 2vcps is a vote for vAwesomeness.

Remember the what not to tweet list? Always fun.
What about the time I ranted some nonsense from VMworld? Here. and here.
Who could forget the time I said some stuff about VMware?

I trust equiped with this information you will now be able to place 2vcps in your list of votes.

All kidding aside, I think it is just cool to be on the ballot.

Thank you to everyone out there that gave any attention to this little blog. I tried to supply new information where I could and also document my journey trying to improve at the esxcfg- commands. In 2009 the traffic increased from less than 100 visits in a week to approaching 500 in a week. That is a slow hour for some of the best vm bloggers (Yellow-Bricks, Scott Lowe) but I am not trying to replace them.

In March I was layed off. I spent 2 months full time looking for work. The VMware community really stepped up. Special thanks go out to a few people that tried to help out or checked in with how the search was going: @jasemccarty @maishk @jasonboche and many others. By the end of April I started a new position as a Virtualization Infrastructure Engineer at Veristor Systems. This job required moving the family from Memphis, Tennessee to Atlanta, Georgia. All the fun of a new job, house hunting and moving to a completely new city. We closed on our new house just a few days after VMworld 2009. My first trip to VMworld was amazing but I would suggest not buying a house while you are in and out of conferences, labs and seminars. Lots of details to fix, I will be happy if this is not the case ever again.

Getting into the VMworld stuff.

More VMworld Stuff

After such a busy year it was only the beginning of September. The rest of the year was filled with VMware Installation projects and VMware health checks. I was all over the Southeast US. Maybe I can come visit some of you in 2010?

Technical/VM/Blogging Goals for 2010:

1. Continually improve the quality of content going to this blog.
2. Schedule, take and pass the VCDX exams.
3. Improve my overall storage knowledge.
4. Renew my expired CCNA (bad move letting this lapse in 2009)

2009 was such a big year personally. I hope and pray for a little more stability in 2010 but I also love the excitement of new challenges, but the VCDX should be enough.

Our Backyard in December

Also, I scheduled a Merry Christmas post, but for some reason I didn’t do what I thought I did. So a late Merry Christmas to all of you out there.

Recently I have spent time re-thinking certain configuration scenarios and asking myself, “Why?” If there is something I do day to day during installs is this still true when it comes to vSphere? or will it still be true when it comes to future versions.
Lately I have questioned how I deploy LUNs/volumes/datastores. I usually deploy multiple moderate size datastores. In my opinion this was always the best way to fit in MOST situations. I also will create datastores based on need afterward. So will create some general use datastores then add a bigger or smaller store based on performance/storage needs. After all the research I have done and asking questions on twitter* I still think this is a good plan in most situations.
I went over a VMworld.com session TA3220 – VMware vStorage VMFS-3 Architectural Advances since ESX 3.0 and read this paper:
http://www.vmware.com/resources/techresources/1059
I also went over some blog posts at Yellow-Bricks.com and Virtualgeek.

An idea occurred to me when it comes to using extents in VMFS, SCSI Reservations/Locks, and VDI “Boot Storms”. First some things a picked up.
1. Extents are not “spill and fill” VMFS places VM files across all the LUNs. Not quite what I would call load balancing, since it does not take IO load into account when placing files. So in situations where all the VM’s have similar loads this won’t be a problem.
2. Only the first LUN in a VMFS span gets locked by “storage and VMFS Administrative tasks” (Scalable Storage Performance pg 9). Not sure if this implies all locks.

Booting 100’s of VM’s for VMware View will cause locking and even though vSphere is much better when it comes to how quickly this process takes. There is still an impact. So I am beginning to think of a disk layout to ease administration for VDI, and possibly lay the groundwork for improved performance. Here is my theory:

Create four LUNs with 200GB each. Use VMFS to extents to group them together. Resulting in an 800 GB datastore with 4 disk queues and only 1 LUN that locks during administrative tasks.

Give this datastore to VMware View and let it have at it. Since the IO load for each VM is mostly the same, and really at the highest during boot other tasks performed on the LUN after the initial boot storm will have even less impact. So we can let desktops get destroyed and rebuilt/cloned all day with only locking that first LUN. This part I still need to confirm in the LAB.

What I have seen in the lab is with same sized clones the data on disk was spread pretty evenly across the LUNs.

Any other ideas? Please leave a comment. Maybe I am way off base.

*(thanks to @lamw @jasonboche and @sakacc for discussing or answering my tweets)

It took just about a year. Which shows I need more consistency with my blog (should have been about 1 month). I finally finished a brief explanation of each esxcfg command. My little self study for the VCDX, this is in no way exhaustive.

Make sure to check out other great resources out there:
Simon Long
Harley Stagner
Both good places to start.

Hopefully my VCDX compilation page can help.

The esxcfg-vswif command allows you to create and modify Service Console ports and their IP information. Many times I have to change stuff after the install process is complete and the only place is via the direct service console because network communication is not possible. This usually happens when the network team changes a vlan in the middle of the install or they change a subnet. Not to disparage network teams many times I am the network team and the virtualization team.
Create a new vswif:
#first add a port group with esxcfg-vswitch
esxcfg-vswitch -A "Service Console Test" vSwitch-Test
#then use esxcfg-vswif to create a new vswif
esxcfg-vswif -a -i 172.16.50.40 -n 255.255.255.0 -p "Service Console Test" vswif1
#List your vswifs
esxcfg-vswif - l
#Example:
[root@esx3 root]# esxcfg-vswif -l
Name Port Group IP Address Netmask Broadcast Enabled DHCP
vswif0 Service Console 172.16.50.50 255.255.255.0 172.16.50.255 true false
vswif1 Service Console Test172.16.50.40 255.255.255.0 172.16.50.255 true false

Modify your Service Console network information:
esxcfg-vswif -i 172.16.50.41 -n 255.255.255.0 vswif1
#example
[root@esx3 root]# esxcfg-vswif -i 172.16.50.41 -n 255.255.255.0 vswif1
Setting IP config
Nothing to flush.
[root@esx3 root]# esxcfg-vswif -l
Name Port Group IP Address Netmask Broadcast Enabled DHCP
vswif0 Service Console 172.16.50.50 255.255.255.0 172.16.50.255 true false
vswif1 Service Console Test172.16.50.41 255.255.255.0 172.16.50.255 true false