2vcps and a Truck » administration

Operational Readiness

Jon Owings — Tue, 04 May 2010 01:16:57 +0000

One thing I am thinking about due to the VCDX application is operational readiness. What does it mean to pronounce this project or solution good-to-go? In my world it would be to test that each feature does exactly what it should be doing. Most commonly this will be failover testing, but could reach into any feature or be as big as DR plan that involves much more than the technical parts doing what they should. Some things I think need to be checked:

Resources

Are the CPU, Memory, Network and Storage doing what they should be? Some load generating programs like IOmeter can be fine to test network and storage performance. CPU busy programs can verify Resource Pools and DRS are behaving the way they should.

Failover

You have redundant links right? Start pulling cables. Do the links failover for Virtual Machines, Service Console, and iSCSI? How about the redundancy of the physical network, even more cable to pull! Also test that the storage controllers failover correctly. Also, I will make sure HA does what it is supposed to, instantly power off a host and make sure some test virtual machines start up somewhere else on the cluster.

Virtual Center Operations

Deploy new virtual machines, host and storage VMotion, deploy from a template, and clone a vm are all things we need to make sure are working. If this is a big enough deployment make sure the customer can use the deployment appliance if you are making use of one. Make sure the alarms send traps and emails too.

Storage Operations

Create new luns, test replication, test storage alarms and make sure the customer understands thin provisioning if it is in use. Make sure you are getting IO as designed from the Storage side. Making use of the SAN tools to be sure the storage is doing what it should.

Applications

You can verify that each application is working as intended within the virtual environment.

There must be something I am missing but the point is trying to test out everything so you can tell that this virtualization solution is ready to be used.

My Fun with the VMware Enterprise Administration and Design Exams

Jon Owings — Thu, 22 Apr 2010 19:44:41 +0000

Sorry I have been missing for a few weeks. I know many were quite worried why I hadn’t blogged for a couple weeks (not really).

Back in February I sat for the Enterprise Administration Exam at PEX in Las Vegas. It was scheduled the day after the Super Bowl, what a bunch of distractions. Thankfully I passed and I want to give my experience so as to not violate any rules or anything I agreed to. This was a technical test. A lot of settings and configurations and information like that. Still multiple choice so at least you know the right answer is on the screen (hopefully, I did have one I thought none of these are right). The lab section was actually as fun as test taking could be. I wish there was more lab practical type things when it comes to these kinds of tests. Overall there is more intricate settings and config questions then you will find on the VCP exam.

At the end of April I took the Design Exam. This was a much different experience. I had a extremely hard time finding a study list of things that would help. Know the Exam Blueprint is all I would say. Also, this I think is where VMware can start finding out who does Architecture work and who may be an Administrator. I could say you could read every PDF on VMware.com and still not know how to pass this test unless you work with the solutions multiple times. The design drawing was a challenge, I wasted too much time reading the requirements document and ran of time, but I feel I was able to get a good portion of what I needed up on the page. Technically the interface was kind of quirky.

I felt both exams were challenging and but were fair to the Exam Blueprints. Nothing on there made me scream, “they didn’t say they would test on THAT!” The design exam needs some technical improvement (matching questions were buggy).

Now begins the harder and more involved process. The Design submission and hopefully an invitation to a defense.

VMware View – User Profile Options

Jon Owings — Sun, 14 Mar 2010 19:59:24 +0000

All the technology and gadgets for managing desktops are worthless if your users complain about their experience with the desktop. Something I learned administering Citrix Presentation Server. Differing methods exist to keep the technical presentation of the desktop usable, for example the mouse being in sync and the right pixels show the right colors. What is also included in the user experience is a consistent environment where their personal data and settings are where they should be. Here are a few methods for managing those bits when using VMware View.

Mandatory Profiles
This profile is kept on the a central file share. The profile is copied to the machine on login, when the user logs out the changes are not kept. Great way to keep a consistent profile on kiosk type and data entry desktops. Where customization is not needed and most likely not wanted mandatory profiles are worth exploring. Main change is you set up the profile just like you want it then rename the NTUSER.dat to NTUSER.man. A lot exists on the internet about setting up man profiles.

Local Profiles
If you go through life never changing a thing in your Windows environment, you are using a Local Profile. Not to say you don’t change settings, save files or customize your background. You just have Windows running as the default. This is an option I will usually discourage because it is hard to backup data that is often kept in the local profile. VMware View will redirect user data to a User Data Disk (or whatever it is called today) on Persistent Desktop Pools. This is a good way to get the data on another VMDK. This introduces problems when looking at data recovery. There is solutions, but just something you will need to remember to look into.

Roaming Profiles
Roaming profiles is a great way to redirect current profiles to a central location. In theory this works great. In a View environment you can keep a local copy on a users desktop profile and the changes are copied back and forth. I have often seen this work just great. Then from time to time, the profile will become corrupt, many times it does not unload correctly when users disconnect, or log out. Then you may have to pick through folders trying to find their “My Documents”. This is why I would suggest using this with Group Policy and Folder redirection which I will cover next.

Redirecting Folders
You may end up using a folder redirection group policy. This will move folders like the Desktop and My Documents for a user to a file server. This slims down the roaming profile as those locations are redirected to another location outside of the profile. This data is not copied from the machine to the server over and over. More information here.

Other Options
Immidio Flex Profiles
I really liked this option it was a way to combine mandatory profiles and a Roaming profile. This program would run some scripts on logon and log off to save files and settings. A really great paper on how to use it can be found here. Just like any great program that takes a new way to solve an annoying old problem, this is now not free.

RTO Virtual Profiles
I have never implemented this solution before. I have used it as part of a few training labs. I liked the feel. Now that VMware has purchased this software from RTO, the website redirects to a transition page. So I am looking for a way to test it in the lab, hoping the next set of bits of View includes RTO. Check this FAQ out for more information.

Maybe once it is built into View this will no longer be a serious issue. Profiles will be one of those things we tell stories to young padawan VM admins about, “We used to have to fight profiles, they were big and slow, and sometimes they would disappear!” Until that day…

VMware View and Xsigo

Jon Owings — Fri, 19 Feb 2010 03:48:34 +0000

*Disclaimer – I work for a Xsigo and VMware partner.

I was in the VMware View Design and Best practices class a couple weeks ago. Much of the class is built on the VMware View Reference Architecture. The picture below is from that PDF.

It really struck me how many IO connections (Network or Storage) it would take to run this POD. Minimum (in my opinion) would be 6 cables per host with ten 8 host clusters that is 480 cables! Let’s say that 160 of those are 4 gb Fiberchannel and the other 320 are 1 gb ethernet. The is 640 gb for storage and 320 for network.

Xsigo currently uses 20 gb infiniband and best practice would be to use 2 cards per server. The same 80 servers in the above cluster would have 3200 gb of bandwidth available. Add in the flexibility and ease of management you get using virtual IO. The cost savings in the number director class fiber switches and datacenter switches you no longer need and the ROI I would think the pays for the Xsigo Directors. I don’t deal with pricing so this is pure contemplation. So I will stick with the technical benefits. Being in the datacenter I like any solution that makes provisioning servers easier, takes less cabling, and gives me unbelievable bandwidth.

So just in the way VMware changed the way we think about the datacenter. Virtual IO will once again change how we deal with our deployments.

Storage Design and VDI

Jon Owings — Wed, 23 Dec 2009 20:42:59 +0000

Recently I have spent time re-thinking certain configuration scenarios and asking myself, “Why?” If there is something I do day to day during installs is this still true when it comes to vSphere? or will it still be true when it comes to future versions.
Lately I have questioned how I deploy LUNs/volumes/datastores. I usually deploy multiple moderate size datastores. In my opinion this was always the best way to fit in MOST situations. I also will create datastores based on need afterward. So will create some general use datastores then add a bigger or smaller store based on performance/storage needs. After all the research I have done and asking questions on twitter* I still think this is a good plan in most situations.
I went over a VMworld.com session TA3220 – VMware vStorage VMFS-3 Architectural Advances since ESX 3.0 and read this paper:
http://www.vmware.com/resources/techresources/1059
I also went over some blog posts at Yellow-Bricks.com and Virtualgeek.

An idea occurred to me when it comes to using extents in VMFS, SCSI Reservations/Locks, and VDI “Boot Storms”. First some things a picked up.
1. Extents are not “spill and fill” VMFS places VM files across all the LUNs. Not quite what I would call load balancing, since it does not take IO load into account when placing files. So in situations where all the VM’s have similar loads this won’t be a problem.
2. Only the first LUN in a VMFS span gets locked by “storage and VMFS Administrative tasks” (Scalable Storage Performance pg 9). Not sure if this implies all locks.

Booting 100′s of VM’s for VMware View will cause locking and even though vSphere is much better when it comes to how quickly this process takes. There is still an impact. So I am beginning to think of a disk layout to ease administration for VDI, and possibly lay the groundwork for improved performance. Here is my theory:

Create four LUNs with 200GB each. Use VMFS to extents to group them together. Resulting in an 800 GB datastore with 4 disk queues and only 1 LUN that locks during administrative tasks.

Give this datastore to VMware View and let it have at it. Since the IO load for each VM is mostly the same, and really at the highest during boot other tasks performed on the LUN after the initial boot storm will have even less impact. So we can let desktops get destroyed and rebuilt/cloned all day with only locking that first LUN. This part I still need to confirm in the LAB.

What I have seen in the lab is with same sized clones the data on disk was spread pretty evenly across the LUNs.

Any other ideas? Please leave a comment. Maybe I am way off base.

*(thanks to @lamw @jasonboche and @sakacc for discussing or answering my tweets)

ESX Commands – Summary

Jon Owings — Wed, 23 Dec 2009 01:20:30 +0000

It took just about a year. Which shows I need more consistency with my blog (should have been about 1 month). I finally finished a brief explanation of each esxcfg command. My little self study for the VCDX, this is in no way exhaustive.

Make sure to check out other great resources out there:
Simon Long
Harley Stagner
Both good places to start.

Hopefully my VCDX compilation page can help.

ESX Commands: esxcfg-vswitch

ESX Commands: esxcfg-vswif

Jon Owings — Tue, 22 Dec 2009 01:17:33 +0000

The esxcfg-vswif command allows you to create and modify Service Console ports and their IP information. Many times I have to change stuff after the install process is complete and the only place is via the direct service console because network communication is not possible. This usually happens when the network team changes a vlan in the middle of the install or they change a subnet. Not to disparage network teams many times I am the network team and the virtualization team.
Create a new vswif:
#first add a port group with esxcfg-vswitch esxcfg-vswitch -A "Service Console Test" vSwitch-Test #then use esxcfg-vswif to create a new vswif esxcfg-vswif -a -i 172.16.50.40 -n 255.255.255.0 -p "Service Console Test" vswif1 #List your vswifs esxcfg-vswif - l #Example: [root@esx3 root]# esxcfg-vswif -l Name Port Group IP Address Netmask Broadcast Enabled DHCP vswif0 Service Console 172.16.50.50 255.255.255.0 172.16.50.255 true false vswif1 Service Console Test172.16.50.40 255.255.255.0 172.16.50.255 true false

Modify your Service Console network information:
esxcfg-vswif -i 172.16.50.41 -n 255.255.255.0 vswif1 #example [root@esx3 root]# esxcfg-vswif -i 172.16.50.41 -n 255.255.255.0 vswif1 Setting IP config Nothing to flush. [root@esx3 root]# esxcfg-vswif -l Name Port Group IP Address Netmask Broadcast Enabled DHCP vswif0 Service Console 172.16.50.50 255.255.255.0 172.16.50.255 true false vswif1 Service Console Test172.16.50.41 255.255.255.0 172.16.50.255 true false

ESX Commands: esxcfg-vswitch

Jon Owings — Sun, 22 Nov 2009 17:52:59 +0000

This is a command I use the most often. It is big when I configure Equalogic and MPIO in vSphere. Additionally, many times I show up on site and the network is not configured prior to my arrival. Even if I sent all the configs ahead of time. People like to wait until the last minute. When building the ESX environment then I may build it with the Service Console in the default vlan or in which ever one it is plugged into in the physical network. So oftern the network guy catches up and sets the dot1q trunk and I lose connectivity and I have to go to the console and set the vlans correctly. Like this:
esxcfg-vswitch -v 8 -p "Service Console" vSwitch0
Speaking of iSCSI you will also set your jumbo frames on the vSwitch from this command
esxcfg-vswitch -m 9000 vSwitch1

Of course there is a ton more you can do from this command. Here is some man page action for you:
esxcfg-vswitch(8) VMware ESX Manual esxcfg-vswitch(8)


NAME

     esxcfg-vswitch - VMware ESX Virtual Switch Configuration tool
SYNOPSIS

     esxcfg-vswitch OPTIONS [VSWITCH]
DESCRIPTION

     esxcfg-vswitch provides an interface for adding, removing, and modifying virtual switches and their settings.  By default,

     there is a single virtual switch called vSwitch0.
OPTIONS

     -a --add

             Add a new virtual switch to the system.  It requires a virtual switch name to be provided.
     -d --delete

             Delete a virtual switch.  This will fail if any ports on the virtual switch are still in use by VMkernel networks,

             vswifs, or VMs.
     -l --list

             List all virtual switches and their portgroups.
     -L --link

             Add an uplink to a virtual switch.  This will attach a new unused physical NIC to a virtual switch.
     -U --unlink

             Remove an uplink from a virtual switch.  This will remove a NIC from the uplink list of a virtual switch. If it is

             the last uplink, physical network connectivity for that switch will be lost.
     -R --restore-uplinks

             Restore uplinks for all virtual switches from config file. This will restore all uplinks for each virtual switch from

             configurtion file. INTERNAL USE ONLY.
     -M --add-pg-uplink

             Add an uplink to the list of uplinks for a portgroup
     -N --del-pg-uplink

             Delete an uplink from the list of uplinks for a portgroup
     -P --add-dvp-uplink

             Add an uplink to a DVPort on a DVSwitch. Must specify --dvp in the same commnad.
     -Q --del-dvp-uplink

             Delete an uplink from a  DVPort on a DVSwitch. Must specify --dvp in the same commnad.
     -p --pg

             Provide the name of the portgroup. For the â--vlanâ option,  "ALL" can be specified to operate on all portgroups of a

             virtual switch.
     -V --dvp

             Provide the DVPort ID of a DVSwitch.

 -v --vlan

             Set the VLAN ID for a specific portgroup of a virtuals switch Using the option "0" will disable VLAN for this port-

             group.  Requires that the --pg option is also specified.
     -c --check

             Check to see if a virtual switch exists. The program prints a "1" if it exists; otherwise it prints "0".
     -A --add-pg

             Add a new portgroup to a virtual switch with the given name.
     -D --del-pg

             Delete a portgroup.  This operation will fail if the portgroup is in use.
     -C --check-pg

             Check whether the specified name is in use for a portgroup.  Prints "1" if the name is in use, prints "0" otherwise.
     -B --set-cdp

             Set the CDP status for a given virtual switch.  To set pass one of "down", "listen", "advertise", "both".
     -b --get-cdp

             Print the current CDP setting for this switch.
     -X --set-maxactive

             Set the max active uplinks for the virtual switch.
     -x --get-maxactive

             Get the max active uplinks for the virtual switch.
     -m --mtu

             Set the MTU for the vswitch.  This affects all the NICs attached to the vswitch.
     -r --restore

             Used at system startup to restore configuration.  INTERNAL USE ONLY.
     -h --help

             Print a simple help message.
EXAMPLES

     Add a Virtual Switch:
     esxcfg-vswitch --add vSwitch1
     Add a Portgroup to vSwitch0:
     esxcfg-vswitch --add-pg="New Portgroup" vSwitch0
COPYRIGHT

     VMware ESX is Copyright 1998-2009 VMware, Inc.  All rights reserved.

VMware ESX 4.0 November 22, 2009 VMware ESX 4.0

Central Syslog Host for ESX – Syslog-ng

Jon Owings — Thu, 18 Dec 2008 20:27:00 +0000

Someone may have already written all this, but oh well.

1. Install something free like Ubuntu Server.

2. I use Ubuntu because I like Debian and apt-get. So run:

#apt-get install syslog-ng

Running Syslog-ng gives you more than the standard syslog daemon.

3. Configure syslog-ng to recieve udp logs.

root@hoth:/# cd /etc/syslog-ng/
root@hoth:/etc/syslog-ng# vi syslog-ng.conf

side note: learn VI

Add this after the main source section:

source s_remote {
udp();
};

After the destinations:

destination df_remote { file(“/var/log/remote.$HOST”); };

The $HOST will sort the logs by IP of the server.

And finally add this to the end of the syslog-ng.conf file:

log {
source(s_remote);
destination(df_remote);
};

Now Restart the syslog-ng service

root@hoth:/etc/syslog-ng# service syslog-ng restart

4. Make changes to the ESX syslog config. Thanks to Tooms.dk I have been using syslog-ng so much I needed to find the commands for standard syslog.

1. In the /etc/syslog.conf file add this line “*.* @172.16.0.202” without the ” and change the ip number to your syslog servers ip

2. Restart the syslog service with the command “service syslog restart”

3. Open the ESX server firewall with this command “esxcfg-firewall -o 514,udp,out,syslog” to allow syslog outgoing trafic

4. Tell the ESX firewall to reload the config with this command “esxcfg-firewall -l”

Now you can tail -r /var/log/remote.10.10.10.2 (or whatever your IP is).