Category Archives: administration

Build IP Management Server with Ubuntu, GIT and Nmap

Have 25 Vlans in your lab and Storage, Servers, Switches and other equipment all over the place?

Build a good Ubuntu box

wpid1098-media_1358298686321.png
If you need more help on this leave some comments. It is pretty straight forward.
Be sure to select openssh server as a role.
After you login:
Remember to
$sudo apt-get update
$sudo apt-get upgrade
#If you are adventurous
$sudo apt-get dist-upgrade

Install VMware Tools (you ARE doing this in a VM right?)

wpid1099-media_1358299813754.png
From your VI Client set the VMware tools install to start, this will insert the disk for VMware Tools, but unfortunately doesn’t automate the install.
$sudo -s
#apt-get install build-essential
#apt-get install
#apt-get install apache2
The last two steps might not be needed but I do them out of habit. They allow the VMware tools to build the Kernel modules needed for VMXNET drivers.#mount /dev/cdrom /mnt/cd
#cp /mnt/cd/VMwareTools-8.6.0-425873.tar.gz
#tar xvzf /tmp/VMwareTools-8.6.0-425873.tar.gz
#cd vmware-tools-distrib
#./vmware-install.pl
Now Run through the install of VMware tools.#apt-get install nmap
#apt-get install gitweb
#exit
You want to do some of the next steps as your normal user not root.
$cd ~
$mkdir nmaps
$git init nmaps
$sudo vi /etc/gitweb.conf
Modify the project root var to match your path to you nmaps.
wpid1100-media_1358302137956.png
Now restart apache.
$sudo /etc/init.d/apache2 force-reload
wpid1101-media_1358302508614.png
Ok great, the webpage will be up at http://yourhost/gitweb
Now to add your network information:
$nmap -sP 10.10.10.0/25 -oG /pathtoyour/nmapsdirectory/file.map
<update> The above command is to scan a subnet with pings then dump the output to a file.</update>

Nmap – great network scanner, the -sP sets it to “Ping” mode, you don’t need to crawl ports or anything so ping is fine, it will resolve the DNS if that is working.
my example:
$nmap -sP 172.10.0.0/16 -oG /home/lab-admin/nmaps/vlan10.map

For what it is worth a /16 network takes forever. If you know the usable ip’s are a way smaller subset you can specify a range.
$nmap -sP 172.10.0.1-100 -oG /home/lab-admin/nmaps/vlan10.map

Next on the first time you must add the file to the git repository
$cd nmaps
$git add /home/lab-admin/nmaps/vlan10.map
$git commit -m “My Network Maps”

Now we have some content on the gitweb page.

wpid1102-media_1358344480271.png
Click tree to see the files you have added.
wpid1103-media_1358344539639.png
Click on blob to see your IP’s in use.

Let’s automate it with crontab

wpid1104-media_1358345701192.png
As you can see it is fairly easy to get rolling. Before you finish, run the nmap command for each subnet and git add each file to the repository.Now we will tell cron to run the nmap scans on a regular interval and git commit the changes.
$vi ip-scanscript.sh
In this file put each ip scan on its own line. As seen above. Save the changes and exit.
$chmod +x ip-scanscript.sh
$./ip-scanscript.sh
This will create all the files you want in one swoop.
$cd nmaps
$git add *
This will add all the new files in the nmaps directory to the git repository.Edit the script one more time.
$vi ip-scanscript.sh
wpid1105-media_1358346482687.png
By adding the git commit line to the script you can now set this up to automatically run with cron.
$crontab -e
[note]I was having issues with the range line in the screenshot above. Still testing.
[2nd note] change the git commit line to include a ‘-a’ example:
git commit -a -m “My IP Lists”
If you need help with crontab this is where I go when I can’t remember.
http://www.adminschoice.com/crontab-quick-reference
http://www.thegeekstuff.com/2011/07/cron-every-5-minutes/
I set mine to run every 4 hours.Hope this is helpful, not much reason to buy a fancy product when this can be up and running so quickly.

More References:
http://gitref.org/basic/

Book Review: Automating vSphere with VMware vCenter Orchestrator

So to be 100% honest I have had this book on my desk for several months. Just staring at me. Calling my name. VMware press provided this copy to me along with Mike Laverick’s SRM book and so I am finally going to review the first one.

Cody Bunch does an amazing job of breaking down one of the most mystifying yet powerful products hidden in the VMware portfolio. VMware vCenter Orchestrator is almost mythical in the promises of automation of typical tasks of a vSphere administrator. While you can bang your head against the wall for weeks trying to figure out how to properly setup the vOrchestrator server and client I was able to use Cody’s guidance to have to operational and running test workflows in just a few hours (I am a slow reader).

I can’t stress enough the need for automation and orchestration in today’s virtual machine environment. The business is demanding more and more from the Virtualization team and in order to deliver vCenter Orchestrator is a good start since you probably already OWN it.

Hopefully soon there will be an update with information on the vApp version of Orchestrator. Check it out here on Amazon or your favorite book reseller.

Thanks again

Leadership in the Cloud (And everywhere else)

This is really a post about leadership in general, but I like to apply it to our industry. I am totally cool if you take these concepts and apply them elsewhere.

In any work environment there is constant posturing, politicing, conflicting, that has nothing to do with the actual cause of the workplace. I am going to offer a few leadership tips for everyone, not just for managers, vp’s and directors. Tips that we can all put to use.

1. It is not all about you. We all know that “guy” (or girl). Using every oppurtunity to push others down and himself up. Using others backs to climb on never lasts. Being the MVP of a losing team is never my goal, make everyone around you better. The skills involved in doing that will take you further than your daily task knowledge. No one ever says, “Wow, Jon sure can deploy a sweet VM.” If you are known for adding value, contributing and making everyone better that is how what you do will last. Valuing your team as something more than tools to make you look good is a good start.

2. Have a Purpose/Mission. I am here to change the world. Personally and Professionally. I have done jobs and have volunteered with people and organizations where no one knows why they do what they do. If you are making Pizza, make life changing pizza. If you are building next-gen datacenters, do it in a way that will alter life for someone.

3. Lead, Even if you aren’t supposed to. Don’t sit around and wait to be asked to do something leadershippy.

4. Have a Strategy. If you don’t know why you do what you do get that first. Then decide how the world will look when you are done. Impact (well good impact) on people will not happen on accident.

5. If you see a problem be part of the Solution. Stop complaining. There is only so much time in the day. Personally, it is natural for me to complain. I am very good at pointing out faults in everything. I have to consciously make the decision to work on the solutions for things I can change and shut up about the other stuff (for now). Some things just need the proper timing.

6. Community. Jump into the deep end of the pool of community. Make this a core tenant of everything you participate in. You can not do it all by yourself. Community substitues like Twitter and Facebook are a start but go meet in person with some real people. Just an idea.

The most cynical of my readers never started reading this. If you got this far, I hope in your mind you see how this applies to you. Of course any comments are welcome.

Some Reality for us Infrastructure Peeps or Apps are cool too

Don’t’ you just love double titles?

For many years I have been an infrastructure guy. I really liked how the cables, and processors and Memory and blinking lights worked. Applications were often the necessary evil tolerated so that I can play with cool technology. During my own journey toward learning about the cloud it becomes increasingly important to consider the function of the application. Six years ago me would totally punch me in the face right now. Traitor. J

1 – Don’t get your App messed up in my resource buckets of awesomeness

 

So the reality check to the Infrastructure geek in me is this: The application teams really think of what you do as the network. That is why when anything is ever wrong it is always “the network’s” fault. What we love to do is getting abstracted more and more. I will still contend that is very important and very hard to do. Whether you are building reference architectures or deploying a converged infrastructure appliance almost no one but us cares. They just want the data to do their jobs. So while we have really great discussions about speeds and feeds, the guy in the picture below just wants the app. From the hypervisor down we need to design with the application in mind or we will risk becoming like that goth dude locked in the server room on IT Crowd.

 

2 Honey badger don’t care about FCoE

My next post will get into what I have been researching regarding what is out there and hopefully help us (infra. peeps) understand our App/Dev brothers better.

You are probably an Infrastructure person if:

  1. You read this blog.
  2. You work mainly with Virtualization
  3. Storage Admin
  4. Network Admin
  5. You like to make fun of DBA’s

 

No clever title – ESXCLI

I have been missing in action for a few weeks. It is time to catch up for all the lost time. One topic I feel many people don’t know too much about is esxcli. I know how to do what I usually do with esxcli. There is a lot more there for us to explore.

First stop and take a look at the virtuallyGhetto article.

It can be run from the Service Console, the ESXi Tech Support Mode command line, or from the vMA. As William points out if you are running these command from the vMA you need to authenticate individually to each host. He goes on to list some articles that go over the most used case of esxcli, swiscsi.

A couple of quick examples I like to use:

esxcli nmp device setpolicy –device naa.6090a07800c2ea66b8c114050000c00d –psp VMW_PSP_RR

This command changes the policy for a storage device to another path selection policy. In this case it is Round Robin. This is great for when you are rebuilding ESX and the storage is already zoned. ESX will add the storage with the default PSP and changing a few dozen datastores on each host one at a time via the GUI can be VERY tedious.

Then how do I change the default PSP?

esxcli nmp satp setdefaultpsp –psp VMW_PSP_RR –satp VMW_SATP_DEFAULT_AA

This can be modified for different array types after the “—satp” tag or different path policies after the “–psp” tag.

For the VCAP-DCA4 exam I am studying for I wonder how much deeper than this they will go? I would feel most Data Center Administrators need to set up swisci settings and possibly change path policies. Anything I am missing? If you check out Duncan’s article here it will be great to know how to list what is available.

Equallogic, VAAI and the Fear of Queues

Previously I posted on how using bigger VMFS volumes helps Equallogic reduce their scalability issues when it comes to total iSCSI connections. There was a comment about does this mean we can have a new best practice for VMFS size. I quickly said, “Yeah, make em big or go home.” I didn’t really say that but something like it. Since the commenter responded with a long response from Equallogic saying VAAI only fixes SCSI locks all the other issues with bigger datastores still remain. ALL the other issues being “Queue Depth.”

Here is my order of potential IO problems on with VMware on Equallogic:

  1. Being spindle bound. You have an awesome virtualized array that will send IO to every disk in the pool or group. Unlike some others you can take advantage of a lot of spindles. Even then, depending on the types of disks some IO workloads are going to use up all your potential IO.
    Solution(s): More spindles is always a good solution if you have unlimited budget. Not always practical. Put some planning into your deployment. Don’t just buy 17TB of SATA. Get some faster disk and break your Group into pools and separate the workloads into something better suited to the IO needs.
  2. Connection Limits. The next problem you will run into if you are not having IO problems is the total iSCSI connections. In an attempt to get all of the IO you can from your array you have multiple vmk ports using MPIO. This multiplies the connections very quickly. When you reach the limit, connections drop and bad things happen.
    Solution: The new 5.02 firmware increases the total maximum connections. Additionally, bigger datastores means less connections. Do the math.
  3. Queue Depth. There are queues everywhere, the SAN ports have queues. Each LUN has a queue. The HBA has a queue. I would need to defer to a this article by Frank Denneman (a much smarter guy than myself.) That balanced storage design is best course of action.
    Solution(s): Refer to problem 1. Properly designed storage is going to give you the best solution for any potential (even though unlikely) queue problems. In your great storage design, make room for monitoring. Equallogic gives you SANHQ USE IT!!! See how your front end queues are doing on all your ports. Use ESXTOP or RESXTOP to see how the queues look on the ESX host. Most of us will find that queues are not a problem when problem one is properly taken care of. If you still have a queuing problem then go ahead and make a new datastore. I would also request Equallogic (and others) release a Path Selection Policy plugin that uses a Least Queue Depth algorithm (or something smarter). That would help a lot.

So I will repeat my earlier statement that VAAI allows you to make bigger datastores and house more VM’s per store. I will add a caveat, if you have a particular application that needs a high IO workload, give it a datastore.

Update Manager Problem after 4.1 Upgrade

A quick note to hopefully publicize a problem I had which I see is discussed in the VMware Community Forums already.

After building a new vCenter Server and Upgrading the vSphere 4.0 databases for vCenter and Update Manager. I noticed I could not scan hosts that were upgraded to 4.1. To be fair, by upgrading I mean rebuilt with a fresh install but with the exact same name and IP addresses. Seems that the process I took to upgrade has some kind of weird effect in the Update Manager Database. The scans fail almost immediately. I searched around the internet and found a couple of posts on the VMware Forums about the subject. One person was able to fix the problem by removing Update Manager and when reinstalling selecting the option to install a new database. I figured I didn’t have anything important in my UM database so I gave it a try and it worked like a champ.

Right now there is not any new patches for vSphere 4.1 but I have some Extension packages that need to be installed (Xsigo HCA Drivers). I wanted to note that I like the ability to upload extensions directly into Update Manager. This is a much cleaner process than loading the patches via the vMA for tracking and change control purposes.

Finding the Fusion OVFTool

The OVFtool is something I wished VMware Fusion had a while back and finally got a chance to use it the other day. I checked google and I found that it was located at:

/Library/Application Support/VMware Fusion/ovftool

As I looked for that path I was surprised it was not there. I upgraded from Fusion 2 to 3 to 3.1 and never recalled a chance or a place to add the OVFtool to my install. I could not find an independent download for the Mac OVFtool. I ended up re-installing the newest version of Fusion and I had to click “Advanced” during the install and turn on the OVFtool to install. Not sure if that is the best way, but that is how I got it to work. :)

Now that the path exists I was able to convert the OVF Appliance to be used on my Mac.

ovftool --help reveals a ton of options. To do a basic conversion though try this:


$mkdir /Users/username/Documents/Virtual\ Machines/ApplianceName
$/Library/Application Support/VMware Fusion/ovftool/ovftool ./Appliance.ovf /Users/username/Documents/Virtual\ Machines/ApplianceName

This will expand and convert the VM to be used with Fusion. Now just select open the VM in Fusion and play away.

Operational Readiness

One thing I am thinking about due to the VCDX application is operational readiness. What does it mean to pronounce this project or solution good-to-go? In my world it would be to test that each feature does exactly what it should be doing. Most commonly this will be failover testing, but could reach into any feature or be as big as DR plan that involves much more than the technical parts doing what they should. Some things I think need to be checked:

Resources

Are the CPU, Memory, Network and Storage doing what they should be? Some load generating programs like IOmeter can be fine to test network and storage performance. CPU busy programs can verify Resource Pools and DRS are behaving the way they should.

Failover

You have redundant links right? Start pulling cables. Do the links failover for Virtual Machines, Service Console, and iSCSI? How about the redundancy of the physical network, even more cable to pull! Also test that the storage controllers failover correctly. Also, I will make sure HA does what it is supposed to, instantly power off a host and make sure some test virtual machines start up somewhere else on the cluster.

Virtual Center Operations

Deploy new virtual machines, host and storage VMotion, deploy from a template, and clone a vm are all things we need to make sure are working. If this is a big enough deployment make sure the customer can use the deployment appliance if you are making use of one. Make sure the alarms send traps and emails too.

Storage Operations

Create new luns, test replication, test storage alarms and make sure the customer understands thin provisioning if it is in use. Make sure you are getting IO as designed from the Storage side. Making use of the SAN tools to be sure the storage is doing what it should.

Applications

You can verify that each application is working as intended within the virtual environment.

There must be something I am missing but the point is trying to test out everything so you can tell that this virtualization solution is ready to be used.

My Fun with the VMware Enterprise Administration and Design Exams

Sorry I have been missing for a few weeks. I know many were quite worried why I hadn’t blogged for a couple weeks (not really).

Back in February I sat for the Enterprise Administration Exam at PEX in Las Vegas. It was scheduled the day after the Super Bowl, what a bunch of distractions. Thankfully I passed and I want to give my experience so as to not violate any rules or anything I agreed to. This was a technical test. A lot of settings and configurations and information like that. Still multiple choice so at least you know the right answer is on the screen (hopefully, I did have one I thought none of these are right). The lab section was actually as fun as test taking could be. I wish there was more lab practical type things when it comes to these kinds of tests. Overall there is more intricate settings and config questions then you will find on the VCP exam.

At the end of April I took the Design Exam. This was a much different experience. I had a extremely hard time finding a study list of things that would help. Know the Exam Blueprint is all I would say. Also, this I think is where VMware can start finding out who does Architecture work and who may be an Administrator. I could say you could read every PDF on VMware.com and still not know how to pass this test unless you work with the solutions multiple times. The design drawing was a challenge, I wasted too much time reading the requirements document and ran of time, but I feel I was able to get a good portion of what I needed up on the page. Technically the interface was kind of quirky.

I felt both exams were challenging and but were fair to the Exam Blueprints. Nothing on there made me scream, “they didn’t say they would test on THAT!” The design exam needs some technical improvement (matching questions were buggy).

Now begins the harder and more involved process. The Design submission and hopefully an invitation to a defense.