If you’re an IT pro, no matter if you’re an admin, and engineer, a consultant, a PC technician, you have a toolbox of useful utilities, scripts, and software that you use to fix problems.  As time goes by, some of those tools get used more and more.  Others are used less and less for various reasons.  But what surprises me is how many tools in my toolbox on the surface have less and less use cases, but I still come back to them even when it seems I never would need to again.

Over the last few weeks, I’ve been working with a customer who has had significant turnover from consultants they’ve used.  They are moving off a troubled disparate datacenter environment that had over time developed numerous problems to a more consolidated environment that various SyCom resources including me have built for them that is functioning properly, has updated software and firmware, etc.  Along the way, we’ve run into numerous challenges that you wouldn’t normally anticipate.  Troubleshooting them to fix the problems often would take too much time to fix,.  Finding a duct tape solution was more expedient.

I wanted to give a few examples just to illustrate that having a wide knowledge of utilities out there and experience with them can help you solve problems.

In this case, the task was seemingly simple – move VMs running on a legacy NetApp array and vSphere 5.1 servers to a new(er) cluster running vSphere 5.5.  The clusters were managed by two different vCenter servers.  These clusters were within the same physical datacenter.  They had network connectivity between them.  They did not have access to the same storage arrays.  The customer allowed downtime to move them.  Therefore, the easiest way was a shared nothing cold migration (we’re running 5.1 on the source side, remember).  Simple, right?

Doing It the Textbook Way

I approached this like how any vSphere resource would.  Get the two clusters into the same vCenter instance, shut the VMs down, and migrate them cold.  How many times have you seen that fail?  Me?  Pretty much never.  Well, it wouldn’t work.  I’ll spare you the troubleshooting details, but trust me, doing it the native way wouldn’t work.

At this point, the time had come to get creative and bust out some useful utilities I hadn’t used in a long time.  We had to get the job done.  Tick tick!

Useful Utilities #1 – Veeam FastSCP

The customer wasn’t a Veeam customer (yet).  While the customer could take some downtime off hours, there was a limit to that.  We had to move about 2TB of data, so we needed to move this data as quickly as possible without a ton of labor to reconfigure the networks to get both environments access to the storage.

Sure, I could use WinSCP to just bulk copy the VMs over, but Veeam FastSCP, built into Veeam Backup and Replication trial, is free, and it moves data quicker as it disables encryption on the data transfer, which was acceptable to the customer.  I hadn’t had any reason to use FastSCP in probably five years because cold migration functionality and exporting VMs to OVFs and what not within vSphere made it unnecessary.  But here I was, using it yet again.

And sure enough, it worked like a champ.  We tested a quick procedure using it on a few development workloads.  We then proceeded moving all but the critical VMs, and it worked great… except for the last VM of course.  Come to find out, that was a critical SQL VM that the customer didn’t realize was using physical Raw Device Mappings.

Well, shoot, how do we do this one in a quick manner?

Useful Utilities #2 – VMware Converter

For numerous reasons, including perhaps sheer circumstance of projects I’ve worked on, I hadn’t until this had a need to use VMware Converter in years.  Virtualization is so prevalent now, that P2V is one of those things for me that’s like, “Hey man, remember that time we had to convert like 100 physical machines to virtual back in the day?  Good times!”

Also, I’ve generally recommended to customers to avoid converting physical to virtual anyway.  It should generally be seen as a shortcut, but never optimal.  If you could just build a fresh new VM and get the data moved, the resulting VM would be cleaner.  It would probably perform better.  There’s less chance of instability from old drivers and what would inevitably be a significant change in hardware for the OS and application.  Obviously, if you’re dealing with a ton of machines, rebuilding them all isn’t practical.  In that case, you might have to turn to a P2V tool.

But if you got a VM with physical RDMs, you can’t clone the VM.  You can’t bulk copy the Virtual Machine files over.  You could create new VMDKs and copy everything out of the RDM disks to those and reassign drive letters.  However, this SQL VM was nasty with complex mount points and drive letters assigned.  We had to get it done the weekend the RDMs were discovered.

Solution?  VMware Converter!  I tried installing it on an admin server and set up the job.  That of course failed because of Murphy’s Law.  The Converter agent wouldn’t install due to insufficient permissions.  I installed it directly on the SQL VM (with the same account I tried to push the agent, mind you), stopped the SQL services to ensure the data was static, and ran it.  Other than it shuffling a few drive letters around on the converted VM that a few mouse clicks fixed, it worked like a champ.

How about you?  Any useful utilities you’ve used recently you haven’t used in awhile?