Monthly Archives: November 2015

vSphere 6.0 Change Block Tracking Patch released

vSphere, ,

Just a heads up, but VMware dropped the public release of the patch to resolve the Change Block Tracking problem in ESXi 6.0.  You can apply the patch using VMware Update Manager, or install it manually.

Logically, remember that you can’t just apply the patch and all is well.  You need to reset CBT data to “start fresh” because all changed blocks reported prior to the patch are still suspect.  Most backup vendors detail how you do this in VMware, but I wanted to share a few tips in this regard.

  1. Change Block Tracking can easily be disabled/enabled on Powered Off VMs.  That’s not an issue.
  2. You can reset Change Block Tracking information on a VM by disabling CBT on the VM, taking a snapshot of the VM, deleting the snapshot, and then re-enable CBT.  This makes for automation of this very easy.  Veeam has a PowerCLI script that can do this as an example, although it is clearly a use at your own risk affair.

Finally, don’t forget to enable CBT on your backup jobs and/or VMs when you’re ready if that was disabled as a workaround.  You can do this using PowerShell if you’re using Veeam.

 

Change VMware MPIO policy via PowerCLI

vSphere, ,

This is one of those one liners I think that I’ll never use again, but once again, I found myself using it to fix MPIO policies on a vSphere 5.0 environment plugging into a Nexsan storage array.  I’ve previously used it on EMC and LeftHand when the default MPIO policy for the array type at the time of ESXi installation is not the recommended one after the fact, or in the case of LeftHand, it is wrong from the get go.

get-vmhost | get-scsilun | where vendor -eq "NEXSAN" | set-scsilun  -MultipathPolicy "RoundRobin"

In this case, it was over 300 LUN objects (LUNs x hosts accessing them), so that’s about 5 mouse clicks per object to fix via GUI.  Translation, you REALLY want to use some kind of scripting to do this, and PowerCLI can do it in one line.

You gotta love PowerShell!

Disable CBT on Veeam jobs via PowerShell

howto, vSphere, , , ,

If you haven’t heard the not so great news, VMware has discovered a bug  in vSphere 6 with Change Block Tracking (CBT) that can cause your backups to be corrupt and therefore invalid.  Currently, they are recommending not to use CBT with vSphere 6 when backing up your VMs.

I was looking for an easy way to disable this on all jobs in Veeam quickly via PowerShell, but it’s not obvious how to do that, so I took some time to figure it out.  Here it is assuming the module is loaded in your PowerShell session.

$backupjobs = get-vbrjob | where jobtype -eq "Backup"
foreach ($job in $backupjobs){
$joboptions = $job | get-vbrjoboptions
$joboptions.visourceoptions.UseChangeTracking = $false
$job | set-vbrjoboptions -options $joboptions
}

Here’s now to enable it again:

$backupjobs = get-vbrjob | where jobtype -eq "Backup"
foreach ($job in $backupjobs){
$joboptions = $job | get-vbrjoboptions
$joboptions.visourceoptions.UseChangeTracking = $true
#$joboptions.visourceoptions.EnableChangeTracking = $true
$job | set-vbrjoboptions -options $joboptions
}

Sorry it’s not pretty on the page, but I wanted to get this out ASAP to help anyone needing to do this quickly and effectively.

One thing to note is in the enable script, there’s a commented line out.  If you have already set your jobs manually and wish to use the script to enable CBT again, be aware that the option to enable CBT within VMware if it is turned off gets disabled if you turn CBT off altogether within the job setup.  If you disable CBT with my script, that doesn’t get touched, so you don’t need to remove the # on that line.   If you want that option enabled again, take out the # before that line, and it’ll enable that option again.

Hope this helps!

Clarifying vSphere Fault Tolerance

vSphere,

I hear a lot of confusion about some of the new enhancements of vSphere 6. One is specifically Fault Tolerance (FT)

In case you do not know what FT is, this is a feature that basically (was supposed to) fit the need for a handful of your most critical VMs that High Availability (HA) didn’t protect well enough.  HA restarts a VM if the ESXi physical host it was running on failed on another host, or if you enable VM Monitoring, a VM that blue screened or locked up.  Note the VM would be down during the restart time of the VM and the boot up of the OS within the VM.  FT effectively runs a second copy of the VM in lockstep on another host, so should the host the live VM runs on fails, the second copy immediately takes over on the other host, with no downtime.

Please note that vSphere 6 nor previous versions of vSphere do not protect against an application crash itself unless the application crashed due to a hardware failure using Fault Tolerance.  It only protects against effectively failures pertaining to hardware, like a host failure.  There is no change there.  If you want protection from application failures, you still should look at application clustering and high availability solutions, like Exchange DAGs, network load balancing, SQL clustering, etc.  On the flip side, I have personally seen many environments actually have MORE downtime because of application clustering solutions, especially when customers don’t know how to manage them properly, but FT is a breeze to manage.

The problem with FT in the past is it had so many limitations.  The disks had to be zero eager thick provisioned for the VM, you could not VMotion the VM or the second copy, and more, but the biggest limitation was the VM could only have 1 vCPU.  If you’re thinking how many critical apps only need 1 vCPU, the answer is pretty much zero.  Almost all need more, so FT became the coolest VMaware feature nobody used.

That changes in vSphere 6.  You can use FT to protect VMs with up to 4 vCPUs.  They can be thin or thick provisioned.

FT protected VMs can now be backed up with whole VM backup products that utilize the VMware Backup APIs, which is all of them that backup whole VMs.  Veeam, VMware Data Protection, etc.  This is a pretty big deal!

You can hot configure FT for a VM on the fly now without disrupting the VM if it is currently running, which is yet also really cool.  Maybe you got a MS two node cluster, and one gets corrupted.  Enable FT on the remaining one to provide extra protection until the second node is rebuilt!

Also, the architecture changed.  This is good and bad.  In the past, FT required all the VMs disks to be on shared storage, and the active and passive VMs used the same Virtual disk files, VM Config files, etc.  This is no longer the case.  Now the storage is replicated as well, and it can be to the same Datastore or different datastores.   Those datastores can be on completely different storage arrays if you want.  On the downside, you need twice the storage for FT protected VMs than you did before, but the good news is a storage failure may not take out both data sets and kill the VM, too!

In my opinion, these changes have finally made FT definitely something that should be considered and will be implemented far more commonly.

So while a lot of the restrictions were lifted, there are still some left, notably:

  • Limit of 4 vCPUs, 64GBs of RAM for a FT protected VM.
  • Far more hardware is supported, but you still need hardware that is officially supported.
  • VM and the FT copy MUST be stored on VMFS volumes.  No NFS, VSAN, or VVOL stored VMs!
  • You cannot replicate the VM using vSphere Replication for a DR solution.
  • No storage DRS support for FT protected VMs
  • 10gb networking is highly recommended.  This is the first resource that runs out when protecting VMs with FT.  So if you were thinking FT with the storage replication would be a good DR solution across sites, uhh, no.
  • Only 4 FT active or passive copies per host.

So, if you’re thinking about a vSphere solution for a customer, and you pretty much dismissed FT, consider it now.  And if you support environments with VMware, get ready to see more FT as vSphere 6 gets adopted!

vSphere 6 NFS 4.1 Gotchas

vSphere,

VMware has added some additional NFS features to v6.  I knew it supported NFS 4.1 as well as 3, but there are some significant ramifications related to this and some gotchas.

  1. vSphere does not support parallel NFS (pNFS)! 
  2. It DOES support multipathing if connecting with NFS 4.1.  What you do is add the multiple NFS target IPs when setting up your NFS mount.
  3. IMPORTANT: This is the biggest gotcha, and something we all need to be aware of.  If your system supports both 4.1 and 3 simultaneously for a mount, you MUST use one or the other for an export used by VMware!  V3 and v4.1 use different locking mechanisms.  Having both enabled for an mount simultaneously and then having different ESXi hosts mounting it with differing versions can corrupt data!!!  Best practice is enable only one of the two protocols for that export, never both.
  4. You can authenticate using Kerberos, which is more secure.

NFS 4.1 support though isn’t all roses.  Here’s what you can’t do…

  • No Storage DRS
  • No SIOC
  • No SRM
  • No vVol
  • No VAAI (bigger deal now that Standard licensing includes VAAI)
  • Must run vSphere Replication 6.1 or higher to use VR to replicate VMs on NFS 4.1 mounts.

vSphere 6 – Certificate Management – Part 2

howto, vSphere, ,

Previously, I posted about certificate management in vSphere 6, which has simplified the process and provided several ways to trust certificates that will be used, while providing flexibility about what will issue the certificates to vSphere internal functionality and client facing functionality as well.

One of the simplest means to allow your clients to trust SSL certificates used by vSphere 6 is to utilize the CA functionality built into vCenter 6, and configure your clients to trust it as a Root CA, which is the focus of this blog post.

Basically, to do this, you need to obtain the root certificates of your vCenter servers, and the install them into your clients Trusted Root CA stores.  Thankfully, this is a relatively straightforward and easy process.

Obtaining Root CA files for vCenter

Obtaining the root CA files for vCenter is very easy.  To do this, simply connect to the FQDN of your vCenter server using HTTPS.  Do not add anything after this, like you may to connect to the vSphere Web Client.  For example, you would use:

https://vcentersvrname.domain.com

Once connected, you will see a link to obtain the vCenter certificates, as you can see below:

Download vCenter root CA certsWhen you download this file, it will be a zip file containing two certificate files for your vCenter server.  If your vCenter server is part of a linked mode installation, you will have two certificate files for every vCenter in the linked mode instance.  The files with an .0 extension are the root CA files you need to import.  Below, you can see the zip file downloaded from a vCenter in a two vCenter server linked mode installation.

vcenterdownloadfile
Extract the contents of the zip file.  Next, rename the .0 files with a .cer extension.  This will allow the files to be easily installed within a Windows machine.  You can then open them to check out the properties of the files if you like.

Installing Root CA file(s)

If you’re familiar with Windows machines, this is pretty straight forward.  You would either import this file into each machine manually, or you can use a Group Policy Object, import the file(s) into it, and refresh your GPO.

That’s it!  Pretty easy to do!  At the very least, you should do this instead of blindly allowing the exception for the untrusted certificate every time, because we all know we aren’t checking the thumbprints of those certs to ensure we’re connecting into the same server, plus this removes the warnings if you’re not creating a permanent exception.