This is the final part of my series on how to deal with compromised vSphere 6.0 certificates. If you are coming here first, I highly recommend reading:
Compromised vSphere 6.0 Certificates – Part 1
Compromised vSphere 6.0 Certificates – Part 2
We pick up where we left off in Part 2. The scenario is there are suspected compromised vSphere 6.0 certificates in your environment that were provided to vCenter issued either via a root certificate within the Platform Services Controller (PSC), or the PSC generated certificates with an installed intermediate certificate from an external Public Key Infrastructure (PKI). Again, VMware does not support certificate revocation when its PSC automatically generated the cert using either its own root or via the external PKI’s issued intermediate. You must then regenerate all certs.
At this point, I assume certificates were regenerated for the PSC’s root or the intermediate certificate, along with all vCenter server certificates. I outlined that process in Part 2. Now, the question is what to do about everything else when dealing with compromised vSPhere 6.0 certificates? What about ESXi servers? What about external products that plug into vCenter like VMware Update Manager? NSX Manager? vRealize Operations Manager?
Let’s get to it!
Compromised vSphere 6.0 Certificates – ESXi servers
After resetting all the certificates within the PSC and vCenter, I have good news when it comes to your ESXi servers. They won’t have any problems. Resetting VCSA certificates has nothing to do with ESXi servers because they do not obtain certificates from VCSA, nor do ESXi servers have any trusts of the certificates that were reset.
Yay!
Compromised vSphere 6.0 Certificates – Most external vCenter dependent solutions from VMware
In most cases, with it comes to external vCenter products that establish relationships with vCenter, these products do often establish a trust of one of the certificates that were reset. However, they do not obtain certificates from a PSC themselves. Therefore, you need to fix these products by simply establishing trust with the new certificate that is now installed with vCenter. And in most cases, this is as easy as it was when you registered it with vCenter in the first place.
I’m not going to show step by step of every product, as I don’t have the time. I will however come back and update this post if/when I need to do this with various products. I am going to use vRealize Operations Manager as an example of a typical product that is fixed the same basic way.
vRealize Operations Manager
Here is what vCOPS looks like following the PSC/vCenter certificate resets:
Note we are checking the same place you go to register the product with vCenter in the first place (Appliance portal > Solutions > VMware vSphere). If we were talking about NSX, you would login to the NSX Manager’s direct portal and navigate to Manage Appliance Settings > NSX Management Service > Configure. This is the same basic concept, even though where you navigate might be different on the product.
Go into settings and re-establish the connection. In vCOPS, that means clicking the gear on that page, and on both the vCenter Adapter and vCenter Python Actions Adapter, click “Test Connection”. Low and behold, a pop up comes up to ask if you wish to trust a new certificate it doesn’t yet trust.
If you click OK to trust, vCOPS adds the new certificate to its trusted store. However, you get an error that you effectively can’t trust two certificates for the same object.
I show this just in case other products share similar behavior. Delete the old trusted certificate from the appliance.
In this case, navigate to Certificates, and delete the trusted certificate.
Hovering over a column gives more specific info for that cert, which can help identify which certificate to delete.
Then, go back and issue the Test Connection command.
Stop and start the collections on the Solutions page as needed.
Click refresh and wait to ensure “Data Receiving” is shown for the collection status. Otherwise, vCOPS is not functioning.
Other products will have their idiosyncrisies, but they have the same basic concept of establishing trust for the new vCenter certificate. You perform this process pretty much where you registered the product with vCenter in the first place.
Compromised vSphere 6.0 Certificates – Abnormal external vCenter dependent solutions from VMware
Some products need specialized procedures to trust the new certificates installed in your vCenter/PSC servers. Here are all of the ones so far I’ve run into, and how to fix those:
vCenter Update Manager (VUM)
I’m hardly shocked VUM would need a specialized procedure. VUM runs on a Windows OS only. It remains 32-bit, as opposed to almost every other VMware product. Plus, it has had esoteric procedures when it came to certificates for a long time.
Navigating to an impacted VUM server within the vSphere Client nets you a pretty immediate error that clearly shows a problem with the SSL certificate.
“sysimage.fault.SSLCertificateError”
Time to fix the trust of the new certificate!
First, remote into the VUM server.
Next, run the VMware vSphere Update Manager Utility under the installation directory for VUM (X:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe, where X is the drive in which you installed the VUM binaries). Login, and select to re-register with vCenter.
Of course, restart vSphere Update Manager service to complete the process.
The error will go away, and VUM will function again.
Summary
Hopefully, this gives everyone enough info to complete the process or point them in the right direction. If you have any insights to other products I didn’t cover, please post in the comments! As I try more products, I will also update this article.
Thanks for reading!