This is the final part of my series on how to deal with compromised vSphere 6.0 certificates.  If you are coming here first, I highly recommend reading:

Compromised vSphere 6.0 Certificates – Part 1

Compromised vSphere 6.0 Certificates – Part 2

We pick up where we left off in Part 2.  The scenario is there are suspected compromised vSphere 6.0 certificates in your environment that were provided to vCenter issued either via a root certificate within the Platform Services Controller (PSC), or the PSC generated certificates with an installed intermediate certificate from an external Public Key Infrastructure (PKI).  Again, VMware does not support certificate revocation when its PSC automatically generated the cert using either its own root or via the external PKI’s issued intermediate.  You must then regenerate all certs.

At this point, I assume certificates were regenerated for the PSC’s root or the intermediate certificate, along with all vCenter server certificates.  I outlined that process in Part 2.  Now, the question is what to do about everything else when dealing with compromised vSPhere 6.0 certificates?  What about ESXi servers?  What about external products that plug into vCenter like VMware Update Manager?  NSX Manager?  vRealize Operations Manager?

Let’s get to it!

Compromised vSphere 6.0 Certificates – ESXi servers

After resetting all the certificates within the PSC and vCenter, I have good news when it comes to your ESXi servers.  They won’t have any problems.  Resetting VCSA certificates has nothing to do with ESXi servers because they do not obtain certificates from VCSA, nor do ESXi servers have any trusts of the certificates that were reset.

Yay!

Compromised vSphere 6.0 Certificates – Most external vCenter dependent solutions from VMware

In most cases, with it comes to external vCenter products that establish relationships with vCenter, these products do often establish a trust of one of the certificates that were reset.  However, they do not obtain certificates from a PSC themselves.  Therefore, you need to fix these products by simply establishing trust with the new certificate that is now installed with vCenter.   And in most cases, this is as easy as it was when you registered it with vCenter in the first place.

I’m not going to show step by step of every product, as I don’t have the time.  I will however come back and update this post if/when I need to do this with various products.  I am going to use vRealize Operations Manager as an example of a typical product that is fixed the same basic way.

vRealize Operations Manager

Here is what vCOPS looks like following the PSC/vCenter certificate resets:

Note we are checking the same place you go to register the product with vCenter in the first place (Appliance portal > Solutions > VMware vSphere).  If we were talking about NSX, you would login to the NSX Manager’s direct portal and navigate to Manage Appliance Settings > NSX Management Service > Configure.  This is the same basic concept, even though where you navigate might be different on the product.

Go into settings and re-establish the connection.  In vCOPS, that means clicking the gear on that page, and on both the vCenter Adapter and vCenter Python Actions Adapter, click “Test Connection”.  Low and behold, a pop up comes up to ask if you wish to trust a new certificate it doesn’t yet trust.

If you click OK to trust, vCOPS adds the new certificate to its trusted store.  However, you get an error that you effectively can’t trust two certificates for the same object.

I show this just in case other products share similar behavior.  Delete the old trusted certificate from the appliance.

In this case, navigate to Certificates, and delete the trusted certificate.

Hovering over a column gives more specific info for that cert, which can help identify which certificate to delete.

Then, go back and issue the Test Connection command.

Stop and start the collections on the Solutions page as needed.

Click refresh and wait to ensure  “Data Receiving” is shown for the collection status.  Otherwise, vCOPS is not functioning.

Other products will have their idiosyncrisies, but they have the same basic concept of establishing trust for the new vCenter certificate.  You perform this process pretty much where you registered the product with vCenter in the first place.

Compromised vSphere 6.0 Certificates – Abnormal external vCenter dependent solutions from VMware

Some products need specialized procedures to trust the new certificates installed in your vCenter/PSC servers.  Here are all of the ones so far I’ve run into, and how to fix those:

vCenter Update Manager (VUM)

I’m hardly shocked VUM would need a specialized procedure.  VUM runs on a Windows OS only.  It remains 32-bit, as opposed to almost every other VMware product.  Plus, it has had esoteric procedures when it came to certificates for a long time.

Navigating to an impacted VUM server within the vSphere Client nets you a pretty immediate error that clearly shows a problem with the SSL certificate.

“sysimage.fault.SSLCertificateError”

Time to fix the trust of the new certificate!

First, remote into the VUM server.

Next, run the VMware vSphere Update Manager Utility under the installation directory for VUM (X:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe, where X is the drive in which you installed the VUM binaries).  Login, and select to re-register with vCenter.

Of course, restart vSphere Update Manager service to complete the process.

The error will go away, and VUM will function again.

Summary

Hopefully, this gives everyone enough info to complete the process or point them in the right direction.  If you have any insights to other products I didn’t cover, please post in the comments!  As I try more products, I will also update this article.

Thanks for reading!

In this second blog article, I discuss what to do with compromised vSphere 6.0 Certificates issues by a PSC to vSphere components.  As mentioned in the previous blog article, you cannot revoke certificates issued by a PSC either using an installed intermediate certificate from an external CA or using its own root.  You must regenerate all certificates instead.

FYI, this post assumes you’re using the VCSA.  Windows installable vCenter is nearly identical, aside from the path to Certificate Manager.

Compromised vSphere 6.0 Certificates – Embedded PSC With Own Root Certificate

If you have compromised vSphere 6.0 certificates automatically generated from an embedded PSC, you must regenerate all certificates.  Yes, you must regenerate even certificates you don’t suspect, too.

To do this:

1. Login as root into your embedded vCenter server via console, SSH, etc.
2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
   1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “VC-ROOTCFG”.  Answering every other question identically won’t hurt.
   2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for the vCenter server.
6. When prompted afterregenerating all certificates, stop and start all services using:
   1. service-control –stop –all
   2. service-control –start –all
7. I would recommend rebooting your vCenter server now.
8. Download your root certificate again and reimport into GPO or however you established trust on the clients for the root originally.
9. Fix all trust issues with external products.  (See part 3 of this series!)

This is probably the one time you might actually want an embedded PSC for vCenter.  This is far simpler than if you have an external PSC.  (I still recommend external PSC’s in all cases for the record!!!)

Compromised vSphere 6.0 Certificates – External PSC(s) With Own Root Certificate

This is somewhat similar.  However, keep in mind each PSC is a CA.  Therefore, you probably should do this on every PSC that’s a part of the same environment if you suspect certificate(s) have been compromised.

To do this:

1. Login as root into your external PSC server via console or SSH.
2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
   1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “PSC1-ROOTCFG”.  Answering every other question identically won’t hurt.
   2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for this server.  Even if it asks for a cert for the web client, do NOT put in the name of the vCenter server.  It will also ask for an optional IP address.  Obviously, if you input one, make sure it’s the correct one.
6. When prompted after regenerating all certificates, stop and start all services using:
   1. service-control –stop –all
   2. service-control –start –all
7. I recommend rebooting the machine when you’ve completed this.
8. To verify the PSC cert reset worked, attempt to go to https://FQDNofPSC.domain.com/psc to ensure you get a login prompt.  If you don’t, the certificate reset failed.  Stop and redo this portion again.  You likely didn’t provide some kind of different answer to one of the questions for each certificate to make them unique.
9. Run Certificate Manager on your vCenter server(s).  Here’s where it gets weird.  VMware says you should run Option 3 – Replace Machine SSL certificater with VMCA Certificate and answer the questions.  Next, run Option 6 – Replace Solution user certificates with VMCA certificates.  That didn’t work for me.  The only way I could get it to work is run Option 8 – Reset all certificates.  That’s the only way I could get it to work.  I found another oddity.  During this process, you are asked: “Performing operation on distributed setup, Please provide valid Infrastructure Server IP.”  If I entered an IP address, and did the rest correctly (remember to answer the questions but provide a different value for name for each certificate!), the process would kick off, get stuck at a long time here and eventually fail:Status : 85% Completed [starting services…]
   Error while starting services, please see log for more details
   Status : 0% Completed [Operation failed, performing automatic rollback]

   Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

   Then the certificates would roll back.  Enter the FQDN of one of your PSC servers instead!  That allows it to continue.

10. Download your root certificate again and re-import into GPO or however you established trust on the clients for the root originally.
11. Fix all trust issues with external products.  (See part 3 of this series!)

This is far more complicated than the first one, but it’s probably the one you’re more likely to need to do.

Compromised vSphere 6.0 Certificates – Intermediate CA

If you installed a now compromised intermediate CA certificate, revoke the intermediate certificate within the external PKI.  You should then request and install a new intermediate certificate within the PSC.  Then proceed with regenerating certificates for all other components. (See above…)

And that’s how you deal with compromised vSphere 6.0 Certificates.  In part 3, I’ll delve into how to fix trust issues with various products that might arise from regenerating these certificates.

As I alluded to in a previous post, I’ve been needing to do some more in depth testing in relation to vSphere 6.0, which I run in VMware Workstation.  Now the cat is out of the bag!  I’m running through scenarios about what with compromised vSphere 6.0 certificates.

After scouring the internet, there are plenty of blog articles about how to go with various certificate management models.  There’s not a lot of information what to do if suspect a compromised vSphere 6.0 certificate.

I wanted to cover the basics here discuss more specifics in later articles.  I’m going to start with a short introduction to vSphere 6.0 Certificate Management, and then the implications of each when it comes to a compromised certificate.

vSphere 6.0 Certificate Management Basics

I’ve posted about this in the past.  Here’s a VERY quick recap.

1. You can have the Platform Services Controller (PSC) act as a root Certificate Authority (CA), and hand out certificates automatically to other vSphere components automatically, which is the easiest to implement and manage.
2. You can have the PSC act as an intermediate CA, and issue certificates using the intermediate certificate you install automatically.  This is arguably the second easiest to implement and manage.
3. You can generate Certificate Service Requests to an external CA manually for various vSphere components, and install those certificates manually.  This is arguably the hardest to implement and manage.

One other note here is you can mix and match these options.  This is usually implemented by having non-client internal vSphere component certificates be issued by the PSC, and client facing certs such as the cert for the vSphere Web Client be issued by an external CA.

Again, the above information is not intended to be a primer for certificate management in vSphere 6.0.  It’s only to facilitate discussion about what to do if a certificate has been compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by an External CA

One advantage of using an external certificate authority to issue the certificates for vSphere is the support for certificate revocation.  If any certificate is compromised that was issued by an external CA, you can simply within that PKI environment revoke the certificate.  Replacing compromised vSphere 6.0 certificates is done the same way the certs was acquired in the first place.

The basic steps would be:

1. Revoke the suspected compromised certificate within the PKI.
2. Go through the process of obtaining a new certificate, and install it.
3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

It’s not so straightforward if the PSC generated the certificate you suspect is compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by a PSC

For all intents and purposes, it doesn’t really matter if compromised vSphere 6.0 certificates were issued by a PSC using its own root certificate or using an installed intermediate certificate obtained from an external CA.  If the PSC in the end generated the certificate used by a vSphere component, any vSphere component, certificate revocation is not supported.

If you suspect a certificate has been compromised, you have no choice but to regenerate all certificates, even certificates that you don’t expect to be compromised.  This should certainly be considered prior to deciding upon which model to use for vSphere 6.0 Certificate Management.

The basic steps would be:

1. Run the Certificate Management Utility on the PSC in question to regenerate all certificates.  If any doubts, do this on all PSCs.
2. Run the Certificate Management Utility on any vCenter server that obtained its certificates from that PSC.  If you’re running embedded PSC with vCenter, you already did this in step one.
3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

If the above seems like it’s just as easy, it isn’t.  For one, documentation on how to do this with external PSC’s is vague and confusing from VMware.  Secondly, it gets more complicated the more PSC and vCenter nodes you have.

I’ll go in more depth on how address compromised vSphere 6.0 Certificates issued by a PSC in Part 2.  In Part 3, I’ll address how to fix trust issues with certificates in various products.