All posts by Aaron Margeson

2v0-641 exam review

VCP6-NV 2V0-641 Exam Review and Resources

Certifications, VCP6-NV,

Sorry the updates haven’t been forth coming.  That’s mainly because my two year requirement for recertification was coming up for my VCP-DCV certification, and I was preparing to meet it by achieving VCP6-NV.  Saturday, I passed, so I wanted to share my experiences and provide a 2V0-641 exam review.

2V0-641 Exam Review – Thoughts on Exam

This exam is no slouch in difficulty.  I’ve ready plenty of reviews that seem to suggest the VCP-NV exam isn’t difficult, but maybe that was the predecessor to 2V0-641.  I made my first attempt a week before, and failed it despite putting a lot of time studying, and soaking in as much as possible using the Hands on Labs from VMware.

My first score was 276, and you must score 300 to pass.  I think part of the reason why was contained in my score report, which lists objectives recommended for review if you fail.  I noticed a string of questions that didn’t pertain to what the exam blueprint described for various network fabrics.  The blueprint had stated to know the challenges of various network designs, and how NSX solved them, but these questions were more about how to configure NSX when you have various network fabric design types.  Sure enough, one objective listed that I needed to review was “Describe considerations for running VMware NSX on physical network fabrics”.  As of the time I’m drafting this blog post, that is nowhere on the exam blueprint.  Needless to say, I was extremely frustrated, with another 2 weeks to go before I lost my VCP status.

After dusting myself off, I went back to studying and gathered information for this new objective, and scored 357 on my second attempt with one more week to go before my VCP-DCV expired.  A little too close for comfort, but I got it done.

2V0-641 Exam Review – How to Prepare

I attended an online version of the 6.2 Install and Configure Course back in August.  Be very aware that this course alone is absolutely not sufficient to pass this exam.  I can’t stress that enough.  You’ll know just by looking at the exam blueprint, as entire major exam objectives are completely not covered at all.  The course covers others, but only scratched the surface.  You absolutely have to know your stuff for this exam!

I highly recommend the VMware Press Official Cert Guide for VCP-6-NV by Elver Sena Sosa.  I had the pleasure of meeting him at VMworld 2016 this year in Las Vegas.  It’s one of the best written IT books I’ve ever read, and comes with simulation exams. However, it too is also not sufficient for preparing for this exam, either.  But it helped me understand how NSX works that I didn’t even realize I was short on despite attending the course.  I found the packet walkthroughs for both the logical switch and logical routers extremely helpful!  However, the book also doesn’t cover some sections for this exam in the blueprint.  The simulation questions also do not reflect the difficulty or the style of questions I found on both attempts on the exam well.  I scored consistently almost perfectly on these simulation exams on the first attempt prior to taking the actual exam, and yet failed anyway.  Still, this is in my opinion the most helpful resource for this exam.  In fact, I’m quite certain I’ll reference this book in future NSX deployments.

Finally, I really wanted to give a shot out to another blogger, Rich Dowling, who helped me immensely with his notes on various portions on the exam objectives listed in the blueprint.  While it’s not for the current exam, most of the exam objectives are the same. Make sure you check them out if you’re taking this exam!

Also, make sure you read carefully through the recommended documents within the exam blueprint.  This is especially the case for NSX Design Guide!  (see surprise objective!)

2V0-641 Exam Review – Resources

Here are some resources to help with your 2V0-641 exam review.

VMware Certified Professional 6 – Network Virtualization Exam page – Contains the exam blueprint

VMware NSX Network Virtualization Design Guide

vSphere Networking Guide

NSX for vSphere Administration Guide

NSX Command Line Interface Reference Guide

VCP6-NV Official Cert Guide (Exam #2V0-641)  (Kindle Version)

2V0-641 Exam Review – Check back for more

I’ll be posting some section reviews where I think my notes would be helpful for others beyond Rich Dowling’s excellent notes to try to help others as Rich helped me.  I hope you’ll find those useful!

Compromised vSphere 6.0 Certificates – Part 3

vCenter, vSphere, ,

This is the final part of my series on how to deal with compromised vSphere 6.0 certificates.  If you are coming here first, I highly recommend reading:

Compromised vSphere 6.0 Certificates – Part 1

Compromised vSphere 6.0 Certificates – Part 2

We pick up where we left off in Part 2.  The scenario is there are suspected compromised vSphere 6.0 certificates in your environment that were provided to vCenter issued either via a root certificate within the Platform Services Controller (PSC), or the PSC generated certificates with an installed intermediate certificate from an external Public Key Infrastructure (PKI).  Again, VMware does not support certificate revocation when its PSC automatically generated the cert using either its own root or via the external PKI’s issued intermediate.  You must then regenerate all certs.

At this point, I assume certificates were regenerated for the PSC’s root or the intermediate certificate, along with all vCenter server certificates.  I outlined that process in Part 2.  Now, the question is what to do about everything else when dealing with compromised vSPhere 6.0 certificates?  What about ESXi servers?  What about external products that plug into vCenter like VMware Update Manager?  NSX Manager?  vRealize Operations Manager?

Let’s get to it!

Compromised vSphere 6.0 Certificates – ESXi servers

After resetting all the certificates within the PSC and vCenter, I have good news when it comes to your ESXi servers.  They won’t have any problems.  Resetting VCSA certificates has nothing to do with ESXi servers because they do not obtain certificates from VCSA, nor do ESXi servers have any trusts of the certificates that were reset.

Yay!

Compromised vSphere 6.0 Certificates – Most external vCenter dependent solutions from VMware

In most cases, with it comes to external vCenter products that establish relationships with vCenter, these products do often establish a trust of one of the certificates that were reset.  However, they do not obtain certificates from a PSC themselves.  Therefore, you need to fix these products by simply establishing trust with the new certificate that is now installed with vCenter.   And in most cases, this is as easy as it was when you registered it with vCenter in the first place.

I’m not going to show step by step of every product, as I don’t have the time.  I will however come back and update this post if/when I need to do this with various products.  I am going to use vRealize Operations Manager as an example of a typical product that is fixed the same basic way.

vRealize Operations Manager

Here is what vCOPS looks like following the PSC/vCenter certificate resets:

compromised vsphere 6.0 certificates vcops statusNote we are checking the same place you go to register the product with vCenter in the first place (Appliance portal > Solutions > VMware vSphere).  If we were talking about NSX, you would login to the NSX Manager’s direct portal and navigate to Manage Appliance Settings > NSX Management Service > Configure.  This is the same basic concept, even though where you navigate might be different on the product.

Go into settings and re-establish the connection.  In vCOPS, that means clicking the gear on that page, and on both the vCenter Adapter and vCenter Python Actions Adapter, click “Test Connection”.  Low and behold, a pop up comes up to ask if you wish to trust a new certificate it doesn’t yet trust.

compromised vSphere 6.0 certificates trust new certs

If you click OK to trust, vCOPS adds the new certificate to its trusted store.  However, you get an error that you effectively can’t trust two certificates for the same object.

compromised vSphere 6.0 certificates vCOPS trust new cert error

I show this just in case other products share similar behavior.  Delete the old trusted certificate from the appliance.

In this case, navigate to Certificates, and delete the trusted certificate.

compromised vSphere 6.0 certificates vCOPS delete old untrusted cert

Hovering over a column gives more specific info for that cert, which can help identify which certificate to delete.

Then, go back and issue the Test Connection command.

Stop and start the collections on the Solutions page as needed.

compromised vSphere 6.0 certificates vCOPS data receiving

Click refresh and wait to ensure  “Data Receiving” is shown for the collection status.  Otherwise, vCOPS is not functioning.

Other products will have their idiosyncrisies, but they have the same basic concept of establishing trust for the new vCenter certificate.  You perform this process pretty much where you registered the product with vCenter in the first place.

Compromised vSphere 6.0 Certificates – Abnormal external vCenter dependent solutions from VMware

Some products need specialized procedures to trust the new certificates installed in your vCenter/PSC servers.  Here are all of the ones so far I’ve run into, and how to fix those:

vCenter Update Manager (VUM)

I’m hardly shocked VUM would need a specialized procedure.  VUM runs on a Windows OS only.  It remains 32-bit, as opposed to almost every other VMware product.  Plus, it has had esoteric procedures when it came to certificates for a long time.

Navigating to an impacted VUM server within the vSphere Client nets you a pretty immediate error that clearly shows a problem with the SSL certificate.

compromised vSphere 6.0 certificates VUM SSL error

“sysimage.fault.SSLCertificateError”

Time to fix the trust of the new certificate!

First, remote into the VUM server.

Next, run the VMware vSphere Update Manager Utility under the installation directory for VUM (X:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe, where X is the drive in which you installed the VUM binaries).  Login, and select to re-register with vCenter.

compromised vSphere 6.0 certificates reregister VUM

Of course, restart vSphere Update Manager service to complete the process.

The error will go away, and VUM will function again.

Summary

Hopefully, this gives everyone enough info to complete the process or point them in the right direction.  If you have any insights to other products I didn’t cover, please post in the comments!  As I try more products, I will also update this article.

Thanks for reading!

vCenter to VCSA migration tool released

vCenter, vSphere,

It’s out!  You can finally migrate a Windows vCenter install to a vCenter appliance VCSA based install using a supported VCSA migration tool!

https://blogs.vmware.com/vsphere/2016/09/vcenter-server-migration-tool-vsphere-6-0-update-2m.html

Just a heads up, here are the requirements:

  • It only migrates from Windows installations to VCSA based installations
  • It only migrates from 5.5 (any revision) to VCSA 6.0 Update 2
  • You must used the embedded database in VCSA
  • It will not migrate VUM if VUM is installed on the same OS as your vCenter (no big deal IMO, not hard to reinstall VUM)

Cool things I noticed about this migration utility according to the link above:

  • Preservation of original vCenter settings including
    • IP address
    • FQDN
    • Certificates
    • Alarm settings (AWESOME!)
  • It doesn’t touch your current vCenter’s stuff, so easy rollback.

Not so cool things:

  • You can’t change deployment models during the migration.  A simple deployment in 5.5 becomes an embedded deployment in VCSA, which you don’t want to do, but you can always transition out of that.

I’m sure you’ll be seeing a blog article here kicking the tires on this thing.  But this will definitely help me convince people to go to the VCSA.

unity

How To Manage Unity Host LUN IDs

Unity

EMC’s new Unity arrays have been out for a bit now.  Every array has its weirdness to it.  I just found my first Unity oddity today.  Remember, the Unity replaces the VNX and the VNXe.  If you’ve never played with a VNXe, EMC simplified the Unisphere interface, and removed and/or hid many options in the process.  With that in mind, the Unity does this, too.  You will find this perhaps first managing Unity Host LUN IDs.

Why might you want to control the host LUN IDs?

  • Consistent LUN Masking for easier troubleshooting.
  • Storage based replication like RecoverPoint best practices want consistent Host LUN IDs for the replicated LUNs for both host access and RPAs.
  • Boot from SAN LUNs are either required to be a specific Host LUN ID such as LUN 0, or you may be running Cisco UCS and need to specificy a specific Host LUN ID for the boot LUN in the boot policy.

In this case, the customer decided to reinstall ESXi on a new boot LUN.  They created a new boot LUN, granted the correct ESXi host access, and then deleted the old one.

Provisioning LUNs

Provisioning LUNs on an EMC Unity array is easy, at least in most cases.  EMC streamlined the interface on these arrays.  You simply go to Storage > the subcategory Block or VMware if it’s for ESXi, and step through the wizard to set your options and grant host access.

unitycreatelun

When you get to the point where you grant access, you simply put checkmarks in the boxes for each host to which you wish to grant access.

unitygrantlunaccess

Do you see anything missing, even if you click that gear and add columns?  If you guessed there’s no way to control what host LUN ID will be used in the LUN masking, go pat yourself on the back.

Fortunately, the Unity does provide a way to set the Host LUN ID once a LUN is created… in most cases…

Managing Unity Host LUN IDs for LUN Masking

Here’s how Unity Host LUN IDs work.  Unity automatically picks a host LUN ID for the host, picking the next available host LUN ID for that host.  If you wish to change the host LUN ID for a LUN mask, simply navigate to Access > Hosts, click on the host you wish the change the Host LUN ID for, the pencil icon to edit.  Next, click LUNs.  Finally, click “Modify LUN IDs”.

change unity host lun id

Easy right?  As long as your host isn’t identified as a VMware host, you’re good!

Managing Unity Host LUN IDs for LUN Masking on VMware Hosts

If you follow those directions but your host is identified as a VMware host within Unity, you’re in for a nasty surprise.  Let’s play the game, “what’s missing?”

unity host lun id missing

If you guessed “no way to change Host LUN IDs”, you’re correct again.  YOU’RE ON FIRE!!!  That’s not all, though.  There’s no LUN access management here at all!  That includes Host LUN IDs.

Don’t bother going into the initiators either.  It’s not there.  It’s not under the LUN in question, either; although, that’s how to manage LUN access in general for VMware hosts.  Other than command line, there’s nothing you can do with identified VMware hosts.

Solution?  Don’t make it an identified VMware host.  Simply remove the vCenter server from Unity.  That places all ESXi servers discovered from this vCenter into the general Hosts group, giving you back the ability to change the Host LUN IDs.

unity vcenter server removed esxi hosts

Now, you can manage the Host LUN IDs again!

Compromised vSphere 6.0 Certificates – Part 2

vCenter, vSphere, ,

In this second blog article, I discuss what to do with compromised vSphere 6.0 Certificates issues by a PSC to vSphere components.  As mentioned in the previous blog article, you cannot revoke certificates issued by a PSC either using an installed intermediate certificate from an external CA or using its own root.  You must regenerate all certificates instead.

FYI, this post assumes you’re using the VCSA.  Windows installable vCenter is nearly identical, aside from the path to Certificate Manager.

Compromised vSphere 6.0 Certificates – Embedded PSC With Own Root Certificate

If you have compromised vSphere 6.0 certificates automatically generated from an embedded PSC, you must regenerate all certificates.  Yes, you must regenerate even certificates you don’t suspect, too.

To do this:

  1. Login as root into your embedded vCenter server via console, SSH, etc.
  2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
  3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
  4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
  5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
    1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “VC-ROOTCFG”.  Answering every other question identically won’t hurt.
    2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for the vCenter server.
  6. When prompted afterregenerating all certificates, stop and start all services using:
    1. service-control –stop –all
    2. service-control –start –all
  7. I would recommend rebooting your vCenter server now.
  8. Download your root certificate again and reimport into GPO or however you established trust on the clients for the root originally.
  9. Fix all trust issues with external products.  (See part 3 of this series!)

This is probably the one time you might actually want an embedded PSC for vCenter.  This is far simpler than if you have an external PSC.  (I still recommend external PSC’s in all cases for the record!!!)

Compromised vSphere 6.0 Certificates – External PSC(s) With Own Root Certificate

This is somewhat similar.  However, keep in mind each PSC is a CA.  Therefore, you probably should do this on every PSC that’s a part of the same environment if you suspect certificate(s) have been compromised.

To do this:

  1. Login as root into your external PSC server via console or SSH.
  2. Enter into shell.  If you didn’t enable shell via the console, you can run “shell.set –enable True” and then run “shell”.
  3. Run the certificate manager utility.  For the VCSA, you simply run /usr/lib/vmware-vmca/bin/certificate-manager
  4. Select option 4 – Regenerate a new VMCA Root Certificate and replace all certificates.
  5. Certificate Manager asks for various pieces of information for each certificate regeneration such as the country, organization, OrgUnit, State, Locality, email, etc. These are cosmetic values mostly, and are only visible if someone really examines the certificate.  Functionally, they make no difference.  However, I wanted to call your attention to a couple of things that are very important. It is VERY CRITICAL you do the following for each certificate, or else the process will fail!
    1. There is a bug in the certificate automation tool, where if you answer identical values for all questions asked, the same certificate will be generated for that cert.  You’ll notice there are multiple certs that end up being regenerated.  You can tell which one is being regenerated with the following line: “Please configure root.cfg with proper values before proceeding to next step.”  That means the root certificate is being regenerated.  You’ll see various certs as well like “machine”, “machine-ssl”, “vpxd.cfg”, etc.   Each one of these certs must actually be unique.  Ensure that you give at least some different value for one of the questions asked for every cert regenerated for a server.  By far, the easiest way to do this is to answer the following question uniquely for every cert: “Enter proper value for ‘Name’ [Default value : CA]”  Simply name it an abbreviated name of the server and the certificate name.  In this case, you could call it “PSC1-ROOTCFG”.  Answering every other question identically won’t hurt.
    2. One question that is more than cosmetic that you must answer correctly is: “Enter proper value for ‘Hostname’ [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com]”.  Make sure this is the actual DNS name for this server.  Even if it asks for a cert for the web client, do NOT put in the name of the vCenter server.  It will also ask for an optional IP address.  Obviously, if you input one, make sure it’s the correct one.
  6. When prompted after regenerating all certificates, stop and start all services using:
    1. service-control –stop –all
    2. service-control –start –all
  7. I recommend rebooting the machine when you’ve completed this.
  8. To verify the PSC cert reset worked, attempt to go to https://FQDNofPSC.domain.com/psc to ensure you get a login prompt.  If you don’t, the certificate reset failed.  Stop and redo this portion again.  You likely didn’t provide some kind of different answer to one of the questions for each certificate to make them unique.
  9. Run Certificate Manager on your vCenter server(s).  Here’s where it gets weird.  VMware says you should run Option 3 – Replace Machine SSL certificater with VMCA Certificate and answer the questions.  Next, run Option 6 – Replace Solution user certificates with VMCA certificates.  That didn’t work for me.  The only way I could get it to work is run Option 8 – Reset all certificates.  That’s the only way I could get it to work.  I found another oddity.  During this process, you are asked: “Performing operation on distributed setup, Please provide valid Infrastructure Server IP.”  If I entered an IP address, and did the rest correctly (remember to answer the questions but provide a different value for name for each certificate!), the process would kick off, get stuck at a long time here and eventually fail:Status : 85% Completed [starting services…]
    Error while starting services, please see log for more details
    Status : 0% Completed [Operation failed, performing automatic rollback]

    Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

    Then the certificates would roll back.  Enter the FQDN of one of your PSC servers instead!  That allows it to continue.

  10. Download your root certificate again and re-import into GPO or however you established trust on the clients for the root originally.
  11. Fix all trust issues with external products.  (See part 3 of this series!)

This is far more complicated than the first one, but it’s probably the one you’re more likely to need to do.

Compromised vSphere 6.0 Certificates – Intermediate CA

If you installed a now compromised intermediate CA certificate, revoke the intermediate certificate within the external PKI.  You should then request and install a new intermediate certificate within the PSC.  Then proceed with regenerating certificates for all other components. (See above…)

And that’s how you deal with compromised vSphere 6.0 Certificates.  In part 3, I’ll delve into how to fix trust issues with various products that might arise from regenerating these certificates.

Compromised vSphere 6.0 Certificates – Part 1

vCenter, vSphere, ,

As I alluded to in a previous post, I’ve been needing to do some more in depth testing in relation to vSphere 6.0, which I run in VMware Workstation.  Now the cat is out of the bag!  I’m running through scenarios about what with compromised vSphere 6.0 certificates.

After scouring the internet, there are plenty of blog articles about how to go with various certificate management models.  There’s not a lot of information what to do if suspect a compromised vSphere 6.0 certificate.

I wanted to cover the basics here discuss more specifics in later articles.  I’m going to start with a short introduction to vSphere 6.0 Certificate Management, and then the implications of each when it comes to a compromised certificate.

vSphere 6.0 Certificate Management Basics

I’ve posted about this in the past.  Here’s a VERY quick recap.

  1. You can have the Platform Services Controller (PSC) act as a root Certificate Authority (CA), and hand out certificates automatically to other vSphere components automatically, which is the easiest to implement and manage.
  2. You can have the PSC act as an intermediate CA, and issue certificates using the intermediate certificate you install automatically.  This is arguably the second easiest to implement and manage.
  3. You can generate Certificate Service Requests to an external CA manually for various vSphere components, and install those certificates manually.  This is arguably the hardest to implement and manage.

One other note here is you can mix and match these options.  This is usually implemented by having non-client internal vSphere component certificates be issued by the PSC, and client facing certs such as the cert for the vSphere Web Client be issued by an external CA.

Again, the above information is not intended to be a primer for certificate management in vSphere 6.0.  It’s only to facilitate discussion about what to do if a certificate has been compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by an External CA

One advantage of using an external certificate authority to issue the certificates for vSphere is the support for certificate revocation.  If any certificate is compromised that was issued by an external CA, you can simply within that PKI environment revoke the certificate.  Replacing compromised vSphere 6.0 certificates is done the same way the certs was acquired in the first place.

The basic steps would be:

  1. Revoke the suspected compromised certificate within the PKI.
  2. Go through the process of obtaining a new certificate, and install it.
  3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

It’s not so straightforward if the PSC generated the certificate you suspect is compromised.

Dealing with Compromised vSphere 6.0 Certificates Issued by a PSC

For all intents and purposes, it doesn’t really matter if compromised vSphere 6.0 certificates were issued by a PSC using its own root certificate or using an installed intermediate certificate obtained from an external CA.  If the PSC in the end generated the certificate used by a vSphere component, any vSphere component, certificate revocation is not supported.

If you suspect a certificate has been compromised, you have no choice but to regenerate all certificates, even certificates that you don’t expect to be compromised.  This should certainly be considered prior to deciding upon which model to use for vSphere 6.0 Certificate Management.

The basic steps would be:

  1. Run the Certificate Management Utility on the PSC in question to regenerate all certificates.  If any doubts, do this on all PSCs.
  2. Run the Certificate Management Utility on any vCenter server that obtained its certificates from that PSC.  If you’re running embedded PSC with vCenter, you already did this in step one.
  3. Fix any trust issues that may occur with the new certificate.  For example, you must  manually fix VUM when you change a vCenter certificate.

If the above seems like it’s just as easy, it isn’t.  For one, documentation on how to do this with external PSC’s is vague and confusing from VMware.  Secondly, it gets more complicated the more PSC and vCenter nodes you have.

I’ll go in more depth on how address compromised vSphere 6.0 Certificates issued by a PSC in Part 2.  In Part 3, I’ll address how to fix trust issues with certificates in various products.

Cisco UCS Manager Firmware Upgrade Procedure

Cisco UCS

I’ve been involved in many a Cisco UCS Manager Firmware Upgrade.   Cisco’s documentation if you don’t find the exact right page is confusing.  If you don’t elect to do an automatic installation, you need to do the components in the proper order.  Personally, I’ve had issues doing it with the automatic deployment.  I do it manually.  If you want to do it manually as well, here’s the correct order and some caveats.

Cisco UCS Manager Firmware Upgrade Procedure

  1. Verify the proper firmware, drivers, etc. for your hardware and the OS (whether it’s Windows, VMware, etc.) your servers run.
  2. Upload the Infrastructure Bundle into UCSM, so it’s ready to deploy.
  3. Determine the primary and subordinate Fabric Interconnect.  I like to SSH into UCSM, and run the following to give me that info plus general cluster health status before proceeding: 
    show cluster extended-state
  4. Go to Admin > Communication Management > Call Home, and turn off call home.  You don’t want Cisco calling you thinking UCSM is on fire, when you’re doing an upgrade, right?
  5. Check alerts and verify the system is healthy before proceeding.  Fix anything that’s potentially a problem.
  6. Take a backup of your UCS configuration.
  7. Activate the new version of UCS Manager.  Verify it completes, and no unexpected errors result.  It is possible that sometimes errors are expected, and it’s OK to proceed.  Here’s an example.  Look them up!
  8. Update the firmware on the IO Modules by going to Equipment > Chassis > Chassis number > IO Modules > IO Module you want to upgrade > General > Update firmware.  Repeat on the second IO Module. You can track the progress on the Update Status portion of the general page.
  9. Activate the firmware on the IO Modules by going to Equipment > Chassis > Chassis number > IO Modules > IO Module you want to upgrade > General > Activate firmware.  Clear the checkbox for “Set Startup Version Only” to have the code change take effect immediately.  If you leave this option enabled, you’ll need to reboot the IOM Module yourself.  I recommend clearing it, and let UCSM reboot it for you. You may also receive the error: “Failed start activation.  Manual upgrade/activation is disallowed because the Default Infrastructure Policy ‘Startup Version’ is set.  Retry the operation after changing the version to ‘Not Set'”  Check out this post for the solution.
  10. Activate Firmware on subordinate Fabric Interconnect by going to Equipment > Installed Firmware.  Right click the subordinate FI, select Activate Firmware, and select the new firmware package.  Verify when the FI comes back up it is running the proper new version, and that your network and storage redundancy is working properly.
  11. Failover the UCSM cluster by connecting to UCSM via SSH, and run the following: 
    connect local-mgmt
cluster lead b
  12. Active the firmware on the formerly primary FI, which is now the subordinate by repeating the above, but do the other FI this time.  Verify it’s running the proper new version, and your network and storage redundancy is working properly.
  13. Validate network connectivity and storage multipathing.
  14. Turn back on CallHome.
  15. Take a backup of the final configuration.

That’s how to do a manual Cisco UCS Manager Firmware Upgrade.

vmware workstation

vCenter 6 VCSA External PSC in VMware Workstation

vCenter, vSphere, ,

I’ve been doing a lot of various oddball testing with vCenter for various scenarios, which have required me to deploy more complex configurations with vCenter 6 recently in my lab.  I found very quickly that there isn’t good consolidated documentation on how to do more advanced vCenter deployments other than directly to ESXi hosts.  It took me quite a bit of time to figure this all out.  I wanted to share this with anyone else who may be doing similar test.  Here’s how to deploy vCenter 6 VCSA with External PSC in VMware Workstation.

And I mean this to be a “proper” lab you can really test vCenter stuff in.  No plain IP addresses for host names!  We want proper FQDNs and what not here!

I am assuming you already have the following up and functioning, along with the following information:

  • DNS server
  • Proper networking for VMware workstation suitable for whatever you’re going to do
  • Document what you will want your host names, IP addresses, DNS IP(s), default gateway IP, and Single Sign-On site names in advance.  Fair warning: this all can get very confusing!  Don’t introduce confusion by deciding these things on the fly.  We’re going to use the following for this article:
    • 1st PSC – vcenter6-2-psc1.vs6lab.local, 192.168.1.61
    • 2nd PSC – vcenter6-2-psc2.vs6lab.local, 192.168.1.62
    • vCenter – vcenter6-2.vs6lab.local, 192.168.1.60
    • DNS server: 192.168.1.80
    • Default Gateway: 192.168.1.1

Deploy vCenter 6 VCSA External PSC in VMware Workstation – Preparation

Here are the things you should get out of the way first:

  1. Download VCSA 6 from VMware if you haven’t already done so.
  2. Extract the VCSA download package to a temporary directory.  For simplicity’s sake, we will assume you extracted the download to c:\VCSA.  Rename the c:\VCSA\vcsa\vmware-vcsa file with an OVA file extension.
  3. Create A AND PTR records for all PSC and vCenter nodes within your lab’s DNS server.

Deploy vCenter 6 VCSA 1st External PSC in VMware Workstation

In order to begin your lab deployment to have an external PSC in VMware Workstation, you must deploy the first PSC.

To deploy the first external PSC in VMware Workstation, do the following:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2-psc1. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed: 
    guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.addr = "192.168.1.61"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
guestinfo.cis.system.vm0.hostname = "vcenter6-2-psc1.vs6lab.local"
guestinfo.cis.vmdir.password = "P@ssw0rd"
guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
guestinfo.cis.deployment.node.type = "infrastructure"
guestinfo.cis.vmdir.first-instance = "true"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad.  Sometimes HTML invisible formating gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2-psc1.vs6lab.local and verify the web page comes up, telling you to sign into a vCenter Management server to manage the PSC.

We’re assuming this completed successfully at this point.  If you encountered issues, correct this before proceeding.

Deploy vCenter 6 VCSA Non-Embedded Server in VMware Workstation

After you deploy the first external PSC in VMware Workstation, you need to deploy the vCenter server itself.

By default, deploying a vCenter 6 server appliance will automatically default to embedded within Workstation.  The line guestinfo.cis.deployment.node.type within the VMX file controls the node type.  As you saw above, setting it to “infrastructure” makes the VCSA instance a Platform Services Controller (PSC).  Let’s make a vCenter server!

To deploy a vCenter Server leveraging the external PSC in VMware Workstation, do the following:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed: 
    guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.addr = "192.168.1.60"
guestinfo.cis.appliance.net.pnid = "vcenter6-2.vs6lab.local"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
guestinfo.cis.system.vm0.hostname = "vcenter6-2-psc1.vs6lab.local"
guestinfo.cis.vmdir.password = "P@ssw0rd"
guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
guestinfo.cis.deployment.node.type = "management"
guestinfo.cis.vmdir.domain-name = "vsphere.local"
guestinfo.cis.vmdir.site-name = "default-first-site"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad, as sometimes HTML invisible formating gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.  Also, note that the vSphere Web Client takes a long time to initialize.  Be patient!
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2.vs6lab.local and login to the vSphere Web Client.  Ensure you can access the administration and inventory sections of the Web Client.  Ensure both the vCenter appliances show up under Administration as healthy.vcenter using external PSC in VMware Workstation check1st external PSC in VMware Workstation check

Still with me?  Awesome!  We’re almost done!  What if you want to add an additional vCenter Platform Services Controller?

Deploy vCenter 6 VCSA Additional External PSC in VMware Workstation

You may be content with just a single external PSC, but additional PSCs can be deployed to test other scenarios as well. Here’s how to deploy an additional external PSC in VMware Workstation:

  1. Double click on c:\VCSA\vcsa\vmware-vcsa.ova
  2. Provide the name for the new virtual machine.  I’m calling mine vcenter6-2-psc2. Also provide the storage location for the virtual machine.
  3. After importing is completed, open the virtual machine’s VMX file before you power the VM up.  You need to add the following lines to the VMX file, adjusting values as needed: 
    guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.addr = "192.168.1.62"
guestinfo.cis.appliance.net.pnid = "vcenter6-2-psc2.vs6lab.local"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.80"
guestinfo.cis.vmdir.password = "P@ssw0rd"
guestinfo.cis.appliance.root.passwd = "P@ssw0rd"
guestinfo.cis.deployment.node.type = "infrastructure"
guestinfo.cis.vmdir.site-name = "default-first-site"
guestinfo.cis.vmdir.domain-name = "vsphere.local"
guestinfo.cis.vmdir.first-instance = "false"
guestinfo.cis.vmdir.replication-partner-hostname = "vcenter6-2-psc1.vs6lab.local"
  4. Ensure that you created both the A and PTR records for this appliance.  If you didn’t create them correctly, the remaining steps are a waste of time, as you’ll have to redeploy the appliance.
  5. Power the virtual machine on.  If you get error messages that the VMX file is corrupt, the above lines likely did not get added properly within the VMX file.  If you copied above from my web page, try retyping it all out in notepad, as sometimes HTML invisible formatting gets copied and pasted that you’re not aware of.  Allow the machine to complete its initialization.
  6. Verify it has completed properly.  To do this, you can open the VM’s console window, verify that it shows the correct name and IP address, and does not show any error messages that say firstboot failed.  If you see this error, you likely did not put in the proper information above, and/or the DNS A and PTR records were not properly created, or a similar issue with name resolution.  Also, you can go to https://vcenter6-2-psc2.vs6lab.local and verify the web page comes up, telling you to sign into a vCenter Management server to manage the PSC.  You should also go into the vSphere Web Client under Administration > System Configuration > Nodes and verify the new PSC shows up, and its services are healthy. additional external PSC in VMware Workstation check

To make a PSC in a different site, change the guestinfo.cis.vmdir.site-name value to a new site.

And there you have it!

powercli

Manage ESXi SSH Using PowerCLI

PowerCLI, vSphere,

Let’s face it. Starting and stopping SSH in ESXi is pain through GUI methods.  I often as a consultant need to connect via SSH to hosts to run data collect scripts, assess NIC and HBA firmware and driver versions, and for troubleshooting purposes, like to run esxtop.  The good news is you can manage ESXi SSH Using PowerCLI.  How cool is that?

Just remember to use get-vmhost to narrow down the specific hosts you want to execute the following commands.

Get the current status of ESXi SSH Using PowerCLI

get-vmhost  hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | select-object vmhost,policy,running

Policy is the start up mode.

  • Automatic = Start automatically if any ports are open, and stop when all ports are closed
  • On = Start and stop with host
  • Off = Start and stop manually

Start ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Stop ESXi SSH Using PowerCLI

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Note the confirm switch.  If you don’t specify that, it will prompt you.

Set startup policy for ESXi SSH Using PowerCLI to start and stop with host

get-vmhost hostname | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | set-vmhostservice -policy "Off"

Be careful if you have any third party products that use SSH.  Nutanix for example comes to mind.  If you goofed and need it set to start and stop with host, just use “On” for the policy parameter.

Log Insight Manager – Install Content Pack

Log Insight Manager,

I’m working a bit with VMware Log Insight Manager for the first time, so I wanted to provide people with a taste of this product.  People don’t tend to know about Log Insight Manager, so hopefully this might ease your worries about it.  It is a pretty easy to use product.  If you’re not familiar with this product, Log Insight Manager is a syslog and event log aggregator and analyzer from VMware that helps to parse, analyze, and alert based on queries.

Installing Content Packs into Log Insight Manager

Content packs are the rule sets of what to look for in syslogs that Log Insight Manager is ingesting.  You can make your own.  You can also install content packs from the marketplace as well.  Finally, you can manually import them.

Installing from Marketplace

Installing from the marketplace is easy.  Just click the settings bar in the top right, click Content Packs, and you’ll be taken to the marketplace (in red).

log insight manager install content pack

Unfortunately, this customer has locked down internet access.  I don’t have time at the moment to show how easy this is.  Basically, you can install the content packs right out of this portal, and you’re off to the races.

Importing a Content Pack

This is also very easy.  First off, you would download the Content Pack file.  Once into the Content Packs portion of the web interface, click “Import Content Pack” in the bottom left (in green box).

Browse to the content pack file you downloaded, and select if this should be installed as a content pack, or your own content space.  In this case, I’m installing VMware’s NSX content pack, so I selected “Install as content pack”.  Click import.

log insight manager select content pack

Content packs often give you further instructions.  In this case, the NSX content pack gave instructions to point all NSX components for syslogging to Log Insight Manager.

You should then see the Content Pack listed under Installed Content Packs.

log insight manager installed content pack

And there you have it!