All posts by Aaron Margeson

Air Console Wired/Wireless Serial Solution Review

WhatsInTheBag

UPDATE: Get-Console noticed my blog article and contacted me to offer my readers 10% off using coupon code JJGH667QS on their orders.

I realized I hadn’t done a WhatsInTheBag post in awhile, and I wanted to share with you a really cool device I recently picked up – the Air Console Mini.

I work with a lot of different data center equipment.  If you do, you know a means to connect via serial to equipment is an absolute must in some cases, or far more convenient in many.  All Cisco networking gear requires this pretty much for initial configuration at least to the point of getting it on the network for you to remote into to finish the configuration.  Data Domains are often initialized via serial.  Cisco UCS C-Series  server and CIMC cards, as I’ve mentioned before, can be configured via serial as well.  Cisco UCS Fabric Interconects, too…

You get the idea – you need a serial connection.  And most laptops these days don’t have serial ports, either.

I started out lugging a Trendnet Serial USB adapter, a long serial cable, and a Cisco console cable.  This for all intents and purposes worked, it was cheap, didn’t need special drivers for Windows, easy to use, but man was it bulky and added weight to my already heavy bag, mainly due to the really long serial cable.  After all, with this solution, I was tethered to equipment I was working on by the length of the cable, so that pretty much required a long serial cable.

I tried a bluetooth to serial adapter, for the sheet simple reason that I wouldn’t be physically chained to the equipment, but it didn’t work, despite having the ability to have serial communications settings that could be changed to work with the equipment.  I quickly gave up on it, and resigned to using the wired cheap solution.

But then, I found the Air Console!  I don’t want to go extensively into all the features and models of them.  You can get that info easily from their site.  I elected to go with the Air Console Mini, which is similar to their other models in functionality, but it doesn’t have an internal rechargeable battery.  I did this to eliminate one extra thing that could fail that might be hard to fix, and reduce the size of the device for my already loaded backpack.  Besides, I carry a power bank, and I stuff a micro-USB cable in the bag I carry it in, which is used to provide it power.  Usually, there’s a USB port nearby from some equipment I can use to power it up, but if not, I got my PowerBank ready.

airconsolecomponents

The Air Console Mini (top left) comes with the blue Cisco Console cable (bottom left), and the white micro-USB cable as well (bottom right).  The adapters adapt the console cable to serial or null modem.  I throw all this into a small draw string bag to keep it together.  I need to throw a mini-USB cable in the kit at well, which allows you to plug the Air Console Mini directly into the front USB console port on applicable Cisco network gear.  The Air console itself has several ports.

airconsolepower

The micro-USB port provides power to the device.  The holes to the left are to hard reset the device back to factory default, and the other is an indicator light or activity and troubleshooting the device itself.

airconsolerj45

The RJ45 port allows you to wire the Air Console into an ethernet network to access its web portal and for Serial over LAN access.  Internally, as I’ve mentioned before, is a WAP and wireless network client, allowing you to join its non-routed wifi network, or you can access it to configure it to connect to an external wireless network.

airconsoleusb

The USB port plugs into the Cisco Console cable to enable connectivity to the serial device with which you wish to connect.  You can also use a mini-USB cable as mentioned before for Cisco network gear with a USB console interface.

The console cable is also very adaptable.

airconsoledongle

The USB end plugs into the USB port of the Air Console, and doubles as a USB wired console cable to your laptop you can use when you need that hard wired connection.  Note you need to download and install their included drivers to do that.  The Cisco console RJ45 end can be adapted to serial or null modem, but those are not included.  On the side, the object protruding on the right side is the Bluetooth receiver for the device, allowing you to connect with mobile bluetooth enabled iOS or Android devices, or your PC using Bluetooth over serial.

There’s a lot of connectivity options here, just to review.  You can connect it serially to Cisco and other serial devices and then access it:

  1. Wired over a LAN
  2. Bluetooth using their proprietary apps
  3. Serial over Bluetooth (yay, use putty if you want!)
  4. Via joining it to a wireless network, and then connecting wifi via Serial over LAN
  5. Joining its non-routable wireless network, and connecting via Serial over the LAN
  6. USB-Hub-Serial cable it comes with if you must be wired only, just install the drivers, plug it up, and you don’t need the network piece of it

For client options:

  1. Standard Serial terminal applications via Bluetooth Serial
  2. Connect to its web portal that has a terminal web app built in
  3. IOS and Android apps

All the typical serial settings like baud rate, stop bits, flow control, parity, and data bits can be customized to work with whatever you need within the IOS app, the web portal’s terminal application, or of course your terminal app on your PC, although with the Cisco and Data Domain gear I tried, the defaults were fine, except Cisco C-Series servers, which use a 115200 baud rate.

I have an iPhone and iPad, as well as a Windows work laptop.  Note that you do have to pay for their IOS app as an additional charge to the device itself.  I tried every option above except using their Android app (I don’t have a suitable device to try that), and every option worked.  I will say the IOS Get Console app did crash anytime I tried to switch to a different app and switched back, which is annoying, but overall, this product worked reliably.  I have tried it with Data Domains, various Cisco switches and routers, UCS Fabric Interconnects, and UCS C-Series servers.  All worked great!

Via bluetooth wireless connectivity, everything is great except that it cannot do baud rates equal to or higher than 38400 due to technical limitations of Bluetooth.  Cisco UCS C-Series servers‘ console ports by default to 115200, just heads up, so in these cases you should connect via Wifi/LAN, not bluetooth.

It is also firmware upgradeable to address bugs and to add additional features, and they do release updates for it, so it seems well supported.

The Air Console Mini runs about $70 USD, and the regular model is $10 more with a built in LION battery, so you don’t have to worry about powering it for quick initial configurations.  They also have an additional XL model with a larger battery.

I love this thing, quite honestly.  It hardly adds any weight to my bag, and I have an extremely flexible device for all things serial.  Who wouldn’t want to do a quick initialization of a Cisco UCS FI cluster with their iPhone?

Configure Dump Collector with PowerCLI in vSphere 6

PowerCLI, vSphere, ,

I had a script to configure Dump Collector settings that I used in previous versions of vSphere.  If you look around the web, you’ll find similar PowerCLI snippets to configure Dump Collector.

If you use that snippet in vSphere 6, it doesn’t work.  You’ll get the following error:

Message: Cannot set 2 server ip parameters.;
InnerText: Cannot set 2 server ip parameters.EsxCLI.CLIFault.summary
At line:4 char:1

This is because ESXCLI now has a parameter for whether to use IPv6, so when using get-esxcli, invoking the method to set requires an additional value.  Remember, esxcli is not intuitive in that “enabled” properties are either true or null, so don’t use $false.

The revised code should now be:

$vcenterip = '192.168.1.10'
foreach($vmhost in Get-VMHost){
	$esxcli = Get-EsxCli -VMHost $vmhost.Name
	$esxcli.system.coredump.network.set($null,"vmk0",$null,$vcenterip,6500)
	$esxcli.system.coredump.network.set($true)
	$esxcli.system.coredump.network.get()
}

Also not something commonly found on the internet – can you test the ESXi netdump configuration?  Yep!

foreach($vmhost in Get-VMHost){
$esxcli = Get-EsxCli -VMHost $vmhost.Name
Write-Host "Checking dump collector on host $vmhost.name"
$esxcli.system.coredump.network.check()
}

And there you have it!

Deploy Cisco UCS C-Series Servers via Serial

Cisco UCS

One thing I really like about Cisco UCS C-Series servers is the CIMC card.  Without paying extra, you get effectively a fully featured out of band management card, all the way down to the integrated KVM within the CIMC web interface.  Cool, right?  Technically, you arguably don’t need a KVM in your server rack if you have these servers.  Just get the CIMC card on the network, and you’re golden!

But what about initial deployments?  You gotta get that CIMC card on the network.

During a deployment today with a customer in a branch site of Cisco UCS C-Series standalone servers, I ran into an interesting scenario.  The customer did not have a keyboard and mouse at the site (it was still under construction).  This was partly because they opted not to purchase a KVM because the CIMC cards in the servers.

Did you know you can configure the BIOS settings and CIMC card via a Serial connection?

Yep, just plug your Cisco console cable into the back console port, or adapt it to serial and connect it into the front VGA/USB/Serial dongle, boot up the server, and enter the keystroke to go into the BIOS or CIMC configuration setup when you see it.

ucs-c-series-serial

The only notable thing is to set your terminal app to 115200 for baud rate, but the rest of the settings are typical for Cisco gear.

So, yeah, with my trusty Air Console serial device (more on that later), I am able to configure the BIOS or CIMC card with my iPhone or iPad.  Look ma, no keyboards or monitors!

VMware ending Enterprise licensing

vSphere

In a surprising move, VMware is moving to end the Enterprise licensing level, leaving only Standard and Enterprise Plus licensing levels, along with the Essentials and Essentials Plus packs.

VMware in fact has already removed Enterprise licensing from their product page.

As a consolation, VMware apparently will be offering existing Enterprise licensed customers special promotion pricing to upgrade to Enterprise Plus.

Also, vCenter Standard licenses will be bundled with 25 OS instance licenses for vRealize Log Insight, which definitely adds value for vCenter customers.  Log Insight is actually a really good product seemingly few people are aware of that aggregates event logs from Windows OS’s and syslogging, and allows for analysis and monitoring for specific events.  I hope this encourages customers to take a good look at Log Insight, because I think it’s a really good product that deserves more attention than it gets.

But all and all, I think these changes are not good.  The elephant in the room is DRS.  I have many customers who thought the price jump from Standard to Enterprise to get DRS was hard enough to swallow.  Some did, and some didn’t.  But now there’s a gaping canyon between Standard and Enterprise Plus in both features and price.  For many customers, all they really wanted in additional features above Standard is DRS, and they’ll now be forced to pay more to get it when many already didn’t due to price, so I don’t see this working out for VMware nor customers in the end, as more customers will opt to either stick to Standard instead of begrudgingly stepping up to Enterprise Plus, or consider other less expensive hypervisors such as Hyper-V or KVM.

It’s impossible to avoid having flashbacks to the vRAM licensing debacle.  If they truly wanted to simplify licensing, in my opinion DRS should be added to Standard even with an increase in licensing costs for Standard. Most environments that cost conscious are typically small and often would have the ability to use Essentials packs.  Plus, EVERYONE can make use of DRS somehow, so at least somebody would pay more for something they could actually use.

SRM installation error – unexpected error -1

SRM, vSphere,

I assisted a colleague today with an issue with reinstalling Site Recovery Manager 5.8 for a customer.  During installation, when requested to input the vCenter FQDN and administrative user, an “unexpected error: -1” pop up would occur.  After a bit of research, I found articles pointing to various certificate problems in SRM 4.X and 5.X.  The odd thing was he attempted to point it to the second vCenter server, and it would proceed, but this was the SRM server for one site, and this vCenter was in another.

I’ve been cognizant new vCenter releases disabling SSLv3, so we checked the build numbers of the two vCenter servers.  Sure enough, the one generating the error was 5.5.Update 3b which disabled SSLv3 support, but the vCenter that didn’t generate the error was 5.5 Update 2, which still supports SSLv3.  We then checked the build of the SRM 5.8 installation file, which was 5.8, NOT 5.8.1.  While this isn’t the same error, it states the fact that SRM 5.8.1 is required to interoperate well with vCenter 5.5 Update 3b, unless you want to enable SSLv3 in your vSphere environment, which isn’t the best thing for security. Even the interoperability guide shows that 5.8.0 as unsupported with vCenter 5.5 Update 3.

Apparently, the customer upgraded one vCenter very recently, but not the other, and also didn’t check SRM interoperability before doing so, which caused the weird behavior.   They also didn’t mention this to us.  vCenter in the second site was upgraded, and SRM 5.8.1 was installed instead of 5.8.0, and this resolved the issue.

So, if you have this error during SRM installation, it’s likely a problem with certificates, so start there, and be cognizant of any changes that might impact certificates or their use.  As always, check your builds, the interoperability matrix, and the upgrade order prior to updating any vSphere component.

In-place upgrading Windows OS on vCenter 6?

vSphere, ,

I recently had a customer with two vCenter VMs running on Windows 2008 R2.  They were vCenter servers upgraded from 5.1 to vCenter 6.0 about six months ago.  They’re both using embedded PSCs, and have vSphere Replication and SRM plugged into them.  To simplify administration, they have embarked on a project to get all servers running Windows Server 2012 R2.

After researching, there really isn’t a great, documented way to transplant a vSphere 6 server from one OS instance to another.  Normally, I’m not a big fan of in place upgrading server operating systems, but this was a special case to meet the customer’s objective, and redeploying two vCenters and then likely having to redeploy/reconfigure SRM wasn’t something I’d want to do, plus any pitfalls with vSphere Replication. But the question is – will vCenter 6, especially with an embedded Platform Services Controller and lots of things plugged into it, work after an in place OS upgrade?

I definitely had my doubts.  The answer though in my lab is surprisingly yes!  I tried it both with an embedded PSC, and then tried it again with a once embedded PSC reconfigured to use an external PSC.  I didn’t encounter any problems whatsoever, although I should point out this was a lab environment with a clean fresh setup prior to the OS in place upgrades.

So I went ahead and did it for the customer’s environment (they aren’t  big enough to have a lab environment), and it worked like a champ as well!

Here are some things I would make sure of before proceeding:

  • You may want to backup the vCenter database.  Warning: the vPostgress Windows backup script said it ran successful for me but generated an empty 0KB backup file.  (This was one of the reasons I didn’t attempt a transplant of vCenter to a new OS instance!)  Check to ensure this database file is valid before counting it as a backup to fallback to if there’s a problem.  This may be a future blog article once I get some answers for why this happened.
  • Verify what version of Windows is running, and ensure you have the required media and license keys.  In particular, if vCenter is running Windows Server 2008 R2 Datacenter, you can’t upgrade to Windows Server 2012 R2 Standard.
  • Verify what database vCenter and VUM are using if on the same box.  vPostgress is fine.  But of it’s Microsoft SQL running on the vCenter server itself, make sure SQL is running something that is supported on Windows Server 2012 R2.  Of specific note on some of these older vCenter VMs, SQL 2008 R2 needs to be SP2 or later.
  • I would recommend stopping all vCenter related services, VUM related services (if on the same OS instance), the database service (if it’s on the same OS instance),  and AV active protection prior to OS upgrade.
  • Make sure the C drive has at least about 15GBs of free space.
  • Reboot the OS prior to starting the upgrade to clear out any cobwebs.
  • Take a snapshot and/or backup vCenter before proceeding.  (Kinda duh…)  What isn’t so duh is before you take the last snapshot, launch the upgrade prior to doing this and verify you don’t need to do anything prior to installing the upgrade.  This is usually stuff like it may require you to reboot the OS prior to the upgrade  If all you see is the warning to check to ensure your applications are compatible, cancel the upgrade, take your snapshot, and start the other precautionary steps below. If there are other things it asks you to do, do those first, THEN snapshot your VM.
  • Don’t forget to kill your snapshot once everything is done, and you’ve confirmed everything is working.

It worked flawlessly using these precautions for both production vCenter servers!

vCenter 6 Reconfigure from Embedded to External PSC

vSphere, , ,

There have been some problems with embedded PSC configurations, so I’ve had requests to move away from the embedded PSC (PSC and vCenter in same OS instance) to external configurations.  Thankfully, vCenter Update 1 and above has a method to do just this!

Transitioning to External PSC

To accomplish this, I first built a new virtual machine running Server 2012 R2, patched it to current, joined it to the domain, and granted the appropriate rights for the vCenter service account.

It’s also important to note that the existing vCenter 6.0 must be running Update 1 or later for this to work.  Obviously, you should deploy a new PSC using the same build as the existing vCenter.  Patch up your current vCenter up to Update 1 or higher obviously if needed.

Also, make sure you have a good rollback plan, like a whole VM backup or snapshots as needed.

This process works just as well for the appliance.

You then install an external PSC joining the existing SSO domain and site.  Now there are two PSCs, but vCenter is still setup in an embedded configuration, so the external PSC isn’t used yet.

At this point, you need to use the cmsso-util utility with the reconfigure option, located in your vCenter installation folder.  It’s typically under C:\Program Files\VMware\vCenter Server\bin folder.

cmsso-util reconfigure –repoint-psc destpsc.vs6lab.local –username administrator –domain-name “vsphere.local” -passwd “P@ssw0rd”

I immediately ran into my first issue…

repointpscdnserror

“The provided Platform Services Controller(PSC) is not a replication partner of the localhost. Please make sure to provide the Primary Network Identifier (PNID) of the PSC.”

A little googling led me quickly to this community post that states the DNS name is apparently case sensitive, so check your DNS records to see if maybe it’s all caps, or what.  Use that, and you’re golden.  In my case, it was DESTPSC.vs6lab.local.

Sit back and be patient.  Mine took probably a solid 10 minutes, but I am running it in a slower lab environment.

When it’s finished, verify the vSphere Web Client is functioning.  Also, verify the PSC has been repointed under your vCenter Server – Manage – Advanced Settings – config.vpxd.sso.admin.uri

confirmpscrepoint

PSC is done!

vSphere Replication 6 – Stopping replication impacts

vSphere, vSphere Replication,

I unfortunately didn’t get a chance to post Thursday, as I came down with a bit of a stomach bug, but I’m back at it!

I found this little interesting tidbit during preparation for VCAP6-DCV Deployment…

Did you know in 6.X that stopping replication on a VM in vSphere Replication 6.X has different behavior depending upon if you used a replication seed?

Just to make sure we’re all clear, a replication seed in vSphere Replication speak is if you copy down a VMDK from the source side, upload it to the target site, and then configure replication for the VM and select the datastore/folder for the VMDK.  When vSphere Replication sees the matching VMDK, it uses the data there and replicates only the changes since the download.

In vSphere Replication 6.X, if this was done, and you stop replication for a VM, the target VMDKs are left in place.  If this wasn’t done, and just let vSphere Replication replicate the initial copy of the VMDK, if you stop replication, the VMDK is DELETED at the target site!  If that’s a large data set, that could be a lot of data that has to be replicated again, and more than likely over a WAN link!

This in particular impacts a somewhat common task when it comes to growing a VMDK for a VM being replicated by vSphere Replication.  To do this, replication must be disabled at some point.

If you used a replication seed, it’s actually easier.  You simply stop replication, grow the VMDK on both sides, and reconfigure replication.  Pretty easy.  The target VMDK would obviously not have been deleted, making this possible.

If the VMDK wasn’t seeded, you need to do a planned failover, stop replication, resize the VMDK on both sides, and reconfigure replication.  This also obviously requires downtime.

I’m still investigating to see if there’s a way to determine if the VMDK was seeded or not, so you would know which way to go.  If you’re unsure though, use the non-seeded method as a precaution unless it’s okay to have to re-replicate the VMDK/whole VM.

Configure PDC Emulator NTP Settings Using GPO

Active Directory,

Active Directory was something I used to spend quite a lot of time working with, but not quite as much these days.  However, this is a trick not many people are aware of.  If you’re familiar with Active Directory, you know that the PDC Emulator FSMO Role Holder in the root domain of the forest needs to be configured to use an external time source.  All other domain controllers should not.  It’s easy enough to do; however, anytime this FSMO role needs to be moved, especially unplanned but even when planned, configuring the new domain controller is often overlooked, as is the old FSMO role holder being reset back if it is to continue running.

However, it’s actually pretty easy to just configure a GPO to target only the PDC Emulator in the root domain to configure NTP settings.  This automates the configuration.  You can simply leave all your domain controllers to default NTP settings, and let this do the work for you.  It’s self-tuning and corrective.  I don’t know about you, but I like stuff that does work for me.

First, create a WMI Filter called “PDC Emulator” that targets the PDC Emulator in the root domain of the forest.  The filter should be:

Select * from Win32_ComputerSystem where DomainRole = 5

Next, create a GPO to accomplish this task, apply the WMI filter above to it, diable the User Configuration portion of the GPO to help reduce login times, and link it to the Domain Controllers container.

Set the following options as follows:

Administrative Templates/System/Windows Time Service/Time Providers

Configure Windows NTP Client – Enabled with the following options:

NTP Server: tock.usno.navy.mil,0x1 ntp-s1.cise.ufl.edu,0x1 ntp.colby.edu,0x1 (Use whatever servers you want, but it’s recommended to use an odd number greater than 1 that are geographically dispersed, but are managed by the same entity.  All three of the above are run by the US Naval Observatory across the east coast.)

Type: NTP

CrossSiteSyncFlags: 2

ResolvePeerBackoffMinutes: 15

ResolvePeerBackoffMaxTimes: 7

SpecialPollInterval: 900 (Default is 3600, but if your PDC Emulator is ever running on a virtual machine, VMware recommends this be set to 900 due to higher clock drifts within VMs.  This simply causes the machine to sync time more often with the NTP server, so it shouldn’t harm physical machines honestly, either.)

EventLogFlags: 0

Enable WIndows NTP Server – Enabled

Allow Active Directory replication to occur or force replication if you like, and refresh the PDC Emulator’s GPOs.

Hope this helps!

VMware network test commands

Automation, vSphere, , ,

I recently ran into an issue with vSphere Replication that involved network connectivity (probably a future post), and I quickly realized that VMware network test commands are not consistent across all their products, so this could be confusing for many people.  I’ll update this post later as I get the commands for other products, but this may help someone looking for how to do VMware network testing and troubleshooting.

ESXi

ESXi has two helpful commands.  For basic connectivity tests, vmkping is awesome because it’s simple to use and to specify which kernel port group you want to test.  Sure, you could use ping, but you can’t specify which vmk interface with it.

To ping 192.168.1.1 with your Management Port group, assuming it’s default, so it’s using vmk0, it’s simply:

vmkping 192.168.1.1 -I vmk0

Another good use is validating jumbo frames, as you can specify the packet size as well and disable packet fragmentation.  To conduct the same test with a packet size of 9000 and ensure the packet doesn’t get fragmented:

vmkping 192.168.1.1 -I vmk0 -s 9000 -d

For testing specific port connectivity, ESXi does support the netcat, aka nc command.  To test port 80 on destination 192.168.1.1:

nc -z 192.168.1.1 80

You can specify UDP mode using -u as well.  Note that at least in my experience -s <source IP> does NOT work, so I don’t believe it’s possible to specifically direct netcat through a specific vmkernel port.  When I tried it for example forcing it through an IP that shouldn’t work, connectivity was still made when it shouldn’t have.

Any VMware Product Running on Windows 2012 or Higher (vCenter, SRM)

Everybody knows ping.  I’m not gonna go over that.  But did you know that PowerShell has a ping cmdlet?  This is useful for documentation of results, using export-csv, and scripting lots of ping tests.

To ping 192.168.1.1:

test-connection 192.168.1.1

Another handy trick is you can remotely have multiple Windows machines ping the same computer and/or specify multiple targets.  For example, if I want server1, server2, to ping 192.168.1.1 and 192.168.1.2:

test-connection -Source Server1,Server2 -ComputerName server3,server4

PowerShell also has cmdlets to test network port connectivity as well.  To test if the local machine can connect to 192.168.1.1 on TCP port 80:

test-netconnection -computername 192.168.1.1 -InformationLevel detailed -port 80

Unfortunately, there isn’t a handy -source parameter, but you could use PowerShell remoting to run this command on multiple remote computers, too.

VMware vCenter Server Appliance

For pinging, there’s the ping command.  That’s easy enough.

If you try to use netcat for port testing, it isn’t there by default.  You have to run the following to temporarily install it on version 6:

/etc/vmware/gss-support/install.sh

Rebooting the VCSA removes it.

You can also use curl if that’s something you’d rather not do:

curl -v telnet://192.168.1.1:80

vSphere Replication Appliance

For pinging, there’s the ping command.  No surprises.

For network port testing, again, netcat isn’t installed, nor is there a supported way to install it to my knowledge.  Instead, use the curl command:

curl -v telnet://192.168.1.1:80

Keep checking back, as I add more.