Monthly Archives: January 2016

Configure PDC Emulator NTP Settings Using GPO

Active Directory,

Active Directory was something I used to spend quite a lot of time working with, but not quite as much these days.  However, this is a trick not many people are aware of.  If you’re familiar with Active Directory, you know that the PDC Emulator FSMO Role Holder in the root domain of the forest needs to be configured to use an external time source.  All other domain controllers should not.  It’s easy enough to do; however, anytime this FSMO role needs to be moved, especially unplanned but even when planned, configuring the new domain controller is often overlooked, as is the old FSMO role holder being reset back if it is to continue running.

However, it’s actually pretty easy to just configure a GPO to target only the PDC Emulator in the root domain to configure NTP settings.  This automates the configuration.  You can simply leave all your domain controllers to default NTP settings, and let this do the work for you.  It’s self-tuning and corrective.  I don’t know about you, but I like stuff that does work for me.

First, create a WMI Filter called “PDC Emulator” that targets the PDC Emulator in the root domain of the forest.  The filter should be:

Select * from Win32_ComputerSystem where DomainRole = 5

Next, create a GPO to accomplish this task, apply the WMI filter above to it, diable the User Configuration portion of the GPO to help reduce login times, and link it to the Domain Controllers container.

Set the following options as follows:

Administrative Templates/System/Windows Time Service/Time Providers

Configure Windows NTP Client – Enabled with the following options:

NTP Server: tock.usno.navy.mil,0x1 ntp-s1.cise.ufl.edu,0x1 ntp.colby.edu,0x1 (Use whatever servers you want, but it’s recommended to use an odd number greater than 1 that are geographically dispersed, but are managed by the same entity.  All three of the above are run by the US Naval Observatory across the east coast.)

Type: NTP

CrossSiteSyncFlags: 2

ResolvePeerBackoffMinutes: 15

ResolvePeerBackoffMaxTimes: 7

SpecialPollInterval: 900 (Default is 3600, but if your PDC Emulator is ever running on a virtual machine, VMware recommends this be set to 900 due to higher clock drifts within VMs.  This simply causes the machine to sync time more often with the NTP server, so it shouldn’t harm physical machines honestly, either.)

EventLogFlags: 0

Enable WIndows NTP Server – Enabled

Allow Active Directory replication to occur or force replication if you like, and refresh the PDC Emulator’s GPOs.

Hope this helps!

VMware network test commands

Automation, vSphere, , ,

I recently ran into an issue with vSphere Replication that involved network connectivity (probably a future post), and I quickly realized that VMware network test commands are not consistent across all their products, so this could be confusing for many people.  I’ll update this post later as I get the commands for other products, but this may help someone looking for how to do VMware network testing and troubleshooting.

ESXi

ESXi has two helpful commands.  For basic connectivity tests, vmkping is awesome because it’s simple to use and to specify which kernel port group you want to test.  Sure, you could use ping, but you can’t specify which vmk interface with it.

To ping 192.168.1.1 with your Management Port group, assuming it’s default, so it’s using vmk0, it’s simply:

vmkping 192.168.1.1 -I vmk0

Another good use is validating jumbo frames, as you can specify the packet size as well and disable packet fragmentation.  To conduct the same test with a packet size of 9000 and ensure the packet doesn’t get fragmented:

vmkping 192.168.1.1 -I vmk0 -s 9000 -d

For testing specific port connectivity, ESXi does support the netcat, aka nc command.  To test port 80 on destination 192.168.1.1:

nc -z 192.168.1.1 80

You can specify UDP mode using -u as well.  Note that at least in my experience -s <source IP> does NOT work, so I don’t believe it’s possible to specifically direct netcat through a specific vmkernel port.  When I tried it for example forcing it through an IP that shouldn’t work, connectivity was still made when it shouldn’t have.

Any VMware Product Running on Windows 2012 or Higher (vCenter, SRM)

Everybody knows ping.  I’m not gonna go over that.  But did you know that PowerShell has a ping cmdlet?  This is useful for documentation of results, using export-csv, and scripting lots of ping tests.

To ping 192.168.1.1:

test-connection 192.168.1.1

Another handy trick is you can remotely have multiple Windows machines ping the same computer and/or specify multiple targets.  For example, if I want server1, server2, to ping 192.168.1.1 and 192.168.1.2:

test-connection -Source Server1,Server2 -ComputerName server3,server4

PowerShell also has cmdlets to test network port connectivity as well.  To test if the local machine can connect to 192.168.1.1 on TCP port 80:

test-netconnection -computername 192.168.1.1 -InformationLevel detailed -port 80

Unfortunately, there isn’t a handy -source parameter, but you could use PowerShell remoting to run this command on multiple remote computers, too.

VMware vCenter Server Appliance

For pinging, there’s the ping command.  That’s easy enough.

If you try to use netcat for port testing, it isn’t there by default.  You have to run the following to temporarily install it on version 6:

/etc/vmware/gss-support/install.sh

Rebooting the VCSA removes it.

You can also use curl if that’s something you’d rather not do:

curl -v telnet://192.168.1.1:80

vSphere Replication Appliance

For pinging, there’s the ping command.  No surprises.

For network port testing, again, netcat isn’t installed, nor is there a supported way to install it to my knowledge.  Instead, use the curl command:

curl -v telnet://192.168.1.1:80

Keep checking back, as I add more.

Preparing for the VCAP6-DCV Deployment Exam

Certifications, VCAP6-DCV Deployment, ,

I received an email recently reminding me that my VCP5-DCV will expire come October.  I knew this was coming, but I was holding out on upgrading my VCP6-DCV because VMware offers a certification upgrade path too good for me to refuse.

I’m a VCAP5-Data Center Design.  In fact, my VCP5 two year clock to renew or upgrade in fact was reset when I passed that exam in October 2014.  VMware has a new certification on your way to VCDX called VCIX (VMware Certified Implementation eXpert) that indicates that you’ve completed all VCAP exams for a track.  For example, within the Data Center Virtualization track, if you pass both the VCAP Data Center Deployment and the Design exams, you get VCIX-DCV certification.

For VCAP5 holders like myself in Data Center Virtualization, if you passed one of the two exams, you can simply pass the other, and you get VCAP6-DCV Design AND Deployment certifications, which also nets you VCIX, AND it upgrades your VCP to version 6 as well.  That’s right, if I pass VCAP6-DCV Deployment (the replacement for VCAP5-DCA), I net four new certifications. It’s hard to say no to that.

The downside of course is if I go this route and don’t complete it by October of 2016, I’ll lose all my VMware certifications.  YIKES!

On top of all that, this exam is brand new.  The objectives are completely rewritten from the VCAP5-DCA exam it replaces.  There’s no books or anything to follow, and I really don’t want to play the waiting game to see if any become available in time for me to get this done, so I’m going to have to make my own study resources, which honestly I do anyway.  And since I blog now, I’m going to share it with my readers and the community.

Expect to see a lot of VCAP6-DCV Deployment material and blog entries coming!  I’m also setting up pages within my site for all of this info here.  Each exam objective will light up as I complete my notes for each one.

Enjoy!

Updating vSphere 6 vCenter Server Appliance

vSphere,

If you skipped the first release of vCenter 6 and deployed Update 1, a new version of vCenter was released for Update 1 with some security fixes among other things.  Many people are opting for the appliance version of vCenter for the first time, and patching it isn’t like the Windows version, so I wanted to document my experience with how to install updates for the vSphere 6 vCenter Server Appliance.

First off, friendly reminder, RTFM with this kind of thing.  I’m screwing around in my lab, so I didn’t as I began and immediately ran into issues, as you’ll see, but it was my fault.

Step 1:  Check interoperability with all vSphere components, third party products, and note upgrade paths.

If you are using any products that interact with vCenter from VMware such as Horizon View, vCenter Operations Manager, Site Recovery Manager, or third party products such as backup products (Veeam, etc.), management products (VMTurbo), etc., ensure you are using versions that are supported with the new version of vCenter you are about to upgrade to, and if not, ensure you map out the proper order and new versions you need to install in order to preserve functionality for all your products and services.  Don’t forget to check support on your external database if you use one, too.

I’m assuming you’ve taken care of all this already.

Step 2: Download all your relevant files you’ll need.

At a minimum, you’ll need to download the patch file from VMware.  This is NOT the full install version of the appliance!  You need to go to:

https://my.vmware.com/group/vmware/patch

Filter for patches for vCenter, the major version of vCenter, and download the applicable patch file for your deployed version of the appliance.

I didn’t RTFM, so I downloaded the VCSA full installable file ISO, and got greeted with the following:

Command> software-packages stage –iso –acceptEulas
[2016-01-09T19:31:01.009] : Staging software update packages from ISO
[2016-01-09T19:31:01.009] : ISO unmounted successfully
[2016-01-09T19:31:01.009] : CD drives do not have valid patch iso.
[2016-01-09T19:31:01.009] : Staging process failed.

Get the patch file!

If you use the Appliance Management Interface to do this, you can have it automatically download the correct file for you.  The upgrade ISO files aren’t the smallest files, so I would encourage you to download it and have it ready.  If you’re curious, the patch file I downloaded for this was 1.5GBs.  You don’t want to eat up your planned downtime by waiting for an ISO.

Step 3:  Ensure a backout plan if it fails.  Take whole VM backups of all relevant vCenter VMs – Platform Services Controller and vCenter.  Take a VM snapshot as well for faster rollback.

The remaining steps are repeated for external PSCs and vCenter servers.  Just ensure you update all external PSCs before you update vCenter server nodes.  Don’t forget to test PSC functionality prior to continuing with the vCenter servers.

Step 4: Mount the patch ISO file into the VM if you are doing this via command line, or which to use a manually downloaded ISO instead of having vCenter download it for you.

Straightforward step here.  If you don’t know how to do this, you probably should stop now. 🙂

Step 5: Initiate the upgrade command

Command line method

Enable SSH on the appliance via the VCSA DCUI, and putty into the VM, and run the following:

software-packages install –iso –acceptEulas

(That’s double hyphens.)

You can seed the install files as well if you like, but I personally don’t see much advantage in doing this.

GUI

Using a web browser, log in to the vCenter Server Appliance Management Interface.  (Port 5480 using https), ensure the repository is configured properly (probably “Use default option”) if you want vCenter to download the patch ISO for you, initiate a check for patches.  Select URL if you want vCenter to download the patch for you, or select Check CDROM if you downloaded the ISO already and mounted it.  Finally, click Install Updates.

Step 6: Monitor the install progress and follow the instructions.

Monitor the installation, and ensure that it succeeds.  It’s completed when you are back to the Command> prompt if you’re using the command line.  You should also see:

Packages upgraded successfully, Reboot is required to complete the installation.

Reboot the VCSA VM if you are instructed to do so using:

shutdown reboot -r “vCenter 6.0 Update <whatever version you’re installing”

If you’re updating with the GUI, you should see a Reboot option under Summary.

If you have errors, review the /var/log/vmware/applmgmt/software-packages.log file.

Step 6: Dismount the ISO

Again, simple stuff.

Step 7 – Verify functionality of vCenter and integrated products

Step 8 – Clear out VM snapshot

Obviously, do not do this until you’re sure you don’t need to rollback.  With that said, do NOT keep the snapshot indefinitely either, as it will degrade vCenter performance, use up additional space on your datastore, and increases the chance of data corruption the longer you wait.

And there you have it!

You’re gonna lose a lot of free OneDrive space!

Miscellanous,

I suspect most of us have personal Microsoft ID’s and probably at the very least store some OneNote notebooks there! You’re about to lose a lot of OneDrive space if you don’t take action.

TLDR version:

You will lose 10GBs or 25GBs of free Microsoft OneDrive storage if you don’t opt in to keeping it by 1/31/16!  Opt to keep it here:

https://preview.onedrive.com/bonus/

So here’s the deal…

Microsoft was offering 15GBs of free storage, plus the camera roll bonus of an additional 15GBs of cloud storage for free if you opted into that, totaling 30GBs of cloud storage.  Microsoft is lowering new accounts and existing accounts if you do not take action to 5GBs and no camera roll bonus, meaning you will lose a free 10GBs or even 25GBs if you do nothing.

First, try to activate the free camera roll phone backup to OneDrive to gain another 15GBs if you haven’t already done this.  Even if you don’t wish for your camera on your phone to automatically upload pics, you still get additional space for whatever you want.  Also, I would recommend if you don’t care about the feature to disable it on your phone to ensure it never takes up cell data allotment trying to upload all your phone camera photos.  If you didn’t do this before, I don’t know if it still works…

http://www.philkasper.net/techphilter/2015/3/31/how-to-get-30gb-of-free-onedrive-cloud-storage-from-microsoft-1

Finally, you need to opt in to keep your OneDrive allotment as it is using this link:

https://preview.onedrive.com/bonus/

If you cannot get the camera roll bonus, you still will net a free +10GBs to keep compared to taking no action at all, which is triple the OneDrive storage!

Spread the word!

Nutanix administration do’s and don’ts

Nutanix, , , ,

As a virtualization consultant, I know there’s a wide variety of technologies at every level – hypervisor, storage, networking, and even server hardware is getting to some degree more complex in terms of what you need to know to manage it effectively.  Everyone can’t be an expert in every single storage technology as an example, and with more and more options that are radically different in their architecture, right now I wanted to make my own little contribution to the world for consultants and admins alike on basic things you should and shouldn’t do with one storage solution – Nutanix.  For us consultants, we often find ourselves within environments with something we’re not totally familiar with, so some helpful concise guidance can go a long way.  Admins, too, may have depended upon a consultant or previous colleagues that no longer work there for implementation and support, but now it’s on them, so I thought this would be helpful.

There are quite a few things everyone should know if they ever are working on a environment with Nutanix that aren’t necessarily obvious.  I can see it being pretty darn easy to blow up a Nutanix environment if you’re not aware of some of these things.

Common stuff

  • Contact Nutanix Support before downgrading licensing or destroying cluster to reclaim licenses (unnecessary if you’re using Starter licensing though). This was repeated many times, so I’m guessing if this isn’t done, you’ll be hating life getting licensing straight.
  • Do NOT delete the Nutanix Controller VM on any Nutanix host (CVM names look like: NTNX-<blockid>-<position>-CVM)
  • Do NOT modify any settings of a Controller VM, all the way down to even the name of the VM.
  • Shutdown/Startup gotchas:
    • It’s probably best to never shutdown/reboot/etc. more than one Nutanix node in a cluster at a time. If you do more, you may cause all hosts in the Nutanix cluster to lose storage connectivity.
    • When shutting down a single host or < the redundancy factor (Nutanix number of hosts it is configured to tolerate failure in a Nutanix cluster), migrate/shutdown all VMs on host EXCEPT the controller VM, THEN shutdown the controller VM.
    • If you are shutting down a number of hosts that exceeds the redundancy factor, you need to shutdown the Nutanix cluster. There’s also a specialized procedure to start up the Nutanix cluster in this situation.  That’s beyond the scope of this email.
    • When booting up a host, do the following:
      • start the Controller VM first that resides on it, and verify it’s services are working by SSH to it using:
        • Ncli cluster status | grep –A 15 <controllerVmIP>
      • Then have it rescan its datastores.
      • Then verify the Nutanix Cluster state using the following to ensure cluster services are all up via same SSH session:
        • cluster status
  • Hypervisor Patching
    • Make sure to patch one hypervisor node and ensure Controller VM comes back up with services are good before proceeding to the next one. Also do one at a time in a Nutanix cluster (see above).
    • Follow shutdown host procedure above.

vSphere

  • NEVER use “Reset System Configuration” command in Nutanix.
  • If resource pools are created, Controller VM (CVM) must have the highest share.
  • Do NOT modify NFS settings.
  • VM swapfile location should be the same folder as the VM. Do NOT place it on a dedicated datastore.
  • Do NOT modify the Controller VM startup/shutdown order.
  • Do NOT modify iSCSI software adapter settings.
  • Do NOT modify vSwitchNutanix standard vSwitch.
  • Do NOT modify Vmk0 interface in port group “Management Network”.
  • Do NOT disable ESXi host SSH.
  • HA configuration recommended settings:
    • Enable admission control and use percentage based policy with value based on number of nodes in cluster
    • Set VM Restart Priority for CVMs to Disabled.
    • Set Host Isolation Response of cluster to Power Off
    • Set Host Isolation Response of CVMs to Leave Powered ON.
    • Disable VM Monitoring for all CVMs
    • Enable Datastore Heartbeating by clicking Select only from my preferred datastores and choosing Nutanix datastores. If cluster has only one datastore (which would be common potentially in Nutanix deployments), add advanced option das.ignoreInsufficientHbDatastore=true to avoid warnings about not having at least two heartbeat datastores.
  • DRS stuff:
    • Disable automation of all CVMs
    • Leave power management disabled (DPM)
  • Enable EVC for lowest processor class in cluster.

Hyper-V

  • Do NOT use Validate Cluster within Failover Clustering nor SCVMM, as it is not supported. Not sure what would happen if you did, but I’m guessing it would be pretty awesome, and you probably should make sure you got popcorn ready if you’re gonna do that.
  • Do NOT modify the Nutanix or Hyper-V cluster name
  • Do NOT modify the external network adapter name
  • Do NOT modify the Nutanix specific virtual switch settings

KVM (the Hypervisor… also assuming this means if you’re using Acropolis Hypervisor from Nutanix since it’s KVM based…)

  • Do NOT modify the Hypervisor configuration, including installed packages
  • Do NOT modify iSCSI settings
  • Do NOT modify the Open vSwitch settings

I hope this proves helpful to people who unexpectedly find themselves working on Nutanix and need a quick primer to ensure they don’t break something!

Desk treadmill activity review – 2015

Fitness, Treadmill Desk, Treadmill Desk

I wanted to start this year off with an update on my desk treadmill/walking update.  I intended to do more frequent progress updates, but it’s hard to do that and blog on technical topics as much as I’d like.  But this kind of update is better than none at all.

As I am drafting this, I’m compiling a list of all my weekly progress of steps tracked by my Lumo Lift, which helps me keep good posture that helps avoid neck pain, but also tracks the number of steps I take surprisingly accurately.  Roughly, 2,000 steps = 1 mile.  My Lumo Lift did break however the last week of the year.  It wasn’t the product’s fault.  It inadvertently got smashed, let’s just say in a manner that it was unsurprising that it didn’t work thereafter.  I just got my replacement today, so I’m back to tracking my steps!

Obviously, not all walking was done on my desk treadmill, but a lot of it was.  When you see dips in weekly steps, that’s likely a business trip, vacation, or I exercised some other way other than anything that would trigger detected steps.  I’m pretty sure before I compile this data there’s going to be some insane amounts here.

Also, for some stupid reason, I deleted some weekly progress emails that I used to track this at the beginning.  I was able to salvage some from when I began, but you’ll see a large gap until I consistently saved them starting in June.  Bottom line is I walked actually a lot more than this.

WeekStepsMilesNotes
3/3016147780TOO HYPED!
4/612789763
4/2713456867
5/412161960
5/2510471552
6/115529177
6/810170750
6/1510968354
6/2211535657
6/2911019655
7/69023945Business trip
7/1314149470
7/2015035475
7/2710805554
8/310184650
8/109336446
8/1710647953
8/248219641Vacation
8/318765543Vacation
9/710529052Vmworld
9/1410877854
9/2111127855
9/2811600858
10/512496862
10/128032840
10/1998444Training
10/263912219Business trip
11/25926929Business trip
11/911779258
11/1610135350
11/2312225661
11/3011869959
12/76902834Business trip
12/148024740Business trip
12/2111209556
12/2811169955
1/33765218Lumo broken
Total38298971896HOLY CRAP!!!

First off, there’s quite a few funny things in the table.  By far is the fact that I walked 80 miles my first week with the treadmill, but never exceeded it, although I did come close on the week of June 1st.  I was pumped to use it, and I was doing work at home conducive to allowing me to walk without issues, and I was excited to get going on it even into the night.

But reality does kick in.  The thing I’ve learned about using the desk treadmill is it’s hard to walk and concentrate REALLY hard on something, like training, unless it’s an urgent thing.  If I’m working with a customer interactively, I’m fine.   If I’m troubleshooting a problem, I’m also fine walking on the treadmill.  If I’m learning something new?  Not fine.  And that’s why the crazy 70+ mile weeks stopped to keep up my learning and lab stuff I did.

But even with me reducing some of the miles I walked, I still pretty darn consistently walked A LOT.  I knew I was racking up miles, but it’s really eye opening tallying it up and seeing almost 2000 miles.  There is absolutely zero doubt in my mind I well exceeded 2000 miles in 2015 walking considering all the weeks I didn’t save the emails for, and I would never have come close to that without the desk treadmill.

I also lost weight, but not nearly as much as I would like.  I did in all drop about 15 pounds, but considering I walked over 2000 miles, that’s disappointing, and I know it’s because I snack too often.  I need to get a better grip on not eating while maintaining this level of physical activity.

But all in all, the desk treadmill has improved my health, and I’m very glad I did it!