Category Archives: Automation

VMware network test commands

Automation, vSphere, , ,

I recently ran into an issue with vSphere Replication that involved network connectivity (probably a future post), and I quickly realized that VMware network test commands are not consistent across all their products, so this could be confusing for many people.  I’ll update this post later as I get the commands for other products, but this may help someone looking for how to do VMware network testing and troubleshooting.

ESXi

ESXi has two helpful commands.  For basic connectivity tests, vmkping is awesome because it’s simple to use and to specify which kernel port group you want to test.  Sure, you could use ping, but you can’t specify which vmk interface with it.

To ping 192.168.1.1 with your Management Port group, assuming it’s default, so it’s using vmk0, it’s simply:

vmkping 192.168.1.1 -I vmk0

Another good use is validating jumbo frames, as you can specify the packet size as well and disable packet fragmentation.  To conduct the same test with a packet size of 9000 and ensure the packet doesn’t get fragmented:

vmkping 192.168.1.1 -I vmk0 -s 9000 -d

For testing specific port connectivity, ESXi does support the netcat, aka nc command.  To test port 80 on destination 192.168.1.1:

nc -z 192.168.1.1 80

You can specify UDP mode using -u as well.  Note that at least in my experience -s <source IP> does NOT work, so I don’t believe it’s possible to specifically direct netcat through a specific vmkernel port.  When I tried it for example forcing it through an IP that shouldn’t work, connectivity was still made when it shouldn’t have.

Any VMware Product Running on Windows 2012 or Higher (vCenter, SRM)

Everybody knows ping.  I’m not gonna go over that.  But did you know that PowerShell has a ping cmdlet?  This is useful for documentation of results, using export-csv, and scripting lots of ping tests.

To ping 192.168.1.1:

test-connection 192.168.1.1

Another handy trick is you can remotely have multiple Windows machines ping the same computer and/or specify multiple targets.  For example, if I want server1, server2, to ping 192.168.1.1 and 192.168.1.2:

test-connection -Source Server1,Server2 -ComputerName server3,server4

PowerShell also has cmdlets to test network port connectivity as well.  To test if the local machine can connect to 192.168.1.1 on TCP port 80:

test-netconnection -computername 192.168.1.1 -InformationLevel detailed -port 80

Unfortunately, there isn’t a handy -source parameter, but you could use PowerShell remoting to run this command on multiple remote computers, too.

VMware vCenter Server Appliance

For pinging, there’s the ping command.  That’s easy enough.

If you try to use netcat for port testing, it isn’t there by default.  You have to run the following to temporarily install it on version 6:

/etc/vmware/gss-support/install.sh

Rebooting the VCSA removes it.

You can also use curl if that’s something you’d rather not do:

curl -v telnet://192.168.1.1:80

vSphere Replication Appliance

For pinging, there’s the ping command.  No surprises.

For network port testing, again, netcat isn’t installed, nor is there a supported way to install it to my knowledge.  Instead, use the curl command:

curl -v telnet://192.168.1.1:80

Keep checking back, as I add more.

Using PowerShell when there isn’t PowerShell support

Automation,

I know many of us work on lots of different technologies, many of which don’t have native PowerShell cmdlets, and that kind of thing.  Sometimes it’s DOS, sometimes, it’s Telnet/SSHing into a command line where you got to run individual command strings to fix a bunch of individual objects.  I know many of you guys end up hacking stuff together using Excel or other tools to basically to assemble a repeated command to fix multiple objects, or create rules or whatever, like…

First part of command object1 second part of command

First part of command object2 second part of command

And you got a list of all your objects you got to do this on.  This can be painful.

Let me give you an example…

Working on an issue with an old version of EMC RecoverPoint, which has no PowerShell integration.

Basically, the customer masked some LUNs to VMAX front end ports that aren’t hooked up, and RecoverPoint is barking because it can’t access those ports.  So the customer has to unmap the front end ports and unmask.  I know for many of you guys, it’s this garbledy gook of tech you don’t work with.  In the end, the specific technology doesn’t matter.

RecoverPoint reports all the volumes that are the problem, like this:

Devices: 2B3B,277F,83D8,2B34,2250,21DD,2774,102A,21E2,281E,102B,281F,83D5,83E1,12B7,83CB,83DC,83DF,2775,83DB,24BB,83CE,818D,83D9,2784,2776,83CD,83DA,12CF,281D,83E3,0FB4,83D0,2B50,83CC,0FA3,8037,0FB3,83D1,2772,8196,83D4,83CF,83E2,83D3,83D7,2773,277E,12CC,12C9,8038,83DE,8036,1518,83D6,83D2,83DD,83E0

The first thing I need is an array of these I can pump into a loop.

This is stupid simple for PowerShell.  Each device is separated by a comma, so I can just use comma as the split character.

(Cut off the long string of devices, you get the idea)

$devicelist = “2B3B,277F,83D8,2B34,2250,21DD”

$devices = $devicelist.split(‘,’)

Now, if you type $devices, you get:

2B3B

277F

83D8

2B34

2250

21DD

Now we have our simple array.

Also, another helpful thing to know is if you have a sequence of numbers, you can use another PowerShell trick.  Say I need an array of objects that’s object1-10.  Also easy:

$objects = 1..10 | foreach-object {“object” + $_}

Type $objects and you get:

object1

object2

object3

object4

object5

object6

object7

object8

object9

object10

Yes, you can do this for IPs.  Say I want an array of all IPs in 192.168.0.0/24, so I can ping them or whatever.

$ips = 1..254 | foreach-object {‘192.168.1.’ + $_}

Maybe port ranges with “TCP” in front for firewall rule statements.

$tcpports = 3000..4000 | foreach-object {“TCP” + $_}

Now, I need to have command string stuff added in front and behind this.  Again, this doesn’t matter what tech you’re working on, just put your garbledy gook that I wouldn’t understand in.  $_ is the instance in the array

$commands = $devices | foreach-object {‘symconfigure -sid 1234 -cmd “unmap dev ‘ + $_ + ‘ from dir ALL:ALL;” commit’}

If I type $commands, I get:

symconfigure -sid 1234 -cmd “unmap dev 2B3B from dir ALL:ALL;” commit

symconfigure -sid 1234 -cmd “unmap dev 277F from dir ALL:ALL;” commit

symconfigure -sid 1234 -cmd “unmap dev 83D8 from dir ALL:ALL;” commit

symconfigure -sid 1234 -cmd “unmap dev 2B34 from dir ALL:ALL;” commit

symconfigure -sid 1234 -cmd “unmap dev 2250 from dir ALL:ALL;” commit

symconfigure -sid 1234 -cmd “unmap dev 21DD from dir ALL:ALL;” commit

BAM!  We got our commands, and we’re rolling.  If I want to save the commands as a text file…

$commands | out-file c:\dir\ourcoolscript.txt

Now I can copy/paste into putty/telnet session, or upload the script file and launch it if that’s possible, whatever I want to do.

WAY faster IMO than using other tools or duct taping a solution using Excel or other weird methods, and far more flexible.

So even if your technologies don’t have PowerShell, you can still use PowerShell!

Taking scripting too far?

Automation

I love scripting, and I am a huge advocate of PowerShell.  I talk about how it can be leveraged seemingly all the time to customers who don’t leverage it.  I encourage customers constantly to make use of it to make them more efficient.

But…  is it possible to take scripting too far?  Of course.

I stumbled across this article about a sysadmin who automated his job to arguably a ridiculous degree.

I shouldn’t say he arguably went too far.  He definitely did.  To me, the worst example in the article is where he automated the rollback of one of his user’s databases based on contents of an email if he received it from a particular end user.

Scripting is so beneficial virtually anyone in the IT field, or more specifically automation.  I applaud almost all efforts to do this.  However, scripting gets dicey when you begin to automate specifically decision making, especially complex decision making.

Don’t get me wrong, decision making is possible and beneficial in scripting, but it shouldn’t always be used.  I’ve many a times included conditional logic in a script, and it was absolutely essential to accomplishing the goal of the script.  However, sometimes decisions are just too complex to make based on limited information.

In this case, I have a lot of problems setting up what he did.  First off, how on earth can you tell just from some keywords in the contents of an email that you should roll back the database, without the end user asking specifically to roll back the database?  Even if the end user requested this, if the end user doesn’t know how to do this, there’s a pretty decent chance that this isn’t the best solution anyway.

Secondly, I seriously doubt the email was authenticated to be from this specific user.  IE, if this type of automation is wide spread given the general security posture of most email systems, it could be trivial to exploit to cause a day’s worth of data loss.

With all this said, I generally have the opposite problem with customers not automating anything, as opposed to customers automating things they shouldn’t, but this does demonstrate it’s possible to go to the opposite extreme.