Unhandled Exception when logging into ESXi Host client

Unhandled Execption Error
Ran into this weird error after powering up the C.R.I.B. and logging into the ESXi Host to boot up VMs. This error popped up while using Safari, Chrome, and Firefox.

Unhandled Exception (1)
Cause: Error: [$rootScope:inprog] http://errors.angularjs.org/1.3.2/$rootScope/inprog?p0=%24digest

The error presents itself with two buttons: Reload and Details. The picture displayed shows the details of the error that I got. You can hit reload, attempt to log back in, and rinse and repeat the process. Or you can select Details, where you can only ‘close’ the error and repeat the process.

A quick check with Professor Google finds this is an issue mentioned in the VMware Communities. Luckily, the entry had an answer attached. (Thanks, “rshell”). It’s not a fix, but it is an answer.

I can’t explain why the error occurs, but I can explain what causes it to pop up. This error occurs when you land on the logon page, enter credentials, and then hit on the keyboard instead of clicking on the button. If you click the login button, you have no errors and vSphere loads normally.

Tool: Microsoft Remote Connectivity

While onsite with a customer yesterday, I was introduced to a new free tool provided by Microsoft called the “Remote Connectivity Analyzer”.

So what does this do? How can it be of use to me?

I’m glad you asked. Our scenario was that the company used Office365 for their emails. I’ve watched companies make the move to cloud provided email for years now. But where this gets to be problemsome, is when you need to configure applications to send/receive email. Our specific scenario involved configuring vRA to use Office365. Now, I already know that if Office365 is set up to use EWS (Exchange Web Services) this won’t work. So boys and girls, if the customer uses Office365 and they are setting up vRA – ask them if O365 is using EWS. If so, stop. Save yourself a headache.

But back to the problem. So when we go to configure the email server (inbound or outbound), we only have the ability to plug in the configuration and click a little “Test Connection” button. If it fails, you get a generic error message and then you have to scour logs to try and decrypt what went wrong. Do you not have enough ports open? Are they blocking subnets? Is the email account enabled? Now you have to try and troubleshoot the problem with multiple teams.

Our first foray was to try and check if we could ping outlook.office365. We SSH’d into the vRA Cafe and pinged. We were good. We attempted telnet by running the command: curl -v telnet://outlook.office365.com:993. We were good. We had the network team verify that traffic was making it to and from. Again, good.

MS Remote Connectivity Analyzer
MS Remote Connectivity Analyzer
The next step was to validate the email configuration. The email team checked over our settings — which surprise, we’re good. Then they introduced us to the Remote Connectivity Analyzer. Using this tool, they were able to see test the connection between the admin’s workstation and outlook.office365. What we found was that the email account was set up as a shared email mailbox, and not an individual user box.

But what was fascinating was that the analyzer ran tests to check EVERYTHING. So the next time you run into issues with trying to configure email, give this a shot.

FINE PRINT: The test machine will need to have internet access to give this a whirl.

Prevent the incrementing of eth devices on Linux systems after guest customization of a cloned VM

I’ve ran into this issue before and found the following article written by Chris Greene from Orchestration.io.

After the guest customization process runs on cloned VMs in some VMware products, you may notice that on your Linux systems the eth device number gets incremented. For example, when the system is first built, the first eth device will be eth0. If the system is cloned & customized, the eth device will become eth1. This may not be a problem on some systems, but people often need/prefer the first eth device to be eth0 or at least to not change after the system is customized.

The issue arises because of old entries in the udev network file – /etc/udev/rules.d/70-persistent-net.rules. After an initial install of a Linux system that has a NIC with a MAC of “00:50:56:02:00:7c”, /etc/udev/rules.d/70-persistent-net.rules will look something like

When you perform a clone & customization (as in creating a new vApp from a template in vCloud), the source VM is cloned and has NIC with a new MAC address. When the cloned VMs boots, udev notices the new NIC and updates /etc/udev/rules.d/70-persistent-net.rules with the new NIC and gives it the name eth1.

A new file named /etc/sysconfig/network-scripts/ifcfg-eth1 will be created that points to the eth1 device

Now when ifconfig is ran, you will see eth1 instead of eth0.

To prevent the issue from occurring, delete the /etc/udev/rules.d/70-persistent-net.rules file before shutting down the VM and turning it into a template. This will cause a new /etc/udev/rules.d/70-persistent-net.rules to be created when the customizing VM boots up. The new file will only contain the NICs on the system and they should be labelled as eth0, eth1, etc.

Another thing you may want do before shutting the VM down to be added as a template is modify /etc/sysconfig/network-scripts/ifcfg-eth0 so that ONBOOT is set to no (ONBOOT=no). I’ve seen issues in vCloud where multiple vApp templates are being deploying onto the same network and the VMs have the same IP (that was initially on the VM before it was turned into a template). Then the systems boot, ifup is ran, which runs arping. I’ve seen arpping return an error in these situations, which causes VMware tools to not start. This then causes guest customization to fail since VMware tools is relied up by guest customization.

ESXi BIOS Power Settings Best Practice

Dell Poweredge R720 Server BIOS Power Settings
Dell Poweredge R720 Server BIOS Power Settings

There is a movement to change the best practice regarding the bios power settings on an ESXi host. In the past, you would set the power settings in the BIOS to max and be done with it. You did this because previous versions of vSphere did not work well with the various C-States of the processor (ie., When the workload on the host would drop, the CPU would drop to a lower power setting. When it would go to wake and draw the CPU back to full power, the host would crap all over itself.).

vSphere 5.5 introduced a new feature that allows VMware to better utilize the C-States of some of the newer processors. Thus allowing the CPUs to change power and speed states without affecting the performance or behavior of the Host. (YEAH!!)

I know we are happy about this, but not so fast…

This link describes the behavior and a couple of performance benefits from the change — however, it should also be noted, that this change is dependent on the type of workloads in your environment. If you have a heavy IO Intense workload or a time sensitive workload, you may still want to have that host on max power, high performance, etc.


Disable Flow Control

HP C7000 Blade Chassis
HP C7000 Blade Chassis
I was onsite this past weekend doing a vSphere 5.1 turnkey install. The turnkey consisted of a C7000 Blade Chassis, and two ESXi 5.1U3 hosts. With this new installation, the customer was adding to a new 10Gbe environment to his network. There were also a few settings that needed to be made, specifically, disabling Flow Control on the ESXi Hosts for Mgmt, Data, and vMotion.

The Flex10 Virtual Connect Modules in the chassis do not have a method of disabling flow control, each ESXi Host would auto-negotiate with Flow Control On for both transmit and receive. Personally, I’ve never had to worry about Flow Control within the environments that I’ve set up. But on this engagement, I was working side by side with a guy who lives, breathes, and retransmits network. Trust me, he knows his stuff. It was his recommendation to disable Flow Control. Realizing I had to figure this out quickly, I turned to the number one tool in my toolbag – Professor Google.

Flexing my “Google-Fu”, I found my answer right away: VMware’s KB Article 1013413. I opened an SSH session, and ran this command

# ethtool –pause tx off rx off

That’s Great! One problem. When I reboot, Flow Control is going to reenable because of the auto-negotiate. So how do I make this persistent/permanent? Another well versed Google search brought me to VMware KB Article 2043564. Once this change was applied, reboots did not reenable Flow Control. My network guy was happy, and hey – I learned something new.

So the next question then, is how do I add this change into a kickstart file. Hmm….. Stay tuned folks. I’ll see if there’s a way to do that.