16 January 2020

Upgrade ESXi host using vCenter Update Manager

VMware Update Manager (VUM) is a component of vCenter has matured a lot as product since its introduction. One of my favorite parts about it is that it is now baked into the vCenter Server Appliance (VCSA). Using it makes the process of updating and patching your hosts a simple centralized task. Here we are going to use VUM to upgrade our ESXi host from v6.0 to v6.7.

1) The First thing we need to do is check the infamous “VMware Product Interoperability Matrices” to find our upgrade path. Looking at the matrix, we can see that we should be able to upgrade from any v6.0 release directly to v6.7u3.

Sometimes, depending on which version of ESXi you are running, you may not be able to jump straight to the latest release. You might need to do an incremental upgrade, stepping up through versions, until you finally land on your ‘final’ version. 

For example, in the picture above we can see that ESXi v5.5 does not have a direct upgrade path to v6.7. You would first have to upgrade the host to v6.0 or v6.5, before you make another jump to v6.7.

2) The next step is to download the bits of your ESXi iso. There are a couple places you can grab this iso from.

a) The first place you can get it from is directly from VMware.  Login into your VMware portal, in the right hand corner under “Quick Links” click on ‘Download Products’. In the search bar at the top of the page enter “ESXi” and you should see what we’re looking for in the auto-suggested dropdown, click on it.

Now look for the keyword “Product Binaries” in the search results, there’s a lot of other stuff like drivers listed, which we’re not looking for right now. Click the link, you’ll see another page with the actual download links.

b) For most folks, the above-mentioned way to find and download the ESXi iso will be fine. However, if you are running Dell hardware, then you have a second option; Download and use the Dell Customized ESXi image. It’s an ESXi iso that has all the Dell specific drivers pre-baked into it.

The Dell customization image includes: modifying files to identify Dell and Dell support, updating the ESXi image with the latest validated asynchronous drivers, and updating patches if it addresses a common issue for Dell systems.

To find this customized iso image, go to https://support.dell.com/ and enter your product serial number.

Click on the “Drivers & Downloads” tab.

Choose “Enterprise Solutions” in the Category field.

Expand the VMware ESXi link and click download on the desired version for your server.

3) Now we are finally ready to start using VUM, and we begin by importing our iso image we downloaded in the previous step. Login into your vCenter Server. From the menu, click on “Update Manager” -> “ESXi Images” -> “Import”.

Click the “Browse” button to navigate to and select the ESXi 6.7 iso image you downloaded previously. This will start uploading the file into VUM.

4) With our iso image now in vCenter, the next step is creating the baseline that vCenter will use to check if hosts are in compliance. Select the image you just imported, and click on “New Baseline”

Give your new baseline a name, such as “ESXi 6.7u3 Upgrade”, and click ‘NEXT’.

Make sure the image you imported is the one you have selected and click ‘NEXT’ again.

Verify all the summary details are correct and click ‘FINISH’.

Select “Baselines”, you will see the baseline that we just finished creating.

5) The next step is to attach the baseline we just created to an object. The two objects we can attach our baseline to is either our cluster or individual host. I’m going to show how to do it on a cluster object, but the process is the same for a host.

From the “Hosts and Clusters” view in vCenter, select either your cluster object. Go to the “Updates” -> “Host Updates” -> “Attached baselines” -> “Attach” -> “Attach Baseline or Baseline Group”

Select the baseline we created earlier and click ‘ATTACH’.

6) With our baseline attached to our cluster or host object, we are finally ready to upgrade the ESXi host itself. Or in VUM terms – “remediate the host”.

Go to the “Updates” -> “Host Updates” -> “Attached baselines”. Select the baseline you want to apply, in our case it’s the “ESXi 6.7u3 Upgrade”, and then click on “Remediate”.

Click on “Accept the license agreement” -> Click on “Next” to accept all the default options before upgrading ESXi 6.x to ESXi 6.7 -> Click on “Finish” to begin the upgrade. Your host should go into “Maintenance Mode” before it applies it’s remediation steps and completes the upgrade, click “OK”.

7) It’s time to sit back and watch the status bar do it’s thing.

8) Once the remediation has completed, you will be able to verify that your host has been upgraded and is now in compliance.

14 January 2020

IT Academy @ Stanly CC

Staying up to date and current in IT work and IT specializations is important. Things are always changing, and vendors are always introducing both new products and new paths to certification on their new products.

If you are paying for your own training and certifications, one great resource you should check out is Stanly Community College’s IT Academy. Through their IT Academy, they offering professional level training at an affordable community college price. Depending on what your looking to learn, you should take a look at what Stanly CC is offering. Stanly CC has partnerships with some big vendors in the IT space and offers online courses that will satisfy the classroom portion on your path to various certifications.

I recently have had some personal experience with Stanly CC. I haven taken two of their VMware courses; “Install, Configure, Manage – vSphere 6.7” and “Optimize and Scale – vSphere 6.7”. The courses themselves were pretty well done. Lots of in-depth videos that dived into the intricacies of vSphere environment and ESXi. They also had lots of really great labs to work through that corresponded to the videos. The hands-on labs, IMHO, were the best part. They really provided the chance to “get your hands dirty” and actually set up things like a VDS switch, configuring tag based storage policies, or running esxtop to determine which virtual machines were experiencing resource contention; just to name a few of the labs.

Overall, I was happy having gone the Stanly CC route. The skills that I learned there definitely helped me towards passing my exams and earning my VMware VCP-DCV certification. I give them a solid recommendation in terms of improving skill sets and advancing your technical knowledge.

24 December 2019

Office 365 – Winmail.dat

After changing over from on-prem Exchange to O365, I had one user where the recipients of their emails would receive any attachment that was sent as the “dreaded” winmail.dat file instead of the .pdf or whatever file the user was actually sending. It was intermittent however, in that some users would get the actual file and some (all external) would get the winmail.dat file.

First thing that I did was check that user was sending their mail as HTML, and not Rich-Text. After changing that value, I check back a week or so later and the user was still experiencing the issue, so it was time to dig a little deeper. After some searching online I was able to find that this was not an uncommon issue.

The issue happens because the receiver’s email client can not interpret the email message that was sent from Outlook in the Rich-Text format. When using Outlook to end an email using the Rich-Text format, a plain text copy of the email is also sent along with an attachment named winmail.dat. This ‘winmail.dat’ attachment is what contains all of the formatting, elements, and other data specific to Rich-Text email messages. This method of sending the email message is called “Transport Neutral Encapsulation Format” or “TNEF” for short.

Unfortunately, many non-Microsoft email programs can not properly open message that are received in TNEF. To fix this, we can use PowerShell to force Exchange Online to convert Rich-text messages to HTML before it sends it off. You can use the commands shown below to set the ‘RemoteDomain’ property “TNEFEnabled” to false on the Default policy.

1) Connect to Exchange Online via Powershell.

2) Get your Default RemoteDomain policy:
Get-RemoteDomain | fl *

3) Set the TNEFEnabled property:
Set-RemoteDomain Default -TNEFenabled $false
Re-run step 2 and you should see that “TNEFEnabled” is set to “False”.

Just in case… Here is how to change it back to a NULL value to undo your change in step 3, and let the Outlook client again decide how it wants to send the message.

4) Set the TNEFEnabled property back to NULL:
Set-RemoteDomain Default -TNEFenabled $nul

20 December 2019

File management – Cisco Flash

When working on Cisco devices, sometimes you have to delete files from the device. It’s moreless the same on all Cisco devices. While there are other file systems available to you on your device (see first command listed), Cisco devices use “flash:” as their default file system. My examples will also use the default “flash:” but note that the commands will work the same on any other available file system, such as “usb0:“.

Show Available File Systems
Device# show file systems

Show Files
Device# dir flash:

Create Directory
Device# mkdir folder

Step into Directory
Device# cd folder

Delete File
Device# delete /force flash:/filename
Device# delete flash:/filename

Delete Directory
Device# delete /force /recursive flash:/folder

Copy File
Device# copy usb0:[/directory]/filename flash0:[/directory]/filename

Category: Cisco | LEAVE A COMMENT
18 December 2019

MDT loading to cmd prompt

I was running an MDT LiteTouch deployment and the computer kept getting stuck at a command prompt after loading into MDT environment. Normally it should have launched right into the LiteTouch wizard, but for whatever reason, it just didn’t want to go beyond this point on its own. I discovered two possible ways to circumvent it.

I had used this same means of deployment on other machines of the same make and model, with the same Task Sequence, so I knew it wasn’t a driver issue as many eluded to in the various forums I looked at. However, I went ahead and did an “ipconfig” and also tried to mount the MDT share using “net use“, just to make sure there was no network issues. net use z:\\<servername>\deploymentshare$ Well, it turned out that I did already have an IP address and the mapped drive was already in use. So how can I now get on with my deployment?

The first and possibly the easiest way was to just manually launch the LiteTouch. It is as straight forwards as it sounds. I entered the command below into my command prompt, and viola! The LiteTouch wizard came right up and I continued on my way. X:\windows\system32> Z:\scripts\LiteTouch.vbs

The second method, really isn’t any more difficult than the first. Sometimes a folder named “MININIT” already exists on the computer’s C: drive and it can cause issues when trying to do a new deployment. The way around that is just to wipe the drive and essentially start back at square one. Warning though – Following this method WILL erase the drive and wipe any data that is on it. While it probably doesn’t matter, as you’re stuck anyways, but it is just something to be aware of.

At the first Wizard window, press F8. Then type:

  1. Diskpart (enter)
  2. List disk (enter)
  3. Select disk 0 (choose your main OS disk) (enter)
  4. Clean (enter)
  5. Reboot and start your imaging process again.

After wiping the disk of any pre-existing partitions your next deployment attempt should go smoothly as it now has a blank HDD to work on.

18 December 2019

Cisco 3850 – Upgrading IOS

Upgrading the IOS version on your Cisco 3850 Catalyst switch is not something to be dreaded. In fact it should probably be rejoiced, as these days it most likely means you’re also catching up on patching some vulnerabilities on your device. While the upgrade process is a little different on the 3850 then older generations of the Catalyst switches, not much has changed. Overall it is an easy process that won’t take you long to complete.

What has changed is that the 3850 switch? Well the “operating system” that the hardware now runs on is called IOS-XE. Whereas previous generations of Cisco hardware ran on IOS software.

The IOS-XE software is delivered as a bundle containing a set of packages which your device will use. These packages, during the install step, will get expanded before they are actually installed on the device. The older generations of devices that ran IOS had a .bin file that was essenitally a “flat” image and not a bundle of packages. While there are other changes too, that is the main difference between the “new” IOS-XE and the “old” IOS.

Okay… Lets get our switch upgraded…

Step 1: Download the latest (or rather the recommended) IOS-XE version from the support portal of Cisco.com. (At the time of me writing this, the recommended release version is “Fuji-16.9.4”)

Step 2: Transfer the .bin file you downloaded in step 1 to your device. There are a few ways to so; TFTP or USB being the most common.

  • Via TFTP

3850-SW# copy tftp flash:
Address or name or remote host []? 10.1.2.3
Source filename[]? cat3k_caa-universalk9.16.09.04.SPA.bin
Destination filename[cat3k_caa-universalk9.16.09.04.SPA.bin]?
Accessing tftp://10.1.2.3/cat3k_caa-universalk9.16.09.04.SPA.bin...
Loading cat3k_caa-universalk9.16.09.04.SPA.bin from 10.1.2.3: !!!!!!!!!!!!!

  • Via USB

3850-SW# copy usbflash0:cat3k_caa-universalk9.16.09.04.SPA.bin flash:

Step 3: Start the Install.

3850-SW# software install file flash:cat3k_caa-universalk9.16.09.04.SPA.bin

Once the software install completes, you will need to reboot the switch so that it can boot into the freshly installed IOS-XE version.

3850-SW# reload

Step 4: Verify the new version. After rebooting the device in the previous step, you will want to verify that you are indeed running the new IOS version. You can run the following command to do so.

3850-SW# show version | i Switch
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, RELEASE SOFTWARE (fc2)
Switch Ports Model SW Version SW Image Mode

Here we can see that the switch is running Fuji, and that it is on version 16.9.4. And now we’re done!

Category: Cisco | LEAVE A COMMENT
13 December 2019

dracut-initqueue

I was updating the firmware on some Dell FC630 servers when I came across this. I really thought that the server hung during the update and I was in for a long night of trying to fix it. Wait and see what the fix was…

So using the DellEMC Repository Manager tool, I created a linux based SmartBootableISO that included the desired updates for my hardware. I then connected to the server’s iDrac virtual console, mounted the iso, and booted the server to the iso image. Everything appeared fine as I watched the server boot up. Then I saw it throw the following message:

dracut-initqueue[686]: mount /dev/sr0/ is write-protected, mounting read-only

Then after waiting and staring for about 5 minutes I started to worry. What’s going on? Did it just freeze?

Well… No, thankfully it had not froze.

It was just mounting a file as read-only, which apparently took longer than you would think it would. After waiting even longer felt right, it finally got past this step, and the server proceeded along with it’s boot-up process. The wait time varied slightly between servers, ranging from about 7 to 10 minutes.

So if you happen to see the “dracut-initqueue” message, don’t panic, your server did not hang. Just wait it out… Grab a coffee or go have a restroom break. Use those few minutes to stretch. Your server will continue chugging along shortly.

4 December 2019

ReFS allocation size

I was reformatting a drive for some Veeam backups and was trying to recall what I had set the ReFS allocation unit size to when I initially setup the drive. Well, I could not remember to save my life. Luckily, with a little command line action, it’s easy enough to find out what it was set it to.

The command line tool to use is fsutils. To see what options are available to us when using fsutils, we can run the following command.fsutils fsinfo /?

Using “E:/” as the drive we are checking out, we can run the following line to discover information about the volume itself. fsutil fsinfo volumeinfo E:

To view the specific ReFS info on this drive, we can run the following line. fsutil fsinfo refsinfo E: Take a look at value for the “Bytes Per Cluster”, this is where we can see that when this drive was formatted, it’s allocation unit size was set to 65k. 65k is also the recommended setting for Veeam destinations if you are using ReFS.

4 December 2019

Office 365 in-“stalled” at 51%

Out of an entire organization, we had one single machine that had issues installing O365, it would always get stuck at “51%”. I even let tried letting it run for an entire weekend. The weird part was that it was a Win10 machine, and it was all up-to-date in terms of applying Windows Updates. The same ODT script worked perfectly fine for all the other machines that were a mix of Win8.1 & Win10.

So I gave the Office 365 uninstaller a whirl. It ran thru pretty quickly and said everything was removed. It ran so quick, it didn’t even seem like it did anything. I went ahead and tried my ODT script again. and voila – It worked!

So my only take away, is if you’re having issues with the installer “stalling” out, try MS’s uninstaller and try your install again.

30 November 2019

Bitnami – Disabling TLS v1.0 & v1.1

I was surprised when I ran the Qualys SSL Labs scan against my website that I got a lower score than I expected. The SSL Labs scan is a FREE deep analysis of the SSL configuration of your public facing website, that returns a score and grade of your server. It also provides some suggestions as to what you can do to improve your server’s score.

Well, it turns out there are two things that were hindering my score. One is easy to fix via a configuration change in Bitnamo, I’ll cover that in this article. The other isn’t necessarily hard, but it involves a DNS record known as a CAA, which I’m not going to cover in this article.

The quick easy fix was to change the versions of TLS that I allowed my server to use, by disabling the older versions of the TLS protocol, v1.0 and v1.1.

I used WinSCP to connect to my server, and went to the file. /opt/bitnami/apache2/conf/bitnami/bitnami.conf

Within the file you can set/remove the “SSL Protocol” directive. In my case, I set it to SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

Once that change has been saved, all that’s left is to restart Apache. You can restart connecting over SSH to your server and issuing the following command sudo /opt/bitnami/ctlscript.sh restart apache

After you see Apache restart, go ahead and re-scan your website. I found that for my website, my score increased and thus my grade was better after disabling the older versions of TLS. Success!