31 March 2022

ESXi Access to resource settings on the host is restricted

I was trying to upload an ova file thru vCenter when I got an error message about a controller type in this particular ova. From prior troubleshooting, I knew that the workaround was to just log directly onto a host and upload the ova directly to the host. I know I had done this before so I was confused for a brief moment when I got the error message:

Access to resource settings on the host is restricted to the server that is managing it: xx.xx.xx.xx.

Okay… so the ESXi host is being managed by vCenter. How do I finagle my upload onto a host? Here’s how…

  • Enable SSH (if it is disabled) on the ESXi host you want to upload/deploy your ova or ovf to.
    • You should know how/where to enable SSH… If you don’t there are plenty of articles you can google.
  • Connect to the host via SSH.
  • We want to stop the service that allows the ESXi host and the vCenter to communicate. To do so we want to run the following commands.

/etc/init.d/vpxa stop
/etc/init.d/hostd restart

  • Deploy your ova or ovf to your host thru the ESXi host’s DCUI WebGUI.
  • After you deploy your virtual machine, restart the VPXA service via SSH on the ESXi host.

/etc/init.d/vpxa start

  • Wait a couple of minutes while the ESXi host and your vCenter re-establish communication between each other.
  • If SSH was previously disabled, re-disable it on your host.
28 January 2022

Nutanix LCM – Insufficient space on ESXi scratch disk

I was running into an issue where I could not run the Nutanix LCM Inventory action on a cluster because the scratch disk on an ESXi host was too small. And it seems reasonable to assume that if you’re reading this article, then you too probably have encountered an with the pre-check “test_esxi_scratch_space“.

I’ve seen the issue a few times now, and in my experience, the ESXi host has always just had its scratch disk set to the wrong disk. The first couple of times I saw this, the fix was to update the scratch disk from within the advanced host settings in VCSA. The part I did not like about fixing the issue this way, was that it required a reboot of the host for the setting to take hold. Having to schedule a maintenance period or create downtime is never an ideal solution. Thankfully I learned that there is another way to repoint the scratch disk that requires no downtime and it only requires a few lines of CLI against the ‘problem’ ESXi host.

Start by connecting via SSH to the ESXi host that is having the issue with the scratch disk.

Run the command “ls -ll /scratch” to find which volume is currently set as the scratch disk

root@ESXi# ls -ll /scratch
lrwxrwxrwx    1 root     root            49 May  8 23:40 /scratch -> /vmfs/volumes/5xyzxyz6-dxyzxyzb-1c73-ac1xyzxyz990

Run the “df -h” command to list all of the disks on the host and their sizes

root@ESXi# df -h

Filesystem   Size   Used Available Use% Mounted on
NFS          1.6T   1.4T    127.4G  92% /vmfs/volumes/OS-XXX-Repoxxx
VMFS-5      52.0G   1.1G     50.9G   2% /vmfs/volumes/NTNX-local-ds-17xyzz340111-B
vfat         4.0G  27.6M      4.0G   1% /vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234
vfat       285.8M 205.8M     80.0M  72% /vmfs/volumes/5xyzxyz6-dxyzxyzb-1c73-ac1xyzxyz990
vfat       249.7M 152.6M     97.2M  61% /vmfs/volumes/58xyzxyz-cdxyzxyz-766a-12xyzxyz1226
vfat       249.7M 145.3M    104.4M  58% /vmfs/volumes/b4xyzxyz-80xyzxyz-9bf2-e5xyzxyzf6d0

Now that we have the current scratch disk and a list of the sizes of all the disks, we can check if the scratch volume is indeed set to the volume that is 4GB in size.

In the example above we can see that the volume “/vmfs/volumes/5xyzxyz6-dxyzxyzb-1c73-ac1xyzxyz990” is only 285MB in size. That means that this current volume is far too small. No wonder we’re getting an error.

We want to set our scratch disk to a volume that is 4GB in size. According to the list above that means we want to use the volume “/vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234”. To set the desired scratch disk we’ll use the command “ln -sfn <volume_id> /scratch”.

root@ESXi# ln -sfn /vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234 /scratch

If we recheck what the scratch disk is on our host, we’ll see that it is now set to the proper disk volume.

root@ESXi# ls -ll /scratch

lrwxrwxrwx    1 root     root            49 May  8 23:40 /scratch -> /vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234

Now that the scratch disk is properly configured on the host we can update it in VCSA and be done.

From the Host, go to Configure, then Advanced Systems Settings, and click “Edit”.
Select “ScratchConfig.CurrentScratchLocation” and set it to the same value that you just manually configured on host. Hit “Apply”, and you’ll see that the VCSA now recognizes the newly configured scratch disk.

Well now we’re done, and we didn’t even need to reboot a single physical host! You can read more about this error in Nutanix’s KB article about it.

1 January 2021

VMware PVSCSI on a new Windows install

If you haven’t already upgraded your Windows servers to Windows 2019, then you will probably be doing so soon enough. That means that it’s time to review the steps you take in building out your virtual machines (VMs). Are you running your VMs from a SAN? Then during this refresh, you should really take the time to consider using the VMware Paravirtual SCSI (PVSCSI) driver.

VMware Paravirtual (PVSCSI) adapters are high-performance storage adapters that can provide greater throughput and lower CPU utilization. They are best suited for environments where hardware or applications drive a very high amount of I/O throughput, such as SAN environments. PVSCSI adapters are not suited for DAS environments.

VMware, https://kb.vmware.com/s/article/1010398

When building new VMs there are four options you can choose from for their SCSI controller. The default LSI Logic SAS driver that is automatically selected for you will work just fine in most environments. That said, when you want to guarantee maximum performance from your VMs you will need to use the PVSCSI. Why wouldn’t you want to allow your VMs their max performance? It’s simple enough to do. Heck, do it and make a “golden image” template so you can easily redeploy it if you don’t want to repeat the steps on each VM everytime. It’s just a couple of clicks now for better performance later. Here we go…

  1. Launch the vSphere Client and log in to an ESXi host or vCenter Server.
  2. Select create a new virtual machine.
  3. In the vSphere Client, right-click on the virtual machine and click Edit Settings.
  4. Click the Hardware tab.
  5. Click Add.
  6. Select Hard Disk.
  7. Click Next.
  8. Choose any one of the available options.
  9. Click Next.
  10. Specify the options you require. Options vary depending on which type of disk you chose.
  11. Choose a Virtual Device Node and specify whether you want to use Independent mode. For data disks, choose a Virtual Device Node between SCSI (1:0)to SCSI (3:15). For a boot disk, choose Virtual Device Node SCSI (0:0) or choose the Virtual Device Node that boots in the order you require.

    Note: To set a disk to use Independent mode there must be no snapshots associated to the virtual disk, if there are existing snapshots commit them before changing the disk type.
     
  12. Click Next.
  13. Click Finish to complete the process and exit the Add Hardware wizard. A new disk and controller are created.
  14. Select the newly created controller and click Change Type.
  15. Click VMware Paravirtual and click OK.
  16. Click OK to exit the Virtual Machine Properties dialog.
  17. Power on the virtual machine.
  18. Install VMware Tools. VMware Tools includes the PVSCSI driver.
  19. If it is a new virtual disk, scan and format the hard disk within the guest operating system.
16 January 2020

Upgrade ESXi host using vCenter Update Manager

VMware Update Manager (VUM) is a component of vCenter has matured a lot as product since its introduction. One of my favorite parts about it is that it is now baked into the vCenter Server Appliance (VCSA). Using it makes the process of updating and patching your hosts a simple centralized task. Here we are going to use VUM to upgrade our ESXi host from v6.0 to v6.7.

1) The First thing we need to do is check the infamous “VMware Product Interoperability Matrices” to find our upgrade path. Looking at the matrix, we can see that we should be able to upgrade from any v6.0 release directly to v6.7u3.

Sometimes, depending on which version of ESXi you are running, you may not be able to jump straight to the latest release. You might need to do an incremental upgrade, stepping up through versions, until you finally land on your ‘final’ version. 

For example, in the picture above we can see that ESXi v5.5 does not have a direct upgrade path to v6.7. You would first have to upgrade the host to v6.0 or v6.5, before you make another jump to v6.7.

2) The next step is to download the bits of your ESXi iso. There are a couple places you can grab this iso from.

a) The first place you can get it from is directly from VMware.  Login into your VMware portal, in the right hand corner under “Quick Links” click on ‘Download Products’. In the search bar at the top of the page enter “ESXi” and you should see what we’re looking for in the auto-suggested dropdown, click on it.

Now look for the keyword “Product Binaries” in the search results, there’s a lot of other stuff like drivers listed, which we’re not looking for right now. Click the link, you’ll see another page with the actual download links.

b) For most folks, the above-mentioned way to find and download the ESXi iso will be fine. However, if you are running Dell hardware, then you have a second option; Download and use the Dell Customized ESXi image. It’s an ESXi iso that has all the Dell specific drivers pre-baked into it.

The Dell customization image includes: modifying files to identify Dell and Dell support, updating the ESXi image with the latest validated asynchronous drivers, and updating patches if it addresses a common issue for Dell systems.

To find this customized iso image, go to https://support.dell.com/ and enter your product serial number.

Click on the “Drivers & Downloads” tab.

Choose “Enterprise Solutions” in the Category field.

Expand the VMware ESXi link and click download on the desired version for your server.

3) Now we are finally ready to start using VUM, and we begin by importing our iso image we downloaded in the previous step. Login into your vCenter Server. From the menu, click on “Update Manager” -> “ESXi Images” -> “Import”.

Click the “Browse” button to navigate to and select the ESXi 6.7 iso image you downloaded previously. This will start uploading the file into VUM.

4) With our iso image now in vCenter, the next step is creating the baseline that vCenter will use to check if hosts are in compliance. Select the image you just imported, and click on “New Baseline”

Give your new baseline a name, such as “ESXi 6.7u3 Upgrade”, and click ‘NEXT’.

Make sure the image you imported is the one you have selected and click ‘NEXT’ again.

Verify all the summary details are correct and click ‘FINISH’.

Select “Baselines”, you will see the baseline that we just finished creating.

5) The next step is to attach the baseline we just created to an object. The two objects we can attach our baseline to is either our cluster or individual host. I’m going to show how to do it on a cluster object, but the process is the same for a host.

From the “Hosts and Clusters” view in vCenter, select either your cluster object. Go to the “Updates” -> “Host Updates” -> “Attached baselines” -> “Attach” -> “Attach Baseline or Baseline Group”

Select the baseline we created earlier and click ‘ATTACH’.

6) With our baseline attached to our cluster or host object, we are finally ready to upgrade the ESXi host itself. Or in VUM terms – “remediate the host”.

Go to the “Updates” -> “Host Updates” -> “Attached baselines”. Select the baseline you want to apply, in our case it’s the “ESXi 6.7u3 Upgrade”, and then click on “Remediate”.

Click on “Accept the license agreement” -> Click on “Next” to accept all the default options before upgrading ESXi 6.x to ESXi 6.7 -> Click on “Finish” to begin the upgrade. Your host should go into “Maintenance Mode” before it applies it’s remediation steps and completes the upgrade, click “OK”.

7) It’s time to sit back and watch the status bar do it’s thing.

8) Once the remediation has completed, you will be able to verify that your host has been upgraded and is now in compliance.