31 March 2022

ESXi Access to resource settings on the host is restricted

I was trying to upload an ova file thru vCenter when I got an error message about a controller type in this particular ova. From prior troubleshooting, I knew that the workaround was to just log directly onto a host and upload the ova directly to the host. I know I had done this before so I was confused for a brief moment when I got the error message:

Access to resource settings on the host is restricted to the server that is managing it: xx.xx.xx.xx.

Okay… so the ESXi host is being managed by vCenter. How do I finagle my upload onto a host? Here’s how…

  • Enable SSH (if it is disabled) on the ESXi host you want to upload/deploy your ova or ovf to.
    • You should know how/where to enable SSH… If you don’t there are plenty of articles you can google.
  • Connect to the host via SSH.
  • We want to stop the service that allows the ESXi host and the vCenter to communicate. To do so we want to run the following commands.

/etc/init.d/vpxa stop
/etc/init.d/hostd restart

  • Deploy your ova or ovf to your host thru the ESXi host’s DCUI WebGUI.
  • After you deploy your virtual machine, restart the VPXA service via SSH on the ESXi host.

/etc/init.d/vpxa start

  • Wait a couple of minutes while the ESXi host and your vCenter re-establish communication between each other.
  • If SSH was previously disabled, re-disable it on your host.
28 January 2022

Nutanix LCM – Insufficient space on ESXi scratch disk

I was running into an issue where I could not run the Nutanix LCM Inventory action on a cluster because the scratch disk on an ESXi host was too small. And it seems reasonable to assume that if you’re reading this article, then you too probably have encountered an with the pre-check “test_esxi_scratch_space“.

I’ve seen the issue a few times now, and in my experience, the ESXi host has always just had its scratch disk set to the wrong disk. The first couple of times I saw this, the fix was to update the scratch disk from within the advanced host settings in VCSA. The part I did not like about fixing the issue this way, was that it required a reboot of the host for the setting to take hold. Having to schedule a maintenance period or create downtime is never an ideal solution. Thankfully I learned that there is another way to repoint the scratch disk that requires no downtime and it only requires a few lines of CLI against the ‘problem’ ESXi host.

Start by connecting via SSH to the ESXi host that is having the issue with the scratch disk.

Run the command “ls -ll /scratch” to find which volume is currently set as the scratch disk

root@ESXi# ls -ll /scratch
lrwxrwxrwx    1 root     root            49 May  8 23:40 /scratch -> /vmfs/volumes/5xyzxyz6-dxyzxyzb-1c73-ac1xyzxyz990

Run the “df -h” command to list all of the disks on the host and their sizes

root@ESXi# df -h

Filesystem   Size   Used Available Use% Mounted on
NFS          1.6T   1.4T    127.4G  92% /vmfs/volumes/OS-XXX-Repoxxx
VMFS-5      52.0G   1.1G     50.9G   2% /vmfs/volumes/NTNX-local-ds-17xyzz340111-B
vfat         4.0G  27.6M      4.0G   1% /vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234
vfat       285.8M 205.8M     80.0M  72% /vmfs/volumes/5xyzxyz6-dxyzxyzb-1c73-ac1xyzxyz990
vfat       249.7M 152.6M     97.2M  61% /vmfs/volumes/58xyzxyz-cdxyzxyz-766a-12xyzxyz1226
vfat       249.7M 145.3M    104.4M  58% /vmfs/volumes/b4xyzxyz-80xyzxyz-9bf2-e5xyzxyzf6d0

Now that we have the current scratch disk and a list of the sizes of all the disks, we can check if the scratch volume is indeed set to the volume that is 4GB in size.

In the example above we can see that the volume “/vmfs/volumes/5xyzxyz6-dxyzxyzb-1c73-ac1xyzxyz990” is only 285MB in size. That means that this current volume is far too small. No wonder we’re getting an error.

We want to set our scratch disk to a volume that is 4GB in size. According to the list above that means we want to use the volume “/vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234”. To set the desired scratch disk we’ll use the command “ln -sfn <volume_id> /scratch”.

root@ESXi# ln -sfn /vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234 /scratch

If we recheck what the scratch disk is on our host, we’ll see that it is now set to the proper disk volume.

root@ESXi# ls -ll /scratch

lrwxrwxrwx    1 root     root            49 May  8 23:40 /scratch -> /vmfs/volumes/5xyzxyz-1234xyzz-12xy-1234xyzz1234

Now that the scratch disk is properly configured on the host we can update it in VCSA and be done.

From the Host, go to Configure, then Advanced Systems Settings, and click “Edit”.
Select “ScratchConfig.CurrentScratchLocation” and set it to the same value that you just manually configured on host. Hit “Apply”, and you’ll see that the VCSA now recognizes the newly configured scratch disk.

Well now we’re done, and we didn’t even need to reboot a single physical host! You can read more about this error in Nutanix’s KB article about it.

6 August 2021

Nutanix password change

If you leave the default passwords on your Nutanix cluster you’ll start to see alerts in Prism that the default password is still in use. It will alert you about it for both the CVM and the physical hosts. This alert is very easy to clear by just updating the password. Here’s how…


To run just the default password health check from your CVM you can use the following command:

nutanix@cvm$ ncc health_checks system_checks default_password_check

Or you can also run the complete set of NCC health checks:

nutanix@cvm$ ncc health_checks run_all

If the health check passes, you’ll see this line in the output:

/health_checks/system_checks/default_password_check              [ PASS ]

If the health check fails you’ll see this in the output and it will tell you which host(s) alerted:

/health_checks/system_checks/default_password_check              [ INFO ]
------------------------------------------------------------------------+
Detailed information for default_password_check:
Node x.x.x.x:

Nutanix Controller VM (CVM) password change

Running this command will prompt you for your new desired password for the ‘nutanix’ user on the CVM:
nutanix@cvm$ sudo passwd nutanix

Once you change the CVM’s password it will replicate to all of the CVMs in your cluster, thus changing the password on all of your CVMs at once.

Hypervisor password change

  • AHV
    To change the local “admin” account password for all AHV hypervisors in the Nutanix cluster, you can run this command from any CVM in the cluster.
    nutanix@cvm$ echo -e "CHANGING ALL AHV HOST ADMIN PASSWORDS. Note - This script cannot be used for passwords that contain special characters ( $ \ { } ^ &)\nPlease input new password: "; read -s password1; echo "Confirm new password: "; read -s password2; if [ "$password1" == "$password2" ] && [[ ! "$password1" =~ [\{\$\^}\&] ]]; then hostssh "echo -e \"admin:${password1}\" | chpasswd"; else echo "The passwords do not match or contain invalid characters (\ $ { } ^ &)"; fi
    To change the local “nutanix” account password for all AHV hypervisors in the Nutanix cluster, you can run this command from any CVM in the cluster.
    nutanix@cvm$ echo -e "CHANGING ALL AHV HOST NUTANIX PASSWORDS. Note - This script cannot be used for passwords that contain special characters ( $ \ { } ^ &)\nPlease input new password: "; read -s password1; echo "Confirm new password: "; read -s password2; if [ "$password1" == "$password2" ] && [[ ! "$password1" =~ [\{\$\^}\&] ]]; then hostssh "echo -e \"nutanix:${password1}\" | chpasswd"; else echo "The passwords do not match or contain invalid characters (\ $ { } ^ &)"; fi

  • VMware ESXi 
    To change the local root password for all ESXi hosts in the cluster, you can run this command from any CVM in the cluster.
    nutanix@cvm$ echo -e "CHANGING ALL ESXi HOST PASSWORDS. Note - This script cannot be used for passwords that contain special characters ( $ \ { }  ^ &)\nPlease input new password: "; read -s password1; echo "Confirm new password: "; read -s password2; if [ "$password1" == "$password2" ] && [[ ! "$password1" =~ [\\\{\$\^\}\&] ]]; then hostssh "echo -e \"${password1}\" | passwd root --stdin"; else echo "The passwords do not match or contain invalid characters (\ $ { } ^ &)"; fi

  • Microsoft Hyper-V 
    To change the local administrator password for all Hyper-V hosts in the cluster, you can run this command from any CVM in the cluster.
    nutanix@cvm$ echo -e "CHANGING ALL HYPER-V HOST PASSWORDS. Note - This script cannot be used for passwords that contain special characters ( $ \ { }  ^)\nPlease input new password: "; read -s password1; echo "Confirm new password: "; read -s password2; if [ "$password1" == "$password2" ] && [[ ! "$password1" =~ [\ \"\'\\\{\$\^\}] ]]; then hostssh "net user administrator $password1"; echo "Updating Host and ManagementServer Entries..."; ncli host ls | grep -i id | grep -Eo "::[0-9]*" | cut -c 3- | while read hID; do ncli host edit id=$hID hypervisor-password=$password1;done  > /dev/null; ncli host ls | grep "Hypervisor Address" | awk '{print $4}' | while read hIP; do ncli managementserver edit name=$hIP password=$password1;done > /dev/null;  else echo "The passwords do not match or contain invalid characters (\ $ { } ^)"; fi

Further info can be found in the following Nutanix KB.

Category: Nutanix | LEAVE A COMMENT
26 March 2020

Howto: Folding@Home – VMware Fling


VMware is doing their part to help make it easy for folks to contribute to the Folding@Home (F@H) project. They have packaged together an appliance as an OVA file on VMware Flings that you can deploy on any of their virtualization products either on your hardware or in a cloud, using Workstation or Fusion, or ESXi hosts. That means that with just a few clicks you can download and deploy a VM running on the super light-weight PhotonOS that has the F@H client pre-installed and is ready-to-go.

You might be asking why this is so great, I mean the client isn’t exactly difficult to setup on other operating systems. Well you are correct. This fling is geared towards VMware virtualization enthusiasts and professionals that already have homelabs or datacenters, with idle compute power they want to contribute. By using those idle resources and dedicating an VM appliance towards contributing, it basically becomes a set-it-and-forget-it deal that will always be chugging along in the background.

If you are new to virtualization, then deploying this appliance can serve as a great way to learn about flings, appliances, and deploying a VM in general while contributing to a cause.

Note: If you intent to deploy this in your company’s data center, or your work pc/laptop, you should make sure to have permission to allow for it from the appropriate people in your organization before deploying, just to cover your ass.

Step 1: Create your Passkey

If you don’t already have a username and passkey, then the first thing you’ll need to do is create your user and get your passkey. You’ll use this later as you deploy the appliance. If you already have yours, then proceed to the next step.

https://apps.foldingathome.org/getpasskey

Just to let you know, when I signed up earlier this week, it took a few hours for me to receive my passkey from F@H. So don’t get upset if you don’t hear from them immediately after clicking “Get Passkey”.

Step 2: Download the Fling

The first thing we need to do is download the OVA from the WMware Flings website.
https://flings.vmware.com/vmware-appliance-for-folding-home

Step 3: Deploy

Workstation

1. Double-click on the OVA file you download to launch VMware Workstation. It will present you a wizard to “Import Virtual Machine”.
Enter a name and file path for your F@H appliance, then click ‘Next’.

2. Now to work down the options from the left pane…

-Enter a hostname
-Enter an IP address (leave blank if DHCP)
-Gateway
-DNS

-Enter password for the appliance; VMware1!
*This is the root password for the appliance

-Enter you F@H username
-Enter your F@H team (Leave as 52737 to contribute as part of VMware’s team)
-Enter your Passkey
-GPU (If using a GPU change to TRUE, if you are using a virtual machine with a GPU, this must be in passthrough mode)
-Enter F@H management networks info (can probably leave alone)
-F@H password defaults to the OS password (VMware1!)

Then click ‘Import’.

Go ahead and use my F@H username & passkey if you really want to fold as me… It just means my F@H user will get credit for any folding you do.

3. Once the import is complete, it should automatically power on. Go ahead and power it off. The first thing I recommend to do is upgrade the VM.

Click on “Upgrade this virtual machine” and follow the wizard to upgrade it to the highest version that is compatible in your environment. For me, it is Workstation 14.x.

Because this is an OVA file and so easy to re-deploy if I screw something up, I choose to just alter the VM, and not make a clone.

4. Next step is to edit the VM and add more CPUs, if desired. Click on “Edit virtual machine settings”.

Click on ‘Processors’. From the “Number of processors” dropdown you can choose how many processors you want to dedicate to this appliance. Then click ‘Ok’.

5. Go ahead and power on your F@H appliance.

ESXi / vCenter

1. In vCenter or on your ESXi host, right click on your Datacenter/Cluster/Host and select “Deploy OVF Template”.

2. Select the OVA file you downloaded earlier, and click ‘Next’.

3. Give your VM appliance a name, and click ‘Next’.

4. Walk thru the rest of the wizard. Choose your computer resource you wish to deploy it on to. Review the details. Select your storage. Select your network.

5. Customize the F@H template setting for your environment.
-Hostname
-IP address
-Gateway
-DNS
-OS ‘root’ password
-F@H username
-F@H passkey
-GPU
-F@H remote management password

Then click ‘Next’ and ‘Finish’ to deploy your new appliance.

6. Once deployed, make sure the vm is powered off. Right click on the vm and select ‘Edit Settings…”. Select CPU and from the dropdown adjust the CPU to the desired number you wish to dedicate to your appliance, and click ‘Ok’.

7. Power on your F@H vm and you are ready to start contributing.

Step 4: Troubleshooting

Once your appliance is up and running, there are a few command that you will find helpful.

Start and Stop
/etc/init.d/FAHClient start
/etc/init.d/FAHClient stop

Restart
/etc/init.d/FAHClient restart
Status
/etc/init.d/FAHClient status
Check the Logs
/etc/init.d/FAHClient log -v
Check CPU stats
top

With the huge growth of contributors to F@H, it has made getting work units more difficult. If you check your logs and see messages similar the what is in my screenshot below, then your appliance IS working, it is just waiting for work.

Leave it running and you’ll eventually see it start chugging along when it gets a work unit.

Also, on the F@H fling website you can also find two PDFs, one about deployment and another with FAQs. Give those a look if you run into any other issues.

16 January 2020

Upgrade ESXi host using vCenter Update Manager

VMware Update Manager (VUM) is a component of vCenter has matured a lot as product since its introduction. One of my favorite parts about it is that it is now baked into the vCenter Server Appliance (VCSA). Using it makes the process of updating and patching your hosts a simple centralized task. Here we are going to use VUM to upgrade our ESXi host from v6.0 to v6.7.

1) The First thing we need to do is check the infamous “VMware Product Interoperability Matrices” to find our upgrade path. Looking at the matrix, we can see that we should be able to upgrade from any v6.0 release directly to v6.7u3.

Sometimes, depending on which version of ESXi you are running, you may not be able to jump straight to the latest release. You might need to do an incremental upgrade, stepping up through versions, until you finally land on your ‘final’ version. 

For example, in the picture above we can see that ESXi v5.5 does not have a direct upgrade path to v6.7. You would first have to upgrade the host to v6.0 or v6.5, before you make another jump to v6.7.

2) The next step is to download the bits of your ESXi iso. There are a couple places you can grab this iso from.

a) The first place you can get it from is directly from VMware.  Login into your VMware portal, in the right hand corner under “Quick Links” click on ‘Download Products’. In the search bar at the top of the page enter “ESXi” and you should see what we’re looking for in the auto-suggested dropdown, click on it.

Now look for the keyword “Product Binaries” in the search results, there’s a lot of other stuff like drivers listed, which we’re not looking for right now. Click the link, you’ll see another page with the actual download links.

b) For most folks, the above-mentioned way to find and download the ESXi iso will be fine. However, if you are running Dell hardware, then you have a second option; Download and use the Dell Customized ESXi image. It’s an ESXi iso that has all the Dell specific drivers pre-baked into it.

The Dell customization image includes: modifying files to identify Dell and Dell support, updating the ESXi image with the latest validated asynchronous drivers, and updating patches if it addresses a common issue for Dell systems.

To find this customized iso image, go to https://support.dell.com/ and enter your product serial number.

Click on the “Drivers & Downloads” tab.

Choose “Enterprise Solutions” in the Category field.

Expand the VMware ESXi link and click download on the desired version for your server.

3) Now we are finally ready to start using VUM, and we begin by importing our iso image we downloaded in the previous step. Login into your vCenter Server. From the menu, click on “Update Manager” -> “ESXi Images” -> “Import”.

Click the “Browse” button to navigate to and select the ESXi 6.7 iso image you downloaded previously. This will start uploading the file into VUM.

4) With our iso image now in vCenter, the next step is creating the baseline that vCenter will use to check if hosts are in compliance. Select the image you just imported, and click on “New Baseline”

Give your new baseline a name, such as “ESXi 6.7u3 Upgrade”, and click ‘NEXT’.

Make sure the image you imported is the one you have selected and click ‘NEXT’ again.

Verify all the summary details are correct and click ‘FINISH’.

Select “Baselines”, you will see the baseline that we just finished creating.

5) The next step is to attach the baseline we just created to an object. The two objects we can attach our baseline to is either our cluster or individual host. I’m going to show how to do it on a cluster object, but the process is the same for a host.

From the “Hosts and Clusters” view in vCenter, select either your cluster object. Go to the “Updates” -> “Host Updates” -> “Attached baselines” -> “Attach” -> “Attach Baseline or Baseline Group”

Select the baseline we created earlier and click ‘ATTACH’.

6) With our baseline attached to our cluster or host object, we are finally ready to upgrade the ESXi host itself. Or in VUM terms – “remediate the host”.

Go to the “Updates” -> “Host Updates” -> “Attached baselines”. Select the baseline you want to apply, in our case it’s the “ESXi 6.7u3 Upgrade”, and then click on “Remediate”.

Click on “Accept the license agreement” -> Click on “Next” to accept all the default options before upgrading ESXi 6.x to ESXi 6.7 -> Click on “Finish” to begin the upgrade. Your host should go into “Maintenance Mode” before it applies it’s remediation steps and completes the upgrade, click “OK”.

7) It’s time to sit back and watch the status bar do it’s thing.

8) Once the remediation has completed, you will be able to verify that your host has been upgraded and is now in compliance.