Performance issues HP Microserver gen8 and VMWARE EXSI 6.5

upgraded ESXI to 6.5 U1 which is version 6.5.0 build 5969303

You are running HPE Customized Image ESXi 6.5.0 version 650.10.1.0.47 released on July 2017 and based on ESXi 6.5.0 Vmkernel Release Build 5310538

unfortunately the HP Microserserver Gen8 is than running with: HPE_bootbank_scsi-hpvsa_5.5.0.102-1OEM.550.0.0.1331820

As I was (and still am but as of writing I am in maintenance mode) seeing spikes in my CPU usages. It might be caused by the bad performance

checking with:
cd /vmfs/volumes/datastore1
time dd if=/dev/zero of=tempfile bs=8k count=1000000

It took a very long time to see some output. (Very long time!)

than I tried to downgrade to the hpvsa-5.5.0-88.zip driver

guidelines:

  1. enter maintenance mode
  2. I do a reboot, but you can do it probably without
  3. copy the downloaded driver to /tmp/ and run the following command
  4. esxcli software vib install -d /tmp/hpvsa-5.5.0-88.zip
  5. the old driver will be removed and the 5.5.0-88 driver installed
  6. Now important: if you do reboot this way: you will not see your DataStores anymore, only your NFS datastores (in my situation) this is caused by VMWARE ESXI as it will be using vmw_ahci driver for the datastore.
  7. so disable the usage of this ‘default’ driver: esxcli system module set –enabled=false –module=vmw_ahci
  8. now you can reboot and your device will be using the 5.5.0-88 driver
  9. you can see this by using; cat /proc/driver/hpvsa/hpvsa0

the speed should have been returned.

 

The hassle of upgrading ESXI 6.0 to ESXI 6.5 on a HP Microserver gen8

Wauw ..

yesterday and today I tried to upgrade my HP Microserver Gen8 from VMWARE ESXI 6.0.0 to 6.5
what a trouble ..

Steps:
shut down all vms (hosts) and enter maintenance mode. If you do this there are a number of online blogs with help to install from online depots but in all my tests it was too slow or I was to impatient to wait to end. But in the end I always ended up with an system with errors:

the transaction is not supported: VIB Hewlett-Packard_bootbank_scsi-hpvsa_5.5.0-88OEM.550.0.0.1331820

Do not try to force the installation. You will end up with an system where it seems that your EXSI is updated to 6.5 but actually is is running in a ramdisk environment. Easy to see cause you have lost your datastores (NFS datastores are still mounted).

If you than reboot: you are back to 6.0.0 .. so how to solve this:

easy: when entering maintenance mode: reboot your machine. In many guidelines this part is not mentioned.

After I rebooted I first tried the online depot installation documentation but ended up waiting and waiting. I got a VMWARE image for update from VMWARE but I got all kinds of different issues again:

“The upgrade contains the following set of conflicting VIB” When using the standard image of VMWARE: many conflicting vibs a no go for me. So I read some blogs and I found out that it is best to keep using your HPE image vmware files. (So in short: if you used the HPE VMWARE ESXI ISO installing Esxi onto your HP Microserver Gen8 keep using the update files with HPE in it and not the plain VMWARE onces. It can give issues !

After I used the HPE image I only got 1 issue: one vib was still complaining. On this blog I read what I needed to do: remove this vib (partner supported, so not native).

After removing this VIB I could upload the various ZIP bundle files like

VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017-depot.zip
VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017-depot.zip

with the command:
esxcli software vib install -d “/vmfs/volumes/datastore1/patch-directory/VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017-depot.zip”

I was able to install finally the 6.5 version. Due to issues of Storage driver I immediately updated to the U1 update
with the command:

esxcli software vib update -d “/vmfs/volumes/datastore1/patch-directory/VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017-depot.zip”

the blog nxhut showed me some info that with the U1 version the storage speed performance issues should be over.

A simple test downloading a file towards a VM showed me a 18MB/s so the 10MB/s barrier was not seen.

A last reboot and get the machine out of maintenance mode. Finally I have a running 6.5U1.