Comments (1)

By admin, October 20, 2010 4:39 pm

Count Down is on the way, only 6 hours left, it is probably the most anticipating problem in VMware ESX backup world in 2010!

I’ve posted a question in Veeam’s forum asking the following and gained lots of knowledge regarding How Veeam backup works over SAN as well as Equallogic stuffs.

Veeam v5 can tell EQL to take a snapshot locally first using VAAI (ie, super fast), then send that Completed snapshot to Veeam Backup server for de-dupe and compression and then finally save to local storage on the backup server.

=====================================
For example, we have three kind of backup running at 6AM daily AT THE SAME TIME.

VMs are all on Equallogic SAN VMFS volume.

1. Acronis True Image backup inside each VM. (ie, File Level, backup time is 5-10 mins per VM)
2. Veeam Backup SAN (ie, Block Level, backup time is 1-5 mins per VM)
3. Equallogic Snapshot (ie, Block Level, backup time is 1-5 seconds for the whole array)

Will this actually create any problem? I mean LOCKING problem due to concurrent access to the same volume?

But Beginning with Equallogic version 5.0, the PS Series Array Firmware supports VMware vStorage APIs for Array Integration (VAAI) for VMware vSphere 4.1 and later. The following new ESX functions are supported:

• Harddware Assisted Locking – Provides an alternative meanns of protecting VMFS cluster file system metadata, improving the scalability of large ESX environments sharing datastores.

So this shouldn’t be a problem any more.

Any hints of recommendation?

I feel very happy now as finally VAAI can be used to greatly increase the snapshot performance and shorten the backup window, together with Veeam’s vStorage API, I am pretty sure the backup time is going to break the records shortly, I will report later after I installed B&R V5 tomorrow.

One more thing, I did try to use ASM/VE to backup and restore VM once for testing, I forgot if it’s with VAAI or not or if the FW is 4.3.7, but it’s quite slow and one thing I also don’t like is to backup snapshot onto the Same Array!

1. Array space is too expensive to place snapshot on it, at least for VM, I am ok with taking array snapshot for lun/volume only though to avoid array failure kind of disaster.

2. It’s not safe to place VM snapshot on the same array, what if the array crashes?

3. As an EQL user, everyone know that “Dirty Bit” problem (ie, once the block is written, there is no way to get it back.) In other words, it’s a great waste of deleted/empty space. Not Until EQL releases the Thin/Thick Space Reclaim feature in the coming FW5.x version, I think the technology is still not mature or ready yet to have VM snapshot to be placed on EQL volume. FYI, 3PAR is the only vendor having true Thin Space Reclaim, HDS reclaim is a “fake ” one, search google about a 3PAR guy commenting about HDS’s similar technology you will see what I mean.

4. Remember EQL use 16MB stripe? It means even there is 1K movement in block, EQL array will take the whole 16MB, so your snapshot is going to grow very very very very huge, what a waste! I really don’t understand why EQL designed their stripe size to 16MB instead of say 1MB, is it because 16MB can give you much better performance?

5. Another bad thing is even you buy a PS6500E with lots of cheap space, but you still can’t use ASM/VM to backup a VM on PS6000XV volume and place the snapshots on PS6500E, it HAS TO be the same array or pool and stay at PS6000XV, so seem there is no solution.

That’s why we finally selected Veeam B&R v5 Enterprise Edition and that’s how I arrived here and encounter all the great storage geeks!
=====================================

Some of the very useful feedbacks: 

 The EQL Snapshot is done on the EQL Hardware Level. The vmware snapshot is done at vmware level, with vaai hardware assisted but not done on the hardware level. (BUT if it is exactly that what you want DELL provides a cool tool named auto snapshot manager vmware edition, this will allow you to trigger a hardware snapshot exactly the same time a vcneter snapshot is triggered – good for volume shadow copy consistency).

The VEEAM Snapshot is done with vCenter Server or ESX/i directly using VAAI or not (dependes on your firmware and esx/i version). Don´t mix around Hardware Snapshots and hardware assisted snapshots, is is NOT the same.

 

That is correct, but at a very small time window, VAAI kicks in, when the snaphot is triggered, at this time VAAI kicks in with the locking mechanism. But then again, don´t mix it up with a hardware snapshot, VAAI can NOT trigger SAN-Vendor-specific hardware snapshots.

You can quite do a lot with Equallogic SAN, if you want it. There are many ways allowing you to trigger hardware snapshots if you want them triggered. So i suggest you take a look at

a) Auto Snapshot Manager VMware Edition
b) latest Host Integration Tools (H.I.T.)

 

Correct, but VEEAM/vmware-triggered snapshots, when doing backups at night, would not really grow that much because the time window for doing the backup is extremely small (when using high speed lan and cbt) – therfor me personally i have no problem with it – and even if it gets a problem: VEEAM has sophisticated mechanisms which allow you to safely break the backup operation if the snapshot grows too huge.

 

Update: Official Answer from Equallogic

Good morning, 

So, the question is does VMware’s ESX v4.1 VAAI API allow you to have one huge volume vs. the standard recommendation for more smaller volumes while still maintaining the same performance?

The answer is NO.  

Reason: The same reasons that made it a good idea before, still remain.   You are still bound by how SCSI works.  Each volume has a negotiated command tag queue depth (CTQ).  VAAI does nothing to mitigate this.   Also, until every ESX server accessing that mega volume is upgraded to ESX v4.1,  SCSI reservations will still be in effect.  So periodically, one node will lock that one volume and ALL other nodes will have to wait their turn.   Multiple volumes also allows you to be more flexible with our storage tiering capabilities.   VMFS volumes, RDMs and storage direct volumes can be moved to the most appropriate RAID member.

i.e. you could storage pools with SAS, SATA or SSD drives, then place the volumes in their appropriate pool based on I/O requirements for that VM.

So do you mean if we are running ESX version 4.1 on all ESX hosts, then we can safely to use one big volume instead of several smaller ones from now on? 

Re: 4.1. No.  The same overall  issue remains.  When all ESX servers accessing a volume are at 4.1, then one previous bottleneck of SCSI reservation and only that issue is removed.  All the other issues I mentioned still remain.   Running one mega volume will not produce the best performance and long term will be the least flexible option possible.   It would similar in concept to taking an eight lane highway down to one lane.

In order to fully remove the SCSI reservation, you need VAAI, so the combination of ESX v4.1 and array FW v5.0.2 or greater will be required.

As a side note, here’s an article which discusses how VMware uses SCSI reservations.  

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005009

Here’s a brief snippet from the KB.

There are two main categories of operation under which VMFS makes use of SCSI reservations.

The first category is for VMFS data-store level operations. These include opening, creating, resignaturing, and expanding/extending of VMFS data-store.

The second category involves acquisition of locks. These are locks related to VMFS specific meta-data (called cluster locks) and locks related to files (including directories). Operations in the second category occur much more frequently than operations in the first category. The following are examples of VMFS operations that require locking metadata:
    * Creating a VMFS datastore
    * Expanding a VMFS datastore onto additional extents
    * Powering on a virtual machine
    * Acquiring a lock on a file
    * Creating or deleting a file
    * Creating a template
    * Deploying a virtual machine from a template
    * Creating a new virtual machine
    * Migrating a virtual machine with VMotion
    * Growing a file, for example, a Snapshot file or a thin provisioned Virtual Disk

Follow these steps to resolve/mitigate potential sources of the reservation: 

a.Try to serialize the operations of the shared LUNs, if possible, limit the number of operations on different hosts that require SCSI reservation at the same time.

b.Increase the number of LUNs and try to limit the number of ESX hosts accessing the same LUN.

c.Reduce the number snapshots as they cause a lot of SCSI reservations.

d.Do not schedule backups (VCB or console based) in parallel from the same LUN.

e.Try to reduce the number of virtual machines per LUN. See vSphere 4.0 Configuration Maximums and ESX 3.5 Configuration Maximums.

f.What targets are being used to access LUNs?

g.Check if you have the latest HBA firmware across all ESX hosts.

h.Is the ESX running the latest BIOS (avoid conflict with HBA drivers)?

i.Contact your SAN vendor for information on SP timeout values and performance settings and storage array firmware.

j.Turn off 3rd party agents (storage agents) and rpms not certified for ESX.

k.MSCS rdms (active node holds permanent reservation). For more information, see ESX servers hosting passive MSCS nodes report reservation conflicts during storage operations (1009287).

l.Ensure correct Host Mode setting on the SAN array.

m.LUNs removed from the system without rescanning can appear as locked.

n.When SPs fail to release the reservation, either the request did not come through (hardware, firmware, pathing problems) or 3rd party apps running on the service console did not send the release. Busy virtual machine operations are still holding the lock.

Note: Use of SATA disks is not recommended in high I/O configuration or when the above changes do not resolve the problem while SATA disks are used. (ie, USE SAS 10K or 15K or EVEN SSD should greatly help!)

An updated review from InfoWorld about New EqualLogic firmware takes a load off VMware

Comments (1)

By admin, October 20, 2010 11:32 am

To show Glbal TCP Parameters:
netsh int tcp show global

1. How to enable and disable TCP Chimney Offload (aka TCP offload) in Windows Server 2008 R2:
netsh int tcp set global chimney=enabled
netsh int tcp set global chimney=disabled

Determine whether TCP Chimney Offload is working, type “netstat –t” the line shows “Offloaded” is with Offloaded feature enabled.

2. How to enable and disable RSS in Windows Server 2008 R2:
netsh int tcp set global rss=enabled

3. Disable TCO Autotuninglevel in Windows Server 2008 R2 for performance gain in iSCSI
netsh interface tcp set global autotuninglevel=disabled

Update Jan-24:

I simple enabled everythingand found there is no difference on Equallogic iSCSI IOMeter performance.

TCP Global Parameters
———————————————-
Receive-Side Scaling State          : enabled
Chimney Offload State               : enabled
NetDMA State                        : enabled
Direct Cache Acess (DCA)            : enabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : ctcp
ECN Capability                      : enabled
RFC 1323 Timestamps                 : disabled

ECN is Explicit Congestion Notification and is enabled by default and is a little more complex as it tweaks TCP protocol when sending a SYN and is mostly used by routers and firewalls. Since the default is enabled, I’d just set it to the default and be done with it.

Comments (0)

By admin, October 19, 2010 12:08 pm

Dell has announced it’s upcoming virtual appliance “Dell Management Plug-In for VMware vCenter (aka DMPVV)”, basically, it’s an integrated OpenManage to vCenter and best of all, we can upgrade BIOS/Firmware directly from vCenter, no more downtime using USC or other method. (ie, works just like in Windows environment, in-line upgrade without shutting down ESX, of course, you need to reboot it after the upgrade is completed.)

I do hope Dell will make it a FREE PRODUCT for all its Poweredge Customers.

Why? Why NOT?

• Most of us already installed OpenManage on ESX, so we can see most of the server hardware data like temperature, warning, configure storage, etc.
• ESX also provides an extensive Hardware Health section
• Using the free Veeam Monitor 5 also can give you the above with tracking as well
• Host Profile is also available in ESX Enterprise Plus version
• Warranty Information can be found easily on-line

So why would someone pay for a nicer GUI ? Ok, you may argue that BIOS/Firmware upgrade thing, well, I don’t mind do it in USC, it’s even safer as I am upgrading BIOS/Firmware at lowest system level.

Update March 6, 2011

Installing and Configuring the Dell Managment vCenter Plugin – The Dell TechCenter

Update July 19, 2011 

Trial Version (Latest v1.0.1)

Comments (0)

By admin, October 18, 2010 1:02 pm

Best Method Learned: DO NOT TRY TO RE-SIZE or CONVERT THE SOURCE DISK TO THIN when using vCenter Converter Standalone, if you want to reduce the size of the original disk, then do this using Storage vMotion after the conversion is completed, see details below:

I HAVE RELALY BEEN THERE AND WENT THOUGH ALL THE PAIN ONCE, now I am happy after I did this intensive research on P2V using backup image.

8:20PM Start, disconnect physical server external network cables connecting to the Internet, leaving internal LAN on for VPN, so I am still able to RDP into the server.

8:36PM Take a final incremental snaphot on the physical server using True Image Server Echo (TIS9.1/TIS9/TIS8 all having some kind of wired problem during conversion, so better upgrade the TIS version on the physical server first)

8:40PM Since I’ve already copied all the previous TIS backup files to the vCenter server beforehand, so it only took me 4 mins to copy this final incremental backup file over.

9:04PM Use vCenter Converter Standalone to import TIS backup files and converted into Equallogic SAN volume (which is Thin-Provisioned), it’s so fast, 120GB in about 24 mins (about 83MB/s over 1Gbps). If you are going to do Re-size or Make it Thin at this stage, the conversion process is going to take at least 3-4 times longer and still failed at the end even you are able to boot the VM (You will either face login failure with correct password or some service will NEVER START for un-known reasons), so my suggest would be just KEEP THE SOURCE AS THICK (aka flat) and do all the THIN later, keep reading please.

10:08PM Finished re-configuring VM, disabled all the unnecessary services, uninstall all no need software and drivers (luckily I still have that physical running side by side in the above for comparison just in case some configuration need to be made), finally the VM is running on Version 7, with VMXet3 driver and using Para-virtualized for Disk Controller (this really boost the performance by at least 3 to 5 times, booting the same W2K3 physical server required 7 mins before, using normal SCSI Parallel is about 4 mins, using Para-virtualized is about 30 seconds!!! Wow!!! Of course, I am not sure if it’s to do with ESX4.1 and EQL FW5.0.2 supports Storage Hardware Acceleration and MEM. Everything is just so much faster than before. However you still have this WASTED big 120GB THICK disk that’s converted form a RAID-5 (36GBx5) Physical Server.

10:20PM Now is where the magic happens. I then did a Storage vMotion (ie, migrate the datastore) to the real VMFS volume where it should be (it was on a temp VMFS just for conversion), AND NOW YOU CAN USE THIN-PROVIIONING, guess how long does it take for Equallogic PS6000XV to do that? Yes, total 12 mins to migrate that 120GB thick and converted to thin (about 30GB)! Amazing! Again, I am pretty sure it’s something to do with ESX4.1 and EQL FW5.0.2 supports Storage Hardware Acceleration and MEM, that VAAI storage hardware acceleration thing is really kicking in!

So the whole thing took about 2 hours, seemed easy right? However I didn’t tell you I’ve spent the whole week about 30 hours to do many trial and errors with different conversion method things like use TIS Echo to convert the TIBs into VMDK (WASTE OF TIME, DON’T TRY IT), Boot TIS rescure media and then directly restore the image into VMDK (waste of time), re-size or thin the source disk and then found out I can no longer login and even I can login, some important services just never start. What about real-time conversation, FORGET IT! It will take FOREVER (so slow) and you will encounter even more problems then using a backup image.

So to put things into summary, here is what you need to do if you have True Image Server or other backup files for P2V conversion. 

1. Use vCenter Converter Standalone and simple import the TIB files, do not make it thin and do not re-size it. This proved to be the FASTEST conversion speed. (about 85-95MB/s over a 1Gbps COS link)
2. Then use Storage vMotion to do the actual Thin-Provisioning, if you have VAAI storage hardware acceleration, it will really help in this stage. You will see under esxtop iSCSI network traffic is really low, but IOPS is really high, so you know VAAI is working hard on the array itself and ESX Host CPU is almost nothing during the Storage vMotion as ESX Host no longer required if VAAI or SAN OFFLOAD is being used.
3. Make sure you use Echo version of True Image Server to ensure the most compatible or safe conversion for your VM. TIS9.1/TIS9/TIS8 just created too many unknown problems

 * One more thing, DO NOT TRY TO CONVERT TWO VM at the same time if the TIS backup images are on the same local harddisk partition, it’s just too much for your server and somehow vCenter Converter will lock TIS files of one VM, so when you try to convert the 2^nd VM, the size will somehow misleading (ie, it’s actually 30GB used, but it will show 120GB fully used, I think it’s related to vCenter Converter can only lock one TIB file at the same time. If you try to force converting the 2^nd VM, strange thing DO HAPPEN, so just take your time, have a cup of coffee of tea, sit back and relax.)

Comments (5)

By admin, October 16, 2010 10:03 am

I am having the following problem on our Virtual Center, if you know how to solve this, please kindly let me know, many many thanks in advance!

EventID 1000[VpxdLdap] Failed to search OU=Instances container.  This may indicate a problem with LDAP permissions for the account running VirtualCenter, or that the schema is not compatible with this version of VirtualCenter.

The error occur on the clock and every 15 mins after the clock (ie, 9am, then 9:15am, then 10am, then 10:15am)

It only happens when
1. Running vSphere Client and leave it on (1-3 times a day)
2. Occur 24 times a day if we have vSphere Client on as well as Veeam Monitor on, seem Veeam Monitor is competing with vSphere Client for pulling resources, so that’s why the error occurs more often.

Then vCenter server alarm section will periodically produce alerts saying vCenter Health Status is in YELLOW due to LDAP server cannot be contacted because I am not joining an AD Domain, this sounds ridiculous.

Btw, the vCenter server DID NOT JOIN A DOMAIN, only using the same server’s Workgroup, I know it’s not right or the best way according to vCenter setup guide, but I really want to keep it simple. (ie, do not want to have another physical server just for AD), I really wish VMware will release a patch for vCenter that allow us to select Domain or Workgroup model during installation or even better allow us to change the option on the fly.

I suspect it’s a client pulling problem and/or the client can’t search through AD/LDAP, so it reports such error?

It’s just a warning error, nothing really affecting operation, so I think I can safely ignore it, but do appreciate if someone came across and solved this strange problem.

Update:

From vCenter Error Log:

[2010-10-24 04:19:24.791 05976 error 'App'] [LDAP Client] Failed to poll search: 0×0 (The call completed successfully.)
[2010-10-24 04:19:24.791 05976 warning 'App'] [LDAP Client] Reinitializing search -1 (ou=Licenses,ou=Licensing,dc=virtualcenter,dc=vmware,dc=int)
[2010-10-24 04:19:24.791 05976 error 'App'] [LDAP Client] Failed to perform asynchronous search for base DN = ou=Licenses,ou=Licensing,dc=virtualcenter,dc=vmware,dc=int: 0×51 (Cannot contact the LDAP server.)

[2010-10-24 08:11:56,116 Timer-4  INFO  com.vmware.vim.jointool.util.ldaphealth.LdapHealthMonitor] Encountered an error when checking domain trust health : error code: $@, result: 1717
From vCenter Health Check:

Ldap domain trust change monitor – Warning – encountered an an error when checking domain trust health: error code: 1717

Solution:

From VMware Communities:

The message “Encountered an eror when checking domain trust health: error code 1717″ is simply an informational message in Virtual Center. The “vCenter Service Status plugin for Virtual Center 4″ runs some LDAP checks including checking for the possibility to perform domain trust lookups. When it cannot perform this domain trust lookup then it will show this message.

This message is simply an informational message and should have no major impact on the running of the Virtual Center Server. The only ways to stop this message from appearing would be joining vCenter Server to a AD Domain. Btw, you CANNOT install AD Domain Controller on the same machine with vCenter, it will not work. Because vCenter 4.1 will install an instance of ADAM (Active Directory Application Mode). It uses this when you use vCenter Linked Mode and ADAM will conflict with its’ own AD services if the server is also a Domain Controller.

From ESX 4.1 vCenter Installation Guide:

The system that you use for your vCenter Server installation must belong to a domain rather than a
workgroup. If assigned to a workgroup, the vCenter Server system is not able to discover all domains and
systems available on the network when using such features as vCenter Guided Consolidation Service. To
determine whether the system belongs to a workgroup or a domain, right-click My Computer and click
Properties and the Computer Name tab. The Computer Name tab displays either a Workgroup label or
a Domain label.

Seemed there is no workaround for running vCenter on standalone Workgroup, but why would I use an extra physical machine for the sole purpose of running an AD Domain Controller? It’s TOTALLY AGAINST VIRTUALIZATION and it’s not Green at all, most of all if I have a small enviornment with less than 5 ESX Host, why would I bother to setup a AD?

My own solution would be disable vCenter Health Check alarm or just simply remove the part saying Health Check changed to Yellow should be fine.

Finally, some people may install vCenter on Windows Server 2008 R2 and encounter the following problem, according to VMware KB1025668.

Installing vCenter Server 4.1 on a Windows 2008 R2 system fails

Symptoms
•Cannot install vCenter Server 4.1 on a Windows 2008 R2 system
•Installing vCenter Server 4.1 on a Windows 2008 R2 system fails
•You see on of these errors:

◦The trust relationship between this workstation and the primary domain failed in the jointool-0.log
◦Setup cannot create vCenter Server directory Services Instance
Resolution
This issue may occur if the Active Directory in your environment is hosted by a Windows 2000 domain controller (THAT’S OLD!!!). This issue occurs because vCenter Server 4.1 is unable to retrieve the security identifier (SID) for an account.
�
To resolve this issue, you must apply a Microsoft hotfix. For more information and to download the hotfix, see the Microsoft Knowledge Base article 976494.
�
Note: You must reboot the system before installing vCenter Server again.

Comments (0)

By admin, October 15, 2010 11:33 am

Just found a great article for ESX Storage Best Practices from Cisco, definitely worth reading for understanding how storage really works in VMware vSphere

At the end, it also mentioned the future: VAAI, well the paper was written in 2009 and one year after, we are already using it in our Equallogic SAN.

There is even a section called “Day After Tomorrow”, future technologies like vMotion between Datacenters, DRS and DPM for storage, etc.

Comments (0)

By admin, October 12, 2010 1:04 pm

Basically, all you need to update the SVGA driver to WDDM driver, but why didn’t VMware include that in its latest VMware Tools?

Troubleshooting SVGA drivers installed with VMware Tools on Windows 7 and Windows 2008 R2 running on ESX 4.0

WDDM and XPDM graphics driver support with ESX 4.x, Workstation 7.0, and Fusion 3.0

Comments (0)

By admin, October 12, 2010 9:26 am

So you have been there and encountered that annoying thing, you’ve called Dell Pro-Support and they replied you there is DEFINITELY NO WAY and you also called Microsoft, finger pointing back to Dell by asking you to contact Dell directly as it’s OEM product. You have asked local Microsoft distributor, they also said there is no way to do it, you have to buy Box set or Open License, your existing Dell OEM license will not allow you to reactivate using the key printed on it.

Well, THEY ARE ALL WRONG!!!

• Dell’s Pro-Support is unprofessional in this case.
• Microsoft is responsible for its own product, NOT!
• Local Microsoft Distributor wants you to pay more, huh?

This is the Official Solution from Dell, hope it’s useful for others, the key point is to use Virtual Key to re-activate, and then either activate on-line or use phone to activate again and finally clone it as master gold image for further deployment.

You cannot automatically pre-activate the Windows Server 2008 operating system installed on VMs by using the product activation code in the Dell OEM installation media. You must use the virtual product key to activate the guest operating system. For more information, see the whitepaper Dell OEM Windows Server 2008 Installation on Virtual Machines using Dell OEM Media at dell.com.

I always thought Virtual Key is for Microsoft’s own Hyper-V only and cannot be used in VMWare enviornment, but I was wrong.

Alternatively, you can force the VM to load the default BIOS containing DELL SLIC 2.1 (supports Windows 7 and Windows Server 2008 R2), which will trick the VM thinking it’s actually a PHYSICAL DELL server.

1. Simply add bios440.filename = “DELL.ROM” to VM configuration parameters by using VC Client, of course you do need to upload DELL.ROM to your VM directory and please don’t ask me where to get that DELL.ROM, goole it around yourself. One draw back is this VM can’t be vMotioned around, as the DELL.ROM won’t get vMotioned. (Update: Solution to vMotion is to put DELL.ROM on every host or simply on SAN such as bios440.filename = “/vmfs/volumes/san/DELL.ROM”.

2. Very importantly, you will also need to find the corresponding certificatedell.XRM-MS, then use slmgr.vbs -ilc c:\dell.XRM-MS to import the certificate.

3. Insert the Key by slmgr.vbs -ipk XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

Finally, some say even by adding SMBIOS.reflect = True will work, but I COULD NEVER get this method working!

Update: The reason I didn’t get it working is because I didn’t use Dell’s W2k8R2 installation disk, see this link from IBM, sounds so simple! Really?

Solution

Edit the virtual machine’s .vmx file to contain the following line:

SMBIOS.reflectHost = “true”

Note: Encoding of the text added to the .vmx file must be in UTF8.

This updates the virtual machine BIOS with the IBM Original Equipment Manufacture (OEM) information required to use IBM-provided Operating System (OS) installation media.

IBM-provided Microsoft Windows 2008 media must be “BIOS Locked” to ensure that the OS will only install on IBM hardware. Virtual machines use a virtual BIOS that does not contain information that identifies the system as being manufactured by IBM.

The installation of Microsoft Windows Server 2008 from IBM OEM media to such a virtual machine will fail until the virtual BIOS has been updated to include this information. Alteration of the virtual machine’s .vmx file to state SMBIOS.reflectHost = “true” performs this function for servers using VMware’s ESX/ESXi technology.

The workaround resolves this issue by using media that is not locked to a specific OEM.

The solution resolves this issue by adding IBM information to the virtual BIOS.

Update Apr-16

Tried again today, the method SMBIOS.reflectHost = “true” is DEFINITELY NOT working! Even loaded with Dell’s OEM w2k8r2 std installation disk and the server is Poweredge R710, it still asked for activation. In additional, I discovered I can install Dell’s OEM w2k8r2 std disk on VM even without SMBIOS.reflectHost = “true”, so this means Dell’s w2k8r2 disk can be used on a non-Dell server.

So only the above two methods are working, but not the last one, if you got the last one working, pls drop me a line, thanks.

Update Apr-17

May be the answer is SMBIOS.reflectHost = “true” WILL ONLY WORK for ESX 3.5 or before, as VMware’s KB didn’t indicate this method apply to ESX 4.0.

Comments (0)

By admin, October 11, 2010 11:15 am

After upgrading to ESX4.1, my VM with latest Version 7, VMware PVSCSI and VMXNet3 starts to show “Safely Remove Hardware” alert in tray, but why would you want to remove your harddisk and NIC? Huh?

Then I found this useful link, case solved!

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012225

Comments (0)

By admin, October 10, 2010 12:42 pm

About a month ago, I was told by an experienced Dell Equallogic Consultant to use Normal (non-thin) on EQL array and Thin on ESX VMFS, I wasn’t exactly sure what did he mean by then.

So I did a simple test on my EQL box:

Create a 10GB volume (non-thin), attach to Windows, write 5GB, then remove 4GB, leave with 1GB, to EQL it’s 5GB used.

Then I write another 4GB, EQL still reports 5GB, then I write 1GB more, now EQL reports 6GB.

However in my Thin Provisioning test for the above same 10GB, case looks completely different now.

Create a 10GB volume, attach to Windows, write 5GB, then remove 4GB, leave with 1GB, to EQL it’s 5GB used.

Then I write another 4GB, EQL somehow EQL volume reports the size continously growing to 5GB, then 6GB, then finallyBbang 9GB. WHY? WHY doesn’t it use the UNUSED SPACE? (actually inside to Windows, it’s still 5GB, you will see later)

HOWEVER, Please note THIS, as I continue to add another 4GB to the volume (now EQL reports 9GB, windows reports 5GB), then EQL reached 10GB max (somehow the volume didn’t go offline? why? I don’t know), but I can still add this 4GB to the volume, and windows reports 9GB/10GB used.

So in a strange way, even EQL reports the volume has been fully used, we can still add data to it at Windows level, but it’s just TOTALLY CONFUSING and false volume is going to full alarm all the way when using Thin Provisioning.

That’s why WAIT UNTIL FW5.0.x or FW5.x coming out with the REAL THIN RECLAMATION feature like what HDS’s or 3PAR’s did a year ago. (Yes, EQL is behind in this particular area) 

We are probably better to NOT USE Thin Prov. in ESX, what I mean is to

Use Thick Prov. in EQL, but Thin in ESX VMFS would be the best way.

For snapshots, just set it to a smaller % during the volume creation (10% would be good, as you can always grow it later), this apply to the volume as well, make your own Thin Provioning, just set the volume to a smaller size when you first create it, then gradually expand it as you need later, then you won’t waste a lot of space from the beginning.

Update:

I did another test and it proved I was wrong above.

The GB are reported in EQL Group Manager under Volume Used Size

  Thick (20GB) Thin (20GB)
——————————————-
1. +5GB  5GB  5GB
2. +5GB  10GB  10GB
3. -5GB  10GB  10GB
4. +5GB  10GB  10GB
5. -10GB 10GB  10GB
6. +15GB 15GB  15GB (Warning as over the default 60%)
7. -5GB  15GB  15GB
8. +5GB  15GB  15GB

So we are safe to use Thin Provisioned VMFS now I think. 

Btw, I also received a reply from EQL indicating they are working on the Re-Thin feature.

In response to “reclaiming unallocated array disk space” on the PS Series arrays:

An enhancement request for this feature (reclaim space that was previously used) has already been submitted.  Firmware version 5.0.2 does not introduce this feature.  Engineering has not updated support as to when such a feature will be available in future firmware releases.

Finally, I looked into details about Hitachi HDS’s Re-Thin feature, a 3PAR guy points out HDS’s Re-Thin in fact is actually a…Migration and the Zero Out the unused blocks, but in 3PAR, they can REALLY, I MEAN REALLY do the Re-Thin in real time, no need to copy the volume to another copy and then zero out the unused block. I do hope Equallogic can have this kind of feature instead of a “Not so real” Re-Thin like HDS ones.

Oct 14, 2010 Some update from Dell Pro-Support regarding NTFS/VMFS can REUSE the touched blocks somehow.

=======================
A similar problem is when the initiator OS reports significantly more space in use than the array does. This can be pronounced in systems like VMWare that create large, sparse files. In VMWare, if you create yourself a 10GB disk for a VM as a VMDK file, VMWare does not write 10GB of zeros to the file. It creates an empty (sparse) 10GB file, and subtracts 10GB from free space. The act of creating the empty file only touches a few MB of actual sectors on the disk. So VMWare says 10GB missing, but the array says, perhaps, only 2MB written to.

Since the minimum volume reserve for any volume is 10%, the filesystem has a long way to go before the MB-scale writes catch up with the minimum reservation of a volume. For instance, a customer with a 100GB volume might create 5 VMs with 10GB disks. That’s 50GB used according to VMWare, but only perhaps 5 x 2MB (10MB) written to the array. Until the customer starts filling the VMDK files with actual data, the array won’t know anything is there. If has no idea what VMFS is; it only knows what’s been written to the volume.

• Example: A file share is thin-provisioned with 1 TB logical size. Data is placed into the volume so that the physical allocation grows to 500 GB. Files are deleted from the file system, reducing the reported file system in use to 100 GB. The remaining 400 GB of physical storage remains allocated to this volume in the SAN.

� This issue can also occur with maintenance operatiions including defragmentation, database re-organization, and other application operations.

In most environments, file systems do not dramatically reduce in size, so this issue occurs infrequently. Also some file systems will not make efficient re-use of previously allocated space, and may not reuse deleted space until it runs out of unused space (this is not an issue for NTFS, VMFS).
=======================

Update Oct-15-2010

If you ask me again now, I would say THIN PROVINTIONING (aka TP) ALL THE WAY, both on Equallogic AND on ESX Datastore is the BEST way to go and it is going to be the trend in storage management world I think, especially if Equallogic will release it’s upcoming Re-Thin or Space Reclaim feature in coming 5.x firmware update. (So far only 3PAR is able to do it I think)

Update Sep-3-2011

Storage APIs for Array Integration (VAAI) has been enhanced to reclaim blocks when a virtual disk is deleted, unlike previously where the storage array is not aware about deleted blocks contains data after deleting virtual disks.

There is a new feature in vSphere 5.0 that may finally solved the problem, but is this only going to work in vSphere 5.0? I really do hope ESX 4.1 can also get this VAAI enhancement after upgrading the Equallogic firmware with such thin provisioning reclaim capability. 

Currently, the only way to reclaim a thin provisioned volume (TP) in Equallogic is to Storage VMotion all existing VMs to a new TP volume and then delete the existing one.