Comments (0)

By admin, February 2, 2011 2:13 pm

The following is from Delltechcenter’s forum, posted by Joe from Dell EQL regarding the confusions about Equallogic’s load balancing between pools with multiple array members.

The following best practices should be considered for storage pools:

1. Do not mix arrays with different drive speeds RPM within a single pool unless they are running a unique RAID policy

2. Do not mix arrays with different drive technologies (SATA, SAS, SSD) within a single pool unless they are running a unique RAID policy.

3. Do not mix arrays with different controller speeds (1GbE, 10GbE) within a single pool unless they are each running unique RAID policies.

4. To override the automated performance decisions for a specific volume, indicate a ―preferred RAID type for that volume. If that RAID type exists within the pool, the volume will reside on those arrays that match the preferred RAID policy for the volume.

For example: if you have two members with different drive technologies (Item #2), so you can create each member with a different RAID preference (as you indicated Array1 w/Raid50, Array2 w/RAID5), and then specify the preferred RAID policy for the volume(s), without any issues.

With volume RAID preferences set to Auto, the volume will span both members in the pool. If using the RAID preference, it isn’t a guarantee that it will adhere to the request and only place the volume on a given member; for instance if the volume is too large to fit on the member with the desired RAID level (you might end up with an 80/20 balance).

Note that when mixing SAS/SATA in the same pool the performance load balancer doesn’t take drive RPM into account in its calculations. A common issue is that if you combine fast SAS with slower SATA (often worst with the SATA setup as RAID 5) the SAS array won’t provide its potential performance, and the load balance may place more data on the larger SATA member (typically SATA Arrays have more space).

In other words, old RAID theory still applies here: DO NOT MIX different RPM disks in a raid group.

Comments (0)

By admin, January 27, 2011 12:43 pm

I just came across this PDF today and found it’s pretty much everything about the latest features of EQL SAN like VAAI, MPIO, Thin-Clone, etc. I’ve already implemented most of them in our production enviornment and seeing performance gained by multiple times especially during SVmotion and depoly a new VM from template, of course the reduction in snapshot time really helped Veeam to run at it’s peak performance during backup window.

Hope everyone will finds this PDF useful!

Comments (5)

By admin, December 28, 2010 2:07 pm

I didn’t know this could happen until last night. I restarted VMware vCenter server after installing Microsoft patches for Windows Serve 2008 R2. Then used vSphere client to connect to vCenter, suddenly it started to pop up with error, also found vCenter Server Service has stopped or crashed to be precious and error logs showed Event 1000 ID, luckily I found this is to do with vCenter port conflicts. I was also able to confirm this by another user’s post in VMware community who’s having the same port conflicting problem with PRTG and ESX 4.

As usual, my memory needs to rewind and back to the point where I installed PRTG Network Monitor 4 days ago, by trial and error, I stopped the PRTG Server and Probe Services, then I was able to restart vCenter server again and login with vSphere client. Finally, I was also able to restart PRTG server without any problem, PRTG somehow was able to bind itself dynamically to another port which vCenter couldn’t.

In additional, I did try to change the port for PRTG and its Probe services, but after restarting the server, somehow PRTG is still conflicting with vCenter’s port (I don’t know which port, but PRTG is using a range of ports above 23560)

The solution is quite simple but a bit troublesome, as it’s a matter of precedence, I just need to start vCenter Server service before starting the PRTG services MANUALLY every time after restarting the server.

If you know how to change the service start dependence or any proper way to solve this, please do let me know, as somehow I still couldn’t figure out the permanent solution for this port conflict problem, thanks!

Comments (1)

By admin, December 23, 2010 1:11 pm

Dell has a new “Build-in OS” called System Service (USC) that you can simple update firmware and deploy different OS using the simple GUI. I like it a lot for OS deployment as it indeed saved me a lot of time in finding different drivers, but until recently, I found the specific USC module that is the Platform (Firmware) update feature isn’t very stable and may cause various system problems.

The following is what I’ve encountered:

I press F10 during boot time and enter USC and select Platform Update for firmware update process, configure IP for USC and then next connect to ftp.us.dell.com to check a list of update firmware, after 5 minutes, as usual, USC found a list of outdated firmware (about 15 of them) and asked me if I would like to continue to update those. Great! then click apply, “Please Wait” is what I get on screen and wait, and wait and wait, I waited for more than 2 hours and definitely sensed some kind of strangeness as my last update using USC for BIOS and IDRAC only took 15 mins.

Immediately I called Dell Pro-Support regarding this issue, they suggested me to wait and they ensure me everything should be ok, as even bad thing happens, I can still use Firmware Roll Back under USC, and if a particular firmware download process isn’t completed then the update won’t carry on.

O.K., then I went to sleep, after 12 hours, I checked the server, it is still showing the annoying “Please Wait”, NOTE there is NO WAY for me to know the status of the update progress as it won’t tell me which module/firmware it has finished downloading or completed the upgrade process for a particular firmware.

I have no choice but to reboot the server, luckily, the server (OS is Windows Server 2008 R2) came back ok, and I checked the firmware information in OpenManage, NOTHING HAS BEEN UPDATED, and worst some of the sensors in OpenManage are gone such as temperature/voltage/fan, etc. Right away I KNEW iDRAC Must Have Been Damaged during the firmware update under USC, so I connected to R610 iDRAC and indeed found all of the sensors showing reading errors. Huh???

Then I thought by updating the iDRAC firmware may help, so I tried to update iDRAC6 to latest 1.5.4 using the standard windows firmware file, immediately error shows “This update package is not compatible with your system configuration”, thank you very much!

After searching Google with no solution found, I suddenly thought of a way to update it by downloading the raw iDRAC6 firmware update file and using iDRAC’s own update feature from Remote Access > Update and I have successfully updated it! Yeah! Then I reboot the server, and checked OpenMange again, thank god, everything is back to normal this time!

Finally, I simply go back to the traditional way of updating firmware on Poweredge which I have been doing for the past 10+ years, that is download the firmware one by one and update my Poweredge R610 BIOS/Network/Raid etc one by one.

So my conclusion is:

1. Platform Update utility under USC is NOT A MATURED PRODUCT!

2. Probably USC FTP process hanged, network issue, but USC won’t alert at all, just showing “Please Wait” that really DOESN’T HELP AT ALL!

3. You NEED TO UPDATE iDRAC first (probably using the above method) before using USC, as USC uses iDRAC to facilitate the rest firmware upgrade process. I remembered ont time USC warned me it will upgrade iDRAC first before anything else, but it didn’t warm me this time.

Dell please correct this bug ASAP in USC Platform Update, at least show us the update process bar or some useful detail information!

If you want to read about my first disappointment regarding firmware update, please refer to Do not update firmware/BIOS from within ESX console.

Update:

Just received an update from Virtualization Buster, which is the proper way to update firmware using an USB method via F10 Unified Server Configuration (USC).

Updating Dell R-Series aka 11-th Generation Servers via USB and Repository Manager

http://www.virtualizationbuster.com/?p=1301

Finally, since there is no way to attach an USB via iDRAC6, so if you need to update Driver thru USC, you can use Dell Repository Manager, export the drivers to an ISO. (Tips from Dell Pro-Support)

Comments (0)

By admin, December 20, 2010 12:31 pm

Saw these two piece of useful information on Veeam’s forum today, mostly contributed by Tom:

Block Size Optimization

In the job properties, on the “Backup Destination” if you hit “Advanced” and then select the “Storage” tab with V5 you can now select to optimize for Local disk, LAN target, and WAN target. What this is really doing is setting the “block size” of the VBK. Previous version of Veeam always used 1MB blocks, which is now the equivalent of the “Local disk” option. LAN target uses 512KB blocks, and WAN target uses 256KB blocks. The smaller blocks sizes typically mean that incrementals are smaller and thus less data is transferred over the LAN or WAN at the cost of some CPU. Because we push our backups across sites we always use the WAN target settings.

Reversed Incremental or Synthetic Full

As a general rule, reverse incrementals are going to use the least amount of space on disk, but the most amount on tape, while the reverse is true of forward incrementals, they will use more disk space, but significantly less tape space. The only exception might be if you have a fairly small retention period.

Assume you have a 100GB full backup with 10GB of changes a day, here’s how the space would break down assuming 4 weeks retention:

Reverse Incremental:
Disk — 100GB Full + 280GB (10GB/day * 28 days) of reverse incrementals = 380GB
Tape — 100GB VBK copied to tape every day * 28 days = 2.8TB

Forward Incremental w/Synthetic Full:
Disk — 400GB (100GB Full * 1 per week) + 240GB (10GB/day incrementals * 24 days) = 640GB
Tape — Same as disk, since you simply copy the full or incremental to tape every day = 640GB

So, the Forward Incremental/Synthetic option in this scenario would use ~70% more disk space, but less than 25% of the tape space. If you’re planning to keep only a short period of disks on retention and use tape for long term storage then forward incrementals will save space, but that’s about the only scenario where it will save space. For the best space savings with on disk retention, reverse incremental are the way to go, but at the cost of a large amount of tape space.

 

Personally I prefer reversed incremental because my main strategy is to get a full working backup to tape every night without the need to merge increments when in an emergency need to get my files back from tape.

Comments (0)

By admin, October 31, 2010 12:03 pm

Found this really useful information in the Veeam’s forum (contributed by Anton), a great place to learn about the product and their staff are all very caring and warm.

The formulas we use for disk space estimation are the following:

Backup size = C * (F*Data + R*D*Data)
Replica size = Data + C*R*D*Data

Data = sum of processed VMs size (actually used, not provisioned)

C = average compression/dedupe ratio (depends on too many factors, compression and dedupe can be very high, but we use 50% – worst case)

F = number of full backups in retention policy (1, unless periodic fulls are enabled)

R = number of rollbacks according to retention policy (14 by default)

D = average amount of VM disk changes between cycles in percent (we use 10% right now, but will change it to 5% in v5 based on feedback… reportedly for most VMs it is just 1-2%, but active Exchange and SQL can be up to 10-20% due to transaction logs activity – so 5% seems to be good average)

Comments (0)

By admin, October 29, 2010 11:56 am

The followings are my own findings about Veeam Backup v5: 

• If you have more than 30-50 VM (average 20-30GB) to backup, it’s better to setup a 2^nd Veeam Backup server to load-balance the job. So eventually, you may have a number of load-balanced Veeam Backup servers to evenly spread out the loading, and this will greatly reduce the queue time and shorten the total backup windows.Don’t Worry, all Veeam Backup servers are managed centrally by Veeam Enterprise Manager (and the best part is additional Veeam Backup servers DOES NOT count towards your Veeam licenses, how thoughtful and care Veeam is, thank you!)It’s recommended by Veeam to create 3-4 jobs per Veeam Backup server due to high CPU usage during the backup windows for de-duplication and compression. However this doesn’t apply Replication which doesn’t use de-duplication and compression.Note: However there is another way to fully utilize your one and only Veeam Backup Server by adding more than 4 concurrent jobs. (See quote below)
• The “Estimated required Space” within Job description is incorrect as it’s a know bug that VMware API or Veeam doesn’t know how to interpret Thin-Provisioning volume yet, so estimate by yourself. Most of the time, it will over-estimate the Total Required backup space by 3-10x more, so don’t worry!
• After each job completes, you will see a Processing rate: (for example 251 MB/s), which means total size of VM divided by total time it took to process the VM. This is affected by time it takes to talk to vCenter, freeze guest OS, create and remove snapshot, backup small VM files (configuration), and backup actual disks.

I am still not sure if the target storage requires very fast I/O or high IOPS device like 4-12 RAID10/RAID50 10K/15K SAS disks as some say the backup process with de-duplication and compress is random, some say it’s sequential, so if it’s random, then we need fast spindle with lots of disks, if it’s sequential, then we only need cheap 7200RPM SATA disks, a 4 RAID 5 2TB will be the most cost effective solution for storing the vbk/vik/vrk images.

Some interesting comments quote from Veeam’s forum relating to my findings:

That’s why we run multiple jobs. Not only that, but when doing incremental backup, a high percentage of the time is spent simply prepping the guests OS, taking the snapshot, removing the snapshot, etc, etc. With Veeam V5 you get some more “job overhead” if you use the indexing feature since to system has to build the index file (can take quite some time on systems with large numbers of files) and then backup the zipped index via the VM tools interface. This time is all calculated in the final “MB/sec” for the job. That means that if you only have a single job running there will be lots of “down time” where no transfer is really occurring, especially with incremental backups because there’s relatively little data transferred for most VM’s compared to the amount of time spent taking and removing the snapshot. Multiple jobs help with this because, while one job may be “between VM’s” handling it’s housekeeping, the other job is likely to be transferring data.

There are also other things to consider as well. If you’re a 24×7 operation, you might not really want to saturate your production storage just to get backups done. This is admittedly less of an issue with CBT based incrementals, but used to be a big deal with ESX 3.5 and earlier, and full backups can still be impacting to your production storage. If I’m pushing 160MB/sec from one of my older SATA Equallogic arrays, it’s I/O latency will shoot to 15-20ms or more, which severely impacts server performance on that system. Might not be an issue if your not a 24×7 shop and you have a backup window where you can hammer you storage as much as you want, but is certainly an issue for us. Obviously we have times that are more quiet than others, and our backup windows coincide with our “quiet” time, but we’re a global manufacturer, so systems have to keep running and performance is important even during backups.

Finally, one thing often overlooked is the backup target. If you’re pulling data at 60MB/sec, can you write the data that fast? Since Veeam is compressing and deduping on the fly, it can have a somewhat random write pattern even when it’s running fulls, but reverse incrementals are especially hard on the target storage since they require a random read, random write, and sequential write for ever block that’s backed up during an incremental. I see a lot of issue with people attempting to write to older NAS devices or 3-4 drive RAID arrays which might have decent throughput, but poor random access. This is not as much of an issue with fulls and the new forward incrementals in Veeam 5, but still has some impact.

No doubt Veeam creates files a lot differently that most vendors. Veeam does not just create a sequential, compressed dump of the VMDK files. 

Veeam’s file format is effectively a custom database designed to store compressed blocks and their hashes for reasonably quick access. The hashes allow for dedupe (blocks with matching hashes are the same), and there’s some added overhead to provide additional transactional safety so that you VBK file is generally recoverable after a crash. That means Veeam files have a storage I/O pattern more like a busy database than a traditional backup file dump.

Comments (0)

By admin, October 26, 2010 11:05 pm

Some operating system SKUs are hard-limited to run on a fixed number of CPUs. For example, Windows Server 2003 Standard Edition is limited to run on up to 4 CPUs. If you install this operating system on an 8-socket physical box, it runs on only 4 of the CPUs. The operating system takes advantage of multi-core CPUs so if your CPUs are dual core, Windows Server 2003 SE runs on up to 8 cores, and if you have quad-core CPUs, it runs on up to 16 cores, and so on.

Virtual CPUs (vCPU) in VMware virtual machines appear to the operating system as single core CPUs. So, just like in the example above, if you create a virtual machine with 8 vCPUs (which you can do with vSphere) the operating system sees 8 single core CPUs. If the operating system is Windows 2003 SE (limited to 4 CPUs) it only runs on 4 vCPUs.

Note: Remember that 1 vCPU maps onto a physical core not a physical CPU, so the virtual machine is actually getting to run on 4 cores.

This is an over simplication, since vCPUs are scheduled on logical CPUs which are hardware execution contexts. These can be a while CPU in the case of a single core CPU, or a single core in the case of CPUs that have only 1 thread per core, or could be just a thread in the case of a CPU that has hyperthreading.

Consider this scenario:

In the physical world you can run Windows 2003 SE on up to 8 cores (using a 2-socket quad-core box) but in a virtual machine they can only run on 4 cores because VMware tells the operating system that each CPU has only 1 core per socket.

VMware now has a setting which provides you control over the number of cores per CPU in a virtual machine.

This new setting, which you can add to the virtual machine configuration (.vmx) file, lets you set the number of cores per virtual socket in the virtual machine.

To implement this feature:

1. Power off the virtual machine.
2. Right-click on the virtual machine and click Edit Settings.
3. Click Hardware and select CPUs.
4. Choose the number of virtual processors.
5. Click the Options tab.
6. Click General, in the Advanced options section.
7. Click Configuration Parameters.
8. Include cpuid.coresPerSocket in the Name column.
9. Enter a value (try 2, 4, or 8 ) in the Value column.Note: Ensure that cpuid.coresPerSocket is divisible by the number of vCPUs in the virtual machine. That is, when you divide cpuid.coresPerSocket by the number of vCPUs in the virtual machine, it must return an integer value. For example, if your virtual machine is created with 8 vCPUs, coresPerSocket can only be 1, 2, 4, or 8. The virtual machine now appears to the operating system as having multi-core CPUs with the number of cores per CPU given by the value that you provided in step 9. 
10. Click OK.
11. Power on the virtual machine.

For example:

Create an 8 vCPU virtual machine and set cpuid.coresPerSocket = 2. Window Server 2003 SE running in this virtual machine now uses all 8 vCPUs. Under the covers, Windows sees 4 dual-core CPUs. The virtual machine is actually running on 8 physical cores.

Note:

• Only values of 1, 2, 4, 8 for the cpuid.coresPerSocket are supported for the multi-core vCPU feature in ESX 4.0.
• In ESX 4.0, if multi-core vCPU is used, hot-plug vCPU is not permitted, even if it is available in the UI.
• Only HV 7 virtual machines support the multi-core vCPU feature.

Important: When using cpuid.coresPerSocket, you should always ensure that you are in compliance with the requirements of your operating system EULA (Regarding the number of physical CPUs on which the operating system is actually running).

Update Apr 19

One good example is Windows Server 2003 Web Edition limited to 2 CPU sockets only, so if you assign 8 vCPUs, it will only see 2, by setting cpuid.coresPerSocket = 4 and assign 8 vCPUs, it means your server will have 2 CPU sockets and each socket will have 4 cores, so this manually override the default and allows you to have 8 CPUs technically speaking 8 Cores with Windows Server 2003 Web Edition which is previously impossible before ESX 4.1.

Comments (4)

By admin, October 22, 2010 10:55 pm

From Equallogic Support:

I believe our (Dell EqualLogic) intent was to either restrict access
on the volume level, not based on the ACL passed by the
initiator.  However, if you can express in writing exactly what you
would like to be added as a feature, why you would like this
feature, and how it would be beneficial to you within your
environment, we will create an enhancement request on your behalf.

Just be aware that submitting an enhancement requests does not
guarantee your request will be honored.  New features are added at
the sole discretion of the engineering team and will be added based
upon the needs of all customers and if the underlying code can/will
support the request.

 

 

My Reply:

It’s because many people use Veeam Backup to backup their VMs and in order for Veeam to use SAN offload feature (ie, directly backup the VM from SAN to backup server without going through ESX host), so we need to mount the VMFS Volume direclty from EQL to Windows Server Host and you may ask it will corrupt the VMFS volume which is concurrently mounted by ESX Host, NO it won’t as before mounting the volume on Windows Host, we issued automount=disable to make sure Windows Server won’t automatically initialize or format the volume by accident. (In fact, I found the mounted Equallogic volume under Disk Manager cannot be initialized, everything is gray out and it won’t show in IOMeter as well, but you can Delete Volume under Windows Server 2008 Disk Manager, strange!)

It will serve as a Double Insurance feature if EQL can implement such READ-ONLY to a specific iSCSI initiator that will greatly improve the protection of the attached volume for use of Veeam SAN Offload Backup.

I am sure there are many Veeam and Equallogic users would love to see this feature added.

Please kindly consider adding this feature in coming firmware release.

Thank you very much in advance!

Comments (0)

By admin, October 21, 2010 12:38 am

To enable Copy & Paste option when using VM Remote Console in ESX 4.1:

1. Log into a vCenter Server system using the vSphere Client and power off the virtual machine.

2. Select the virtual machine, on the Summary tab click Edit Settings.

3. Navigate to Options > Advanced > General and click Configuration Parameters.

4. Click Add Row and type the following values in the Name and Value columns:

isolation.tools.copy.disable – false
isolation.tools.paste.disable – false