James' USENIX 2007 notes: The Guru Is In: VMware Virtual Infrastructure Tools and Techniques

VMware Virtual Infrastructure Tools and Techniques
John Arrasjid, Shridhar Deuskar, Irfan Ahmad, and team, VMware

We will be providing an overview of the current technologies available from VMware. We will take questions from the participants with a focus around the VI3 environment and optimization and disaster recovery related questions. Several additional VMware team members will be present with specific areas of expertise for this in-depth session. We will provide feedback on VI3 (ESX Server and VirtualCenter), Lab Manager, desktop products (Workstation, Player, Fusion), VMware Server, and VDI.

The ESX Server now boots directly from the bootloader; it no longer bootstraps via RHEL. The service console is based on RHEL; it's a super stripped-down version.

Benchmarking tools should generally be outside of the VM, otherwise you probably aren't benchmarking what you think you are. For example: the CPU occupation within a VM might only be 25%, but the physical CPU hosting the VMs might be saturated (depending on how many VMMs are running on it.)

If there's something you want, be sure to open an explicit feature request for it, even if you can hack around the absence of what you want. This is because VMware uses feature requests to decide where to allocate development effort.

John fielded a question about what servers are good (or bad) candidates for virtualization. The specific servers that were being questioned were a centralized log host, a Windows domain server, and an Apache httpd server?

John doesn't like to put machines that are doing load generation or performance benchmarking inside virtual machines, because if the virtual server is resource starved, your benchmarks will be invalid. (And unfortunately, what usually happens is that the benchmark overestimates your capacity.)

At VMware, one of their Exchange servers runs on itself on a single ESX box, just so that they can migrate it to perform seamless upgrades.

You cannot perform hot migrations between different instruction sets. Unfortunately, instruction sets are very specifically defined. For example, a virtual machine that was started on physical host which had SSE2 but not SSE3 instructions can't be migrated to a physical host that has SSE3, because the presence of SSE3 counts as a new instruction set.

Actually, for this example, you can get around this restriction by masking out the SSE3 capabilities of the new physical host, which would make the instruction sets identical. But there are cases in which you cannot make the instruction sets equivalent—for example, an Opteron processor will never be equivalent to a Xeon processor. (In fact, different steppings of the same processor might not be equivalent, either.)

VMware licenses by socket, not by core. So although quad core CPUs are (currently) more expensive than dual-core CPUs, you'll get more bang for the same (VMware licensing) buck.

Customers who are doing rendering farms don't virtualize, because they're clustering, and using the entire CPU. If a node fails, they just replace it.

Let VMotion do its job unless it's screwing up. It normally does a pretty good job.

If there's a VMM that's really resource-intensive, VMotion will try to rebalance by picking a more lightweight VMM and migrating it instead. This is because migrating a non-intensive VMM is more efficient than migrating an intensive one.

Apple doesn't want you to run MacOS in a VMM, because Apple is a fundamentally a hardware shop, and they're afraid virtualization will undercut hardware sales.

Although VMware doesn't official make hardware recommendations, the current sweet spot for many customers seems to be dual/quad-CPU, dual/quad-core systems.

For most customers, the number of VMMs that can be hosted on a single host is bound by the amount of physical memory, not by the quantity/capability of the CPUs. Therefore, if you're buying systems, make damn sure you have plenty of expansion capability when it comes to memory.

VMware will share memory pages across VMMs. The more similar the guest OSes are to each other, the more memory sharing is possible (e.g., identical versions of glibc). But VMware will cheerfully share identical memory pages across different operating systems


You can go to the index of my Usenix notes.