Recently my company purchased 2x new servers for in the office to replace the old and wildly outdated servers which were already there (these machines will get a new lease of life eventually).
The original servers were built before I started (nearly 2 years ago) and were single operating system installations running on the full physical machines.
Since I have started at the company I have been promoting the use of Linux and Virtualisation in the office and run all our production servers on virtualised hardware in the cloud, so getting these new machines gave me the ability to build them up as virtualisation hosts.
Once built however, I had a problem.. The machines were running the guests fine, but the hosts thenselves had MASSIVE load averages. CPU utilisation was running about 2% on the hosts, but Load Average was about 50.0 across 1m, 5m and 15m timeframes. Nothing I could do would change this. I thought I had badly installed Ubuntu hosts and considered a rebuild.
It wasn't until I came across a blog post (will try to find the link) discussing performance of KVM virtual disks that I discovered the VirtIO interface.
Always use VirtIO interfaces for linux guests! Modern linux guest kernels have had the VirtIO drivers for some time now so you should not have any trouble.
Why VirtIO?As far as I understand VirtIO changes the paradigm for virtual machines. Under virtualisation conditions it should be nearly impossible for the guests to know they're in a virtual environment meaning they should believe they have full physical hardware to themselves. Obviously this is not true in virtualisation. VirtIO however tells the virtual machines that they're in a virtual environment and asks them to be patient and behave when it comes to accessing physical resource.
The result:Load average dropped from 50.0 to 2.0 for 4 running virtual machines on a physical host. Performance of the machines tripled at least and everything went happy.
Moral of the story: Always use VirtIO interfaces! Let the VMs know they're in a virtual environment and ask them to behave :-)