UArk links

DVC links

Other links

Using Xenoprofile with CentOS 5

Virtual machines are cool and popular. Unfortunately, they are often seen as a silver bullet.

The list goes on and on. Most of the time, using VMs is unecessary at best, and stupid at worst. People often forget that using a VM will decrease performance. Performance loss isn't a big deal for some applications, but for a lot of other applications it is a huge deal. Thus we come to my motivation for profiling applications running inside VMs: If we can accurately characterize VM application performance (for certain classes of applications), we can (heuristically) determine when using VMs is a good idea. The rest of this page is dedicated to the trials and tribulations of getting Xenoprofile to work with CentOS 5 (in a Rocks Cluster).

Links

Configuration

Hardware Software

Beginning Notes

CentOS 5 uses Xen 3.0.3 for virtualization. Since the latest version of Xen is 3.2.1, I wanted to upgrade (of course). Woe be unto me for attempting such a foolhardy task. For some reason I couldn't get Xen 3.2 with their kernel to work. Networking problems galore. I soon abandoned trying to use 3.2. I then tried 3.1. The installation was successful, but xenoprofile wasn't enabled by default. This would mean recompiling the kernel, and although kernel compiling is a pretty standard procedure, I didn't want to do that much work. So I stuck with the stock CentOS 5 Xen kernel.

Oprofile Setup

  1. Download oprofile-0.9.3 and the xenoprofile-0.9.3 patch
  2. Untar oprofile-0.9.3.tar.gz
  3. cd oprofile-0.9.3
  4. patch -p1 < ../oprofile-0.9.3-xen-r2.patch
  5. Then configure, make, make install (I configured with 
    ./configure --prefix=/root/oprofile --with-kernel-support

That was the easy part. If you try to use oprofile, you will most likely get reports that say 


... vmlinux         (no symbols)
... vmlinux-unknown (no symbols)

To get rid of these symbols, you will have to install 

kernel-debuginfo
RPMs. To see if you have debugging symbols available, do this: 


# cd /boot/
# file xen-syms-2.6.18-53.el5 
xen-syms-2.6.18-53.el5: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), statically linked, stripped

xen-syms-2.6.18-53.el5 is the symbol file for the xen hypervisor. At the end of the line, "stripped" tells us that there aren't any useful symbols for us to look at. To check that, use "nm" or "objdump --all" on xen-syms-2.6.18-53.el5. Both of those should tell us that there are no symbols.

To fix that, we have to make non-stripped symbol files. To do that, we have to find the source RPM for the linux kernel we want to use. Since we are using CentOS5, check vault.centos.org. The file I got was kernel-2.6.18-53.el5.src.rpm.

Now that we have the right source RPM, build it.

  1. rpmbuild --rebuild kernel-2.6.18-53.el5.src.rpm (Note: You have have to install some other RPMs like "rpm-build")
  2. Once the build process is finished (it takes a while), there should be several files in /usr/src/redhat/RPMS/x86_64
  3. Install kernel-debuginfo-common, kernel-xen-debuginfo, and kernel-xen-devel
  4. The symbols files we want will be installed into /usr/lib/debug/
  5. The xen symbol file will/should be at /usr/lib/debug/boot/xen-syms-2.6.18-53.el5.debug
  7. The dom0 kernel should be at /usr/lib/debug/lib/modules/2.6.18-53.el5xen/vmlinux

We now have enough data to really use oprofile and xenoprofile. From here, we follow the steps listed in the tutorial above

  1. opcontrol --init
  2. opcontrol --setup --xen=/usr/lib/debug/boot/xen-syms-2.6.18-53.el5.debug --vmlinux=/usr/lib/debug/lib/modules/2.6.18-53.el5xen/vmlinux ...
  3. opcontrol --start
  4. Run tests
  5. opcontrol --stop
  6. opcontrol --shutdown
  7. opreport -l

Hopefully this is enough to get people started.

Various notes

Attempting to use the default kernel for guests failed miserably. The trick was to regenerate the initrd for the guest with mkinitrd --with=xennet --preload=xenblk /boot/initrd-2.6.18-53.el5xen.img 2.6.18-53.el5xen. After this was done, a new problem cropped up. On boot, /etc/sysconfig/network-scripts/ifcfg-eth0 would be overwritten (I suspect that any ifcfg-eth* would be overwritten, but I only tested eth0). This is a problem since my guests change ifcfg-eth* on startup based on kernel arguments. After a reboot, ifcfg-eth0 would remain as it was (if I changed it), but if I cold rebooted the guest, ifcfg-eth0 would be overwritten again. This behavior didn't occur with any other kernel and ramdisk I tried. I found out that the cause was the ramdisk. Regenerating the ramdisk with mkinitrd --preload=xenblk /boot/initrd-2.6.18-53.el5xen.img 2.6.18-53.el5xen made everything go back to the way I expected.

Update: (10/08/2009)

Note: I'm not even going to spellcheck this update, so please forgive any grammatical/semantic errors in the writing. If something isn't clear, shoot me an email.

When I originally wrote this page, all of my Xenoprofile experiences were with old-ish hardware (the single-core Noconas listed above). Since then I have been running on much more modern systems (Intel Harpertown - E5430, and Intel Nehalem - Gainestown X5550). On the Noconas, Xenoprofile 0.9.3 with Redhat's 2.6.18 kernel worked very well. Unfortunately, it wasn't ready for the Harpertowns and definitely wasn't ready for the Gainstowns. For the Harpertowns, the fix was pretty simple (though tedious). In the Linux kernel sources, I had to do this:

  1. Find the kernel sources in /usr/src/redhat/SOURCES/
  2. Unpack those sources to a directory of my choosing (/tmp worked pretty well)
  3. Find nmi_int.c (in arch/i386/oprofile/ I think).
  4. Edit the "ppro_init" function in nmi_int.c so that Harpertowns and Nehalems are recognized as having a "core_2" architecture. For the Harpertowns, the code should be 23. The Nehalems code should be 26.
  5. With that changed, re-tar the unpacked directory and replace the original tar file in /usr/src/redhat/SOURCES/ with your modified one.
  6. After that, rebuild the RPMS and install them.

The Xen sources were a little more difficult even though the procedure is basically the same.

  1. Find the Xen sources in /usr/src/redhat/SOURCES/
  2. Unpack those sources to a directory of my choosing (/tmp worked pretty well)
  3. Find nmi_int.c (in arch/x86/oprofile/ I think).
  4. Edit the "ppro_init" function in nmi_int.c so that Harpertowns and Nehalems are recognized as having a "core_2" architecture. For the Harpertowns, the code should be 7. The Nehalems code should be 10.
  5. With that changed, re-tar the unpacked directory and replace the original tar file in /usr/src/redhat/SOURCES/ with your modified one.
  6. After that, rebuild the RPMS and install them.

For anyone who didn't catch that, Xen doesn't report the CPU model properly for the Harpertowns or Nehalems. I don't know why (I haven't even tried to find out), but that caused quite some consternation since the first thing I tried was duplicating the Linux method (i.e. using model numbers 23 and 26). When that failed, I was very confused. Eventually, I resorted to "printk" debugging and found out what was going on. Once I figured out the proper CPU codes to use, oprofile started working (mostly).

I say "mostly" working because there are some issues on the Nehalems. The Harpertowns work very well. Every event I have gathered has returned data. The Nehalem's don't work so well. Some events (CPU_CLK_UNHALTED, INST_RETIRED, etc.) work just fine. Others don't (L2_LINES_IN, L1I_*, PAGE_WALKS*, ...). I don't know why they don't work, and I'm probably just going to switch to a new kernel that will work. At this point, I am almost done (I think) with my use of Xenoprofile. I'm currently populating a database with the results so that I can do an analysis of them to make some performance predictions. Hopefully it will work. If not, I just lost about 6 months of time.