Debugging provider crashes on SFCB with gdb

I’ve never had to debug a multi-process program with GDB before.  Mostly because I do a lot of my debugging with printf() and elbow-grease.  However, a CIM provider is in an environment where a crash happen within the broker, which means you can’t (easily) add print statements to the appropriate place.  If you pass something broken to a CBFoo() call, the CIMOM will crash a ways down the line.

With Pegsus, you can turn off provider processes, which forces thread-only mode.  This means that the whole CIMOM is vulnerable to a bad provider, but it also means that running the cimserver in gdb is easy and straightforward.  Doing the same in SFCB, however, is not as simple.  Your bad provider ends up in a child of the main process, which crashes without GDB’s full attention, which isn’t very helpful.

While helping to debug a particularly nasty crash with an Indication provider, I learned a (relatively) easy way to catch the crash with GDB.  First, I started SFCB and subscribed to the indication.  This forked off a process to host the provider.  While in the simple case, the PID of the new process will be the highest-numbered sfcbd process, you can also find out which one it is with something like this:

% PROV=libVirtComputerSystemIndication
% for i in $(ps ax | grep sfcbd | awk ‘{print $1}’; do
> grep -q $PROV /proc/$i/maps && echo $i;
> done

That checks all of the currently running sfcbd processes to see which has your provider loaded and prints the PID.  Next, I attached to the process with gdb and allowed it to continue:

% S gdb /usr/local/sbin/sfcbd
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “x86_64-suse-linux”…
Using host libthread_db library “/lib64/libthread_db.so.1”.
(gdb) attach 5832
<snip>
(gdb) c

Next, I triggered the crash and GDB caught it, allowing me to get a stack trace and examime the situation a little.  After that, figuring out the issue was easy.