Ok, at first I was going to start blogging about how to debug linux kernel crashes, OOPses, and hangs by giving some tips and background info to help in your troubleshooting efforts. While coming up with the blog post, I decided to start writing a generic and dynamic kprobe/jprobe module and explain how it would be useful for debugging. And because Murphy likes a good laugh, while creating this module, I created kernel panics, OOPses, and hangs :)
So I decided to change my strategy for the blog post. I am now going to deliver a series of blogs that will try to kill two birds with one stone:
1. Design a module from the ground up
2. Debug along the way
Debugging is an art unto itself, and I think that there's not a whole lot of info out there about it. It doesn't help that debugging sometimes requires knowledge of assembly, or the tedium of obtaining a stack trace via redirecting the console to the serial port. Sometimes it requires knowledge of the ELF specification, or using somewhat obscure tools like objdump, nm, addr2line or how to disassemble C code. Also, when you're dealing with device drivers, there's the added complication of dealing with the IRQ stack as well as the process stack (and possibly thread stacks). And hopefully, you won't have to deal with dreaded latency/timing issues where, like a revenge of Heisenberg, it's impossible to reproduce the problem with debugging turned on.
But the major points to debugging comes down to these elements:
1. Understanding assembly and the machine architecture
2. Obtaining the coredump or OOPs stack trace
3. Getting symbols and offsets from a kernel module
4. Knowing your debugging tools
5. Lots of insight
Perhaps this is why Alan Perlis said:
"It is easier to write an incorrect program than to understand a correct one".
I think there's a tendency for engineers and perhaps managers to think that creating new functionality is the hard part, while the "fix-engine" part is the easy job. Too often, instead of actually figuring out why something doesn't work, a "workaround" is made instead. Hopefully with these debugging blogs, people can start to truly understand what their code is really doing.
I'll be going into deeper depth into each of the above topics in separate posts, but for now, let me give a rough overview of how all the steps fit together. So I will start explaining what you need to know from a 20,000' view, and then finally start looking at examples. Theory first, application later. As I mentioned at the beginning, I will also cover the creation of a debug helper module so that you can practice on your own. So our first step in debugging linux kernel crashes or hangs is understanding how instructions are actually run.
The Stack:
It's vital to understand how the stack works in linux if you are going to troubleshoot an OOPs stack trace, or to investigate processes in a coredump using backtrace. So what is the stack? A stack is a region of memory either 4Kb or 8Kb (depending on how your kernel is setup but usually 8kb) in size that is used to keep track of functions and local (automatic) variables. On x86 architectures, the stack grows down from a higher memory address to a lower address. Understanding what the stack contains is important in helping debug issues.
As functions are called, stack frames are pushed onto this region of memory known as the Stack. Depending on your architecture and how the kernel was configured, this size can vary, but on x86 systems, it is usually 8kb in size. Now you know why recursive functions can "blow the stack", because on every recursion, a new stack frame is placed on the stack. Later, we will look into what exactly goes on in the stack by developing a toy program.
The important part here for debugging is almost always figuring the chain of calls (and the arguments passed to the functions) that led to the problem or where in the current function something blew up. When you get a kernel OOPs or a crashdump, you'll immediately want to see what function was being executed by the kernel when the problem occurred.
A foray into assembly
But how is the stack used, and how does it help us debug a problem? One of the most fundamental things to know in helping trouble shoot a problem is to know where exactly your program crashed. In regular user-land programs, tools like gdb are used to obtain a backtrace, or to set a breakpoint and walk through the program. With the kernel, this is not easy to do. Afterall, the thing you are debugging is the system itself (imagine a neuro-surgeon operating on his own brain). One way around this is to use a second system (kgdb) and step through the system, or to use kexec to launch a second kernel. In more recent versions of the kernel, this is possible, but in some cases, you may not have all the prerequisites to use the newly merged kdb/kgdb interface. Besides, learning how the OS controls the machine is a good thing to know.
So how does knowing assembly help us understand the stack? And how does even knowing what the stack does and contains help us troubleshoot a problem? Hopefully, I can help illustrate this with a tiny sample program that I will introduce shortly.
Getting symbols and offsets- OOPS
If you are able to obtain an OOPS, the easy part is done for you. The OOPs should tell you the name of the offending function and module. What is harder to determine is specifically where in the function it blew up. The key is doing two things:
1. Getting an assembly output to get byte offsets
2. Finding the equivalent line of code in C where the byte offset occurred
Once you have 1 and 2, you (most likely) know where in your source file the problem occurred.
Knowing your debugging tools
There are several tools that will greatly help you in your debugging efforts. Tools like objdump and nm are invaluable for peeking inside object files, and crash is a really nice tool to examine coredumps. But you'll get more debugging bang for your buck if you know more than just hich command line switches to use for your tool. If you understand what these tools are really doing, then you'll have a deeper understanding of how your source file gets translated into objects which in turn actually get executed. And don't underestimate the good-old fashion use of pr_info to log what's going on.
Lots of insight
Sometimes, you have to rely on your intuition and the big picture in order to puzzle out what is really happening. This can especially be true when realizing that things just don't "make sense". For example, you could be following all of the debugging steps fine, but when you look at the call trace and examine the arguments, things look impossible. That's when you may eventually realize that you forgot to compile with debugging flags, or perhaps the tool you are using has a bug. These are hair-pulling moments, but sometimes you have to just trust your gut.
So stay tuned for the next installment where I will begin designing a module to help debug other modules. There will be bugs along the way, and I'll start going over techniques to help solve the problem.
