1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
|
<!--
The FreeBSD Documentation Project
$FreeBSD: doc/en_US.ISO_8859-1/books/handbook/kerneldebug/chapter.sgml,v 1.30 2001/04/09 00:33:52 dd Exp $
-->
<chapter id="kerneldebug">
<title>Kernel Debugging</title>
<para><emphasis>Contributed by &a.paul; and &a.joerg;</emphasis></para>
<sect1>
<title>Debugging a Kernel Crash Dump with <command>gdb</command></title>
<para>Here are some instructions for getting kernel debugging working on a
crash dump. They assume that you have enough swap space for a crash
dump. If you have multiple swap partitions and the first one is too
small to hold the dump, you can configure your kernel to use an
alternate dump device (in the <literal>config kernel</literal> line), or
you can specify an alternate using the
&man.dumpon.8; command. The best way to use &man.dumpon.8; is to set
the <literal>dumpdev</literal> variable in
<filename>/etc/rc.conf</filename>. Typically you want to specify one of
the swap devices specified in <filename>/etc/fstab</filename>. Dumps to
non-swap devices, tapes for example, are currently not supported. Config
your kernel using <command>config <option>-g</option></command>. See <link
linkend="kernelconfig">Kernel Configuration</link> for details on
configuring the FreeBSD kernel.</para>
<para>Use the &man.dumpon.8; command to tell the kernel where to dump to
(note that this will have to be done after configuring the partition in
question as swap space via &man.swapon.8;). This is normally arranged
via <filename>/etc/rc.conf</filename> and <filename>/etc/rc</filename>.
Alternatively, you can hard-code the dump device via the
<literal>dump</literal> clause in the <literal>config</literal> line of
your kernel config file. This is deprecated and should be used only if
you want a crash dump from a kernel that crashes during booting.</para>
<note>
<para>In the following, the term <command>gdb</command> refers to
the debugger <command>gdb</command> run in <quote>kernel debug
mode</quote>. This can be accomplished by starting the
<command>gdb</command> with the option <option>-k</option>. In
kernel debug mode, <command>gdb</command> changes its prompt to
<prompt>(kgdb)</prompt>.</para>
</note>
<tip>
<para>If you are using FreeBSD 3 or earlier, you should make a stripped
copy of the debug kernel, rather than installing the large debug
kernel itself:</para>
<screen>&prompt.root; <userinput>cp kernel kernel.debug</userinput>
&prompt.root; <userinput>strip -g kernel</userinput></screen>
<para>This stage isn't necessary, but it is recommended. (In
FreeBSD 4 and later releases this step is performed automatically
at the end of the kernel <command>make</command> process.)
When the kernel has been stripped, either automatically or by
using the commands above, you may install it as usual by typing
<command>make install</command>.</para>
<para>Note that older releases of FreeBSD (up to but not including
3.1) used a.out kernels by default, which must have their symbol
tables permanently resident in physical memory. With the larger
symbol table in an unstripped debug kernel, this is wasteful.
Recent FreeBSD releases use ELF kernels where this is no longer a
problem.</para>
</tip>
<para>If you are testing a new kernel, for example by typing the new
kernel's name at the boot prompt, but need to boot a different one in
order to get your system up and running again, boot it only into single
user state using the <option>-s</option> flag at the boot prompt, and
then perform the following steps:</para>
<screen>&prompt.root; <userinput>fsck -p</userinput>
&prompt.root; <userinput>mount -a -t ufs</userinput> # so your file system for /var/crash is writable
&prompt.root; <userinput>savecore -N /kernel.panicked /var/crash</userinput>
&prompt.root; <userinput>exit</userinput> # ...to multi-user</screen>
<para>This instructs &man.savecore.8; to use another kernel for symbol
name extraction. It would otherwise default to the currently running
kernel and most likely not do anything at all since the crash dump and
the kernel symbols differ.</para>
<para>Now, after a crash dump, go to
<filename>/sys/compile/WHATEVER</filename> and run
<command>gdb <option>-k</option></command>. From <command>gdb</command> do:
<screen><userinput>symbol-file kernel.debug</userinput>
<userinput>exec-file /var/crash/kernel.0</userinput>
<userinput>core-file /var/crash/vmcore.0</userinput></screen>
and voila, you can debug the crash dump using the kernel sources just
like you can for any other program.</para>
<para>Here is a script log of a <command>gdb</command> session
illustrating the procedure. Long lines have been folded to improve
readability, and the lines are numbered for reference. Despite this, it
is a real-world error trace taken during the development of the pcvt
console driver.</para>
<screen> 1:Script started on Fri Dec 30 23:15:22 1994
2:&prompt.root; <userinput>cd /sys/compile/URIAH</userinput>
3:&prompt.root; <userinput>gdb -k kernel /var/crash/vmcore.1</userinput>
4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel
...done.
5:IdlePTD 1f3000
6:panic: because you said to!
7:current pcb at 1e3f70
8:Reading in symbols for ../../i386/i386/machdep.c...done.
9:<prompt>(kgdb)</prompt> <userinput>where</userinput>
10:#0 boot (arghowto=256) (../../i386/i386/machdep.c line 767)
11:#1 0xf0115159 in panic ()
12:#2 0xf01955bd in diediedie () (../../i386/i386/machdep.c line 698)
13:#3 0xf010185e in db_fncall ()
14:#4 0xf0101586 in db_command (-266509132, -266509516, -267381073)
15:#5 0xf0101711 in db_command_loop ()
16:#6 0xf01040a0 in db_trap ()
17:#7 0xf0192976 in kdb_trap (12, 0, -272630436, -266743723)
18:#8 0xf019d2eb in trap_fatal (...)
19:#9 0xf019ce60 in trap_pfault (...)
20:#10 0xf019cb2f in trap (...)
21:#11 0xf01932a1 in exception:calltrap ()
22:#12 0xf0191503 in cnopen (...)
23:#13 0xf0132c34 in spec_open ()
24:#14 0xf012d014 in vn_open ()
25:#15 0xf012a183 in open ()
26:#16 0xf019d4eb in syscall (...)
27:<prompt>(kgdb)</prompt> <userinput>up 10</userinput>
28:Reading in symbols for ../../i386/i386/trap.c...done.
29:#10 0xf019cb2f in trap (frame={tf_es = -260440048, tf_ds = 16, tf_\
30:edi = 3072, tf_esi = -266445372, tf_ebp = -272630356, tf_isp = -27\
31:2630396, tf_ebx = -266427884, tf_edx = 12, tf_ecx = -266427884, tf\
32:_eax = 64772224, tf_trapno = 12, tf_err = -272695296, tf_eip = -26\
33:6672343, tf_cs = -266469368, tf_eflags = 66066, tf_esp = 3072, tf_\
34:ss = -266427884}) (../../i386/i386/trap.c line 283)
35:283 (void) trap_pfault(&frame, FALSE);
36:<prompt>(kgdb)</prompt> <userinput>frame frame->tf_ebp frame->tf_eip</userinput>
37:Reading in symbols for ../../i386/isa/pcvt/pcvt_drv.c...done.
38:#0 0xf01ae729 in pcopen (dev=3072, flag=3, mode=8192, p=(struct p\
39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403)
40:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
41:<prompt>(kgdb)</prompt> <userinput>list</userinput>
42:398
43:399 tp->t_state |= TS_CARR_ON;
44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */
45:401
46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200)
47:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
48:404 #else
49:405 return ((*linesw[tp->t_line].l_open)(dev, tp, flag));
50:406 #endif /* PCVT_NETBSD || (PCVT_FREEBSD >= 200) */
51:407 }
52:<prompt>(kgdb)</prompt> <userinput>print tp</userinput>
53:Reading in symbols for ../../i386/i386/cons.c...done.
54:$1 = (struct tty *) 0x1bae
55:<prompt>(kgdb)</prompt> <userinput>print tp->t_line</userinput>
56:$2 = 1767990816
57:<prompt>(kgdb)</prompt> <userinput>up</userinput>
58:#1 0xf0191503 in cnopen (dev=0x00000000, flag=3, mode=8192, p=(st\
59:ruct proc *) 0xf07c0c00) (../../i386/i386/cons.c line 126)
60: return ((*cdevsw[major(dev)].d_open)(dev, flag, mode, p));
61:<prompt>(kgdb)</prompt> <userinput>up</userinput>
62:#2 0xf0132c34 in spec_open ()
63:<prompt>(kgdb)</prompt> <userinput>up</userinput>
64:#3 0xf012d014 in vn_open ()
65:<prompt>(kgdb)</prompt> <userinput>up</userinput>
66:#4 0xf012a183 in open ()
67:<prompt>(kgdb)</prompt> <userinput>up</userinput>
68:#5 0xf019d4eb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi =\
69: 2158592, tf_esi = 0, tf_ebp = -272638436, tf_isp = -272629788, tf\
70:_ebx = 7086, tf_edx = 1, tf_ecx = 0, tf_eax = 5, tf_trapno = 582, \
71:tf_err = 582, tf_eip = 75749, tf_cs = 31, tf_eflags = 582, tf_esp \
72:= -272638456, tf_ss = 39}) (../../i386/i386/trap.c line 673)
73:673 error = (*callp->sy_call)(p, args, rval);
74:<prompt>(kgdb)</prompt> <userinput>up</userinput>
75:Initial frame selected; you cannot go up.
76:<prompt>(kgdb)</prompt> <userinput>quit</userinput>
77:&prompt.root; <userinput>exit</userinput>
78:exit
79:
80:Script done on Fri Dec 30 23:18:04 1994</screen>
<para>Comments to the above script:</para>
<variablelist>
<varlistentry>
<term>line 6:</term>
<listitem>
<para>This is a dump taken from within DDB (see below), hence the
panic comment <quote>because you said to!</quote>, and a rather
long stack trace; the initial reason for going into DDB has been a
page fault trap though.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 20:</term>
<listitem>
<para>This is the location of function <function>trap()</function>
in the stack trace.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 36:</term>
<listitem>
<para>Force usage of a new stack frame; this is no longer necessary
now. The stack frames are supposed to point to the right
locations now, even in case of a trap.
From looking at the code in source line 403, there is a
high probability that either the pointer access for
<quote>tp</quote> was messed up, or the array access was out of
bounds.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 52:</term>
<listitem>
<para>The pointer looks suspicious, but happens to be a valid
address.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>line 56:</term>
<listitem>
<para>However, it obviously points to garbage, so we have found our
error! (For those unfamiliar with that particular piece of code:
<literal>tp->t_line</literal> refers to the line discipline of
the console device here, which must be a rather small integer
number.)</para>
</listitem>
</varlistentry>
</variablelist>
</sect1>
<sect1>
<title>Debugging a Crash Dump with DDD</title>
<para>Examining a kernel crash dump with a graphical debugger like
<command>ddd</command> is also possible. Add the <option>-k</option>
option to the <command>ddd</command> command line you would use
normally. For example;</para>
<screen>&prompt.root; <userinput>ddd -k /var/crash/kernel.0 /var/crash/vmcore.0</userinput></screen>
<para>You should then be able to go about looking at the crash dump using
<command>ddd</command>'s graphical interface.</para>
</sect1>
<sect1>
<title>Post-Mortem Analysis of a Dump</title>
<para>What do you do if a kernel dumped core but you did not expect it,
and it is therefore not compiled using <command>config -g</command>? Not
everything is lost here. Do not panic!</para>
<para>Of course, you still need to enable crash dumps. See above on the
options you have to specify in order to do this.</para>
<para>Go to your kernel config directory
(<filename>/usr/src/sys/<replaceable>arch</replaceable>/conf</filename>)
and edit your configuration file. Uncomment (or add, if it does not
exist) the following line</para>
<programlisting>makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols</programlisting>
<para>Rebuild the kernel. Due to the time stamp change on the Makefile,
there will be some other object files rebuild, for example
<filename>trap.o</filename>. With a bit of luck, the added
<option>-g</option> option will not change anything for the generated
code, so you will finally get a new kernel with similar code to the
faulting one but some debugging symbols. You should at least verify the
old and new sizes with the &man.size.1; command. If there is a
mismatch, you probably need to give up here.</para>
<para>Go and examine the dump as described above. The debugging symbols
might be incomplete for some places, as can be seen in the stack trace
in the example above where some functions are displayed without line
numbers and argument lists. If you need more debugging symbols, remove
the appropriate object files and repeat the <command>gdb <option>-k</option></command>
session until you know enough.</para>
<para>All this is not guaranteed to work, but it will do it fine in most
cases.</para>
</sect1>
<sect1>
<title>On-Line Kernel Debugging Using DDB</title>
<para>While <command>gdb <option>-k</option></command> as an off-line debugger provides a very
high level of user interface, there are some things it cannot do. The
most important ones being breakpointing and single-stepping kernel
code.</para>
<para>If you need to do low-level debugging on your kernel, there is an
on-line debugger available called DDB. It allows to setting
breakpoints, single-stepping kernel functions, examining and changing
kernel variables, etc. However, it cannot access kernel source files,
and only has access to the global and static symbols, not to the full
debug information like <command>gdb</command>.</para>
<para>To configure your kernel to include DDB, add the option line
<programlisting>options DDB</programlisting>
to your config file, and rebuild. (See <link
linkend="kernelconfig">Kernel Configuration</link> for details on
configuring the FreeBSD kernel.</para>
<note>
<para>Note that if you have an older version of the boot blocks, your
debugger symbols might not be loaded at all. Update the boot blocks;
the recent ones load the DDB symbols automagically.)</para>
</note>
<para>Once your DDB kernel is running, there are several ways to enter
DDB. The first, and earliest way is to type the boot flag
<option>-d</option> right at the boot prompt. The kernel will start up
in debug mode and enter DDB prior to any device probing. Hence you can
even debug the device probe/attach functions.</para>
<para>The second scenario is a hot-key on the keyboard, usually
Ctrl-Alt-ESC. For syscons, this can be remapped; some of the
distributed maps do this, so watch out. There is an option available
for serial consoles that allows the use of a serial line BREAK on the
console line to enter DDB (<literal>options BREAK_TO_DEBUGGER</literal>
in the kernel config file). It is not the default since there are a lot
of crappy serial adapters around that gratuitously generate a BREAK
condition, for example when pulling the cable.</para>
<para>The third way is that any panic condition will branch to DDB if the
kernel is configured to use it. For this reason, it is not wise to
configure a kernel with DDB for a machine running unattended.</para>
<para>The DDB commands roughly resemble some <command>gdb</command>
commands. The first thing you probably need to do is to set a
breakpoint:</para>
<screen><userinput>b function-name</userinput>
<userinput>b address</userinput></screen>
<para>Numbers are taken hexadecimal by default, but to make them distinct
from symbol names; hexadecimal numbers starting with the letters
<literal>a-f</literal> need to be preceded with <literal>0x</literal>
(this is optional for other numbers). Simple expressions are allowed,
for example: <literal>function-name + 0x103</literal>.</para>
<para>To continue the operation of an interrupted kernel, simply
type:</para>
<screen><userinput>c</userinput></screen>
<para>To get a stack trace, use:</para>
<screen><userinput>trace</userinput></screen>
<note>
<para>Note that when entering DDB via a hot-key, the kernel is currently
servicing an interrupt, so the stack trace might be not of much use
for you.</para>
</note>
<para>If you want to remove a breakpoint, use</para>
<screen><userinput>del</userinput>
<userinput>del address-expression</userinput></screen>
<para>The first form will be accepted immediately after a breakpoint hit,
and deletes the current breakpoint. The second form can remove any
breakpoint, but you need to specify the exact address; this can be
obtained from:</para>
<screen><userinput>show b</userinput></screen>
<para>To single-step the kernel, try:</para>
<screen><userinput>s</userinput></screen>
<para>This will step into functions, but you can make DDB trace them until
the matching return statement is reached by:</para>
<screen><userinput>n</userinput></screen>
<note>
<para>This is different from <command>gdb</command>'s
<command>next</command> statement; it is like <command>gdb</command>'s
<command>finish</command>.</para>
</note>
<para>To examine data from memory, use (for example):
<screen><userinput>x/wx 0xf0133fe0,40</userinput>
<userinput>x/hd db_symtab_space</userinput>
<userinput>x/bc termbuf,10</userinput>
<userinput>x/s stringbuf</userinput></screen>
for word/halfword/byte access, and hexadecimal/decimal/character/ string
display. The number after the comma is the object count. To display
the next 0x10 items, simply use:</para>
<screen><userinput>x ,10</userinput></screen>
<para>Similarly, use
<screen><userinput>x/ia foofunc,10</userinput></screen>
to disassemble the first 0x10 instructions of
<function>foofunc</function>, and display them along with their offset
from the beginning of <function>foofunc</function>.</para>
<para>To modify memory, use the write command:</para>
<screen><userinput>w/b termbuf 0xa 0xb 0</userinput>
<userinput>w/w 0xf0010030 0 0</userinput></screen>
<para>The command modifier
(<literal>b</literal>/<literal>h</literal>/<literal>w</literal>)
specifies the size of the data to be written, the first following
expression is the address to write to and the remainder is interpreted
as data to write to successive memory locations.</para>
<para>If you need to know the current registers, use:</para>
<screen><userinput>show reg</userinput></screen>
<para>Alternatively, you can display a single register value by e.g.
<screen><userinput>p $eax</userinput></screen>
and modify it by:</para>
<screen><userinput>set $eax new-value</userinput></screen>
<para>Should you need to call some kernel functions from DDB, simply
say:</para>
<screen><userinput>call func(arg1, arg2, ...)</userinput></screen>
<para>The return value will be printed.</para>
<para>For a &man.ps.1; style summary of all running processes, use:</para>
<screen><userinput>ps</userinput></screen>
<para>Now you have now examined why your kernel failed, and you wish to
reboot. Remember that, depending on the severity of previous
malfunctioning, not all parts of the kernel might still be working as
expected. Perform one of the following actions to shut down and reboot
your system:</para>
<screen><userinput>panic</userinput></screen>
<para>This will cause your kernel to dump core and reboot, so you can
later analyze the core on a higher level with <command>gdb</command>. This command
usually must be followed by another <command>continue</command>
statement.</para>
<screen><userinput>call boot(0)</userinput></screen>
<para>Which might be a good way to cleanly shut down the running system,
<function>sync()</function> all disks, and finally reboot. As long as
the disk and file system interfaces of the kernel are not damaged, this
might be a good way for an almost clean shutdown.</para>
<screen><userinput>call cpu_reset()</userinput></screen>
<para>is the final way out of disaster and almost the same as hitting the
Big Red Button.</para>
<para>If you need a short command summary, simply type:</para>
<screen><userinput>help</userinput></screen>
<para>However, it is highly recommended to have a printed copy of the
&man.ddb.4; manual page ready for a debugging
session. Remember that it is hard to read the on-line manual while
single-stepping the kernel.</para>
</sect1>
<sect1>
<title>On-Line Kernel Debugging Using Remote GDB</title>
<para>This feature has been supported since FreeBSD 2.2, and it is
actually a very neat one.</para>
<para>GDB has already supported <emphasis>remote debugging</emphasis> for
a long time. This is done using a very simple protocol along a serial
line. Unlike the other methods described above, you will need two
machines for doing this. One is the host providing the debugging
environment, including all the sources, and a copy of the kernel binary
with all the symbols in it, and the other one is the target machine that
simply runs a similar copy of the very same kernel (but stripped of the
debugging information).</para>
<para>You should configure the kernel in question with <command>config
-g</command>, include <option>DDB</option> into the configuration, and
compile it as usual. This gives a large blurb of a binary, due to the
debugging information. Copy this kernel to the target machine, strip
the debugging symbols off with <command>strip -x</command>, and boot it
using the <option>-d</option> boot option. Connect the serial line
of the target machine that has "flags 080" set on its sio device
to any serial line of the debugging host.
Now, on the debugging machine, go to the compile directory of the target
kernel, and start <command>gdb</command>:</para>
<screen>&prompt.user; <userinput>gdb -k kernel</userinput>
GDB is free software and you are welcome to distribute copies of it
under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.16 (i386-unknown-freebsd),
Copyright 1996 Free Software Foundation, Inc...
<prompt>(kgdb)</prompt> </screen>
<para>Initialize the remote debugging session (assuming the first serial
port is being used) by:</para>
<screen><prompt>(kgdb)</prompt> <userinput>target remote /dev/cuaa0</userinput></screen>
<para>Now, on the target host (the one that entered DDB right before even
starting the device probe), type:</para>
<screen>Debugger("Boot flags requested debugger")
Stopped at Debugger+0x35: movb $0, edata+0x51bc
<prompt>db></prompt> <userinput>gdb</userinput></screen>
<para>DDB will respond with:</para>
<screen>Next trap will enter GDB remote protocol mode</screen>
<para>Every time you type <command>gdb</command>, the mode will be toggled
between remote GDB and local DDB. In order to force a next trap
immediately, simply type <command>s</command> (step). Your hosting GDB
will now gain control over the target kernel:</para>
<screen>Remote debugging using /dev/cuaa0
Debugger (msg=0xf01b0383 "Boot flags requested debugger")
at ../../i386/i386/db_interface.c:257
<prompt>(kgdb)</prompt></screen>
<para>You can use this session almost as any other GDB session, including
full access to the source, running it in gud-mode inside an Emacs window
(which gives you an automatic source code display in another Emacs
window) etc.</para>
</sect1>
<sect1>
<title>Debugging Loadable Modules Using GDB</title>
<para>When debugging a panic that occurred within a module, or
using remote GDB against a machine that uses dynamic modules,
you need to tell GDB how to obtain symbol information for those
modules.</para>
<para>First, you need to build the module(s) with debugging
information:</para>
<screen>&prompt.root; <userinput>cd /sys/modules/linux</userinput>
&prompt.root; <userinput>make clean; make COPTS=-g</userinput></screen>
<para>If you are using remote GDB, you can run
<command>kldstat</command> on the target machine to find out
where the module was loaded:</para>
<screen>&prompt.root; <userinput>kldstat</userinput>
Id Refs Address Size Name
1 4 0xc0100000 1c1678 kernel
2 1 0xc0a9e000 6000 linprocfs.ko
3 1 0xc0ad7000 2000 warp_saver.ko
4 1 0xc0adc000 11000 linux.ko
</screen>
<para>If you are debugging a crash dump, you'll need to walk the
<literal>linker_files</literal> list, starting at
<literal>linker_files->tqh_first</literal> and following the
<literal>link.tqe_next</literal> pointers until you find the
entry with the <literal>filename</literal> you are looking for.
The <literal>address</literal> member of that entry is the load
address of the module.</para>
<para>Next, you need to find out the offset of the text section
within the module:</para>
<screen>&prompt.root; <userinput>objdump --section-headers /sys/modules/linux/linux.ko | grep text</userinput>
3 .rel.text 000016e0 000038e0 000038e0 000038e0 2**2
10 .text 00007f34 000062d0 000062d0 000062d0 2**2</screen>
<para>The one you want is the <literal>.text</literal> section,
section 10 in the above example. The fourth hexadecimal field
(sixth field overall) is the offset of the text section within
the file. Add this offset to the load address of the module to
obtain the relocation address for the module's code. In our
example, we get 0xc0adc000 + 0x62d0 = 0xc0ae22d0. Use the
<command>add-symbol-file</command> command in GDB to tell the
debugger about the module:</para>
<screen><prompt>(kgdb)</prompt> <userinput>add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0</userinput>
add symbol table from file "/sys/modules/linux/linux.ko" at text_addr = 0xc0ae22d0?
(y or n) <userinput>y</userinput>
Reading symbols from /sys/modules/linux/linux.ko...done.
<prompt>(kgdb)</prompt></screen>
<para>You should now have access to all the symbols in the
module.</para>
</sect1>
<sect1>
<title>Debugging a Console Driver</title>
<para>Since you need a console driver to run DDB on, things are more
complicated if the console driver itself is failing. You might remember
the use of a serial console (either with modified boot blocks, or by
specifying <option>-h</option> at the <prompt>Boot:</prompt> prompt),
and hook up a standard terminal onto your first serial port. DDB works
on any configured console driver, of course also on a serial
console.</para>
</sect1>
</chapter>
<!--
Local Variables:
mode: sgml
sgml-declaration: "../chapter.decl"
sgml-indent-data: t
sgml-omittag: nil
sgml-always-quote-attributes: t
sgml-parent-document: ("../book.sgml" "part" "chapter")
End:
-->
|