Not to be fixed
Created: Mar 10, 2013
Updated: Apr 19, 2018
Resolved Date: Apr 17, 2018
Previous ID: LIN4-14509
Found In Version: 6.0
Severity: Severe
Applicable for: Wind River Linux 6
Component/s: Kernel
Architecture: MIPS
Use the nlm_xlp_64_be BSP, and kdump/kexec, after a kernel crash triggered and the dump-capture kernel boot, there is no vmcore in /proc generated.
And there is a warning at startup:
Warning: Core image elf header not found
Kdump: vmcore not initialized
The customer followed the insructions in the BSP README, and the Kernel options are all identical with the BSP README.
They configured the project with the following options:
$ /opt/WindRiver4.3/wrlinux-4/wrlinux/../ldat/configure --enable-board=nlm_xlp_64_be --enable-kernel=cgl --enable-rootfs=glibc_cgl --enable-jobs=2 --enable-ccache=yes --with-product-dir=/opt/WindRiver4.3/wrlinux-4/wrlinux --with-template=cpu/mips64_el_n32
Here is the full log of the dump-capture kernel:
[root@SCM3] /mnt # echo c > /proc/sysrq-trigger SysRq : Trigger a crash
CPU 5 Unable to handle kernel paging request at virtual address 0000000000000000, epc == ffffffff8128ec8c, ra == ffffffff8128f2e4
Oops[#1]:
Cpu 5
$ 0 : 0000000000000000 0000000000000000 0000000000000001 ffffffff83510000
$ 4 : 0000000000000063 0000000000000000 ffffffffffffffff 0000000000005234
$ 8 : ffffffffffffffff 0000000000007fff 0000000000000001 00000000101da8d5
$12 : 0000000000000000 ffffffffc0000008 ffffffff81142e88 0000000000000070
$16 : 0000000000000063 ffffffff83660000 0000000000000000 ffffffff81570000
$20 : ffffffff81596d70 0000000000000001 0000000000000007 ffffffff81590000
$24 : 0000000000000018 ffffffff81039e30
$28 : a8000001087a4000 a8000001087a7d18 000000000000006e ffffffff8128f2e4
Hi : 0000000000000000
Lo : 000002e35568c000
epc : ffffffff8128ec8c sysrq_handle_crash+0x14/0x20
Not tainted
ra : ffffffff8128f2e4 __handle_sysrq+0x164/0x218
Status: 7400f8e2 KX SX UX KERNEL EXL
Cause : c080800c
BadVA : 0000000000000000
PrId : 000c1004 (XLP432 Rev B1)
Modules linked in: hdml scm3 softdog cmt cmm pio clock linux_kernel_bde linux_uk_proxy linux_bcm_core mpt2sas
Process sh (pid: 338, threadinfo=a8000001087a4000, task=a8000001004ee808, tls=00000000102293a0)
Stack : 0000000000000000 ffffffff8100ffd4 a8000001042db540 a800000104705900
ffffffff8128f398 0000000000000001 0000000000000030 0000000000000063
0000000000000001 000000000000005c 000000000000006e ffffffff8128f3fc
0000000000000002 ffffffff8100fa94 a8000001042db540 ffffffff8119d34c
000000002aaa8000 0000000000000002 a8000001087a7e10 a800000104705900
a800000104705900 000000002aaa8000 a8000001087a7e10 ffffffff81142ca4
000000001022343c 0000000010223030 a800000104705900 000000002aaa8000
0000000000000002 ffffffff81142ee0 0000000000000000 0000000000000000
000000001022343c 0000000010223030 0000000000000002 000000001021c800
000000002aaa8000 0000000000000002 0000000000000030 ffffffff81003824
...
Call Trace:
[<ffffffff8128ec8c>] sysrq_handle_crash+0x14/0x20
[<ffffffff8128f2e4>] __handle_sysrq+0x164/0x218
[<ffffffff8128f3fc>] write_sysrq_trigger+0x64/0x70
[<ffffffff8119d34c>] proc_reg_write+0x8c/0xd8
[<ffffffff81142ca4>] vfs_write+0xb4/0x148
[<ffffffff81142ee0>] SyS_write+0x58/0x148
[<ffffffff81003824>] handle_sysn32+0x44/0x94
Code: ac629d3c 0000000f 000000c0 <03e00008> a0020000 00000000 2482ffd0 2c43000a 14600006
Disabling lock debugging due to kernel taint
Fatal exception: panic in 5 seconds
Kernel panic - not syncing: Fatal exception
Will call new kernel at 040063f0
Bye ...
Linux version 2.6.34.13-grsec-WR4.3.0.0_cgl (gcc version 4.4.1 (Wind River Linux Sourcery G++ 4.4a-409) ) #38 SMP PREEMPT Thu Feb 28 11:21:22 CST 2013
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 0
bootconsole [early0] enabled
Cached handle for FDT @ ffffffff80100000
FDT Cmdline: kexec mem=130036K@64512K elfcorehdr=194548K savemaxmem=4752M
FDT: cpu_cells: 1 onlinemask[0]: 11111111
xlp_napi_vc_mask 0xb
fdt_cpumask: 00000000,00000000,00000000,11111111
ICI config not enabled
MSGRING_NAPI: Initializing NLM NAPI subsystem
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
User-defined physical RAM map:
memory: 0000000007efd000 @ 0000000003f00000 (usable)
Wasting 903168 bytes for tracking 16128 unused pages
Initrd not found or empty - disabling initrd
Zone PFN ranges:
DMA 0x00003f00 -> 0x00080000
Normal empty
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0x00003f00 -> 0x0000bdfd
Online mask ffffffff,845d65c8,ffffffff,845d65b8
Present CPU map 00000000,00000000,00000000,000000ff
Possible CPU map 00000000,00000000,00000000,000000ff
Detected 8 Slave CPU(s)
PERCPU: Embedded 10 pages/cpu @a800000004dab000 s10432 r8192 d22336 u65536
pcpu-alloc: s10432 r8192 d22336 u65536 alloc=16*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7
Built 1 zonelists in Zone order, mobility grouping off. Total pages: 32064
Kernel command line: kexec mem=130036K@64512K elfcorehdr=194548K savemaxmem=4752M root=/dev/ram console=ttyS0,115200
PID hash table entries: 512 (order: 0, 4096 bytes)
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Primary instruction cache 64kB, 2-way, linesize 32 bytes.
Primary data cache 32kB 2-way, linesize 32 bytes.
mmu_init: cpu = 0, tlbsize = 2176
Setting up NMI Handler
Memory: 114500k/130036k available (1883k kernel code, 15536k reserved, 1473k data, 2400k init, 0k highmem)
Hierarchical RCU implementation.
NR_IRQS:1536
mips_hpt_frequency = 1500006000
console [ttyS0] enabled, bootconsole disabled
console [ttyS0] enabled, bootconsole disabled
Calibrating delay loop... 1495.04 BogoMIPS (lpj=2990080)
Mount-cache hash table entries: 256
Checking for the daddi bug... no.
Init trace_clock_cyc2ns: precalc_mult = 170, precalc_shift = 8
nlm_boot_secondary: logical cpu 1 physical cpu 0
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 1
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 1, tlbsize = 2176
nlm_boot_secondary: logical cpu 2 physical cpu 4
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 2
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 2, tlbsize = 2176
nlm_boot_secondary: logical cpu 3 physical cpu 8
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 3
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 3, tlbsize = 2176
nlm_boot_secondary: logical cpu 4 physical cpu 12
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 4
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 4, tlbsize = 2176
nlm_boot_secondary: logical cpu 5 physical cpu 16
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 5
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 5, tlbsize = 2176
nlm_boot_secondary: logical cpu 6 physical cpu 24
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 6
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 6, tlbsize = 2176
nlm_boot_secondary: logical cpu 7 physical cpu 28
Enabling XLP CPU (XLP432 Rev B1): pr id 0xc1004 smp id 7
CPU revision is: 000c1004 (XLP432 Rev B1)
FPU revision is: 00770000
mmu_init: cpu = 7, tlbsize = 2176
Brought up 8 CPUs
Synchronize counters across 8 CPUs: done.
checking TSC synchronization across all online CPUs:
Measured 4294967165 cycles TSC offset between CPUs, turning off TSC clock.
NET: Registered protocol family 16
check_res_of_trace_clock: sched_clock() low resolution
check_res_of_trace_clock: trace_clock_read64_ns() has high resolution
LTSSM state is 0x0. Fn 1 link not up
LTSSM state is 0x0. Fn 2 link not up
LTSSM state is 0x0. Fn 3 link not up
node_irqmap[0][3] PLC 3
p->farray[0] : int 86, sec 1, sub 8
p->farray[1] : int 87, sec 0, sub 0
p->farray[2] : int 88, sec 0, sub 0
p->farray[3] : int 89, sec 0, sub 0
bio: create slab <bio-0> at 0
SCSI subsystem initialized
pci 0000:00:01.0: PCI bridge to [bus 01-ff]
pci 0000:01:00.0: PCI bridge to [bus 02-ff]
pci 0000:02:01.0: PCI bridge to [bus 03-ff]
pci 0000:02:02.0: PCI bridge to [bus 04-ff]
pci 0000:02:03.0: PCI bridge to [bus 05-ff]
pci 0000:02:0b.0: PCI bridge to [bus 06-ff]
pci 0000:02:0d.0: PCI bridge to [bus 07-ff]
pci 0000:02:0f.0: PCI bridge to [bus 08-ff]
pci 0000:00:01.1: PCI bridge to [bus 09-ff]
pci 0000:00:01.2: PCI bridge to [bus 0a-ff]
pci 0000:00:01.3: PCI bridge to [bus 0b-ff]
Switching to clocksource MIPS
created nlm_timer proc fs entry
init_vdso successfull
Warning: Core image elf header not found
Kdump: vmcore not initialized
msgmni has been set to 223
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
serial8250: ttyS0 at MMIO 0xffffffffb8030100 (irq = 141) is a 16550A
serial8250.0: ttyS0 at MMIO 0xffffffffb8030100 (irq = 141) is a 16550A
serial8250.0: ttyS1 at MMIO 0xffffffffb8031100 (irq = 142) is a 16550A
serial8250.1: ttyS1 at MMIO 0xffffffffb8031100 (irq = 142) is a 16550A
brd: module loaded
i2c /dev entries driver
rtc-m41t80 0-0068: chip found, driver version 0.05
rtc-m41t80 0-0068: rtc core: registered m41t11 as rtc0
rtc-m41t80 0-0068: setting system clock to 2013-02-28 13:57:29 UTC (1362059849)
Could not find a match for specified cpumask
Freeing unused kernel memory: 2400k freed
Please press Enter to activate this console.
/ #
1. configure a project with the following options:
$ /opt/WindRiver4.3/wrlinux-4/wrlinux/../ldat/configure --enable-board=nlm_xlp_64_be --enable-kernel=cgl --enable-rootfs=glibc_cgl --enable-jobs=2 --enable-ccache=yes --with-product-dir=/opt/WindRiver4.3/wrlinux-4/wrlinux --with-template=cpu/mips64_el_n32
2. set the kernel options for the first kernek and the dump-capture kernel, and boot the first krenel.
3. trigger a kernel crash by echo c > /proc/sysrq-trigger SysRq
