Wind River Support Network

HomeDefectsLIN7-1827
Not to be fixed

LIN7-1827 : fsl-t2xxx kdump caught issue with calltrace after scp vmcore

Created: Nov 18, 2014    Updated: Apr 19, 2018
Resolved Date: Apr 17, 2018
Found In Version: 7.0
Severity: Standard
Applicable for: Wind River Linux 7
Component/s: BSP

Description

/oot@t2080rdb:~# scp /proc/vmcore xxx@xxx:/mnt/t2xxx/rt/kdump 


vmcore                                          0%    0     0.0KB/s   --:-- ETAUnable to handle kernel paging request for data at address 0x8458b5118458bb06
Faulting instruction address: 0xc0000000028e8820
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 0 PID: 18 Comm: ksoftirqd/0 Not tainted 3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e1e3100 ti: c00000000e1fc000 task.ti: c00000000e1fc000
NIP: c0000000028e8820 LR: c0000000028e89a8 CTR: c000000002935220
REGS: c00000000e1ff4c0 TRAP: 0300   Not tainted  (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080029000 <CE,EE,ME>  CR: 24ad4e24  XER: 00000000
DEAR: 8458b5118458bb06 ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000028e89a8 c00000000e1ff740 c00000000326b850 0000000000000017 
GPR04: 0000000000000000 0000000000000000 c0000000067781e0 c0000000066ccc00 
GPR08: 0000000000000004 8458b5118458bab9 c000000006849c00 c00000000e1c0000 
GPR12: 0000000024adbe22 c00000000fffc000 c000000002096f78 c00000000e00c6c0 
GPR16: 0000000000000000 c00000000e215490 c00000000e215890 c00000000e215c90 
GPR20: c00000000e215090 c000000003277380 0000000000200200 c00000000e1ffa90 
GPR24: c00000000e1fc000 c000000006778000 0000000000000000 c000000003253ea8 
GPR28: 0000000000000000 0000000000000000 0000000000000001 c000000006778000 
NIP [c0000000028e8820] .tcp_enter_loss+0x178/0x390
LR [c0000000028e89a8] .tcp_enter_loss+0x300/0x390
Call Trace:
[c00000000e1ff740] [c0000000028e89a8] .tcp_enter_loss+0x300/0x390 (unreliable)
[c00000000e1ff7d0] [c0000000028f3374] .tcp_retransmit_timer+0x448/0x81c
[c00000000e1ff860] [c0000000028f3820] .tcp_write_timer_handler+0xd8/0x204
[c00000000e1ff8e0] [c0000000028f3a38] .tcp_write_timer+0xec/0xf4
[c00000000e1ff970] [c000000002079f80] .call_timer_fn+0x70/0x22c
[c00000000e1ffa20] [c00000000207a3e0] .run_timer_softirq+0x270/0x378
[c00000000e1ffb20] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e1ffc00] [c00000000206fd24] .run_ksoftirqd+0x40/0x7c
[c00000000e1ffc80] [c0000000020a20bc] .smpboot_thread_fn+0x2c4/0x418
[c00000000e1ffd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e1ffe30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
40be0030 4800023c 409a004c 554ae8be 554a183e 9949004d e9290000 7fa93040 
419e0068 e95f0390 7faa4800 419e005c <8949004d> 7948ffe3 4182000c 90bf0830 
vmcore                                          0% 3840KB 640.4---[ end trace 0000000000000002 ]---

KB/s - stalled -BUG: sleeping function called from invalid context att2xxx/rt/kdump/bitbake_build/tmp/work/fsl_t2xxx-wrs-linux/linux-windriver/3.14-r0/linux/kernel/locking/rtmutex.c:1626
in_atomic(): 0, irqs_disabled(): 1, pid: 18, name: ksoftirqd/0
CPU: 0 PID: 18 Comm: ksoftirqd/0 Tainted: G      D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Call Trace:
[c00000000e1fee90] [c00000000200a76c] .show_stack+0x168/0x278 (unreliable)
[c00000000e1fef80] [c0000000029e023c] .dump_stack+0x9c/0x108
[c00000000e1ff000] [c0000000020a682c] .__might_sleep+0x148/0x1b0
[c00000000e1ff090] [c0000000029dbe04] .rt_mutex_lock+0x30/0x9c
[c00000000e1ff110] [c0000000020c7c5c] .rt_down_read+0x20/0x34
[c00000000e1ff190] [c000000002080d68] .exit_signals+0x28/0x178
[c00000000e1ff220] [c00000000206b83c] .do_exit+0xc4/0xb10
[c00000000e1ff320] [c000000002014fac] .die+0x2f8/0x410
[c00000000e1ff3d0] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e1ff450] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at .tcp_enter_loss+0x178/0x390
    LR = .tcp_enter_loss+0x300/0x390
[c00000000e1ff7d0] [c0000000028f3374] .tcp_retransmit_timer+0x448/0x81c
[c00000000e1ff860] [c0000000028f3820] .tcp_write_timer_handler+0xd8/0x204
[c00000000e1ff8e0] [c0000000028f3a38] .tcp_write_timer+0xec/0xf4
[c00000000e1ff970] [c000000002079f80] .call_timer_fn+0x70/0x22c
[c00000000e1ffa20] [c00000000207a3e0] .run_timer_softirq+0x270/0x378
[c00000000e1ffb20] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e1ffc00] [c00000000206fd24] .run_ksoftirqd+0x40/0x7c
[c00000000e1ffc80] [c0000000020a20bc] .smpboot_thread_fn+0x2c4/0x418
[c00000000e1ffd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e1ffe30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
BUG: Bad page state in process irq/126-QMan po  pfn:06448
page:c00000000355afc0 count:-1 mapcount:0 mapping:          (null) index:0x0
page flags: 0x0()
page dumped because: nonzero _count
Modules linked in:
vmcore                                          0% 3840KB  11.7KB/s - stalled -CPU: 6 PID: 112 Comm: irq/126-QMan po Tainted: G      D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Call Trace:
[c00000000e743030] [c00000000200a76c] .show_stack+0x168/0x278 (unreliable)
[c00000000e743120] [c0000000029e023c] .dump_stack+0x9c/0x108
[c00000000e7431a0] [c00000000215f490] .bad_page+0x14c/0x1a8
[c00000000e743240] [c00000000215fbf4] .get_page_from_freelist+0x4e0/0xa08
[c00000000e7433a0] [c000000002161554] .__alloc_pages_nodemask+0x1c0/0xc68
[c00000000e743560] [c0000000028694f4] .__netdev_alloc_frag+0x218/0x318
[c00000000e743620] [c0000000026f0f00] ._dpa_bp_add_8_bufs+0x70/0x320
[c00000000e743720] [c0000000026f1340] .dpaa_eth_refill_bpools+0x68/0x8c
[c00000000e7437b0] [c0000000026ef8b0] .priv_rx_default_dqrr+0xa0/0x26c
[c00000000e743860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e743920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7439b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e743a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e743b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e743be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e743c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e743d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e743e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Unable to handle kernel paging request for data at address 0x9b4384d12b57cc7e
Faulting instruction address: 0xc0000000026f1738
Oops: Kernel access of bad area, sig: 11 [#2]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 0 PID: 106 Comm: irq/138-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e66a4c0 ti: c00000000e704000 task.ti: c00000000e704000
NIP: c0000000026f1738 LR: c0000000026f1700 CTR: c000000002018738
Unable to handle kernel paging request for data at address 0xe6660000000000d0
REGS: c00000000e707410 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
Faulting instruction address: 0xc0000000026f1738
MSR: 0000000080029000 <CE,EE,ME>  CR: 24ad4e28  XER: 00000000
DEAR: 9b4384d12b57cc7e ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000026ef8dc c00000000e707690 c00000000326b850 c00000000ed36010 
GPR04: 0000000006dfa638 0000000000000680 0000000000000000 0000000000000000 
GPR08: 000000000000013c 0800005e00000000 c000000002018738 c00000000e1c0000 
GPR12: 0000000024adbe24 c00000000fffc000 0000000000000080 c00000000e700140 
GPR16: 80000800801e9190 0000000000000000 c00000000316f300 0000000000000000 
GPR20: 0000000000000040 c000000003803b80 0000000000000000 ffffffffffffffff 
GPR24: c00000000efc4800 c00000000efc4800 c00000000efc4000 c00000000380a2f8 
GPR28: 0000000006dfa638 c00000000380a8a8 c000000006dfa638 9b4384d12b57cbae 
NIP [c0000000026f1738] ._dpa_rx+0x124/0x780
LR [c0000000026f1700] ._dpa_rx+0xec/0x780
Call Trace:
[c00000000e707690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e7077b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e707860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e707920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7079b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e707a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e707b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e707be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e707c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e707d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e707e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
7b9c0600 e9300008 3bc0ffff 7bde0044 792e65e0 7ffff214 7fff7214 7c00fa2c 
79270085 40820198 7fdcf214 ebfefff8 <e95f00d0> 7fe3fb78 813f00bc 7d297214 
Unable to handle kernel paging request for data at address 0xb392f772be8f9a66
Faulting instruction address: 0xc0000000026f1738
Unable to handle kernel paging request for data at address 0xe6660000000000d0
Faulting instruction address: 0xc0000000026f1738
Unable to handle kernel paging request for data at address 0x80c0dc31ee7e851e
Faulting instruction address: 0xc0000000026f1738
Unable to handle kernel paging request for data at address 0x000000d0
Faulting instruction address: 0xc0000000026f1738
---[ end trace 0000000000000003 ]---

Oops: Kernel access of bad area, sig: 11 [#3]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 4 PID: 110 Comm: irq/130-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e66d5c0 ti: c00000000e738000 task.ti: c00000000e738000
NIP: c0000000026f1738 LR: c0000000026f1700 CTR: c000000002018738
Unable to handle kernel paging request for data at address 0xffffffffffffffd8
REGS: c00000000e73b410 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
Faulting instruction address: 0xc000000002097540
MSR: 0000000080029000 <CE,EE,ME>  CR: 22ad4e28  XER: 00000000
DEAR: 80c0dc31ee7e851e ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000026ef8dc c00000000e73b690 c00000000326b850 c00000000ed36010 
GPR04: 000000000644dfb0 0000000000000680 0000000000000000 0000000000000000 
GPR08: 000000000000013c 0800009c00000000 c000000002018738 c00000000e1d8000 
GPR12: 0000000022adbe24 c00000000fffde00 0000000000000080 c00000000e700640 
GPR16: 80000800801c91d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: 0000000000000040 c000000003883b80 0000000000000000 ffffffffffffffff 
GPR24: c00000000efc4800 c00000000efc4800 c00000000efc4000 c00000000388a2f8 
GPR28: 000000000644dfb0 c00000000388a8a8 c00000000644dfb0 80c0dc31ee7e844e 
NIP [c0000000026f1738] ._dpa_rx+0x124/0x780
LR [c0000000026f1700] ._dpa_rx+0xec/0x780
Call Trace:
[c00000000e73b690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e73b7b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e73b860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e73b920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e73b9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e73ba90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e73bb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e73bbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e73bc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e73bd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e73be30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
7b9c0600 e9300008 3bc0ffff 7bde0044 792e65e0 7ffff214 7fff7214 7c00fa2c 
79270085 40820198 7fdcf214 ebfefff8 <e95f00d0> 7fe3fb78 813f00bc 7d297214 
---[ end trace 0000000000000004 ]---

Oops: Kernel access of bad area, sig: 11 [#4]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 6 PID: 112 Comm: irq/126-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Unable to handle kernel paging request for data at address 0xffffffffffffffd8
task: c00000000e66ee40 ti: c00000000e740000 task.ti: c00000000e740000
NIP: c0000000026f1738 LR: c0000000026f1700 CTR: c000000002018738
Faulting instruction address: 0xc000000002097540
REGS: c00000000e743410 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080029000 <CE,EE,ME>  CR: 24ad4e28  XER: 00000000
DEAR: b392f772be8f9a66 ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000026ef8dc c00000000e743690 c00000000326b850 c00000000ed36010 
GPR04: 000000000644e558 0000000000000680 0000000000000000 0000000000000000 
GPR08: 000000000000013c 0800005e00000000 c000000002018738 c00000000e1e8000 
GPR12: 0000000024adbe24 c00000000fffed00 0000000000000080 c00000000e7008c0 
GPR16: 80000800801b9190 0000000000000000 c00000000316f300 0000000000000000 
GPR20: 0000000000000040 c0000000038c3b80 0000000000000000 ffffffffffffffff 
GPR24: c00000000efc4800 c00000000efc4800 c00000000efc4000 c0000000038ca2f8 
GPR28: 000000000644e558 c0000000038ca8a8 c00000000644e558 b392f772be8f9996 
NIP [c0000000026f1738] ._dpa_rx+0x124/0x780
LR [c0000000026f1700] ._dpa_rx+0xec/0x780
Call Trace:
[c00000000e743690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e7437b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e743860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e743920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7439b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e743a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e743b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e743be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e743c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e743d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e743e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
7b9c0600 e9300008 3bc0ffff 7bde0044 792e65e0 7ffff214 7fff7214 7c00fa2c 
79270085 40820198 7fdcf214 ebfefff8 <e95f00d0> 7fe3fb78 813f00bc 7d297214 
---[ end trace 0000000000000005 ]---

Oops: Kernel access of bad area, sig: 11 [#5]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
Unable to handle kernel paging request for data at address 0xffffffffffffffd8
CPU: 5 PID: 111 Comm: irq/128-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Faulting instruction address: 0xc000000002097540
task: c00000000e66e200 ti: c00000000e73c000 task.ti: c00000000e73c000
NIP: c0000000026f1738 LR: c0000000026f1700 CTR: c000000002018738
REGS: c00000000e73f410 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080029000 <CE,EE,ME>  CR: 24ad4e28  XER: 00000000
DEAR: e6660000000000d0 ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000026ef8dc c00000000e73f690 c00000000326b850 c00000000ed36010 
GPR04: 000000000676e8ce 0000000000000680 0000000000000000 0000000000000000 
GPR08: 000000000000013c 0800009c00000000 c000000002018738 c00000000e1dc000 
GPR12: 0000000024adbe24 c00000000fffe580 0000000000000080 c00000000e700780 
GPR16: 80000800801c11d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: 0000000000000040 c0000000038a3b80 0000000000000000 ffffffffffffffff 
GPR24: c00000000efc4800 c00000000efc4800 c00000000efc4000 c0000000038aa2f8 
GPR28: 000000000676e8ce c0000000038aa8a8 c00000000676e8ce e666000000000000 
NIP [c0000000026f1738] ._dpa_rx+0x124/0x780
LR [c0000000026f1700] ._dpa_rx+0xec/0x780
Call Trace:
[c00000000e73f690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e73f7b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e73f860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e73f920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e73f9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e73fa90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e73fb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e73fbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e73fc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e73fd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e73fe30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
7b9c0600 e9300008 3bc0ffff 7bde0044 792e65e0 7ffff214 7fff7214 7c00fa2c 
79270085 40820198 7fdcf214 ebfefff8 <e95f00d0> 7fe3fb78 813f00bc 7d297214 
---[ end trace 0000000000000006 ]---

Oops: Kernel access of bad area, sig: 11 [#6]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
Unable to handle kernel paging request for data at address 0xffffffffffffffd8
CPU: 3 PID: 109 Comm: irq/132-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Faulting instruction address: 0xc000000002097540
task: c00000000e66c980 ti: c00000000e72c000 task.ti: c00000000e72c000
NIP: c0000000026f1738 LR: c0000000026f1700 CTR: c000000002018738
REGS: c00000000e72f410 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080029000 <CE,EE,ME>  CR: 24ad4e28  XER: 00000000
DEAR: 00000000000000d0 ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000026ef8dc c00000000e72f690 c00000000326b850 c00000000ed36010 
GPR04: 00000000067698ce 0000000000000680 0000000000000000 0000000000000000 
GPR08: 000000000000013c 0800005600000000 c000000002018738 c00000000e1d4000 
GPR12: 0000000024adbe24 c00000000fffd680 0000000000000080 c00000000e700500 
GPR16: 80000800801d11d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: 0000000000000040 c000000003863b80 0000000000000000 ffffffffffffffff 
GPR24: c00000000efc4800 c00000000efc4800 c00000000efc4000 c00000000386a2f8 
GPR28: 00000000067698ce c00000000386a8a8 c0000000067698ce 0000000000000000 
NIP [c0000000026f1738] ._dpa_rx+0x124/0x780
LR [c0000000026f1700] ._dpa_rx+0xec/0x780
Call Trace:
[c00000000e72f690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e72f7b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e72f860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e72f920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e72f9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e72fa90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e72fb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e72fbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e72fc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e72fd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e72fe30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
7b9c0600 e9300008 3bc0ffff 7bde0044 792e65e0 7ffff214 7fff7214 7c00fa2c 
79270085 40820198 7fdcf214 ebfefff8 <e95f00d0> 7fe3fb78 813f00bc 7d297214 
---[ end trace 0000000000000007 ]---

Oops: Kernel access of bad area, sig: 11 [#7]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 4 PID: 110 Comm: irq/130-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e66d5c0 ti: c00000000e738000 task.ti: c00000000e738000
NIP: c000000002097540 LR: c0000000020d281c CTR: c0000000020d27e4
REGS: c00000000e73ad50 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080021000 <CE,ME>  CR: 48ad4e82  XER: 20000000
DEAR: ffffffffffffffd8 ESR: 0000000000000000 SOFTE: 0 
GPR00: c0000000020d281c c00000000e73afd0 c00000000326b850 c00000000e66d5c0 
GPR04: 0000000000000389 0000000000000000 c00000000316b850 c00000000e66d5c0 
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000000 
GPR12: 0000000028ad4e82 c00000000fffde00 0000000000000080 c00000000e700640 
GPR16: 80000800801c91d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: c00000000e66d5c0 c000000003883b80 c00000000e66e060 ffffffffffffffff 
GPR24: c00000000e73b1e0 0000000000000004 c00000000e66e060 c000000002b10e90 
GPR28: c00000000e66d5c0 c00000000e66e008 c00000000e66d5c0 c00000000e66d5c0 
NIP [c000000002097540] .kthread_data+0x20/0x38
LR [c0000000020d281c] .irq_thread_dtor+0x38/0x104
Call Trace:
[c00000000e73afd0] [c0000000029dc808] ._raw_spin_unlock_irqrestore+0x60/0xf0 (unreliable)
[c00000000e73b050] [c0000000020d281c] .irq_thread_dtor+0x38/0x104
[c00000000e73b0d0] [c000000002092378] .task_work_run+0xf0/0x144
[c00000000e73b170] [c00000000206bb08] .do_exit+0x390/0xb10
[c00000000e73b270] [c000000002014fac] .die+0x2f8/0x410
[c00000000e73b320] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e73b3a0] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at ._dpa_rx+0x124/0x780
    LR = ._dpa_rx+0xec/0x780
[c00000000e73b690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e73b7b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e73b860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e73b920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e73b9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e73ba90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e73bb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e73bbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e73bc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e73bd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e73be30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
38210070 e8010010 7c0803a6 4e800020 7c0802a6 fbe1fff8 f8010010 f821ff81 
7c7f1b78 60000000 60000000 e93f03c8 <e869ffd8> 38210080 e8010010 ebe1fff8 
---[ end trace 0000000000000008 ]---

Oops: Kernel access of bad area, sig: 11 [#8]
Fixing recursive fault but reboot is needed!
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 6 PID: 112 Comm: irq/126-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e66ee40 ti: c00000000e740000 task.ti: c00000000e740000
NIP: c000000002097540 LR: c0000000020d281c CTR: c0000000020d27e4
REGS: c00000000e742d50 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080021000 <CE,ME>  CR: 48ad4e82  XER: 20000000
DEAR: ffffffffffffffd8 ESR: 0000000000000000 SOFTE: 0 
GPR00: c0000000020d281c c00000000e742fd0 c00000000326b850 c00000000e66ee40 
GPR04: 0000000000000389 0000000000000000 c00000000316b850 c00000000e66ee40 
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000000 
GPR12: 0000000028ad4e82 c00000000fffed00 0000000000000080 c00000000e7008c0 
GPR16: 80000800801b9190 0000000000000000 c00000000316f300 0000000000000000 
GPR20: c00000000e66ee40 c0000000038c3b80 c00000000e66f8e0 ffffffffffffffff 
GPR24: c00000000e7431e0 0000000000000006 c00000000e66f8e0 c000000002b10e90 
GPR28: c00000000e66ee40 c00000000e66f888 c00000000e66ee40 c00000000e66ee40 
NIP [c000000002097540] .kthread_data+0x20/0x38
LR [c0000000020d281c] .irq_thread_dtor+0x38/0x104
Call Trace:
[c00000000e742fd0] [c0000000029dc808] ._raw_spin_unlock_irqrestore+0x60/0xf0 (unreliable)
[c00000000e743050] [c0000000020d281c] .irq_thread_dtor+0x38/0x104
[c00000000e7430d0] [c000000002092378] .task_work_run+0xf0/0x144
[c00000000e743170] [c00000000206bb08] .do_exit+0x390/0xb10
[c00000000e743270] [c000000002014fac] .die+0x2f8/0x410
[c00000000e743320] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e7433a0] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at ._dpa_rx+0x124/0x780
    LR = ._dpa_rx+0xec/0x780
[c00000000e743690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e7437b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e743860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e743920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7439b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e743a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e743b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e743be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e743c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e743d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e743e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
38210070 e8010010 7c0803a6 4e800020 7c0802a6 fbe1fff8 f8010010 f821ff81 
7c7f1b78 60000000 60000000 e93f03c8 <e869ffd8> 38210080 e8010010 ebe1fff8 
---[ end trace 0000000000000009 ]---

Fixing recursive fault but reboot is needed!
Oops: Kernel access of bad area, sig: 11 [#9]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 7 PID: 113 Comm: irq/124-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e748000 ti: c00000000e744000 task.ti: c00000000e744000
NIP: c0000000026f1738 LR: c0000000026f1700 CTR: c000000002018738
REGS: c00000000e747410 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080029000 <CE,EE,ME>  CR: 28ad4e28  XER: 00000000
DEAR: e6660000000000d0 ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000026ef8dc c00000000e747690 c00000000326b850 c00000000ed36010 
GPR04: 000000000674a0ce 0000000000000680 0000000000000000 0000000000000000 
GPR08: 000000000000013c 0800005e00000000 c000000002018738 c000000003268000 
GPR12: 0000000028adbe24 c00000000ffff480 0000000000000080 c00000000e700a00 
GPR16: 80000800801b11d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: 0000000000000040 c0000000038e3b80 0000000000000000 ffffffffffffffff 
GPR24: c00000000efc4800 c00000000efc4800 c00000000efc4000 c0000000038ea2f8 
GPR28: 000000000674a0ce c0000000038ea8a8 c00000000674a0ce e666000000000000 
NIP [c0000000026f1738] ._dpa_rx+0x124/0x780
LR [c0000000026f1700] ._dpa_rx+0xec/0x780
Call Trace:
[c00000000e747690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e7477b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e747860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e747920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7479b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e747a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e747b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e747be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e747c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e747d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e747e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
7b9c0600 e9300008 3bc0ffff 7bde0044 792e65e0 7ffff214 7fff7214 7c00fa2c 
79270085 40820198 7fdcf214 ebfefff8 <e95f00d0> 7fe3fb78 813f00bc 7d297214 
---[ end trace 000000000000000a ]---

Oops: Kernel access of bad area, sig: 11 [#10]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
Unable to handle kernel paging request for data at address 0xffffffffffffffd8
CPU: 5 PID: 111 Comm: irq/128-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Faulting instruction address: 0xc000000002097540
task: c00000000e66e200 ti: c00000000e73c000 task.ti: c00000000e73c000
NIP: c000000002097540 LR: c0000000020d281c CTR: c0000000020d27e4
REGS: c00000000e73ed50 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080021000 <CE,ME>  CR: 48ad4e82  XER: 20000000
DEAR: ffffffffffffffd8 ESR: 0000000000000000 SOFTE: 0 
GPR00: c0000000020d281c c00000000e73efd0 c00000000326b850 c00000000e66e200 
GPR04: 0000000000000389 0000000000000000 c00000000316b850 c00000000e66e200 
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000000 
GPR12: 0000000028ad4e82 c00000000fffe580 0000000000000080 c00000000e700780 
GPR16: 80000800801c11d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: c00000000e66e200 c0000000038a3b80 c00000000e66eca0 ffffffffffffffff 
GPR24: c00000000e73f1e0 0000000000000005 c00000000e66eca0 c000000002b10e90 
GPR28: c00000000e66e200 c00000000e66ec48 c00000000e66e200 c00000000e66e200 
NIP [c000000002097540] .kthread_data+0x20/0x38
LR [c0000000020d281c] .irq_thread_dtor+0x38/0x104
Call Trace:
[c00000000e73efd0] [c0000000029dc808] ._raw_spin_unlock_irqrestore+0x60/0xf0 (unreliable)
[c00000000e73f050] [c0000000020d281c] .irq_thread_dtor+0x38/0x104
[c00000000e73f0d0] [c000000002092378] .task_work_run+0xf0/0x144
[c00000000e73f170] [c00000000206bb08] .do_exit+0x390/0xb10
[c00000000e73f270] [c000000002014fac] .die+0x2f8/0x410
[c00000000e73f320] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e73f3a0] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at ._dpa_rx+0x124/0x780
    LR = ._dpa_rx+0xec/0x780
[c00000000e73f690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e73f7b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e73f860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e73f920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e73f9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e73fa90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e73fb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e73fbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e73fc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e73fd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e73fe30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
38210070 e8010010 7c0803a6 4e800020 7c0802a6 fbe1fff8 f8010010 f821ff81 
7c7f1b78 60000000 60000000 e93f03c8 <e869ffd8> 38210080 e8010010 ebe1fff8 
---[ end trace 000000000000000b ]---

Fixing recursive fault but reboot is needed!
Oops: Kernel access of bad area, sig: 11 [#11]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 7 PID: 113 Comm: irq/124-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e748000 ti: c00000000e744000 task.ti: c00000000e744000
NIP: c000000002097540 LR: c0000000020d281c CTR: c0000000020d27e4
REGS: c00000000e746d50 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080021000 <CE,ME>  CR: 48ad4e82  XER: 20000000
DEAR: ffffffffffffffd8 ESR: 0000000000000000 SOFTE: 0 
GPR00: c0000000020d281c c00000000e746fd0 c00000000326b850 c00000000e748000 
GPR04: 0000000000000389 0000000000000000 c00000000316b850 c00000000e748000 
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000000 
GPR12: 0000000028ad4e82 c00000000ffff480 0000000000000080 c00000000e700a00 
GPR16: 80000800801b11d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: c00000000e748000 c0000000038e3b80 c00000000e748aa0 ffffffffffffffff 
GPR24: c00000000e7471e0 0000000000000007 c00000000e748aa0 c000000002b10e90 
GPR28: c00000000e748000 c00000000e748a48 c00000000e748000 c00000000e748000 
NIP [c000000002097540] .kthread_data+0x20/0x38
LR [c0000000020d281c] .irq_thread_dtor+0x38/0x104
Call Trace:
[c00000000e746fd0] [c0000000029dc808] ._raw_spin_unlock_irqrestore+0x60/0xf0 (unreliable)
[c00000000e747050] [c0000000020d281c] .irq_thread_dtor+0x38/0x104
[c00000000e7470d0] [c000000002092378] .task_work_run+0xf0/0x144
[c00000000e747170] [c00000000206bb08] .do_exit+0x390/0xb10
[c00000000e747270] [c000000002014fac] .die+0x2f8/0x410
[c00000000e747320] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e7473a0] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at ._dpa_rx+0x124/0x780
    LR = ._dpa_rx+0xec/0x780
[c00000000e747690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e7477b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e747860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e747920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7479b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e747a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e747b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e747be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e747c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e747d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e747e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
38210070 e8010010 7c0803a6 4e800020 7c0802a6 fbe1fff8 f8010010 f821ff81 
7c7f1b78 60000000 60000000 e93f03c8 <e869ffd8> 38210080 e8010010 ebe1fff8 
---[ end trace 000000000000000c ]---

Fixing recursive fault but reboot is needed!
Oops: Kernel access of bad area, sig: 11 [#12]
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 0 PID: 106 Comm: irq/138-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e66a4c0 ti: c00000000e704000 task.ti: c00000000e704000
NIP: c000000002097540 LR: c0000000020d281c CTR: c0000000020d27e4
REGS: c00000000e706d50 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080021000 <CE,ME>  CR: 48ad4e82  XER: 20000000
DEAR: ffffffffffffffd8 ESR: 0000000000000000 SOFTE: 0 
GPR00: c0000000020d281c c00000000e706fd0 c00000000326b850 c00000000e66a4c0 
GPR04: 0000000000000389 0000000000000000 c00000000316b850 c00000000e66a4c0 
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000000 
GPR12: 0000000028ad4e82 c00000000fffc000 0000000000000080 
Unable to handle kernel paging request for data at address 0xffffffffffffffd8
c00000000e700140 
GPR16: 
Faulting instruction address: 0xc000000002097540
80000800801e9190 0000000000000000 c00000000316f300 0000000000000000 
GPR20: c00000000e66a4c0 c000000003803b80 c00000000e66af60 ffffffffffffffff 
GPR24: c00000000e7071e0 0000000000000000 c00000000e66af60 c000000002b10e90 
GPR28: c00000000e66a4c0 c00000000e66af08 c00000000e66a4c0 c00000000e66a4c0 
NIP [c000000002097540] .kthread_data+0x20/0x38
LR [c0000000020d281c] .irq_thread_dtor+0x38/0x104
Call Trace:
[c00000000e706fd0] [c0000000029dc808] ._raw_spin_unlock_irqrestore+0x60/0xf0 (unreliable)
[c00000000e707050] [c0000000020d281c] .irq_thread_dtor+0x38/0x104
[c00000000e7070d0] [c000000002092378] .task_work_run+0xf0/0x144
[c00000000e707170] [c00000000206bb08] .do_exit+0x390/0xb10
[c00000000e707270] [c000000002014fac] .die+0x2f8/0x410
[c00000000e707320] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e7073a0] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at ._dpa_rx+0x124/0x780
    LR = ._dpa_rx+0xec/0x780
[c00000000e707690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e7077b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e707860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e707920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e7079b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e707a90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e707b70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e707be0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e707c70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e707d30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e707e30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
38210070 e8010010 7c0803a6 4e800020 7c0802a6 fbe1fff8 f8010010 f821ff81 
7c7f1b78 60000000 60000000 e93f03c8 <e869ffd8> 38210080 e8010010 ebe1fff8 
---[ end trace 000000000000000d ]---

Oops: Kernel access of bad area, sig: 11 [#13]
Fixing recursive fault but reboot is needed!
PREEMPT SMP NR_CPUS=8 CoreNet Generic
Modules linked in:
CPU: 3 PID: 109 Comm: irq/132-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
task: c00000000e66c980 ti: c00000000e72c000 task.ti: c00000000e72c000
NIP: c000000002097540 LR: c0000000020d281c CTR: c0000000020d27e4
REGS: c00000000e72ed50 TRAP: 0300   Tainted: G    B D       (3.14.23-rt9-WR7.0.0.0_preempt-rt)
MSR: 0000000080029000 <CE,EE,ME>  CR: 48ad4e82  XER: 20000000
DEAR: ffffffffffffffd8 ESR: 0000000000000000 SOFTE: 1 
GPR00: c0000000020d281c c00000000e72efd0 c00000000326b850 c00000000e66c980 
GPR04: 0000000000000389 0000000000000000 c00000000316b850 c00000000e66c980 
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000000 
GPR12: 0000000028ad4e82 c00000000fffd680 0000000000000080 c00000000e700500 
GPR16: 80000800801d11d0 0000000000000000 c00000000316f300 0000000000000000 
GPR20: c00000000e66c980 c000000003863b80 c00000000e66d420 ffffffffffffffff 
GPR24: c00000000e72f1e0 0000000000000003 c00000000e66d420 c000000002b10e90 
GPR28: c00000000e66c980 c00000000e66d3c8 c00000000e66c980 c00000000e66c980 
NIP [c000000002097540] .kthread_data+0x20/0x38
LR [c0000000020d281c] .irq_thread_dtor+0x38/0x104
Call Trace:
[c00000000e72efd0] [c0000000029dc848] ._raw_spin_unlock_irqrestore+0xa0/0xf0 (unreliable)
[c00000000e72f050] [c0000000020d281c] .irq_thread_dtor+0x38/0x104
[c00000000e72f0d0] [c000000002092378] .task_work_run+0xf0/0x144
[c00000000e72f170] [c00000000206bb08] .do_exit+0x390/0xb10
[c00000000e72f270] [c000000002014fac] .die+0x2f8/0x410
[c00000000e72f320] [c000000002030210] .bad_page_fault+0xe0/0x130
[c00000000e72f3a0] [c00000000201f1f0] storage_fault_common+0x40/0x44
--- Exception: 300 at ._dpa_rx+0x124/0x780
    LR = ._dpa_rx+0xec/0x780
[c00000000e72f690] [c0000000026f17d4] ._dpa_rx+0x1c0/0x780 (unreliable)
[c00000000e72f7b0] [c0000000026ef8dc] .priv_rx_default_dqrr+0xcc/0x26c
[c00000000e72f860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e72f920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e72f9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e72fa90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e72fb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e72fbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e72fc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e72fd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e72fe30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
Instruction dump:
38210070 e8010010 7c0803a6 4e800020 7c0802a6 fbe1fff8 f8010010 f821ff81 
7c7f1b78 60000000 60000000 e93f03c8 <e869ffd8> 38210080 e8010010 ebe1fff8 
---[ end trace 000000000000000e ]---

Fixing recursive fault but reboot is needed!
[sched_delayed] sched: RT throttling activated
vmcore                         BUG: Bad page state in process irq/134-QMan po  pfn:06ff7
page:c000000003583e08 count:-1 mapcount:0 mapping:          (null) index:0x0
page flags: 0x0()
page dumped because: nonzero _count
Modules linked in:
vmcore       CPU: 2 PID: 108 Comm: irq/134-QMan po Tainted: G    B D      3.14.23-rt9-WR7.0.0.0_preempt-rt #1
Call Trace:
[c00000000e72b030] [c00000000200a76c] .show_stack+0x168/0x278 (unreliable)
[c00000000e72b120] [c0000000029e023c] .dump_stack+0x9c/0x108
[c00000000e72b1a0] [c00000000215f490] .bad_page+0x14c/0x1a8
[c00000000e72b240] [c00000000215fbf4] .get_page_from_freelist+0x4e0/0xa08
[c00000000e72b3a0] [c000000002161554] .__alloc_pages_nodemask+0x1c0/0xc68
[c00000000e72b560] [c0000000028694f4] .__netdev_alloc_frag+0x218/0x318
[c00000000e72b620] [c0000000026f0f00] ._dpa_bp_add_8_bufs+0x70/0x320
[c00000000e72b720] [c0000000026f1340] .dpaa_eth_refill_bpools+0x68/0x8c
[c00000000e72b7b0] [c0000000026ef8b0] .priv_rx_default_dqrr+0xa0/0x26c
[c00000000e72b860] [c00000000281a734] .qman_p_poll_dqrr+0x1b8/0x2c0
[c00000000e72b920] [c0000000026efb60] .dpaa_eth_poll+0x34/0x80
[c00000000e72b9b0] [c000000002881a64] .net_rx_action+0x258/0x3ec
[c00000000e72ba90] [c00000000206f770] .do_current_softirqs+0x290/0x490
[c00000000e72bb70] [c00000000206fa14] .__local_bh_enable+0xa4/0xa8
[c00000000e72bbe0] [c0000000020d2494] .irq_forced_thread_fn+0x74/0xd0
[c00000000e72bc70] [c0000000020d2788] .irq_thread+0x154/0x1b0
[c00000000e72bd30] [c000000002097070] .kthread+0xf8/0x118
[c00000000e72be30] [c000000002000a08] .ret_from_kernel_thread+0x5c/0xd4
nfs: server 128.224.162.213 not responding, still trying

Workaround

Leave 1G for first kernel, then the vmcore could be dumped.

Steps to Reproduce

1) configure --enable-board=fsl-t2xxx --enable-kernel=preempt-rt --enable-rootfs=glibc-std --with-template=feature/kexec,feature/kdump --enable-parallel-pkgbuilds=16
2) make all
3) deploy images.
4) boot target with 'crashkernel=256M@32M'
5) boot second kernel
root@t2080rdb:~# kexec -p ./vmlinux  --append="`cat args`"
my_load:669: do
Modified cmdline:root=/dev/nfs rw nfsroot=tftpboot/t2xxx/rootfs ip=xxxx:xxxx:xxxx:255.255.254.0:t2080rdb:eth2:off console=ttyS0,115200 ramdisk_size=960000 elfcorehdr=53228K 
segment[0].mem:0x2000000 memsz:20885504
segment[1].mem:0x33eb000 memsz:65536
segment[2].mem:0x33fb000 memsz:4096
segment[3].mem:0x33fc000 memsz:36864
segment[4].mem:0x11fec000 memsz:81920
root@t2080rdb:~# 
root@t2080rdb:~# 
root@t2080rdb:~# 
root@t2080rdb:~# echo c > /proc/sysrq-trigger 


root@t2080rdb:~# 
root@t2080rdb:~# ls /proc/vmcore  -l
-r-------- 1 root root 4026544128 Nov 18 09:22 /proc/vmcore
root@t2080rdb:~# ls /proc/vmcore  -lh
-r-------- 1 root root 3.8G Nov 18 09:22 /proc/vmcore

6) copy vmcore to server
# scp /proc/vmcore wgao@128.224.178.244:/home/wgao/t2xxx/rt/kdump 

Check the issue logs.
Live chat
Online