Fixed
Created: Aug 16, 2017
Updated: Dec 3, 2018
Resolved Date: Oct 22, 2017
Previous ID: LIN8-7471
Found In Version: 8.0.0.18
Fix Version: 8.0.0.23
Severity: Severe
Applicable for: Wind River Linux 8
Component/s: Kernel, unknown
A kernel panic happens several times when running an idle load in which some applications are exchanging state data through TIPC.
This seems to happen after a few hours of uptime.
Aug 10 00:33:47 typhoon-base-unit0 kernel: general protection fault: 0000 [#1] PREEMPT SMP
Aug 10 00:33:47 typhoon-base-unit0 kernel: Modules linked in: tipc ip6_udp_tunnel udp_tunnel bonding ipmi_devintf ipmi_si ipmi_msghandler
Aug 10 00:33:47 typhoon-base-unit0 kernel: CPU: 6 PID: 175879 Comm: kworker/u481:0 Not tainted 4.1.21-WR0.1_cgl #5
Aug 10 00:33:47 typhoon-base-unit0 kernel: Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.01.4002.110820131116 11/08/2013
Aug 10 00:33:47 typhoon-base-unit0 kernel: Workqueue: tipc_rcv tipc_recv_work [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: task: ffff8800b4a14950 ti: ffff8807f2d60000 task.ti: ffff8807f2d60000
Aug 10 00:33:47 typhoon-base-unit0 kernel: RIP: 0010:[<ffffffffa00609d3>] [<ffffffffa00609d3>] subscr_del+0x53/0xa0 [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: RSP: 0018:ffff8807f2d63c78 EFLAGS: 00210292
Aug 10 00:33:47 typhoon-base-unit0 kernel: RAX: dead000000000200 RBX: ffff88081477b380 RCX: 0000000000000202
Aug 10 00:33:47 typhoon-base-unit0 kernel: RDX: dead000000000100 RSI: 0000000000000201 RDI: ffff88081477b380
Aug 10 00:33:47 typhoon-base-unit0 kernel: RBP: ffff8807f2d63c98 R08: ffff8800b6a5ff20 R09: 0000000000000101
Aug 10 00:33:47 typhoon-base-unit0 kernel: R10: ffffffffa0060b79 R11: 00000000000d23be R12: ffff880e211f0000
Aug 10 00:33:47 typhoon-base-unit0 kernel: R13: ffffffff81f09a40 R14: ffff8807fa5970e0 R15: ffff8807fa597070
Aug 10 00:33:47 typhoon-base-unit0 kernel: FS: 0000000000000000(0000) GS:ffff88081e180000(0000) knlGS:0000000000000000
Aug 10 00:33:47 typhoon-base-unit0 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 10 00:33:47 typhoon-base-unit0 kernel: CR2: 00000000081e09b0 CR3: 0000000001e0d000 CR4: 00000000000406e0
Aug 10 00:33:47 typhoon-base-unit0 kernel: Stack:
Aug 10 00:33:47 typhoon-base-unit0 kernel: ffff8807f2d63c98 ffff8807fa597070 ffff8807fa5970e4 ffff8807fa5970e8
Aug 10 00:33:47 typhoon-base-unit0 kernel: ffff8807f2d63cd8 ffffffffa0060a8e 0000000000000005 ffff88081477a3c0
Aug 10 00:33:47 typhoon-base-unit0 kernel: ffff880e1f9c72c0 ffff8807fa5975a0 ffff880e1f9c72c0 ffff8807f2d63d30
Aug 10 00:33:47 typhoon-base-unit0 kernel: Call Trace:
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffffa0060a8e>] subscr_conn_shutdown_event+0x6e/0xa0 [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffffa006b719>] tipc_close_conn+0xb9/0xc0 [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffffa006bb35>] tipc_receive_from_sock+0x105/0x110 [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffffa006b861>] tipc_recv_work+0x31/0x60 [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c05cb>] process_one_work+0x13b/0x4d0
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c09a8>] worker_thread+0x48/0x4c0
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c0960>] ? process_one_work+0x4d0/0x4d0
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c0960>] ? process_one_work+0x4d0/0x4d0
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c6b69>] kthread+0xc9/0xe0
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c6aa0>] ? flush_kthread_worker+0x70/0x70
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff8199d4a2>] ret_from_fork+0x42/0x70
Aug 10 00:33:47 typhoon-base-unit0 kernel: [<ffffffff810c6aa0>] ? flush_kthread_worker+0x70/0x70
Aug 10 00:33:47 typhoon-base-unit0 kernel: Code: 10 0c 00 00 41 83 ec 01 4d 63 e4 4e 8b 64 e0 18 e8 e3 7a 09 e1 48 89 df e8 5b 17 00 00 48 8b 83 80 00 00 00 48 89 df 48 8b 53 78 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 78
Aug 10 00:33:47 typhoon-base-unit0 kernel: RIP [<ffffffffa00609d3>] subscr_del+0x53/0xa0 [tipc]
Aug 10 00:33:47 typhoon-base-unit0 kernel: RSP <ffff8807f2d63c78>
Aug 10 00:33:47 typhoon-base-unit0 kernel: ---[ end trace 183cd6d9bf1a1cd4 ]---
Aug 10 00:33:47 typhoon-base-unit0 kernel: Kernel panic - not syncing: Fatal exception in interrupt
Second instance:
Aug 09 19:42:31 typhoon-base-unit0 kernel: general protection fault: 0000 [#1] PREEMPT SMP
Aug 09 19:42:31 typhoon-base-unit0 kernel: Modules linked in: tipc ip6_udp_tunnel udp_tunnel bonding ipmi_devintf ipmi_si ipmi_msghandler
Aug 09 19:42:31 typhoon-base-unit0 kernel: CPU: 9 PID: 4408 Comm: kworker/u482:1 Not tainted 4.1.21-WR0.1_cgl #5
Aug 09 19:42:31 typhoon-base-unit0 kernel: Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.01.4002.110820131116 11/08/2013
Aug 09 19:42:31 typhoon-base-unit0 kernel: Workqueue: tipc_rcv tipc_recv_work [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: task: ffff880e1c25b0e0 ti: ffff880e1f0e8000 task.ti: ffff880e1f0e8000
Aug 09 19:42:31 typhoon-base-unit0 kernel: RIP: 0010:[<ffffffffa00609d3>] [<ffffffffa00609d3>] subscr_del+0x53/0xa0 [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: RSP: 0018:ffff880e1f0ebc78 EFLAGS: 00210292
Aug 09 19:42:31 typhoon-base-unit0 kernel: RAX: dead000000000200 RBX: ffff880d9ae3b800 RCX: 0000000000000202
Aug 09 19:42:31 typhoon-base-unit0 kernel: RDX: dead000000000100 RSI: 0000000000000201 RDI: ffff880d9ae3b800
Aug 09 19:42:31 typhoon-base-unit0 kernel: RBP: ffff880e1f0ebc98 R08: 0000000000019320 R09: ffff88081dc075c0
Aug 09 19:42:31 typhoon-base-unit0 kernel: R10: ffffffffa0060900 R11: 0000000000000000 R12: ffff880e210ec000
Aug 09 19:42:31 typhoon-base-unit0 kernel: R13: ffffffff81f09a40 R14: ffff880e1e360a00 R15: ffff880e1e360990
Aug 09 19:42:31 typhoon-base-unit0 kernel: FS: 0000000000000000(0000) GS:ffff880e25840000(0000) knlGS:0000000000000000
Aug 09 19:42:31 typhoon-base-unit0 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 09 19:42:31 typhoon-base-unit0 kernel: CR2: 00000000f7604000 CR3: 0000000808f9b000 CR4: 00000000000406e0
Aug 09 19:42:31 typhoon-base-unit0 kernel: Stack:
Aug 09 19:42:31 typhoon-base-unit0 kernel: ffff880e1f0ebc98 ffff880e1e360990 ffff880e1e360a04 ffff880e1e360a08
Aug 09 19:42:31 typhoon-base-unit0 kernel: ffff880e1f0ebcd8 ffffffffa0060a8e 000000000000001c ffff880d9ae3a780
Aug 09 19:42:31 typhoon-base-unit0 kernel: ffff880e21200b40 ffff880e1f426e00 ffff880e21200b40 ffff880e1f0ebd30
Aug 09 19:42:31 typhoon-base-unit0 kernel: Call Trace:
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffffa0060a8e>] subscr_conn_shutdown_event+0x6e/0xa0 [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffffa006b719>] tipc_close_conn+0xb9/0xc0 [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffffa006bb35>] tipc_receive_from_sock+0x105/0x110 [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffffa006b861>] tipc_recv_work+0x31/0x60 [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff810c05cb>] process_one_work+0x13b/0x4d0
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff810c09a8>] worker_thread+0x48/0x4c0
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff810c0960>] ? process_one_work+0x4d0/0x4d0
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff810c6b69>] kthread+0xc9/0xe0
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff810c6aa0>] ? flush_kthread_worker+0x70/0x70
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff8199d4a2>] ret_from_fork+0x42/0x70
Aug 09 19:42:31 typhoon-base-unit0 kernel: [<ffffffff810c6aa0>] ? flush_kthread_worker+0x70/0x70
Aug 09 19:42:31 typhoon-base-unit0 kernel: Code: 10 0c 00 00 41 83 ec 01 4d 63 e4 4e 8b 64 e0 18 e8 e3 7a 09 e1 48 89 df e8 5b 17 00 00 48 8b 83 80 00 00 00 48 89 df 48 8b 53 78 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 78
Aug 09 19:42:31 typhoon-base-unit0 kernel: RIP [<ffffffffa00609d3>] subscr_del+0x53/0xa0 [tipc]
Aug 09 19:42:31 typhoon-base-unit0 kernel: RSP <ffff880e1f0ebc78>
Aug 09 19:42:31 typhoon-base-unit0 kernel: ---[ end trace f2a7921c6bc757e3 ]---
Aug 09 19:42:31 typhoon-base-unit0 kernel: Kernel panic - not syncing: Fatal exception in interrupt
Create a project with this configure parameters
$installDir/configure --enable-addons=wr-cgp --with-custom-distro=gcgl --enable-board=intel-x86-64 --enable-kernel=cgl --enable-rootfs=glibc-cgl --enable-multilib=lib32 --enable-bootimage=iso --enable-jobs=4 --with-rcpl-version=0018
Their applications seem to do a lot of open/close on TIPC sockets.
Seems to happen every few hours...
