Fixed
Created: Nov 22, 2016
Updated: Dec 9, 2019
Resolved Date: Dec 1, 2016
Previous ID: LIN6-11991
Found In Version: 7.0.0.20
Fix Version: 7.0.0.22
Severity: Standard
Applicable for: Wind River Linux 7
Component/s: Kernel
Architecture: IA64
While powering up the machine, kernel bug occurred at block/cfq-iosched.c:3278 with the IP at cfq_dispatch_requests+0x60e/0xba0.
The exact place of this Kernel bug was at “BUG_ON(RB_EMPTY_ROOT(&cfqq->sort_list))”, which is in cfq_dispatch_request function.
Generally I/O Scheduler while operate on pending requests, do the following:
- Merge requests for adjacent sectors
- Sort pending requests based on LBA
- Time when requests are submitted
In Issue case, suspecting that issue had occurred while sorting the pending requests.
cfq_select_queue function validates the next queue to dispatch. So passing the cfqq as argument to cfq_dispatch_request function.
BUG_ON(RB_EMPTY_ROOT(&cfqq->sort_list));
Here RB_EMPTY_ROOT is
#define RB_EMPTY_ROOT(root) ((root)->rb_node == NULL)
So, suspecting that somehow list had freed or deleted and so value was NULL and hence Kernel Bug had raised.
Below is the call stack for the issue:
[ 38.360830] ------------[ cut here ]------------
[ 38.360834] kernel BUG at block/cfq-iosched.c:3278!
[ 38.360839] invalid opcode: 0000 [#1] PREEMPT SMP
moc_memdrv(O) moc_ipsec(O) sha256_generic scdrv(O) loopMocana(O) moc_crypto(PO)
[ 38.360854] CPU: 3 PID: 816 Comm: irq/104-ahci Tainted: P W O 3.10.62 #1
[ 38.360856] Hardware name: Insyde explorer60 X64/7, BIOS explorer60.13.07 07/28/2016
[ 38.360858] task: ffff880272a03150 ti: ffff8802703ba000 task.ti: ffff8802703ba000
[ 38.360868] RIP: 0010:[<ffffffff812d617e>] [<ffffffff812d617e>] cfq_dispatch_requests+0x60e/0xba0
[ 38.360870] RSP: 0018:ffff8802703bbae0 EFLAGS: 00010246
[ 38.360871] RAX: 0000000000000000 RBX: ffff88026ff29bc8 RCX: 0000000000000000
[ 38.360873] RDX: 00000000fffc01fd RSI: ffff88026ff29bc8 RDI: ffff8802703508d0
[ 38.360874] RBP: ffff8802703bbb30 R08: 0000000000000001 R09: 0000000000000000
[ 38.360875] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88026ff29bc8
[ 38.360877] R13: ffff880270350800 R14: 0000000000000001 R15: ffff880270350c00
[ 38.360879] FS: 0000000000000000(0000) GS:ffff88027fd80000(0000) knlGS:0000000000000000
[ 38.360880] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 38.360881] CR2: 00000000f720aff0 CR3: 0000000001c10000 CR4: 00000000001007e0
[ 38.360883] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 38.360884] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 38.360885] Stack:
[ 38.360890] 00000000001521ed ffff880272a03898 ffff8802703bbb60 ffffffff8109d399
[ 38.360894] ffff880272a03898 ffff88026fe98000 ffff88026fe98000 ffff8802703bbfd8
[ 38.360897] ffff8802703bbfd8 ffff880272aa3000 ffff8802703bbb70 ffffffff812c2718
[ 38.360901] ffff8802703bbfd8 ffff88026fdc3000 ffff88026fe98000 ffff88026fe98000
[ 38.360904] ffff8802703bbc20 ffff880272aa3000 ffff8802703bbbd0 ffffffff81459f89
[ 38.360907] 0000000000000000 0000000000000246 0000000081458d8f ffff880272aa31d8
[ 38.360911] ffff880272aa30c8 ffff88026fe98000 ffff88026fdc3000 ffff88026fe98000
[ 38.360914] ffff8802703bbc20 ffff88026fdc30b8 ffff8802703bbbe8 ffffffff812bc663
[ 38.360918] ffff88026fe98000 ffff8802703bbc00 ffffffff812bf356 ffff8802703bbc20
[ 38.360921] ffff8802703bbc58 ffffffff81458e8b ffffffff81452f7c ffff8802729a4200
[ 38.360925] ffff8802703bbc20 ffff8802703bbc20 ffff880272aa31d8 ffff8802729a4200
[ 38.360928] ffff88026fe98000 ffff8802729a4200 ffff880270356ac0 ffff8802703bbc80
[ 38.360932] ffffffff8145a8db 0000000000000000 0000000000000000 0000000000000000
[ 38.360935] ffff8802703bbce0 ffffffff8145ab8d ffff880270356ac0 ffff88026fe98000
[ 38.360938] 000000006fdc3000 ffff88026fe98000 ffff880270350028 ffff8802729a4200
[ 38.360942] 0000000000001000 ffff880270350028 ffff88026fdc3000 0000000000000004
[ 38.360945] ffff8802703bbd10 ffffffff814525af ffff8802729a4200 0000000000007530
[ 38.360949] 0000000000000005 ffff880272a03150 ffff8802703bbd38 ffffffff8145a85d
[ 38.360952] ffff8802703bbd48 0000000000000010 0000000000000004 ffff8802703bbd60
[ 38.360955] ffffffff812c8086 ffff8802703bbd48 ffff8802703bbd48 0000000000000004
[ 38.360959] ffff8802703bbdd0 ffffffff8104366e 00000001703bbd90 ffffffff81c045a0
[ 38.360962] 0020804000000003 000000000000b940 ffff880272a03150 ffff880200000000
[ 38.360966] 000000000000b5a0 ffff880273328b00 ffff8802729a6000 0000000000000001
[ 38.360969] ffff880272a03150 ffffffff810c4060 ffff8802703bbde0 ffffffff810438fc
[ 38.360973] ffff8802703bbe08 ffffffff810c409b ffff880273328b00 ffff880272a03150
[ 38.360976] ffff8802729a6000 ffff8802703bbe68 ffffffff810c42a1 ffff880272a03150
[ 38.360979] 0000000000000000 0000000000000000 0000000000000000 ffffffff810c42e0
[ 38.360983] ffff8802730bba60 ffff880273328b00 ffffffff810c4190 0000000000000000
[ 38.360986] 0000000000000000 ffff8802703bbf48 ffffffff8106368f 0000000000000000
[ 38.360990] ffff880272a03150 ffff880273328b00 ffff880200000000 dead4ead00000000
[ 38.360993] ffff8802ffffffff ffffffffffffffff ffffffff81e4b920 0000000000000000
[ 38.360996] 0000000000000000 ffffffff819f6edf ffff8802703bbed0 ffff8802703bbed0
[ 38.361000] 0000000000000000 dead4ead7fd90000 ffff8802ffffffff ffffffffffffffff
[ 38.361003] ffffffff81e4b920 0000000000000000 0000000000000000 ffffffff819f6edf
[ 38.361007] ffff8802703bbf20 ffff8802703bbf20 ffffffff810635c0 0000000000000000
[ 38.361010] 0000000000000000 ffff8802730bba60 ffffffff8167a0f8 0000000000000000
[ 38.361013] 0000000000000000 0000000000000000 0000000000000000 ffff8802730bba60
[ 38.361016] ffffffff810635c0 0000000000000000 0000000000000000 0000000000000000
[ 38.361019] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 38.361023] 0000000000000000 0000000000000000 ffffffffffffffff 0000000000000000
[ 38.361026] 0000000000000010 0000000000000202 ffff8802703bbf58 0000000000000018
[ 38.361027] Call Trace:
[ 38.361034] [<ffffffff8109d399>] ? __lock_acquire+0x3a9/0xc00
[ 38.361038] [<ffffffff812c2718>] blk_peek_request+0x158/0x270
[ 38.361043] [<ffffffff81459f89>] scsi_request_fn+0x59/0x540
[ 38.361047] [<ffffffff812bc663>] __blk_run_queue+0x33/0x40
[ 38.361050] [<ffffffff812bf356>] blk_run_queue+0x26/0x40
[ 38.361055] [<ffffffff81458e8b>] scsi_run_queue+0x15b/0x310
[ 38.361058] [<ffffffff81452f7c>] ? __scsi_put_command+0x6c/0xb0
[ 38.361061] [<ffffffff8145a8db>] scsi_next_command+0x3b/0x50
[ 38.361065] [<ffffffff8145ab8d>] scsi_io_completion+0x23d/0x730
[ 38.361068] [<ffffffff814525af>] scsi_finish_command+0x9f/0xd0
[ 38.361071] [<ffffffff8145a85d>] scsi_softirq_done+0x10d/0x130
[ 38.361076] [<ffffffff812c8086>] blk_done_softirq+0x96/0xb0
[ 38.361081] [<ffffffff8104366e>] do_current_softirqs+0x24e/0x470
[ 38.361087] [<ffffffff810c4060>] ? irq_thread_fn+0x50/0x50
[ 38.361090] [<ffffffff810438fc>] local_bh_enable+0x6c/0x90
[ 38.361093] [<ffffffff810c409b>] irq_forced_thread_fn+0x3b/0x70
[ 38.361097] [<ffffffff810c42a1>] irq_thread+0x111/0x150
[ 38.361100] [<ffffffff810c42e0>] ? irq_thread+0x150/0x150
[ 38.361104] [<ffffffff810c4190>] ? wake_threads_waitq+0x40/0x40
[ 38.361108] [<ffffffff8106368f>] kthread+0xcf/0xe0
[ 38.361112] [<ffffffff810635c0>] ? kthread_worker_fn+0x1d0/0x1d0
[ 38.361118] [<ffffffff8167a0f8>] ret_from_fork+0x58/0x90
[ 38.361121] [<ffffffff810635c0>] ? kthread_worker_fn+0x1d0/0x1d0
see the following link for patch details : https://patchwork.kernel.org/patch/9164581/
