Wind River Support Network

Description

When using the -rt kernel with Kmemleak enabled you may encounter a panic as shown below. Either disable Kmemleak or use a non-rt kernel until the problem is resolved.

mke2fs 1.44.3 (10-July-2018)
64-bit filesystem support is not enabled.  The larger fields afforded by this feature enable full-strength checksumming.  Pass -O 64bit to rectify.
Discarding device blocks: done                            
Creating filesystem with 43956230 4k blocks and 10993664 inodes
Filesystem UUID: 73c3760f-f8ba-42b4-bd88-e1545a054686
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done     

mke2fs 1.44.3 (10-July-2018)
/dev/sda1 contains a ext4 file system
        created on Fri Nov  2 03:34:46 2018
64-bit filesystem support is not enabled.  The larger fields afforded by this feature enable full-strength checksumming.  Pass -O 64bit to rectify.
Discarding device blocks: ------------[ cut here ]------------
kernel BUG at kernel/locking/rtmutex.c:1048!
invalid opcode: 0000 [#1] PREEMPT SMP PTI
CPU: 5 PID: 689 Comm: mkfs.ext4 Not tainted 4.18.14-rt8-yocto-preempt-rt #1
Hardware name: Intel Corp. Harcuvar/Server, BIOS HAVLCRB1.X64.0015.D62.1708310404 08/31/2017
RIP: 0010:rt_spin_lock_slowlock_locked+0x277/0x2a0
Code: e8 5e 64 61 ff e9 bc fe ff ff e8 54 64 61 ff e9 b7 fe ff ff 0f 0b e8 98 57 53 ff e9 43 fe ff ff e8 8e 57 53 ff e9 74 ff ff ff <0f> 0b 0f 0b 0f 0b 48 8b 43 10 48 85 c0 74 06 48 3b 58 38 75 0b 49 
RSP: 0018:ffff936846d4f3b0 EFLAGS: 00010046
RAX: ffff8e3680361e00 RBX: ffffffff83a8b240 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff8e3680361e00 RDI: ffffffff83a8b258
RBP: ffff936846d4f3e8 R08: ffff8e3680361e01 R09: ffffffff82adfdf0
R10: ffffffff827ede18 R11: 0000000000000000 R12: ffff936846d4f3f8
R13: ffff8e3680361e00 R14: ffff936846d4f3f8 R15: 0000000000000246
FS:  00007fc8b6bfd780(0000) GS:ffff8e369f340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055fb5659e000 CR3: 00000007fdd14000 CR4: 00000000003406e0
Call Trace:
 ? preempt_count_add+0x74/0xc0
 rt_spin_lock_slowlock+0x57/0x90
 ? __kernel_text_address+0x12/0x40
 ? __save_stack_trace+0x75/0x100
 __rt_spin_lock+0x26/0x30
 __write_rt_lock+0x23/0x1a0
 rt_write_lock+0x2a/0x30
 create_object+0x17d/0x2b0
 kmemleak_alloc+0x34/0x50
 kmem_cache_alloc+0x146/0x220
 ? mempool_alloc_slab+0x15/0x20
 mempool_alloc_slab+0x15/0x20
 mempool_alloc+0x65/0x170
 sg_pool_alloc+0x21/0x60
 __sg_alloc_table+0x101/0x160
 ? sg_free_table_chained+0x30/0x30
 sg_alloc_table_chained+0x8b/0xb0
 scsi_init_sgtable+0x31/0x90
 scsi_init_io+0x44/0x130
 sd_setup_write_same16_cmnd+0xef/0x150
 sd_init_command+0x6bf/0xaa0
 ? cgroup_base_stat_cputime_account_end.isra.0+0x26/0x60
 ? elv_rb_del+0x2a/0x40
 scsi_setup_cmnd+0x8e/0x140
 scsi_prep_fn+0x5d/0x140
 blk_peek_request+0xda/0x2f0
 scsi_request_fn+0x33/0x550
 ? cfq_rb_erase+0x23/0x40
 __blk_run_queue+0x43/0x60
 cfq_insert_request+0x2f3/0x5d0
 __elv_add_request+0x160/0x290
 blk_flush_plug_list+0x204/0x230
 schedule+0x87/0xe0
 __write_rt_lock+0x18b/0x1a0
 rt_write_lock+0x2a/0x30
 create_object+0x17d/0x2b0
 kmemleak_alloc+0x34/0x50
 __kmalloc_node+0x1cd/0x340
 alloc_request_size+0x30/0x70
 mempool_alloc+0x65/0x170
 ? ioc_lookup_icq+0x54/0x70
 get_request+0x4e3/0x8d0
 ? wait_woken+0x80/0x80
 blk_queue_bio+0x153/0x470
 generic_make_request+0x1dc/0x3f0
 submit_bio+0x49/0x140
 ? next_bio+0x38/0x40
 submit_bio_wait+0x59/0x90
 blkdev_issue_discard+0x7a/0xd0
 ? _raw_spin_unlock_irqrestore+0x18/0x50
 blk_ioctl_discard+0xc7/0x110
 blkdev_ioctl+0x57e/0x960
 ? __wake_up+0x13/0x20
 block_ioctl+0x3d/0x50
 do_vfs_ioctl+0xa8/0x610
 ? vfs_write+0x166/0x1b0
 ksys_ioctl+0x67/0x90
 __x64_sys_ioctl+0x1a/0x20
 do_syscall_64+0x4d/0xf0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7fc8b6d0ea37
Code: 00 00 90 48 8b 05 49 84 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 19 84 0c 00 f7 d8 64 89 01 48 
RSP: 002b:00007ffdd15b9738 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000080000 RCX: 00007fc8b6d0ea37
RDX: 00007ffdd15b9740 RSI: 0000000000001277 RDI: 0000000000000003
RBP: 000055fb56594a40 R08: 0000000000000003 R09: 0000000000000000
R10: 00007fc8b6e9fa80 R11: 0000000000000206 R12: 00007ffdd15b97d0
R13: 00000000029eb806 R14: 0000000001581000 R15: 0000000000000000
Modules linked in: intel_rapl iTCO_wdt iTCO_vendor_support pnd2_edac intel_powerclamp coretemp crct10dif_pclmul crct10dif_common pcc_cpufreq aesni_intel aes_x86_64 crypto_simd cryptd glue_helper matroxfb_base matroxfb_g450 i2c_i801 matroxfb_accel matroxfb_DAC1064 g450_pll matroxfb_misc i2c_ismt acpi_cpufreq
---[ end trace 0000000000000002 ]---
RIP: 0010:rt_spin_lock_slowlock_locked+0x277/0x2a0
Code: e8 5e 64 61 ff e9 bc fe ff ff e8 54 64 61 ff e9 b7 fe ff ff 0f 0b e8 98 57 53 ff e9 43 fe ff ff e8 8e 57 53 ff e9 74 ff ff ff <0f> 0b 0f 0b 0f 0b 48 8b 43 10 48 85 c0 74 06 48 3b 58 38 75 0b 49 
RSP: 0018:ffff936846d4f3b0 EFLAGS: 00010046
RAX: ffff8e3680361e00 RBX: ffffffff83a8b240 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff8e3680361e00 RDI: ffffffff83a8b258
RBP: ffff936846d4f3e8 R08: ffff8e3680361e01 R09: ffffffff82adfdf0
R10: ffffffff827ede18 R11: 0000000000000000 R12: ffff936846d4f3f8
R13: ffff8e3680361e00 R14: ffff936846d4f3f8 R15: 0000000000000246
FS:  00007fc8b6bfd780(0000) GS:ffff8e369f340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055fb5659e000 CR3: 00000007fdd14000 CR4: 00000000003406e0
note: mkfs.ext4[689] exited with preempt_count 1

Steps to Reproduce

target : barcode: 24977
------------------
Project Build Path:
------------------
pek-lpgtest19.wrs.com:/buildarea1/lyang0/bug-reproduce/build-rt

To Get The Testing Layer:
------------------------
# git clone git://git.wrs.com/git/projects/wassp-repos/testcases/wrlinux
# cd wrlinux;git checkout master

Project Build:
----------------------
    --machines intel-x86-64 --dl-layers
    Build Steps:
    . oe-init-build-env
    Add BTS layer
    1.bitbake-layers add-layer <path to wr-testing/bts-dev>
    2.echo "require templates/feature/bts/template.conf" >> conf/local.conf
    3.echo 'PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-rt"' >> conf/local.conf
    4.bitbake wrlinux-image-glibc-std

This bug is reported by bts, to simplify the test steps you can do: 

on target, with a script with below contents 

for i in {1..10}
do
echo scan > /sys/kernel/debug/kmemleak
mkfs.ext4 -F /dev/sda1
echo clear > /sys/kernel/debug/kmemleak
done