Wind River Support Network

HomeDefectsLIN9-2110
Not to be fixed

LIN9-2110 : intel-x86-32: storage_aio_stress (keyword: MMC)failed on preempt-rt kernel(compute stick))

Created: Nov 10, 2016    Updated: Mar 6, 2019
Resolved Date: Feb 19, 2019
Found In Version: unknown
Severity: Standard
Applicable for: Wind River Linux 9
Component/s: BSP

Description

root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# fdisk -l
Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram2: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram3: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram4: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram5: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram6: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram7: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram8: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram9: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram10: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram11: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram12: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram13: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram14: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram15: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mmcblk0: 29.1 GiB, 31268536320 bytes, 61071360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes




Disk /dev/mmcblk0boot1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mmcblk0boot0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 14.8 GiB, 15854469120 bytes, 30965760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa679c06c

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1          2048  4196351  4194304    2G 83 Linux
/dev/sda2       4196352 30965759 26769408 12.8G 83 Linux


Disk /dev/sdb: 3.8 GiB, 4022337024 bytes, 7856127 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdc332d7d

Device     Boot Start     End Sectors  Size Id Type
/dev/sdb1        2048 7856126 7854079  3.8G 83 Linux
DEVLINKS=/dev/disk/by-id/mmc-DF4032_0x2b103d18 /dev/disk/by-path/platform-80860F14:00
DEVNAME=/dev/mmcblk0
DEVPATH=/devices/pci0000:00/80860F14:00/mmc_host/mmc0/mmc0:0001/block/mmcblk0
DEVNAME=/dev/mmcblk0boot1
DEVPATH=/devices/pci0000:00/80860F14:00/mmc_host/mmc0/mmc0:0001/block/mmcblk0/mmcblk0boot1
DEVNAME=/dev/mmcblk0boot0
DEVPATH=/devices/pci0000:00/80860F14:00/mmc_host/mmc0/mmc0:0001/block/mmcblk0/mmcblk0boot0
/dev/mmcblk0 will be used by testing
the previous partion is
re-partion the /dev/mmcblk0
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# dd if=/dev/zero of=/dev/mmcblk0 bs=1024 count=2
2+0 records in
2+0 records out
2048 bytes (2.0 kB, 2.0 KiB) copied, 0.00279729 s, 732 kB/s
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# (echo o; echo n; echo p; echo 1; echo ; echo; echo w) | fdisk /dev/mmcblk0 || return 0

Welcome to fdisk (util-linux 2.28.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x39356396.

Command (m for help): Created a new DOS disklabel with disk identifier 0x7cca59ec.

Command (m for help): Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): Partition number (1-4, default 1): First sector (2048-61071359, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-61071359, default 61071359): 
Created a new partition 1 of type 'Linux' and of size 29.1 GiB.

Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
new_partition for test device is /dev/mmcblk0p1
the previous partion is /dev/mmcblk0p1
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# mkfs.ext4 /dev/mmcblk0p1 || return 0
mke2fs 1.43 (17-May-2016)
Discarding device blocks:    4096/7633664 528384/76336642625536/76336645246976/7633664               done                            
Creating filesystem with 7633664 4k blocks and 1908736 inodes
Filesystem UUID: 8878ca2f-1084-4f9a-ad4a-68ab3e0532fe
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000

Allocating group tables:   0/233       done                            
Writing inode tables:   0/233       done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:   0/233       done
<< [do_test1] start aio-stress>>
********************
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# mount /dev/mmcblk0p1 /24997

root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# aio-stress -O -I 50 -s 1024m -r 256k -i 1024 -b 1024 /24997/aio-stress-file
dropping io_iter to 64
file size 1024MB, record size 256KB, depth 64, ios per iteration 64
max io_submit 1024, buffer alignment set to 4KB
threads 1 files 1 contexts 1 context offset 2MB verification off
global num pending is 64
Running single thread version 
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# sync

root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# rm /24997/.file -rf

root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# umount /24997

root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# rm -rf /24997

<<end aio-stress>>
********************
<<clean subcase>>
********************

<< [do_check_call_trace] start check testcase call trace>>
********************
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# dmesg |grep -i -w -E -B10 -A50 'Call Trace|BUG:|WARNING:' ||return 0
[ 3290.592668] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3290.593360] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3290.607438] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3296.116686] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[ 3300.612022] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3300.613135] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3300.638955] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3300.639614] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3300.653225] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3310.331862] ------------[ cut here ]------------
[ 3310.331878] WARNING: CPU: 0 PID: 20695 at /buildarea1/lyang0/x86-32-project-rt/build-bts/tmp/work-shared/intel-x86-32/kernel-source/arch/x86/include/asm/fpu/internal.h:534 fpu__restore+0xff/0x2e0
[ 3310.331927] Modules linked in: deflate ctr twofish_generic twofish_common camellia_generic serpent_generic blowfish_generic blowfish_common cast5_generic cast_common xcbc sha512_generic af_key bnep intel_rapl iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass aesni_intel xts aes_i586 lrw gf128mul ablk_helper iwlmvm cryptd mac80211 efivars btusb btrtl btbcm btintel iwlwifi cfg80211 lpc_ich thermal hdmi_audio battery acpi_pad ac tpm_tis tpm_tis_core softdog efivarfs [last unloaded: kmtest]
[ 3310.331933] CPU: 0 PID: 20695 Comm: sh Tainted: G           O    4.8.3-rt2-WR9.0.0.0_preempt-rt #2
[ 3310.331935] Hardware name: Intel Corporation STK1AW32SC/Cherry Trail CR, BIOS SCCHTAX5.86A.0014.2015.1119.1410 11/19/2015
[ 3310.331943]  00000286 00000286 d5767e54 c13fe2e9 00000000 c1c7f6cc d5767e84 c105935a
[ 3310.331949]  c1c8f054 00000000 000050d7 c1c7f6cc 00000216 c1026bdf 00000216 d42cc8c0
[ 3310.331955]  00000000 bfb995d0 d5767e98 c105942a 00000009 00000000 00000000 d5767eb0
[ 3310.331956] Call Trace:
[ 3310.331965]  [<c13fe2e9>] dump_stack+0x47/0x5e
[ 3310.331970]  [<c105935a>] __warn+0xea/0x110
[ 3310.331974]  [<c1026bdf>] ? fpu__restore+0xff/0x2e0
[ 3310.331977]  [<c105942a>] warn_slowpath_null+0x2a/0x30
[ 3310.331981]  [<c1026bdf>] fpu__restore+0xff/0x2e0
[ 3310.331985]  [<c1028524>] __fpu__restore_sig+0x154/0x710
[ 3310.331991]  [<c1028dbf>] fpu__restore_sig+0x2f/0x50
[ 3310.331996]  [<c101afb5>] restore_sigcontext.isra.1+0xd5/0x100
[ 3310.332000]  [<c101b9cc>] sys_sigreturn+0x9c/0xb0
[ 3310.332004]  [<c1001999>] do_int80_syscall_32+0x59/0xb0
[ 3310.332010]  [<c1a3da0e>] entry_INT80_32+0x2a/0x2a
[ 3310.332013] ---[ end trace 0000000000000002 ]---
[ 3310.658098] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3310.658958] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3310.688419] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3310.689324] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)
[ 3310.704645] genirq: Flags mismatch irq 4. 00002008 (serial) vs. 00002088 (8086228E:00)

root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# cat /sys/kernel/debug/kmemleak |grep -E -B10 -A50 'kmemleak_alloc|kmem_cache_alloc' ||return 0

Steps to Reproduce

                                                            
#################################################################################################################################### 

Test case Author:  lei.yang
------------------
Project Build Path:
------------------
pek-lpgtest18.wrs.com:/buildarea1/lyang0/x86-32-project-rt/build-bts

To Get The Testing Layer:
------------------------
# git clone git://git.wrs.com/git/projects/wassp-repos/testcases/wrlinux
# cd wrlinux;git checkout master

Project Build:
----------------------
    --machines intel-x86-32 --kernel preempt-rt --dl-layers --base-url git://pek-git.wrs.com
    Build Steps:
    . environment-setup-x86_64-wrlinuxsdk-linux
    . oe-init-build-env
    Add BTS layer  
    1.bitbake-layers add-layer <path to wr-testing/bts-dev>
    2.echo 'require templates/feature/bts/template.conf' >> conf/local.conf
    3.bitbake wrlinux-image-glibc-std

Build Configuration:
----------------------------
BB_VERSION        = '1.32.0'
BUILD_SYS         = 'x86_64-linux'
NATIVELSBSTRING   = 'universal'
DISTRO            = 'wrlinux-small'
DISTRO_VERSION    = '9.0.0.0'
MACHINE           = 'intel-x86-32'
DEFAULTTUNE       = 'corei7-32'
TARGET_SYS        = 'i686-wrs-linux'
TUNE_FEATURES     = 'm32 corei7'
TARGET_FPU        = ''
wr-base           = 'HEAD:ef0ac383012b0d8aedb61b7d972920f4aa397aaf'
wr-fixes          = 'HEAD:86020b2b2b961e3bb447b54184e817ac131eb7a9'
wrlinux           = 'HEAD:e068db50fb007201dc1a18a9b7cc8c60a4ea7a26'
intel-x86         = 'HEAD:28a64947ddb766cc553aa6e644f8335581c7e9b0'
meta              = 'HEAD:1d4a98d14d8dfa051f76c9dac5092f05582a9c6f'
meta-initramfs    
meta-python       
meta-networking   
meta-oe           
meta-perl         
meta-webserver    = 'HEAD:1d11beae430625a1ee73ea2f2d20c4737de0b5d0'
wr-template       = 'HEAD:d1f165f234f11376a31b99ec2aaef4843d9c6ded'
wr-kernel         = 'HEAD:6de4024f5bc7a84c2b1c29d602bdfd7bfd481dd0'
meta-filesystems  = 'HEAD:1d11beae430625a1ee73ea2f2d20c4737de0b5d0'
meta-realtime     
meta-virtualization = 'HEAD:6de4024f5bc7a84c2b1c29d602bdfd7bfd481dd0'
wr-base-dl        = 'HEAD:aa1eab16a8c351551912bc3b20a3d41896036cbc'
oe-core-dl-2.2    = 'HEAD:c4e4e415289e6bee73aebc12a68b0feb7609bd93'
meta-initramfs-dl-2.2 = 'HEAD:1347989107c2f97644b18d862f850fd4dc0e104e'
meta-python-dl-2.2 = 'HEAD:b7397cb44fb80d3a19b6ef92f4a44de7ffddf53c'
meta-networking-dl-2.2 = 'HEAD:39bd9c511d73eaaba0ab150c65058c0ba563f8e0'
meta-oe-dl-2.2    = 'HEAD:43e3918dd5340ebc899956a466d1bad02d6827ce'
meta-perl-dl-2.2  = 'HEAD:4d65c8cf72a198d849104cd3f213b6f3777f7431'
meta-webserver-dl-2.2 = 'HEAD:ec10bc34d57bd4f8f1a8ff4d6fff993d45c754a0'
wr-kernel-dl      = 'HEAD:f9d20574ab2f63288ecdca850573e0d372577e36'
meta-filesystems-dl-2.2 = 'HEAD:ffa26ce28c1a1ddd181aedc6dbc95c2b539f35a4'
local             = 'master:2a616e6991be2d6f82c7c6e651380c8e5912a1d6'
bts-dev           = 'master:9330ff483b34d5fde05211bfe3bf99dba8272bf0'


Kernel source top 10 commits:
----------------------------
Yang Shi 05358ac nfs4: Restore the original errno for version check Wed Nov 9 16:29:43 2016 -0800
yzhu1 0c2d330 ipv4: net namespace does not inherit network configurations Mon Nov 7 17:24:53 2016 -0800
Paul Gortmaker 8ab1ff2 cryptodev: stomp dynamic version numbering for in tree builds Mon Nov 7 09:36:41 2016 -0800
Liwei Song 42ef2cf cryptodev: fix compile error when enable CONFIG_CRYPTODEV Mon Nov 7 09:36:33 2016 -0800
Kexin(Casey) Chen a07b56d kgdb: protect find_task_by_pid_ns() with rcu read lock Fri Nov 4 17:09:01 2016 -0700
Kexin(Casey) Chen 48d865c x86, kgdb: correct kgdb_arch_remove_breakpoint Fri Nov 4 17:08:57 2016 -0700
Kexin(Casey) Chen ebfa512 pmac_zilog,console_poll: Implement the poll_rx_cb API Fri Nov 4 17:08:52 2016 -0700
Kexin(Casey) Chen c85dac9 kgdboc, tty: use tty open to start low level drivers Fri Nov 4 17:08:44 2016 -0700
Kexin(Casey) Chen b607c929 usb,keyboard,kdb: Implement HID keyboard polling Fri Nov 4 17:08:23 2016 -0700
Kexin(Casey) Chen 8aed4ab kgdboc: fix lost characters with scheduled breakpoint Fri Nov 4 17:08:20 2016 -0700

Kernel config enabled by the testing layer:
------------------------------------------
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=5000
CONFIG_IOSCHED_BFQ=y
CONFIG_INPUT_EVDEV=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_DMATEST=m
CONFIG_DMA_API_DEBUG=y
CONFIG_USB_SERIAL_PL2303=y
CONFIG_PM_TEST_SUSPEND=m
CONFIG_IOMMU_STRESS=m
CONFIG_OPROFILE=y
CONFIG_ACPI_DEBUG=y
CONFIG_DEBUG_HIGHMEM=y
CONFIG_DEBUG_HOTPLUG_CPU0=y
CONFIG_DMADEVICES_DEBUG=y
CONFIG_PCI_DEBUG=y
CONFIG_USB_GADGET=m
CONFIG_USB_MASS_STORAGE=m
CONFIG_CMA_DEBUG=y
CONFIG_IOSCHED_BFQ=y

Kernel Boot Parameter:
---------------------
root@localhost:/opt/wr-test/testcases/bts/storage_aio_stress# cat /proc/cmdline
fs0:\EFI\wrl\bzImage.efi root=/dev/sda2 rw rootwait intel_iommu=off 

To reproduce it:
---------------
# /opt/wr-test/testcases/bts/storage_aio_stress/runtest.sh -k MMC
Test Env:
   -storage device is MMC
   -MMC's /dev/mmcblk0 is for testing 
-----------------------------------------------
Live chat
Online