Wind River Support Network

HomeDefectsLIN9-2285
Not to be fixed

LIN9-2285 : intel-skylake-avx512-64: There are some call trace during running rt_pi_stress case on preempt-rt kernel

Created: Nov 17, 2016    Updated: Feb 22, 2019
Resolved Date: Feb 20, 2019
Found In Version: unknown
Severity: Standard
Applicable for: Wind River Linux 9
Component/s: BSP

Description

root@128.224.179.153:/opt/wr-test/testcases/bts/rt_pi_stress# pi_stress --duration=1000 > rt_pi_stress_tmp.log

Starting PI Stress Test
Number of thread groups: 191
Duration of test run: 1000 seconds
Number of inversions per group: unlimited
     Admin thread SCHED_FIFO priority 4
191 groups of 3 threads will be created
      High thread SCHED_FIFO priority 3
       Med thread SCHED_FIFO priority 2
       Low thread SCHED_FIFO priority 1
Current Inversions: 5767018

...

[ 5720.645012] Task dump for CPU 175:
[ 5720.645012] swapper/175     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000003 ffff885fdede1610 ffff885fde1efec0 ffffffff818700fa
[ 5720.645012]  ffff885fdede1610 0000000300000000 ffff885fdede1610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fde1f0000 ffff885fde1efed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 177:
[ 5720.645012] swapper/177     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000002 ffff885fdee61610 ffff885fde1d7ec0 ffffffff818700fa
[ 5720.645012]  ffff885fdee61610 0000000200000000 ffff885fdee61610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fde1d8000 ffff885fde1d7ed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 178:
[ 5720.645012] swapper/178     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000002 ffff885fdeea1610 ffff885fc8673ec0 ffffffff818700fa
[ 5720.645012]  ffff885fdeea1610 0000000200000000 ffff885fdeea1610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fc8674000 ffff885fc8673ed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 182:
[ 5720.645012] swapper/182     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000003 ffff885fdefa1610 ffff885fdc82bec0 ffffffff818700fa
[ 5720.645012]  ffff885fdefa1610 0000000300000000 ffff885fdefa1610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fdc82c000 ffff885fdc82bed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 183:
[ 5720.645012] swapper/183     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000003 ffff885fdefe1610 ffff885fdc82fec0 ffffffff818700fa
[ 5720.645012]  ffff885fdefe1610 0000000300000000 ffff885fdefe1610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fdc830000 ffff885fdc82fed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 184:
[ 5720.645012] swapper/184     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000003 ffff885fdf021610 ffff885fdc80bec0 ffffffff818700fa
[ 5720.645012]  ffff885fdf021610 0000000300000000 ffff885fdf021610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fdc80c000 ffff885fdc80bed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 187:
[ 5720.645012] swapper/187     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000003 ffff885fdf0e1610 ffff885fdc81fec0 ffffffff818700fa
[ 5720.645012]  ffff885fdf0e1610 0000000300000000 ffff885fdf0e1610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fdc820000 ffff885fdc81fed0
[ 5720.645012] Call Trace:
[ 5720.645012]  [<ffffffff818700fa>] ? cpuidle_enter_state+0xfa/0x320
[ 5720.645012]  [<ffffffff81870357>] ? cpuidle_enter+0x17/0x20
[ 5720.645012]  [<ffffffff810a0597>] ? call_cpuidle+0x37/0x50
[ 5720.645012]  [<ffffffff810a09bd>] ? cpu_startup_entry+0x2dd/0x3d0
[ 5720.645012]  [<ffffffff8103eed0>] ? start_secondary+0xf0/0x100
[ 5720.645012] Task dump for CPU 188:
[ 5720.645012] swapper/188     R  running task        0     0      1 0x00200000
[ 5720.645012]  0000000000000003 ffff885fdf121610 ffff885fc8683ec0 ffffffff818700fa
[ 5720.645012]  ffff885fdf121610 0000000300000000 ffff885fdf121610 ffffffff82141800
[ 5720.645012]  ffffffff820cbc20 0000000000000000 ffff885fc8684000 ffff885fc8683ed0
... 

Steps to Reproduce

####################################################################################################################################

Test case Author:  lei.yang
------------------
Project Build Path:
------------------
pek-lpgtest16.wrs.com.wrs.com:/buildarea1/jhu2/wrl9/intel-skylake/build

To Get The Testing Layer:
------------------------
# git clone git://git.wrs.com/git/projects/wassp-repos/testcases/wrlinux
# cd wrlinux;git checkout master

Project Build:
----------------------
    --machines=intel-skylake-avx512-64 --kernel=preempt-rt --distros=wrlinux-std-sato --base-url git://pek-git.wrs.com --dl-layers
    Build Steps:
    . environment-setup-x86_64-wrlinuxsdk-linux
    . oe-init-build-env
    Add BTS layer
    1.bitbake-layers add-layer <path to wr-testing/bts-dev>
    2.echo 'require templates/feature/bts/template.conf' >> conf/local.conf
    3.bitbake wrlinux-image-glibc-std

Build Configuration:
----------------------------
BB_VERSION        = '1.32.0'
BUILD_SYS         = 'x86_64-linux'
NATIVELSBSTRING   = 'CentOSLinux-7.2.1511'
DISTRO            = 'wrlinux-std-sato'
DISTRO_VERSION    = '9.0.0.0'
MACHINE           = 'intel-skylake-avx512-64'
DEFAULTTUNE       = 'skylake-avx512-64'
TARGET_SYS        = 'x86_64-wrs-linux'
TUNE_FEATURES     = 'm64 haswell haswell-ep skylake mpx avx512'
TARGET_FPU        = ''
lib32:  DEFAULTTUNE       = 'skylake-avx512-32'
lib32:  TARGET_SYS        = 'i686-wrsmllib32-linux'
lib32:  TUNE_FEATURES     = 'm32 haswell haswell-ep skylake mpx avx512'
wr-base           = 'HEAD:2c735d7b8a43f84c1e68d44ff7c90dc52e6dc368'
wr-fixes          = 'HEAD:86020b2b2b961e3bb447b54184e817ac131eb7a9'
wrlinux           = 'HEAD:67765f6534b125121593bd943334a7b48e770078'
intel-skylake     = 'HEAD:04e0a6cb8a4e071df4372299c572dbc258882f9b'
meta              = 'HEAD:2583e6212b08650242ae5d570b2c684e9ced4bf3'
meta-initramfs
meta-python
meta-networking
meta-oe
meta-perl
meta-webserver    = 'HEAD:62c6dadb1aa8900f787a664a8c2bfe68734c2a6c'
wr-template       = 'HEAD:d1f165f234f11376a31b99ec2aaef4843d9c6ded'
wr-kernel         = 'HEAD:9066d22ea9d9e274d72f04d2e055ec6404670e2b'
intel-x86         = 'HEAD:04e0a6cb8a4e071df4372299c572dbc258882f9b'
meta-filesystems  = 'HEAD:62c6dadb1aa8900f787a664a8c2bfe68734c2a6c'
meta-realtime
meta-virtualization = 'HEAD:9066d22ea9d9e274d72f04d2e055ec6404670e2b'
wr-base-dl        = 'HEAD:aa1eab16a8c351551912bc3b20a3d41896036cbc'
oe-core-dl-2.2    = 'HEAD:c4e4e415289e6bee73aebc12a68b0feb7609bd93'
meta-initramfs-dl-2.2 = 'HEAD:1347989107c2f97644b18d862f850fd4dc0e104e'
meta-python-dl-2.2 = 'HEAD:b7397cb44fb80d3a19b6ef92f4a44de7ffddf53c'
meta-networking-dl-2.2 = 'HEAD:39bd9c511d73eaaba0ab150c65058c0ba563f8e0'
meta-oe-dl-2.2    = 'HEAD:d4fa23b0fb27000142f6468c8fb72761fd0b53c6'
meta-perl-dl-2.2  = 'HEAD:4d65c8cf72a198d849104cd3f213b6f3777f7431'
meta-webserver-dl-2.2 = 'HEAD:ec10bc34d57bd4f8f1a8ff4d6fff993d45c754a0'
wr-kernel-dl      = 'HEAD:f9d20574ab2f63288ecdca850573e0d372577e36'
meta-filesystems-dl-2.2 = 'HEAD:ffa26ce28c1a1ddd181aedc6dbc95c2b539f35a4'
local             = 'master:f51d166e869a5fc65db0e13613a524f7b51c8371'
bts-dev           = 'master:bf481b51d81fe4b8b652d21d2630f2a755efd2ce'


Kernel source top 10 commits:
----------------------------
Paul Gortmaker c8b3e7d bfq-iosched: fix compile warning about possibly used uninitialized Wed Nov 16 15:55:59 2016 -0800
Li Zhou ad55401 Revert 'net: sysrq via icmp' Wed Nov 16 13:15:21 2016 -0800
Zhang Xiao 5ee68c8 preempt-rt: workqueue use normal rcu Wed Nov 16 13:15:21 2016 -0800
Qi Hou f9e3a2e tcp: substitute raw spinlock for spinlock in jprobe Wed Nov 16 13:15:20 2016 -0800
Li Zhou 6a61fa3 locking-selftest: change some rwlock tests' expected results for rt kernel Wed Nov 16 13:15:20 2016 -0800
Li Zhou 58c6539 trace: use raw_local_irq_save/restore to avoid trace code loop Wed Nov 16 13:15:19 2016 -0800
Qi Hou fa165cd fix failure to build the preempt_rt kernel of axxiaarm with DEBUG_LOCKING_API_SELFTESTS enabled. Wed Nov 16 13:15:19 2016 -0800
Jianchuan Wang a248aa7 locking_selftest: Save/restore migrate_disable_atomic in locking selftest Wed Nov 16 13:15:18 2016 -0800
Liwei Song 37655ae events/core: make event activation/deactivation RT safe Wed Nov 16 13:15:18 2016 -0800
He Zhe 4c95017 kdb: disable kdb_summary command in preempt context Wed Nov 16 13:15:17 2016 -0800

Kernel config enabled by the testing layer:
------------------------------------------
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=5000
CONFIG_IOSCHED_BFQ=y
CONFIG_DMATEST=m
CONFIG_USB_SERIAL_PL2303=y
CONFIG_CMA_DEBUG=y
CONFIG_UIO_PCI_GENERIC=m
CONFIG_USB_USBNET=y
CONFIG_USB_NET_DM9601=y
CONFIG_IOSCHED_BFQ=y

Kernel Boot Parameter:
---------------------
root@128.224.179.153:/opt/wr-test/testcases/bts/rt_pi_stress# cat /proc/cmdline
kernel console=ttyS0,115200 root=/dev/nfs nfsroot=128.224.165.20:/export/pxeboot/vlm-boards/24854/rootfs rw ip=dhcp selinux=0 enforcing=0 intel_iommu=off

To reproduce it:
---------------
# /opt/wr-test/testcases/bts/rt_pi_stress/runtest.sh
=============
Extra Info:
=============
Live chat
Online