Fixed
Created: May 7, 2018
Updated: Dec 9, 2019
Resolved Date: Jun 19, 2018
Found In Version: 8.0.0.25
Fix Version: 8.0.0.27
Severity: Critical
Applicable for: Wind River Linux 8
Component/s: Kernel
core is always heavily loaded with kworker/ksoftirqd after system is up
Description
Seeing an issue on a new HW blade with WR8 only. When system is up one core is heavily loaded with kworker/ksoftirqd process RES intr. This is not seen on WR6. I've attached the /proc/config.gz, dmesg, and lspci output. If we disable usb via kernel cmd line option (nousb) then we do not see this issue.
Some info we have found:
After moving to WRL8 (kernel 4.1), we are seeing one core is always heavily loaded with kworker/ksoftirqd after system is up. Appears to be handling reschedule interrupts. For output below you can see RES count is high on core 19.
root@Firepower-module1:~# cat /proc/interrupts | grep RES
RES: 27484 534 653 521 502 617 500 379 572 703 681 541 697 603 634 646 457 583 467 89784 630
root@Firepower-module1:~# top
……
501 root 20 0 0 0 0 R 53.2 0.0 17:03.34 kworker/19:1
84 root 20 0 0 0 0 R 31.2 0.0 9:56.57 ksoftirqd/19
The USB driver, xhci_hcd, is used for both WRL6 (kernel 3.10) and WRL8 base images:
Flags: bus master, medium devsel, latency 0, IRQ 373
Memory at 51a80000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: xhci_hcd
However, the following USB patch was added to WRL8:
- usb-add-owl-usb2.0-device-driver.patch
Not sure whether this patch has caused the issue.
Also notice the difference in kernel messages for USB in WRL6 and WRL8 base images:
WRL6 (good):
root@Firepower-module2:~# dmesg | grep xhci
[ 24.239964] xhci_hcd 0000:00:14.0: setting latency timer to 64
[ 24.239967] xhci_hcd 0000:00:14.0: xHCI Host Controller
[ 24.240031] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[ 24.241172] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[ 24.241174] xhci_hcd 0000:00:14.0: supports USB remote wakeup
[ 24.241236] xhci_hcd 0000:00:14.0: irq 373 for MSI/MSI-X
[ 24.241293] usb usb1: Manufacturer: Linux 3.10.62-ltsi-WR6.0.0.29_standard xhci_hcd
[ 24.241393] xHCI xhci_add_endpoint called for root hub
[ 24.241395] xHCI xhci_check_bandwidth called for root hub
[ 24.250064] xhci_hcd 0000:00:14.0: xHCI Host Controller
[ 24.250129] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[ 24.250134] xhci_hcd 0000:00:14.0: supports USB remote wakeup
[ 24.250158] usb usb2: Manufacturer: Linux 3.10.62-ltsi-WR6.0.0.29_standard xhci_hcd
[ 24.250242] xHCI xhci_add_endpoint called for root hub
[ 24.250243] xHCI xhci_check_bandwidth called for root hub
[ 24.560899] usb 1-10: new high-speed USB device number 2 using xhci_hcd
[ 24.891252] usb 1-10.1: new high-speed USB device number 3 using xhci_hcd
WRL8 (bad)
root@Firepower-module3:~# dmesg | grep xhci
[ 19.973032] xhci_hcd 0000:00:14.0: xHCI Host Controller
[ 19.973076] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[ 19.974199] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00009810
[ 19.974205] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[ 19.984178] xhci_hcd 0000:00:14.0: xHCI Host Controller
[ 19.984207] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[ 20.296535] usb 1-10: new high-speed USB device number 2 using xhci_hcd
[ 20.737011] usb 1-10.1: new high-speed USB device number 3 using xhci_hcd
[ 61.918442] usb 1-10.1: reset high-speed USB device number 3 using xhci_hcd
[ 62.488422] usb 1-10.1: new high-speed USB device number 4 using xhci_hcd
We saw the ‘usb 1-10.1: reset high-speed USB device number 3 using xhci_hcd’ message on several of the new blades.
