Wind River Support Network

HomeDefectsLIN5-3185
Fixed

LIN5-3185 : random xilinx_zynq network timeout

Created: Aug 28, 2013    Updated: Dec 19, 2017
Resolved Date: Jun 18, 2014
Found In Version: 5.0.1
Fix Version: 5.0.1.11
Severity: Severe
Applicable for: Wind River Linux 5
Component/s: BSP

Description

The network transmit queue seems to time out randomly on the xilinx_zynq target.

This was not seen with RCPL3, but has been seen on RCPL5 and 6. Status of RCPL4 is unknown. So it looks like a problem was introduced in either RCPL4 or 5.

WARNING: at /opt/wrl_501/xilinx-zynq/bitbake_build/tmp/work/xilinx_zynq-wrs-linux-gnueabi/linux-windriver-3.4-r0/linux/net/sched/sch_generic.c:256 dev_watchdog+0x3e4/0x3f4()
NETDEV WATCHDOG: eth0 (xemacps): transmit queue 0 timed out
Modules linked in: minix
[<800185f4>] (unwind_backtrace+0x0/0x104) from [<805f0088>] (dump_stack+0x20/0x24)
[<805f0088>] (dump_stack+0x20/0x24) from [<80029e08>] (warn_slowpath_common+0x64/0x74)
[<80029e08>] (warn_slowpath_common+0x64/0x74) from [<80029ed4>] (warn_slowpath_fmt+0x40/0x48)
[<80029ed4>] (warn_slowpath_fmt+0x40/0x48) from [<8050bf00>] (dev_watchdog+0x3e4/0x3f4)
[<8050bf00>] (dev_watchdog+0x3e4/0x3f4) from [<80039ccc>] (run_timer_softirq+0x15c/0x404)
[<80039ccc>] (run_timer_softirq+0x15c/0x404) from [<800318c8>] (__do_softirq+0xf8/0x278)
[<800318c8>] (__do_softirq+0xf8/0x278) from [<80031f10>] (irq_exit+0xb0/0xcc)
[<80031f10>] (irq_exit+0xb0/0xcc) from [<8000f828>] (handle_IRQ+0x6c/0xbc)
[<8000f828>] (handle_IRQ+0x6c/0xbc) from [<80008548>] (gic_handle_irq+0x38/0x68)
[<80008548>] (gic_handle_irq+0x38/0x68) from [<805fa7c0>] (__irq_svc+0x40/0x70)
Exception stack(0x80843ea8 to 0x80843ef0)
3ea0:                   00000000 521c8fe1 01a6eb55 00069bad 8115b430 00000000
3ec0: 80842000 8089c1fc 00000000 8094b2f4 00000000 80843f24 19d01d14 80843ef0
3ee0: 80070650 8002563c 20000013 ffffffff
[<805fa7c0>] (__irq_svc+0x40/0x70) from [<8002563c>] (xilinx_enter_idle+0x48/0xb8)
[<8002563c>] (xilinx_enter_idle+0x48/0xb8) from [<804a5164>] (cpuidle_enter+0x24/0x28)
[<804a5164>] (cpuidle_enter+0x24/0x28) from [<804a5858>] (cpuidle_idle_call+0xfc/0x3dc)
[<804a5858>] (cpuidle_idle_call+0xfc/0x3dc) from [<8000fee4>] (cpu_idle+0xe0/0x12c)
[<8000fee4>] (cpu_idle+0xe0/0x12c) from [<805e20bc>] (rest_init+0x90/0x94)
[<805e20bc>] (rest_init+0x90/0x94) from [<807f68a0>] (start_kernel+0x314/0x320)
---[ end trace 1b47f0991320af54 ]---
eth0 transmit timeout 3000 ms, reseting...
------------[ cut here ]------------
kernel BUG at /opt/wrl_501/xilinx-zynq/bitbake_build/tmp/work/xilinx_zynq-wrs-linux-gnueabi/linux-windriver-3.4-r0/linux/mm/vmalloc.c:1537!
Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
Modules linked in: minix
CPU: 0    Tainted: G        W     (3.4.43-WR5.0.1.6_standard #1)
PC is at vunmap+0x38/0x3c
LR is at 0x80843cc0
pc : [<80113f3c>]    lr : [<80843cc0>]    psr: 20000113
sp : 80843cc0  ip : 80842000  fp : 80843ccc
r10: 3f22b000  r9 : 80952000  r8 : 00000000
r7 : 80868778  r6 : c0806000  r5 : 00001000  r4 : c0864000
r3 : 80842000  r2 : 00000104  r1 : 00000100  r0 : c0864000
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 18c5387d  Table: 3eec804a  DAC: 00000015
Process swapper/0 (pid: 0, stack limit = 0x808422f0)
Stack: (0x80843cc0 to 0x80844000)
3cc0: 80843d0c 80843cd0 8001cbfc 80113f10 8011fb64 805f22dc 0000943e 80843ce8
3ce0: 804e19bc bfa55500 00000000 00000000 80868794 8089b61f ffffffff 80401fd0
3d00: 80843d34 80843d10 80400f54 8001ca20 00000000 80843d20 bfa55000 bfa55500
3d20: bfa555a0 bfa555b8 80843d5c 80843d38 80402074 80400c78 00000009 bfa55000
3d40: bfa55254 00000000 808440c0 8089b61f 80843d94 80843d60 8050bc80 80401fdc
3d60: 807249a8 00000000 80843d84 80843db8 808bf400 bfa55268 80842000 00000000
3d80: 808440c0 00000000 80843dec 80843d98 80039ccc 8050bb28 670aea42 00000968
3da0: 80843e04 80843db8 bfa55000 8050bb1c 8089b439 00000102 80843db8 80843db8
3dc0: 670aea42 00000041 80844084 80842000 0000001d 00000001 8094b2f4 00000000
3de0: 80843e44 80843df0 800318c8 80039b7c 80843e14 80843e00 808bf240 0000000a
3e00: 00000000 8083db78 80844080 80863b90 8089bc84 00000102 81161934 80842000
3e20: 808403bc 80842000 0000001d 00000000 8094b2f4 00000000 80843e5c 80843e48
3e40: 80031f10 800317dc 00000000 80863b90 80843e84 80843e60 8000f828 80031e6c
3e60: 00000000 80862440 fe003100 80843ea8 80843edc 00000000 80843ea4 80843e88
3e80: 80008548 8000f7c8 80070650 8002563c 20000013 ffffffff 80843f24 80843ea8
3ea0: 805fa7c0 8000851c 00000000 521c8fe1 01a6eb55 00069bad 8115b430 00000000
3ec0: 80842000 8089c1fc 00000000 8094b2f4 00000000 80843f24 19d01d14 80843ef0
3ee0: 80070650 8002563c 20000013 ffffffff 00002bec 521c8fe1 000674de 521c8fe1
3f00: 00069bad 00979c95 0091e000 8115b430 8089c198 80842000 80843f34 80843f28
3f20: 804a5164 80025600 80843f74 80843f38 804a5858 804a514c 8089e114 8089e11c
3f40: 80843f5c 80869cf8 805f9be0 80842000 8089dd48 8086241c 8060420c 80867d00
3f60: 413fc090 808629e0 80843f9c 80843f78 8000fee4 804a5768 00000002 ffffffff
3f80: 8089dc80 80821df4 81158240 00000000 80843fb4 80843fa0 805e20bc 8000fe10
3fa0: 80822704 80863978 80843ff4 80843fb8 807f68a0 805e2038 ffffffff ffffffff
3fc0: 807f632c 00000000 00000000 80821df4 00000000 18c5387d 80862418 80821df0
3fe0: 80867cf4 0000406a 00000000 80843ff8 00008044 807f6598 00000000 00000000
[<80113f3c>] (vunmap+0x38/0x3c) from [<8001cbfc>] (arm_dma_free+0x1e8/0x1ec)
[<8001cbfc>] (arm_dma_free+0x1e8/0x1ec) from [<80400f54>] (xemacps_descriptor_free+0x2e8/0x344)
[<80400f54>] (xemacps_descriptor_free+0x2e8/0x344) from [<80402074>] (xemacps_tx_timeout+0xa4/0x198)
[<80402074>] (xemacps_tx_timeout+0xa4/0x198) from [<8050bc80>] (dev_watchdog+0x164/0x3f4)
[<8050bc80>] (dev_watchdog+0x164/0x3f4) from [<80039ccc>] (run_timer_softirq+0x15c/0x404)
[<80039ccc>] (run_timer_softirq+0x15c/0x404) from [<800318c8>] (__do_softirq+0xf8/0x278)
[<800318c8>] (__do_softirq+0xf8/0x278) from [<80031f10>] (irq_exit+0xb0/0xcc)
[<80031f10>] (irq_exit+0xb0/0xcc) from [<8000f828>] (handle_IRQ+0x6c/0xbc)
[<8000f828>] (handle_IRQ+0x6c/0xbc) from [<80008548>] (gic_handle_irq+0x38/0x68)
[<80008548>] (gic_handle_irq+0x38/0x68) from [<805fa7c0>] (__irq_svc+0x40/0x70)
Exception stack(0x80843ea8 to 0x80843ef0)
3ea0:                   00000000 521c8fe1 01a6eb55 00069bad 8115b430 00000000
3ec0: 80842000 8089c1fc 00000000 8094b2f4 00000000 80843f24 19d01d14 80843ef0
3ee0: 80070650 8002563c 20000013 ffffffff
[<805fa7c0>] (__irq_svc+0x40/0x70) from [<8002563c>] (xilinx_enter_idle+0x48/0xb8)
[<8002563c>] (xilinx_enter_idle+0x48/0xb8) from [<804a5164>] (cpuidle_enter+0x24/0x28)
[<804a5164>] (cpuidle_enter+0x24/0x28) from [<804a5858>] (cpuidle_idle_call+0xfc/0x3dc)
[<804a5858>] (cpuidle_idle_call+0xfc/0x3dc) from [<8000fee4>] (cpu_idle+0xe0/0x12c)
[<8000fee4>] (cpu_idle+0xe0/0x12c) from [<805e20bc>] (rest_init+0x90/0x94)
[<805e20bc>] (rest_init+0x90/0x94) from [<807f68a0>] (start_kernel+0x314/0x320)
Code: e3d110ff 1a000001 ebffffb1 e89da800 (e7f001f2) 
---[ end trace 1b47f0991320af55 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
[<800185f4>] (unwind_backtrace+0x0/0x104) from [<805f0088>] (dump_stack+0x20/0x24)
[<805f0088>] (dump_stack+0x20/0x24) from [<80015ad4>] (handle_IPI+0x194/0x1a4)
[<80015ad4>] (handle_IPI+0x194/0x1a4) from [<80008574>] (gic_handle_irq+0x64/0x68)
[<80008574>] (gic_handle_irq+0x64/0x68) from [<805fa7c0>] (__irq_svc+0x40/0x70)
Exception stack(0xbf8a5ee8 to 0xbf8a5f30)
5ee0:                   00000000 521c8fe2 00071d9b 00001c76 81164430 00000000
5f00: bf8a4000 8089c1fc 00000000 8094b2f4 00000001 bf8a5f64 006f2e9d bf8a5f30
5f20: 80070650 8002563c 20000013 ffffffff
[<805fa7c0>] (__irq_svc+0x40/0x70) from [<8002563c>] (xilinx_enter_idle+0x48/0xb8)
[<8002563c>] (xilinx_enter_idle+0x48/0xb8) from [<804a5164>] (cpuidle_enter+0x24/0x28)
[<804a5164>] (cpuidle_enter+0x24/0x28) from [<804a5858>] (cpuidle_idle_call+0xfc/0x3dc)
[<804a5858>] (cpuidle_idle_call+0xfc/0x3dc) from [<8000fee4>] (cpu_idle+0xe0/0x12c)
[<8000fee4>] (cpu_idle+0xe0/0x12c) from [<805ecc3c>] (secondary_start_kernel+0x114/0x118)
[<805ecc3c>] (secondary_start_kernel+0x114/0x118) from [<005ec534>] (0x5ec534)

Steps to Reproduce

$ configure --enable-board=xilinx-zynq --enable-rootfs=glibc-small+initramfs-integrated --enable-kernel=standard
$ make

Deploy to target using NFS or ramdisk.

Create two scripts on the target, similar to this:

while true
do                                                                              
        echo -n "#"
        rm -f xilinx-zynq-zynq-zc702-sd.dtb-WR5.0.1.6_standard
        tftp -g 128.224.199.201 -r xilinx-zynq-zynq-zc702-sd.dtb-WR5.0.1.6_standard
done

Use two different files, I used the kernel image as shown in one script and the dtb in the other script as they were convenient files.

Run both scripts in parallel:

root@localhost:~# ./test_k.sh &
root@localhost:~# ./test_d.sh &

Eventually the error:

  NETDEV WATCHDOG: eth0 (xemacps): transmit queue 0 timed out

is shown and the kernel halts.

The time it takes for this to happen seems random, sometimes just a few minutes but other times it can take several hours. My last attempt took about 90 minutes.

Other Downloads


Live chat
Online