Wind River Support Network

Description

When we try to reboot from inside fortigate VNF spun on porter-3 (running with the software components WRL8, RCPL-25, Qemu 2.7), the VNF gets stuck randomly. To recover from this state, we need to destroy and create the VNF again. When the VNF is stuck, we are seeing two scenarios: 

1. VNF is in running state and the cpu usage for this VNF is 100% 
2. VNF is in paused state and the cpu usage for this VNF is 0% 

Following are backtraces for the two scenarios: 

Scenario 1 - VNF is in running state and the cpu usage for this VNF is 100% 
========================================================== 

(gdb) info th 
 Id Target Id Frame 
 3 Thread 0x7f0e508e8700 (LWP 21849) "kvm" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 
 2 Thread 0x7f0e4f1c2700 (LWP 21853) "kvm" 0x00007f0e513f7307 in ioctl () at ../sysdeps/unix/syscall-template.S:81 
* 1 Thread 0x7f0e53f76a40 (LWP 21848) "kvm" 0x00007f0e513f3571 in __GI_ppoll (fds=0x7f0e55584a40, nfds=8, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 
(gdb) bt 
#0 0x00007f0e513f3571 in __GI_ppoll (fds=0x7f0e55584a40, nfds=8, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 
#1 0x00007f0e53b3714c in ?? () 
#2 0x00007f0e53b36af2 in ?? () 
#3 0x00007f0e5389711f in ?? () 
#4 0x00007f0e513370d0 in __libc_start_main (main=0x7f0e538957d0, argc=64, argv=0x7fff9c5cbe58, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff9c5cbe48) at libc-start.c:289 
#5 0x00007f0e5389c2ec in ?? () 
(gdb) t 2 
[Switching to thread 2 (Thread 0x7f0e4f1c2700 (LWP 21853))] 
#0 0x00007f0e513f7307 in ioctl () at ../sysdeps/unix/syscall-template.S:81 
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) 
(gdb) bt 
#0 0x00007f0e513f7307 in ioctl () at ../sysdeps/unix/syscall-template.S:81 
#1 0x00007f0e538de011 in ?? () 
#2 0x00007f0e538de16f in ?? () 
#3 0x00007f0e538cc308 in ?? () 
#4 0x00007f0e52656284 in start_thread (arg=0x7f0e4f1c2700) at pthread_create.c:313 
#5 0x00007f0e513feafd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 
(gdb) th 3 
Ambiguous command "th 3": thbreak, thread. 
(gdb) bt 
#0 0x00007f0e513f7307 in ioctl () at ../sysdeps/unix/syscall-template.S:81 
#1 0x00007f0e538de011 in ?? () 
#2 0x00007f0e538de16f in ?? () 
#3 0x00007f0e538cc308 in ?? () 
#4 0x00007f0e52656284 in start_thread (arg=0x7f0e4f1c2700) at pthread_create.c:313 
#5 0x00007f0e513feafd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 
(gdb) t 3 
[Switching to thread 3 (Thread 0x7f0e508e8700 (LWP 21849))] 
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 
38 cmpq $-4095, %rax /* Check %rax for error. */ 
(gdb) bt 
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 
#1 0x00007f0e53bc4115 in ?? () 
#2 0x00007f0e53bd230e in ?? () 
#3 0x00007f0e52656284 in start_thread (arg=0x7f0e508e8700) at pthread_create.c:313 
#4 0x00007f0e513feafd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 
(gdb) 

Scenario 2 - VNF is in paused state and the cpu usage for this VNF is 0% 
======================================================== 

(gdb) info threads 
 Id Target Id Frame 
 3 Thread 0x7f62172d5700 (LWP 21256) "kvm" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 
 2 Thread 0x7f6215baf700 (LWP 21260) "kvm" pthread_cond_wait () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 
* 1 Thread 0x7f621a963a40 (LWP 21255) "kvm" 0x00007f6217de0571 in __GI_ppoll (fds=0x7f621b9ba080, nfds=6, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 
(gdb) bt 
#0 0x00007f6217de0571 in __GI_ppoll (fds=0x7f621b9ba080, nfds=6, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 
#1 0x00007f621a52414c in ?? () 
#2 0x00007f621a523af2 in ?? () 
#3 0x00007f621a28411f in ?? () 
#4 0x00007f6217d240d0 in __libc_start_main (main=0x7f621a2827d0, argc=64, argv=0x7ffe6d600628, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe6d600618) at libc-start.c:289 
#5 0x00007f621a2892ec in ?? () 
(gdb) thread 2 
[Switching to thread 2 (Thread 0x7f6215baf700 (LWP 21260))] 
#0 pthread_cond_wait () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 
185 62: movl (%rsp), %edi 
(gdb) bt 
#0 pthread_cond_wait () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 
#1 0x00007f621a5b0e09 in ?? () 
#2 0x00007f621a2b9273 in ?? () 
#3 0x00007f6219043284 in start_thread (arg=0x7f6215baf700) at pthread_create.c:313 
#4 0x00007f6217debafd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 
(gdb) thread 3 
[Switching to thread 3 (Thread 0x7f62172d5700 (LWP 21256))] 
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 
38 cmpq $-4095, %rax /* Check %rax for error. */ 
(gdb) bt 
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 
#1 0x00007f621a5b1115 in ?? () 
#2 0x00007f621a5bf30e in ?? () 
#3 0x00007f6219043284 in start_thread (arg=0x7f62172d5700) at pthread_create.c:313 
#4 0x00007f6217debafd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 
(gdb) 

This issue is not happening with Qemu 2.4 but happening with Qemu 2.7

Steps to Reproduce

•Edit the VNF image path in the XML and create the VNF using "virsh create vFW_fortigate.xml"
•Connect to the VNF using "virsh console fortigate"
•Login using the following credentials•Username – admin
•Password - <no password> (press enter)

•Execute the following command to reboot from inside VM.•“execute reboot”

•After 10 to 15 seconds you will get login prompt
•Repeat the reboot command until you will see the console stuck after the following log•“Restarting system.”