Wind River Support Network

Description

Boot time increase issue in LTS22

In LTS22 qemu-system is taking more time and it is affecting our entire boot time.

PFA for the attachments for both the bootcharts from LTS19 and LTS22.

bootchart-20240222-1838-lts19.svg

bootchart-20240222-1727-lts22.svg

=====

We ran set of tests on WR reference board with both LTS19 and LTS22 OVP and observed that qemu-system-x86_64 takes longer to execute during boot up in case of LTS22 when compared with LTS19.

PFA boot chart from WR tests.

bootchart_lts19_WR.svg

IO utilization - read0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sIO utilization - write0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sCPU[overall] utilization0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sCPU[overall] wait0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sProcesses0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0s [2]5.9ms [5]49.0ms [7]88.1ms [9]170.6ms [10]67.5ms [11]53.6ms [14]2.3ms [17]208.8ms [19]64.9ms [20]109.3ms [24]220.3ms [26]77.1ms [27]112.0ms [31]234.6ms [33]69.3ms [34]106.4ms [37]4.3ms [40]6.2ms [75]76.0ms [77]58.6ms [78]37.4ms [84]372.6ms [108]85.5ms [109]106.4ms [110]35.0ms [111]100.8ms [112]83.9ms[113] 104.3ms[120] 5.0ms[125] 3.939s[126] 130.4ms[131] 2.1ms[135] 1.441s[137] 47.9ms[138] 3.6ms[139] 161.5ms[140] 15.7ms[141] 130.0ms[142] 724.7ms[161] 30.2ms[174] 97.9ms[180] 116.8ms[187] 34.5ms[204] 86.7ms[273]1.2ms[288]1.0ms[208] 13.0ms[210] 7.4ms[211] 12.1ms[213] 37.7ms[217]5.6ms[270] 4.1ms[291] 87.4ms[439] 9.1ms[569] 101.0ms[878]1.4ms[622] 26.7ms[1040] 3.6ms[1043] 1.5ms[154] 4.1ms[155] 2.6ms[156] 2.3ms[157] 2.4ms[158] 2.8ms[159]25.369s[160]3.105s[162]170.4ms[163]20.3ms[166]6.3ms[173]1.0ms[169]33.0ms[175]14.2ms[176] 3.8ms[178] 3.5ms[179]6.5ms[182] 3.2ms[183]495.5ms[189]124.6ms[230]4.5ms[314]4.0ms[328]1.4ms[371]4.7ms[453]5.4ms[825]3.5ms[875] 4.0ms[1028] 6.0ms[190]53.9ms[239]4.5ms[244]1.1ms[254]1.4ms[265]2.0ms[282]1.3ms[310]4.2ms[373]4.5ms[397] 2.0ms[438]4.3ms[191]58.8ms[242]4.3ms[259]2.2ms[276]1.0ms[302]4.7ms[327]1.0ms[359]4.3ms[192]56.9ms[277]5.0ms[343]5.4ms[193]73.6ms[206] 5.8ms[216]3.8ms[271]4.8ms[294] 1.6ms[333]5.0ms[381]1.2ms[390]4.1ms[425]3.5ms[448]6.9ms[817]1.1ms[824]3.7ms[194]121.1ms[195]57.7ms[241] 4.7ms[253]1.8ms[308]4.7ms[379]4.3ms[196]41.6ms[261]5.2ms[309]1.0ms[326]4.3ms[382]4.3ms[418]1.3ms[433]4.9ms[197]67.4ms[272]4.2ms[349]5.0ms[426] 5.2ms[430]3.4ms[198]49.0ms[275]5.0ms[344]5.2ms[366]1.0ms[432]4.3ms[184]91.7ms[185] 1.6ms[186] 41.8ms[188] 26.9ms[199]65.8ms[200] 6.5ms[203] 2.2ms[205] 3.1ms[214]8.5ms[226] 2.4ms[228]4.7ms[246]99.2ms[256]4.0ms[257]71.6ms[260] 5.2ms[269]1.6ms[285]1.0ms[286]1.3ms[287]1.1ms[262] 5.1ms[266] 4.2ms[274] 5.7ms[280]59.2ms[451] 5.2ms[452]3.4ms[454] 2.6ms[455]35.7ms[456] 9.7ms[457]19.2ms[458]43.6ms[459] 6.4ms[460]2.0ms[461]191.1ms[462] 2.3ms[463] 6.1ms[464] 6.7ms[465]33.1ms /run/openvswitch/useropts; fi ]]>[467]2.0ms[469]22.1ms[497] 1.1ms[500] 2.9ms[473]5.7ms[474]68.8ms[498] 10.1ms[499]34.2ms[513]17.7ms[540]3.3ms[541]36.5ms[550] 2.2ms[551]395.7ms[740]171.8ms[806]3.3ms[815]1.9ms[830]1.1ms[872]1.7ms[903]1.4ms[993]13.3ms[552]6.9ms[553]1.8ms[554]4.9ms[555]4.6ms[556]8.0ms[557] 4.2ms[558] 7.5ms[560]6.8ms[561]1.6ms[565]13.1ms[588]57.3ms[566]4.4ms[570]15.3ms[568] 5.6ms[573]9.9ms[574]11.5ms[576]2.2ms[577]1.2ms[578]1.4ms[579]1.2ms[580]1.1ms[581]1.1ms[898]1.3ms[575]34.2ms[586]6.7ms[587]4.0ms[595]18.7ms[597]1.5ms[599]1.3ms[600]1.2ms[601]9.7ms[615]6.7ms[617]1.7ms[619]4.1ms[620]1.1ms[623] 112.4ms[629]5.1ms[631]4.3ms[632]1.2ms[635]7.0ms[641]7.0ms[727]1.5ms[728]7.0ms[730]1.8ms[731]4.0ms[732]5.8ms[734]6.9ms[735]6.7ms[736]6.6ms[738]6.1ms[739]3.4ms[756]7.3ms[757]8.0ms[759]10.2ms[794]6.6ms[795]9.5ms[800]9.6ms[801]9.6ms[603]361.4ms[749]50.3ms[799]6.7ms[807]32.3ms[826]6.5ms[831]1.5ms[840]8.8ms[874]5.9ms[904]5.5ms[912]5.6ms[941]6.2ms[977] 6.5ms[994]7.6ms[1003]10.5ms[1004]6.7ms[1011]44.3ms[1026]22.5ms[1035]28.0ms[1045]31.3ms[614]33.3ms[616]8.2ms[621]2.1ms[798]9.8ms[952]25.9ms[965]11.7ms[1059]10.4ms[1007]4.4ms[1008]5.3ms[1012]3.5ms[1062]4.7ms[1027]15.8ms[1037] 7.475s[1047]68.3msBootchart for intel-x86-64 - Wed, 06 Mar 2024 06:44:43 +0000System: Linux 5.2.60-rt15-yocto-preempt-rt #1 SMP PREEMPT RT Thu Feb 22 09:52:49 UTC 2024 x86_64CPU: Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHzBoot options: console=ttyS0,115200 ip=dhcp crashkernel=512M nfsroot=172.25.61.11:/export/pxeboot/vlm-boards/81730/rootfs,vers=3,tcp rw oops=panic panic=1 initcall_blacklist=init_nsc intel_iommu=on iommu=ptBuild: Wind River Linux OVP 19.45 Update 27Log start time: 5.479sIdle time: Not detectedGraph data: 25.000 samples/sec, recorded 500 total, dropped 0 samples, 1016 processes, 145 filteredTop CPU consumers:3938.6ms -  [125]3105.4ms - [160]1567.9ms - [159]1441.4ms -  [135]724.7ms -  [142]495.5ms - [183]372.6ms -  [84]359.4ms - [603]295.1ms -  [1037]234.6ms -  [31]

bootchart_lts22_WR.svg

================

IO utilization - read0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sIO utilization - write0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sCPU[overall] utilization0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sCPU[overall] wait0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0sProcesses0.0s1.0s2.0s3.0s4.0s5.0s6.0s7.0s8.0s9.0s10.0s11.0s12.0s13.0s14.0s15.0s16.0s17.0s18.0s19.0s20.0s [2]7.8ms [9]607.3ms[423]5.3ms [14]43.6ms [15]56.8ms [16]1.9ms [17]94.4ms [18]1.4ms [19]9.0ms [22]5.6ms [23]202.8ms [24]131.1ms [25]306.7ms [29]3.4ms [30]208.3ms [31]195.4ms [32]55.3ms [36]5.5ms [37]205.9ms [38]156.9ms [39]41.5ms [42]4.9ms [44]13.0ms [88]4.3ms [94]8.6ms [95]49.3ms[203]4.4ms[98] 1.227s[100] 120.1ms[106] 4.6ms[116] 1.1ms[118] 1.1ms[120] 1.2ms[124] 66.6ms[125] 84.1ms[126] 71.8ms[127] 151.4ms[131] 1.1ms[134] 5.9ms[140] 4.062s[141] 9.7ms[142] 100.8ms[147] 3.2ms[148] 1.5ms[151] 1.985s[152] 52.8ms[153] 679.7ms[156] 75.5ms[157] 87.7ms[160] 78.9ms[161] 41.6ms[162] 157.4ms[164] 80.0ms[165] 41.0ms[200] 4.0ms[206] 111.0ms[207] 145.9ms[208] 20.5ms[209] 130.1ms[277]8.8ms[232] 33.4ms[234] 13.6ms[236] 12.0ms[283] 2.7ms[296] 35.8ms[300] 25.6ms[329] 173.1ms[628]1.6ms[1017]3.0ms[349] 91.7ms[351] 86.8ms[496]2.7ms[366] 104.3ms[373] 138.3ms[379] 58.6ms[381] 117.3ms[382] 29.2ms[390] 9.7ms[421] 3.5ms[438] 190.8ms[505] 26.8ms[564] 57.9ms[565] 21.5ms[668] 2.3ms[181] 4.0ms[182] 2.8ms[183] 3.8ms[184] 3.9ms[185] 4.3ms[186] 4.3ms[187] 4.9ms[188] 4.3ms[189] 8.3ms[190]10.6ms[191]3.757s[192]319.4ms[193]45.8ms[194] 11.6ms[199] 65.1ms[201] 170.0ms[202]4.6ms[204] 4.0ms[205]3.6ms[210]11.2ms[211]11.7ms[212]25.7ms[215] 4.5ms[216]225.9ms[219]40.4ms[273]6.6ms[314]1.4ms[319]2.5ms[345]1.7ms[358]1.9ms[372]6.8ms[410] 2.5ms[418]1.1ms[442]1.2ms[449] 1.5ms[465]6.5ms[497] 1.5ms[514]2.1ms[528]2.0ms[538]1.2ms[550]2.0ms[563]1.8ms[220]47.2ms[268]6.5ms[313]1.9ms[321]2.1ms[344]2.2ms[355]1.8ms[371]6.7ms[392] 1.6ms[408] 2.8ms[436]1.7ms[461]1.9ms[475]6.0ms[511] 1.9ms[566]7.0ms[221]55.6ms[279]6.4ms[315]1.3ms[320]1.9ms[343]1.7ms[367]7.2ms[415]1.1ms[428] 2.6ms[455]6.3ms[484]2.3ms[527]1.3ms[533]1.4ms[540]1.7ms[222]59.1ms[317]6.1ms[340]1.6ms[346]2.1ms[350]1.2ms[361]1.2ms[370]1.5ms[387]6.6ms[411] 2.0ms[434]1.3ms[459]1.3ms[481]1.4ms[512]3.9ms[561]8.4ms[223]37.3ms[272]6.8ms[284]1.3ms[312]1.2ms[323]2.0ms[341]2.5ms[354]2.5ms[369]6.7ms[413]2.1ms[425]2.3ms[456]1.6ms[483]6.4ms[515]2.7ms[535]1.8ms[539]1.4ms[555]2.4ms[224]94.6ms[316]6.5ms[335]1.5ms[347]1.3ms[360]1.5ms[375]6.6ms[409] 1.7ms[433]1.5ms[452]1.9ms[458]1.0ms[466]1.9ms[477]6.3ms[536]1.8ms[573]1.1ms[576]2.4ms[597]1.8ms[225]91.1ms[238]6.3ms[240]6.1ms[247]1.8ms[416]6.9ms[447]2.6ms[463]1.1ms[495]2.1ms[530]2.0ms[226]82.1ms[318]5.9ms[330]2.1ms[348]1.1ms[352]1.4ms[359]1.8ms[374]6.1ms[388]1.5ms[412] 3.3ms[424]1.8ms[439]1.7ms[451] 1.8ms[464]6.2ms[492]2.1ms[529]1.2ms[594]9.3ms[227]44.2ms[264]7.4ms[311]1.4ms[322]1.9ms[342]2.4ms[368]6.7ms[393]2.5ms[429]1.3ms[443]2.1ms[454]1.1ms[474]2.1ms[531]8.9ms[562]5.0ms[575]1.1ms[589] 1.8ms[228]158.3ms[417]7.2ms[437]2.4ms[453]2.7ms[476]2.1ms[501]2.1ms[532]5.1ms[552]1.1ms[877]5.7ms[886]18.4ms[1023]3.1ms[217]267.8ms[218]45.9ms[229]54.887s[230]107.5ms[231] 6.9ms[250] 2.0ms[251] 3.1ms[254] 7.0ms[274] 3.3ms[275]4.8ms[278]199.8ms[282] 1.0ms[286]5.4ms[287]219.3ms[288]10.4ms[307]2.1ms[331]5.4ms[332]1.8ms[362]1.3ms[364]2.0ms[365]1.7ms[384]1.2ms[414]6.9ms[290]55.2ms[291] 5.6ms[292] 4.4ms[293] 3.9ms[294] 6.4ms[299]88.3ms[301]220.4ms[303]56.7ms[304] 9.2ms[308]97.5ms[309] 7.1ms[328]2.2ms[310]100.2ms[431]1.1ms[435]91.7ms[440]1.1ms /run/openvswitch.useropts ]]>[448]3.3ms> /run/openvswitch.useropts; fi ]]>[460]2.0ms[462]25.4ms[507] 1.2ms[524]1.1ms[525] 3.0ms[542]7.4ms[549] 4.3ms[553] 12.7ms[556]1.2ms[569]5.5ms[571] 4.7ms[580]4.1ms[583]1.2ms[544]130.5ms[572]4.3ms[593]23.3ms[598]17.5ms[626]3.7ms[627]24.9ms[638] 3.5ms[639] 10.5ms[640]3.7ms[641] 10.4ms[642]98.4ms[643] 5.0ms[644]1.5ms[645]7.6ms[650] 4.0ms[651]91.9ms[652]248.7ms[654]338.6ms[655]31.0ms[679]83.1ms[656]4.7ms[658]12.6ms[657]4.8ms[659]51.0ms[660]3.9ms[661]13.2ms[662]2.3ms[673]1.8ms[663]1.8ms[664]2.1ms[665]1.5ms[666]1.8ms[667]1.5ms[672]1.2ms[669]5.8ms[670]4.4ms[671]18.0ms[686]4.9ms[691]6.4ms[694]7.4ms[695]1.3ms[696]105.7ms[699]1.6ms[700]5.1ms[719]8.8ms[721] 6.9ms[723]3.9ms[725]7.0ms[726]5.0ms[728]4.7ms[729]5.3ms[730]1.5ms[731]4.8ms[823]1.6ms[824]1.1ms[825] 2.1ms[826]1.4ms[835]6.8ms[838]8.5ms[862]4.2ms[863]8.7ms[864]8.0ms[865]8.1ms[697]42.7ms[698]9.5ms[704]2.9ms[706]2.035s[837]66.1ms[874]50.3ms[880]3.7ms[928]2.7ms[931]2.5ms[954]3.1ms[955]6.4ms[959]59.6ms[969]69.9ms[971]5.3ms[973]203.5ms[978]39.6ms[980] 3.8ms[982]44.9ms[989]38.9ms[991] 3.5ms[998]43.3ms[1007]71.9ms[1008]60.4ms[1024]63.2ms[1037]64.6ms[1041]4.1ms[867]13.1ms[873]27.9ms[1055]15.4ms[957]2.0ms[960]9.5ms[993]208.0ms[1000] 4.6ms[1002]58.7ms[1011]21.4ms[1026] 1.246s[1060] 7.2msBootchart for intel-x86-64 - Wed, 06 Mar 2024 06:51:14 +0000System: Linux 5.15.136-rt61-yocto-preempt-rt #1 SMP PREEMPT_RT Mon Oct 23 20:38:43 UTC 2023 x86_64CPU: Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHzBoot options: console=ttyS0,115200 ip=dhcp crashkernel=512M nfsroot=172.25.61.11:/export/pxeboot/vlm-boards/81730/rootfs,vers=3,tcp rw oops=panic panic=1 initcall_blacklist=init_nsc intel_iommu=on iommu=ptBuild: Wind River Linux OVP LTS 22.33 Update 13Log start time: 8.203sIdle time: Not detectedGraph data: 25.000 samples/sec, recorded 499 total, dropped 5 samples, 1100 processes, 140 filteredTop CPU consumers:4062.3ms -  [140]3757.5ms - [191]1996.2ms - [706]1985.0ms -  [151]1227.0ms -  [98]821.8ms -  [1026]679.7ms -  [153]607.3ms -  [9]319.4ms - [192]306.7ms -  [25]

Workaround

NA

Steps to Reproduce

LTS22 RCPL0013 OVP Host.

-> LTS19 and LTS22 OVP Guest images are shared in OneDrive folder - 00148607 (https://windriversystems-my.sharepoint.com/:f:/r/personal/prashant_chikhalkar_windriver_com/Documents/shared/juniper/vmhost/lts22/00148607?csf=1&web=1&e=rCunzZ)

 

-> Project configuration:
$ --dl-layers --distros wrlinux-ovp --machines intel-x86-64 --templates feature/initramfs-integrated,feature/initramfs-install,feature/kdump,feature/kexec,feature/sysklogd,feature/dpdk,feature/package-management, feature/tpm2

$ . ./environment-setup-x86_64-wrlinuxsdk-linux
$ . ./oe-init-build-env build

$ vi conf/local.conf
DISTRO = "wrlinux-ovp"
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-rt"

$ bitbake wrlinux-image-ovp-kvm

-> Deploy kernel and rootfs on intel-x86-64 target.

 

root@intel-x86-64:~# uname -a
Linux intel-x86-64 5.15.136-rt61-yocto-preempt-rt #1 SMP PREEMPT_RT Mon Oct 23 20:38:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@intel-x86-64:~# cat /etc/os-release
ID=wrlinux-ovp
NAME="Wind River Linux OVP LTS"
VERSION="10.22.33.13"
VERSION_ID=10.22.33.13
PRETTY_NAME="Wind River Linux OVP LTS 22.33 Update 13"
root@intel-x86-64:~#

 

-> Create guest vm using LTS22 OVP guest image.

 

root@intel-x86-64:~# mkdir -p /opt/windriver
-> Create libvirt guest XML configuration.
root@intel-x86-64:~# cat /opt/windriver/qemu.xml
<domain type='kvm'>
                <name>vm_qemu</name>
                <memory>512000</memory>
                <currentMemory>512000</currentMemory>
                <vcpu>1</vcpu>
                <os>
                                <type arch='x86_64' machine='pc'>hvm</type>
                                <boot dev='hd' />
                                <kernel>/opt/windriver/bzImage-qemux86-64.bin</kernel>
                                <cmdline>console=ttyS0 root=/dev/vda rw ip=dhcp</cmdline>
                </os>
                <devices>
                                <disk type='file' device='disk'>
                                                <source file='/opt/windriver/wrlinux-image-ovp-guest-qemux86-64.ext4'/>
                                                <target dev='vda' bus='virtio' />
                                </disk>
                                <serial type='pty'>
                                                <source path='/dev/pts/0'/>
                                                <target port='0'/>
                                </serial>
                                <console type='pty' tty='/dev/pts/0'>
                                                <source path='/dev/pts/0'/>
                                                <target port='0'/>
                                </console>
                                <interface type='bridge'>
                                                <model type='virtio'/>
                                                <source bridge='virbr0'/>
                                                <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
                                </interface>
                </devices>
</domain>
root@intel-x86-64:~#

 

 

-> Enable "{*}autostart{*}" for libvirt so that qemu vm can launch once Host OS is up after reboot.

 

root@intel-x86-64:/opt/windriver# virsh define qemu.xml
Domain vm_qemu defined from qemu.xml
 
root@intel-x86-64:/opt/windriver# virsh autostart vm_qemu
Domain vm_qemu marked as autostarted
 
root@intel-x86-64:/opt/windriver# ls -lrt
total 1959936
-rw-r--r-- 1 root root    8577696 Mar  1 18:20 bzImage-qemux86-64.bin
-rw-r--r-- 1 root root 1998377984 Mar  1  2024 wrlinux-image-ovp-guest-qemux86-64.ext4
-rw-r--r-- 1 root root       1763 Mar  1  2024 qemu.xml
root@intel-x86-64:/opt/windriver#

 

 

-> Reboot Host OS:

root@intel-x86-64:/opt/windriver# reboot
 

-> Check for svg logfiles after reboot.

 

root@intel-x86-64:/opt/windriver# ls -lrt /run/log/
total 308
drwxr-sr-x 3 root systemd-journal     60 Mar  1 17:44 journal
-rw-r--r-- 1 root root            315392 Mar  1 17:45 bootchart-20240301-1745.svg
root@intel-x86-64:/opt/windriver#

 

 

-> Compare  SVG files from both lts19/lts22 tests and check the execution time for \{*}qemu-system-x86_64{*}.