Wind River Support Network

Description

We have found an issue in the intel-socfpga BSP on WR Linux LTS-21 RCPL 19.

We run into problems when running the following two (stress) tests:

while true; do fcs_client -G /dev/null > /dev/null 2>&1; done &
while true; do cat /sys/class/hwmon/hwmon0/in0_input > /dev/null 2>&1; done &
The *"fcs_client -G"* command above reads out the provisioning data.

After some time we are starting to see the following kernel printouts:

[29444.608380] svc_normal_to_secure_thread: STATUS_ERROR
[29444.613528] soc64-hwmon soc:firmware:hwmon: soc64_readtemp_smc_callback returned 0x0
[29444.621933] svc_normal_to_secure_thread: STATUS_ERROR
[29444.627057] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29444.646499] svc_normal_to_secure_thread: STATUS_ERROR
[29444.651666] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29444.670445] svc_normal_to_secure_thread: STATUS_ERROR
[29444.676012] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29535.544008] svc_normal_to_secure_thread: STATUS_ERROR
[29535.549296] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x72c
[29535.573340] svc_normal_to_secure_thread: STATUS_ERROR
[29535.579170] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29535.601150] svc_normal_to_secure_thread: STATUS_ERROR
[29535.606450] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29535.628635] svc_normal_to_secure_thread: STATUS_ERROR
[29535.635218] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29535.655873] svc_normal_to_secure_thread: STATUS_ERROR
[29535.661683] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29535.683740] svc_normal_to_secure_thread: STATUS_ERROR
[29535.690255] intel-fcs soc:firmware:svc:fcs: error, mbox_error=0x3
[29686.243536] soc64-hwmon soc:firmware:hwmon: timeout waiting for SMC call
[29716.227544] soc64-hwmon soc:firmware:hwmon: timeout waiting for SMC call
 

...and the fcs_client -G command returns only zero data and not the correct/expected provisioning data, see below:

 

root@air3229:~# fcs_client -G prov-data.txt -p
ioctl return status=0
W0:Provision Status Code: 0x0
W1:Key Cancellation Status: 0x0
W2:Co-Sign Status: 0
W2:RootHash0 Cancel Status: 0
W2:RootHash1 Cancel Status: 0
W2:RootHash2 Cancel Status: 0
W2:Number of Hashes: 1
W2:Type of Hash: None
number of hashes is 1
Correct/Expected fcs_client -G output should be:

root@air3229:~# fcs_client -G prov-data.txt -p
ioctl return status=0
W0:Provision Status Code: 0x0
W1:Key Cancellation Status: 0x7
W2:Co-Sign Status: 1
W2:RootHash0 Cancel Status: 0
W2:RootHash1 Cancel Status: 0
W2:RootHash2 Cancel Status: 0
W2:Number of Hashes: 3
W2:Type of Hash: secp384r1
number of hashes is 3
0000: 61 aa 56 65 a9 a0 c2 fe
0008: 8d ff 38 af 3d 9b 69 20
0010: 9b 96 ee ea 9c 29 b6 64
0018: 18 c5 a6 47 87 6e ab 70
0020: d2 2a 81 a1 cd fd 85 8b
0028: 4a 78 2e fd 91 6e d3 12
KCS[0]: 0x0
0000: 0a 78 f7 6f 1f 29 96 4e
0008: 9f 3d 2f f3 c7 75 61 7f
0010: dd 31 d9 0a 8c 65 27 e9
0018: f9 81 27 5b c2 7d fc e1
0020: 1f 0d ee 87 73 b8 7b 82
0028: 68 ad ab bc 4e ce d4 95
KCS[1]: 0x0
0000: a4 c8 ea 41 65 8d 09 05
0008: 77 4c 6a 37 2c 67 a0 16
0010: 0a 77 fc 67 1e 34 91 b3
0018: 7a 66 49 b2 5e 8e c1 66
0020: 42 68 17 0d ac 91 b4 ce
0028: 9b 8e 63 f3 ca 49 16 f7
KCS[2]: 0x0
C1:Big Counter Base: 0x1
C1:Big Counter Value: 0x0
C2:SVN Counter Value3: 0x0
C2:SVN Counter Value2: 0x0
C2:SVN Counter Value1: 0x0
C2:SVN Counter Value0: 0x0
Service Root Key #1 Fuse Status: 0x0
Service Root Key #0 Fuse Status: 0x0
We found that the below kernel commit seems to introduce the problem. If we revert this commit we don't see any problems running these tests.
The commit might solve some kernel module loading issue, but it seems it also introduce a concurrency issue when running multiple SVC clients in parallel.

_commit 94415527742e10e8935eda52e29d11095407cb6a_
_Author: Meng Li <Meng.Li@windriver.com>_
_Date: Fri Dec 17 12:17:39 2021 +0800_

_firmware: stratix10-svc: release mutex sdm_lock after processing command_

_When loading intel-fcs.ko kernel module, system hangs up. This issue_
_is introduced by sdk commit fc23818c8dfa("HSD #14015013554: firmware:_
_stratix10-svc: support up to 4 SVC clients "). This commit intends to declear_
_that stratix10-svc driver can support 4 SVC clients by allocating their own_
_channel to communicate with ATF, and use a mutex to control thread_
_synchronization. But the mutex sdm_lock is not released after processing_
_command, and cause a deadlock issue. So, add the mutex release code after_
_processing command._

_Signed-off-by: Meng Li <Meng.Li@windriver.com>_