Fixed
Created: Apr 22, 2015
Updated: Dec 3, 2018
Resolved Date: Apr 28, 2015
Previous ID: LIN5-20355
Found In Version: 6.0.0.18
Fix Version: 6.0.0.20
Severity: Severe
Applicable for: Wind River Linux 6
Component/s: Kernel
Customer's hardware is based on Eurotech Catalyst TC, which is very similar to the intel_atom_eg20t_pch and somewhat similar to the intel_atom_z530 target/bsp
Customer has tried this on multiple setups and it has reproduce on all so it does not seam to be related to hardware.
The issue seems to be isolated around larger imports to the ext4 partition. So what they are seeing is they do a reprogramming event with a standard set of customer data that at the start of the program is backed up and then at the end the customer data has been wipe clean then restored from the backup. At this point the system is functional, but if the end customer soon after performs another programming session (this one a simple one / just additional/new apps) is where they see the failure. This second programming session locks up.
If you are logged into the system before the system locks up, you can query the system. Otherwise, you can’t.
‘top’ reports out the following:
root@PCGU6UA130009:/root> top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7825 root 20 0 2988 1156 888 R 2 0.1 0:00.05 top
582 root 20 0 6876 2972 2240 D 0 0.1 0:01.26 syslog-ng
830 root 20 0 0 0 0 D 0 0.0 0:00.87 flush-179:0
2754 root 20 0 413m 79m 61m D 0 4.0 0:19.72 yum
2769 root 20 0 2308 628 528 D 0 0.0 0:00.00 pidof
2787 root 37 17 2488 936 804 D 0 0.0 0:00.04 logrotate
3449 root 20 0 2332 316 248 D 0 0.0 0:00.00 touch
Below is the stack trace for process which are in D status:
root@PCGU6UA130009:/root> cat /proc/830/stack
[<c11e3a43>] __jbd2_log_wait_for_space+0xc3/0x1b0
[<c11dece7>] start_this_handle.isra.6+0x267/0x480
[<c11df0eb>] jbd2__journal_start+0xbb/0x100
[<c11df142>] jbd2_journal_start+0x12/0x20
[<c11b4b50>] ext4_journal_start_sb+0x60/0x1b0
[<c11991ed>] __ext4_journalled_writepage+0x6d/0x150
[<c119993f>] ext4_writepage+0x11f/0x2b0
[<c10e1060>] __writepage+0x10/0x40
[<c10e1442>] write_cache_pages+0x162/0x430
[<c10e174d>] generic_writepages+0x3d/0x60
[<c10e2e85>] do_writepages+0x35/0x40
[<c1133133>] writeback_single_inode+0x123/0x3c0
[<c11336fb>] writeback_sb_inodes+0x15b/0x210
[<c1133834>] __writeback_inodes_wb+0x84/0xb0
[<c1133c7a>] wb_writeback+0x1fa/0x360
[<c1135313>] wb_do_writeback+0x1a3/0x210
[<c11353fd>] bdi_writeback_thread+0x7d/0x290
[<c104f39e>] kthread+0x6e/0x80
[<c16a5d3a>] kernel_thread_helper+0x6/0x10
[<ffffffff>
root@PCGU6UA130009:/root> cat /proc/2787/stack
[<c11e3a43>] __jbd2_log_wait_for_space+0xc3/0x1b0
[<c11dece7>] start_this_handle.isra.6+0x267/0x480
[<c11df0eb>] jbd2__journal_start+0xbb/0x100
[<c11df142>] jbd2_journal_start+0x12/0x20
[<c11b4b50>] ext4_journal_start_sb+0x60/0x1b0
[<c11991ed>] __ext4_journalled_writepage+0x6d/0x150
[<c119993f>] ext4_writepage+0x11f/0x2b0
[<c10e1060>] __writepage+0x10/0x40
[<c10e1442>] write_cache_pages+0x162/0x430
[<c10e174d>] generic_writepages+0x3d/0x60
[<c10e2e85>] do_writepages+0x35/0x40
[<c1133133>] writeback_single_inode+0x123/0x3c0
[<c11336fb>] writeback_sb_inodes+0x15b/0x210
[<c1133834>] __writeback_inodes_wb+0x84/0xb0
[<c1133c7a>] wb_writeback+0x1fa/0x360
[<c1135313>] wb_do_writeback+0x1a3/0x210
[<c11353fd>] bdi_writeback_thread+0x7d/0x290
[<c104f39e>] kthread+0x6e/0x80
[<c16a5d3a>] kernel_thread_helper+0x6/0x10
[<ffffffff>] 0xffffffff
root@PCGU6UA130009:/root> cat /proc/2754/stack
[<c113a75d>] sleep_on_buffer+0xd/0x20
[<c113a690>] __wait_on_buffer+0x50/0x110
[<c11e361c>] jbd2_log_do_checkpoint+0x14c/0x4b0
[<c11e3a03>] __jbd2_log_wait_for_space+0x83/0x1b0
[<c11dece7>] start_this_handle.isra.6+0x267/0x480
[<c11df0eb>] jbd2__journal_start+0xbb/0x100
[<c11df142>] jbd2_journal_start+0x12/0x20
[<c11b4b50>] ext4_journal_start_sb+0x60/0x1b0
[<c119d051>] ext4_dirty_inode+0x21/0x50
[<c1132d2f>] __mark_inode_dirty+0x2f/0x1f0
[<c1125b93>] file_update_time+0xa3/0x100
[<c10f7d25>] handle_pte_fault+0x2b5/0xa90
[<c10f9484>] handle_mm_fault+0xa4/0xe0
[<c16a21c5>] do_page_fault+0x1c5/0x600
[<c169f7bb>] error_code+0x5f/0x64
[<ffffffff>] 0xffffffff
root@PCGU6UA130009:/root> cat /proc/2754/stack
[<c107e063>] rt_down_read+0x23/0x30
[<c10f9b29>] __access_remote_vm+0x29/0x180
[<c10f9cfc>] access_process_vm+0x4c/0x70
[<c115a725>] proc_pid_cmdline+0x75/0xf0
[<c115b7df>] proc_info_read+0x7f/0xc0
[<c110ebe9>] vfs_read+0x89/0x150
[<c110ecef>] sys_read+0x3f/0xf0
[<c16a5810>] sysenter_after_call+0x0/0x19
[<ffffffff>] 0xffffffff
root@PCGU6UA130009:/root> cat /proc/2769/stack
[<c11e3a43>] __jbd2_log_wait_for_space+0xc3/0x1b0
[<c11dece7>] start_this_handle.isra.6+0x267/0x480
[<c11df0eb>] jbd2__journal_start+0xbb/0x100
[<c11df142>] jbd2_journal_start+0x12/0x20
[<c11b4b50>] ext4_journal_start_sb+0x60/0x1b0
[<c119abbb>] ext4_setattr+0x2ab/0x480
[<c1127d3b>] notify_change+0x1ab/0x330
[<c110cce7>] do_truncate+0x57/0x90
[<c111c525>] do_last.isra.33+0x475/0x760
[<c111c8af>] path_openat+0x9f/0x360
[<c111cd72>] do_filp_open+0x32/0x80
[<c110dae0>] do_sys_open+0xf0/0x240
[<c110dc63>] sys_open+0x33/0x40
[<c16a5810>] sysenter_after_call+0x0/0x19
[<ffffffff>] 0xffffffff
Here is how the system is configured:
-bash-3.2# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# the device names in this file were setup during the install process.
/dev/mmcblk0p2 / ext4 ro,noatime 0 1
/dev/mmcblk0p1 /boot ext2 ro,noatime 0 0
/dev/mmcblk0p3 /opt/persistent ext4 rw,noatime,data=journal 0 2
/dev/mmcblk0p5 /opt/GSix ext4 ro,noatime 0 2
/dev/mmcblk0p6 /var ext4 rw,noatime,data=journal 0 2
/dev/mmcblk0p7 /opt/usr_data ext4 rw,noatime,data=journal 0 2
tmpfs /run tmpfs rw,nodev,nosuid,noexec,mode=1777 0 0
tmpfs /logging tmpfs size=12M,nodev,nosuid,noexec,mode=1777 0 0
Here are the kernel parameters. Attached is the kernel config.
KERNEL_PARAMETERS="root=/dev/${ROOTFSPARTITION} rootfstype=ext4 ro rootwait notsc quiet loglevel=1 console=tty4 vt.global_cursor_default=0 pci=routeirq video=vesafb"
