Wind River Support Network

Description

While debugging we were thinking that max_dgram_qlen's default value is 10 and due to this may be logs are missing in syslogd.

So, we changed its value to max_dgram_qlen=512 but still having the problem.

Could any one have the idea why logs are missing in syslogd and present in journal.

Description of problem:

All CPP system log entries are also routed to default system log i.e. journal.

Snapshot of CPP System log (/c/logfiles/systemlog/syslog):

Nov 1 12:27:08 2016 oamhost local1.info SYS-EV-LOG[2258]: PIU Device unoperational ( dev = 1, smn = 0, apn = 3 ) 
 Nov 1 12:27:08 2016 oamhost local1.info SYS-EV-LOG[2258]: PIU Device unoperational ( dev = 2, smn = 0, apn = 3 ) 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 1 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 2 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 3 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 4 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 5 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 6 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 7 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 8 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 9 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 10 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 11 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:27:10 2016 oamhost local1.info SYS-EV-LOG[2258]: Switch RPU 12 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 Nov 1 12:28:00 2016 oamhost local1.info SYS-EV-LOG[2258]: MP ( smn = 0, apn = 3 ) starting, administrative state = UNLOCKED 
 Nov 1 12:28:00 2016 oamhost local1.info SYS-EV-LOG[2258]: Program CXC1732001%3_P94A004 on PIU 0.3 started 
 Nov 1 12:28:00 2016 oamhost local1.info SYS-EV-LOG[2258]: Program CXC1737213_RSYSCD on PIU 0.3 started

Snapshot of Journal:

0025: Nov 01 12:27:08 002500 SYS-EV-LOG[2258]: PIU Device unoperational ( dev = 1, smn = 0, apn = 3 ) 
 0025: Nov 01 12:27:08 002500 SYS-EV-LOG[2258]: PIU Device unoperational ( dev = 2, smn = 0, apn = 3 ) 
 0025: Nov 01 12:27:08 002500 postgres[8764]: [3-1] PG 10.32.0.48(43294) LOG: unexpected EOF on standby connection 
 0025: Nov 01 12:27:10 002500 TRI_SERVER[706]: pthread join success : 0x000300 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 1 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 2 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 3 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 4 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 5 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 6 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 7 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 8 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 9 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 10 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 11 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 12 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 ----------- Missing entries in CPP system log but present here ----------------------------- 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 13 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 14 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 15 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 16 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 17 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 18 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 19 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 20 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: Switch RPU 21 from MP (smn = 0, apn = 3 ) to MP (smn = 0, apn = 25 ). Cause: board restart/failure. 
 0025: Nov 01 12:27:10 002500 SYS-EV-LOG[2258]: MP (smn = 0, apn = 3 ) down 
 ----------- Missing entries end here ----------------------------- 
 0025: Nov 01 12:27:10 002500 systemd-journal[292]: Forwarding to syslog missed 10 messages. 
 0025: Nov 01 12:27:10 002500 systemd[1]: Starting Periodically store time in /pramfs... 
 0025: Nov 01 12:27:10 002500 systemd[1]: Started Periodically store time in /pramfs. 
 0025: Nov 01 12:27:10 002500 systemd[1]: Stopping busybox ifplugd daemon... 
 ------ Few non SYS-EV-LOGs -------------

0025: Nov 01 12:28:00 002500 SYS-EV-LOG[2258]: MP ( smn = 0, apn = 3 ) starting, administrative state = UNLOCKED 
 0025: Nov 01 12:28:00 002500 SYS-EV-LOG[2258]: Program CXC1732001%3_P94A004 on PIU 0.3 started 
 0025: Nov 01 12:28:00 002500 SYS-EV-LOG[2258]: Program CXC1737213_RSYSCD on PIU 0.3 started

Highlighted 10 entries are missing in CPP system log where as they are successfully written to journal.

If syslogd could not handle the messages, shouldn't the traces to journal be dropped ?

Workaround

As I was analysing this situation, I came across this link here :

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762700;msg=32

The attached patch seems to work OK for the customer, the fix will also need to be pushed on the wrl8