Discussion:
Journalctl exits under heavy logs
Add Reply
Ravindran Shanmugam
2021-03-20 19:50:36 UTC
Reply
Permalink
Hi,



Bug / Issue:

Under heavy logs entering the systemd-journald, journalctl Exits with one
of the following error messages:



Failed to iterate through journal: Bad message

Failed to get realtime timestamp: Bad message

Failed to get monotonic timestamp: Bad message



The systemd is at version 243



Is this an Upstream bug which was fixed.?



N.B:-

As of now, not planning to upgrade to latest systemd version(247) for some
legacy reasons.





Repro steps:-

1)Change the following 2 options in the file "/etc/systemd/journald.conf"
from



RateLimitIntervalSec=1s

RateLimitBurst=2000

to

RateLimitIntervalSec=0

RateLimitBurst=0



to turn off rate limiting, and then kill the "systemd-journald" process so
that the new one

will be spawned and picks up the new jounald.conf







2) Start ‘journalctl’ with this command:

"/bin/journalctl -o json
--output-fields=_SOURCE_REALTIME_TIMESTAMP,__REALTIME_TIMESTAMP,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT,SYSLOG_FACILITY,MESSAGE,PRIORITY
-f --no-tail > /dev/null 2>&1 &"







3)Create following bash script and run it as "test.sh 100 1"

--------------------

#! /bin/bash



if [[ ! -e /tmp/lines1.txt ]]; then

for i in $(seq 1 50001); do echo "abc1-$i" >> /tmp/lines1.txt; done

for i in $(seq 1 50001); do echo "abc2-$i" >> /tmp/lines2.txt; done

for i in $(seq 1 50001); do echo "abc3-$i" >> /tmp/lines3.txt; done

for i in $(seq 1 50001); do echo "abc4-$i" >> /tmp/lines4.txt; done

for i in $(seq 1 50001); do echo "abc5-$i" >> /tmp/lines5.txt; done

fi



loggernum=${1}

if [[ -z ${loggernum} ]]; then

loggernum=10

fi

loopnum=$((loggernum/5+1))



foreverloop='no'

if [[ -n "$2" ]]; then

foreverloop='yes'

fi



while [[ 1 ]]; do

for i in $(seq 1 ${loopnum}); do

logger -p local0.3 -f /tmp/lines1.txt &

logger -p local0.3 -f /tmp/lines2.txt &

logger -p local0.3 -f /tmp/lines3.txt &

logger -p local0.3 -f /tmp/lines4.txt &

logger -p local0.3 -f /tmp/lines5.txt &

done



sleep 1

ps aux | grep logger | wc -l



if [[ "${foreverloop}" == "yes" ]]; then

echo 'wait for loggers to exit'

wait

journalctl_pid=$(pidof journalctl)

if [[ -z "${journalctl_pid}" ]]; then

echo "journalctl dead, exit"

exit 1

fi

else

exit 0

fi

done





4) Wait for until the message "journalctl dead, exit" appears on the
console, this means journalclt exited.





Rgds,

--Ravi

Loading...