Re: Local sources seem not to be working

Alexandre Santos <[email protected]> Thu, 5 May 2022 14:31:13 +0000
Newsgroups gmane.comp.syslog-ng
Message-ID <CO6PR10MB5602DB8C7C426919C740C46DD0C29@CO6PR10MB5602.namprd10.prod.outlook.com>
Hi Szilard,

The logs being written in /var/log/linecard.log are the ones coming from the ‘syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no));’ source.
Because log messages received in syslog() source, have always local4 facility.

I am sending you in attachment the write_with_rotation.sh script.

Thanks,
Alex

From: syslog-ng <[email protected]> On Behalf Of Szilard Parrag (sparrag)
Sent: 5 de maio de 2022 07:32
To: [email protected]
Subject: Re: [syslog-ng] Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Alex,

After checking the stats, you have sent we can see that there had been some writes:
from
dst.program;d_localfile_linecard#0;/opt/machine/local/bin/write_with_rotation.sh /var/log/linecard.log 10 10;a;written;4518
to

dst.program;d_localfile_linecard#0;/opt/machine/local/bin/write_with_rotation.sh /var/log/linecard.log 10 10;a;written;4549

·         we do not see increase in the counters of /var/log destinatons, but only on one destination

·         we could only see an increase in syslog-udp processed counters

·         there are no dropped/queued counters



We would guess this could be due to flow-control, but for that we would need to see non-zero queued counter values, which is not the case. It could happen that one destination hangs/can't send messages out, which leads to suspended sources due to flow-control, but the syslog() source is not affected since it doesn't send messages to the hanged destination(s).

Based on the stats, only "d_localfile_linecard" is active (~30 messages in 15 minutes), maybe the syslog() source would be affected too without the filtering.



We should see more internal logs, which is problematic, since internal() source seems to be stopped too. For that I would recommend extracting internal() source from the s_src statement and putting it in a separate log path with a simple file destination.
Also, if possible, could you please share your `write_with_rotation.sh` script? It is unlikely that it interferes with syslog-ng, but a double check would be nice. 🙂

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
write_with_rotation.sh (application/octet-stream, 3.4 KB)
#!/bin/bash

# Log filename
OUTPUT_FILE="$1"

# File Maximum size before rotate in MB
FILE_SIZE_MAX="$2"

# Number of files rotated
MAX_ROTATION_FILES="$3"

# @brief: Function to print and log errors
function err_msg() {
  echo "$0 Error: $*"
  logger "$0 Error: $*"
}

# @brief This function call the logrotate with a custom made temporary configuration file.
function logrotate_wrap() {
  # logrotate script
  # param 1 ($1) - the rotation count
  # param 2 ($2) - log file name
  # param 3 ($3) - size
  local count_nr="$1"
  local filename="$2"
  local max_size="$3"

  # create a temporary file
  temp_file=$(mktemp)

  # set contents of the temporary configuration file
  echo -e "${filename} { \n  rotate ${count_nr}\n  size ${max_size}\n  copytruncate\n  missingok\n  compress" >> "$temp_file"
  echo -e "  lastaction" >> "$temp_file"
  echo -e "    if ! gunzip --test ${filename}.1.gz ; then" >> "$temp_file"
  echo -e "      logger -p local7.err \"Integrity of log file ${filename}.1.gz not ok\"" >> "$temp_file"
  echo -e "      rm -f ${filename}.1.gz" >> "$temp_file"
  echo -e "    fi" >> "$temp_file"
  echo -e "  endscript" >> "$temp_file"
  echo -e "}" >> "$temp_file"

  # rotate the log
  /usr/sbin/logrotate "$temp_file"

  # removes the temporary file
  rm -f "$temp_file"
}

# @brief This function moves the saved log that was not compress in logrotate <filename.log.1>, to the beginning of the current log file <filename.log>.
function append_saved_log() {
  # param 1 ($1) - the log file name
  local filename="$1"

  if [[ -f "$filename" ]]; then
    # create a temporary file
    temp_file=$(mktemp)

    # append saved log to the beginning of filename
    cat "$filename.1" "$filename" > "$temp_file"
    mv -f "$temp_file" "$filename"

    # removes the temporary file
    rm -f "$filename.1"
  else
    # just move the file
    mv -f "$filename.1" "$filename"
  fi
}

function check_and_rotate()
{
  local filesize="$1"
  local file_size_max_bytes="$2"

  if [[ $filesize -gt $file_size_max_bytes ]]; then
    # Call the logrotate wrapper function
    logrotate_wrap "$MAX_ROTATION_FILES" "$OUTPUT_FILE" "$file_size_max_bytes"
    # Drop the file caching
    sync; echo 3 > /proc/sys/vm/drop_caches >& /dev/null
    return 0
  fi

  return 1
}

############# MAIN #################

# Check if the number of arguments is correct
if [[ $# -ne 3 ]]; then
  echo "Usage: $0 <log filename> <log file maxsize in MB> <rotation files>"
  exit 1
fi

# Check if 2nd parameter is a number
re='^[0-9]+$'
if ! [[ $2 =~ $re ]] ; then
  err_msg "$2 is not a number"
  exit 1
else
  # convert to MB
  file_size_max_bytes=$((FILE_SIZE_MAX * 1024 * 1024))
fi

# Check if 3rd parameter is a number
if ! [[ $3 =~ $re ]] ; then
  err_msg "$3 is not a number"
  exit 1
fi

if [[ -f "$OUTPUT_FILE.1" ]]; then
  append_saved_log "$OUTPUT_FILE"
fi

# get the size of the log-file
if [[ -f "$OUTPUT_FILE" ]]; then
  filesize=$(stat -c%s "$OUTPUT_FILE")
  if check_and_rotate "$filesize" "$file_size_max_bytes" ; then
    filesize=0
  fi
else
  touch "$OUTPUT_FILE"
  filesize=0
fi

while IFS= read -r line ; do
  # Send mesage to the log-file
  echo "$line" >> "$OUTPUT_FILE"

  # get the size of the log-file
  linesize=${#line}
  # plus one for CR
  filesize=$((filesize + linesize + 1))

  if check_and_rotate "$filesize" "$file_size_max_bytes" ; then
    filesize=0
    if [[ -f "$OUTPUT_FILE.1" ]]; then
      append_saved_log "$OUTPUT_FILE"
      filesize=$(stat -c%s "$OUTPUT_FILE")
    fi
  fi

done