System freezing

"Winston Gutkowski" <[email protected]>
Newsgroups gmane.linux.redhat.release.enigma
Message-ID <[email protected]>
Hi all,

I posted a missive to this forum on the 27th March about problems we were having with our servers freezing up. I include the
original text below for anyone who missed it.

Since then I have managed to duplicate the problem on one of our test machines, and had the luck to have both test and production
servers go down on the same night at the same time. It would appear that the culprit job is *not* the cron.daily as I first assumed,
but a script written by me which runs at very near the same time.

It is meant to create backups of files from the same or another machine by piping the result of a tar to an untar. As stated before,
the problem is not consistent, but periodically it locks up the machine.

I enclose the script and its configuration file for anyone who would care to try their teeth on it. I am still mystified as to what
the error is. The only explanation that should be needed is the "squeeze" command which creates TAB-delimited "word" columns from
input (in this case, the config file).

Original message is below FYI. Anyone out there who can help, I will be eternally grateful.

Thanks

Winston Gutkowski


Bulk of message from March 27th:
...
We have 3 generic Intel servers running RH7.2, and 2 of them have major stability problems.
They are as follows:
1. firewall - no problems.
This is a small (1Gb memory) machine with 2 software RAID 1'd IDE drives, performing firewall and DNS services.
2. Application routing server - constant problems
Medium-sized (2Gb memory) machine with 2 software RAID 1'd IDE drives, running Jetty HTTP server serving Java servlets.
3. Database server - intermittent problems
Big server (3Gb memory) + Disk Array: 2 onboard and 6 external SCSI disks, all software RAID 1'd, running Jetty HTTP server and
Oracle 8.1.7.

In addition to the above mentioned software, all machines also run tripwire intrusion detection software as well.

On both machines that have problems, the system either freezes or crashes just after 4AM, making me think that it may have something
to do with the cron.daily jobs. I found some pages on the Web from people who had problems with machines crashing when makewhatis
runs, but supposedly the problem was fixed in 7.1. On the database server I would sometimes get messages saying that the system did
not have enough memory to execute a fork() shortly before the system froze (it didn't halt; the console merely stopped responding).
On the Application server, the system would simply power itself down (badly; as though someone had simply turned it off at the wall)
shortly after 4AM every 3rd or 4th day.
...
ezfiles.conf.txt (text/plain, 1.5 KB)
# This file contains the list of files to be copied or backed up to
# tape by the toolkit script 'ezfiles'. Any box listed in column
# 1 must be valid for ssh access by the executing user.
#
# The format of a file specification line is as follows:
#				Run	Run	Keep
# Box	File/Directory		Level	Days	[# Comments]
#
# Box name must start in position 1.
# File/Directory cannot be "/".
# Run Level is a number between 0 and 9, which indicates the level at
# which the file/directory is included in the action to be performed.
# Levels are specified by the -l parameter to ezfiles and a line will
# be included if its Run Level is <= the -l value (hence 0 = always).
# Keep days specifies the number of days to keep copied files in the
# target /backup directory on the host machine (based on find...-mtime).
#
# In addition, this file can contain any number of script-style comments.
#
#				Run	Keep
# Machine	File/Dir	Level	Days	# Notes
#
yyzap0	/home			0	999999	# User stuff
yyzap0	/usr/java		0	999999	# Java API
yyzap0	/usr/cvsroot		0	999999	# Java source
yyzap0	/usr/Jetty		0	999999	# All Jetty versions
yyzap0	/etc			1	999999
# Files/directories for database server
yyzdb0	/home/oracle/bin	0	999999	# Oracle utility scripts
yyzdb0	/home/oracle/dba	0	999999
yyzdb0	/home/oracle/[0-9].*	0	999999	# EZText DB version SQL
yyzdb0	/backup/oracle/ezp	0	5	# Production DB dumps
yyzdb0	/usr/local/eztext	1	999999	# Miscellaneous utility jars
yyzdb0	/usr/Jetty		1	999999	# All Jetty versions
yyzdb0	/usr/java		1	999999	# All java API versions
yyzdb0	/etc			1	999999
ezfiles.txt (text/plain, 2.8 KB)
#!/bin/sh
# This script is designed to copy/back up eztext system files.
# and directories. In "copy" mode, files are copied from a remote
# machine to /backup/<machine>, while in "backup" mode previously
# "copy"d files are backed up to tape. Files to be copied are
# specified in a config file (Default=$0.conf).
#
# Syntax: ezfiles [-c configfile][-l level] action [machine]
#	where action is one of:
# find	List files/directories in /backup (Uses find ... -ls).
# list	List files/directories in config file.
# copy	Copy files from machines as specified in config file.
#	USER MUST HAVE AN SSH VERSION 1 KEY WITH NO PASSPHRASE SET
#	UP ON ALL SPECIFIED MACHINES.
# backup Back up files from /backup.
# purge	Purge files in /backup as specified in the config file.
#	(Uses the "Keep Days" column to determine what to delete)
#
# Specifying <machine> limits <action> to files from <machine> only.
#
# -c	Determines config file to use. Default=$0.conf.
# -l	Run level. 0-9. Determines which files to copy/backup/list.
#	The higher the level the more files are included. Default=0.
#
FLAGS=h:l:
. tkflags

test -f ${val_c:=$0.conf} || exec tkerror $0 Config file $val_c not found

ARCHIVE=/dev/st0	# SCSI tape device
THISBOX=`hostname -s`
SSHCMD="ssh -1"
SSH=

# Print an underline or overline for a message
msgline(){ echo "$*"|tr -c '\012' '*' ;}

# Print a message with a date
message(){
	msg="`date "+%Y/%m/%d %H:%M:%S"` $*"
	msgline "$msg"
	echo "$msg"
	msgline "$msg"
}

# List file lines from the config file for the relevant machine
# and level, removing any lines which specify the root directory.
# Output is TAB-delimited columns. Syntax: list [machine]
list(){ squeeze -sc1- $val_c|grep -i "^${1:-[^[:blank:]]\{1,\}}	[^[:blank:]]\{1,\}	[0-${val_l:-0}]\>"|egrep -v '	/	' ;}

# Convert output from list() above, usually to shell commands.
output(){ sed "s~\([^[:blank:]]\{1,\}\)	\([^[:blank:]]\{1,\}\)	\([^[:blank:]]\{1,\}\)	\([^[:blank:]]\{1,\}\)\>.*~$*~" ;}

# Create a tar archive.
ftar(){ $SSH tar cvf $ARCHIVE $* ;}

backup(){ ftar `list $1|output '\1\2'` ;}

purge(){ list $1|output 'find \1\2 -type f -mtime +\4 -atime +\4 -ctime +\4 -exec rm {} \\\;' ;}

find(){ list $1|output 'find /backup/\1\2 -type f -ls'|sh -x ;}

copy(){
	ARCHIVE="-"	# make ftar go to stdout
	for machine in `list $1|squeeze -sc1|sort -u`
	do
		test $machine = "$THISBOX" && SSH= || SSH="$SSHCMD $machine"
		mkdir -p /backup/$machine
		test -w /backup/$machine || exec tkerror $0 Cannot write to /backup/$machine
		cd /backup/$machine
		set -x
		ftar `list $machine|squeeze -sc2`|tar xf -
		test "$TKDEBUG" || set +x
	done
}

cd /backup

message "Started $0 $*"

case ${1:-list} in
	list)	list "$2"|squeeze -c1-2 ;;
	find|copy|backup|purge)	$1 "$2" ;;
	*)	$* ;;
	# *)	exec tkerror $0 Invalid action $1
esac

message "Finished $0 $*"
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.