Work around rpm deadlocking with "setlock2"
"Peter Wolfenden" <[email protected]>
| Newsgroups | gmane.comp.djb.syslog |
|---|---|
| Message-ID | <[email protected]> |
Hello,
I've been using the daemontools package for over a year now,
and have found it very valuable both for building trustworthy
software on our in-house linux platforms and for implementing
workarounds for broken software that comes from out-of-house.
For example, I recently ran into a deadlocking problem with the
'rpm' program, which we use to manage software on our in-house
linux machines. Unfortunately, RedHat doesn't seem anxious to
fix the problem - details are online:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=115152
But this one-line bash script "wraps" the rpm binary with a call
to setlock that prevents multiple processes from attempting to
call rpm at the same time, preventing the deadlock:
#!/bin/bash
setlock /tmp/.rpm_lock /bin/rpm "$@"
All our in-house systems have an rpm user which owns the local
rpm database, and a user must belong to the rpm group in order
to run rpm commands. So the first security improvement to the
bash wrapper looks like this:
#!/bin/bash
if [ root = `/usr/bin/whoami` ]; then
/usr/local/bin/setuidgid rpm /usr/local/bin/setlock \
/var/run/.rpm_lock /bin/rpm "$@"
else
/usr/local/bin/setlock /var/run/.rpm_lock /bin/rpm "$@"
fi
Our Ops team doesn't want the rpm operations to block forever,
since then a "stuck" rpm process would cause all subsequent rpm
calls to "stack up". So I wrote an enhanced version of the
setlock2 program (source attatched) which provides the following
new options (the -b option is for testing):
-p pidfile: Write PID,time to the indicated pidfile (where
PID is the process ID of the running process
and time is the current UTC time), after the
lockfile is locked.
-t timeout: If the lockfile is locked by another process,
wait up to timeout seconds to obtain the lock
before printing an error message and exiting
with a nonzero status code.
-b blocksec: After obtaining the lock (and writing the PID,
if the -p option is specified), sleep blocksec
before running program.
-s: Write error messages to syslog before writing
them to STDERR (also works for non-root users)
The second improvement to the bash script now looks like this
(we give up on the lock after waiting 10 minutes):
#!/bin/bash
if [ root = `/usr/bin/whoami` ]; then
/usr/local/bin/setuidgid rpm /usr/local/bin/setlock2 \
-p /var/run/.rpm_pid -t 600 /var/run/.rpm_lock /bin/rpm "$@"
else
/usr/local/bin/setlock2 -p /var/run/.rpm_pid -t 600 \
/var/run/.rpm_lock /bin/rpm "$@"
fi
The third improvement is to run a separate program at regular
intervals (say once an hour) to kill any running program whose
pid appears in the pidfile and which has held the indicated
lock for longer than, say, an hour (this catches any invocations
of rpm which hang forever). I wrote a Perl script to take care
of this (it first tries to obtain the lock, and then looks at
the pidfile and the process table to decide what should be done).
Does this type of problem seem sufficiently commonplance to
justify including a version of the 'setlock2' program in a
future version of the daemontools package? Or is there a better
(cleaner, faster, simpler, etc) way to get this done?
Cheers,
Peter Wolfenden
Untitled.txt
(text/plain, 10.5 KB)
#include <unistd.h>
#include "lock.h"
#include "open.h"
#include "strerr.h"
#include "string.h"
#include "pathexec.h"
#include "sgetopt.h"
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <time.h>
#include <syslog.h>
/* This program is a modified version of the setlock program
distributed with daemontools-0.76:
setlock [ -nNxX ] lockfile program [ arg ... ]
The options for the original setlock program are as follows
(see also http://cr.yp.to/daemontools/setlock.html):
-n: No delay. If the lockfile is locked by another process
then setlock immediately prints error message and exits
with a nonzero status code.
-N: (Default.) Delay. If the lockfile is locked by another
process, setlock waits until it can obtain a new lock.
-x: If the lockfile cannot be opened (or created) or locked,
setlock exits zero.
-X: (Default.) If the lockfile cannot be opened (or created)
or locked, setlock prints an error message to STDERR and
exits nonzero.
This program differs from the original setlock in that it
has these additional options:
-p pidfile: Write PID,time to the indicated pidfile (where
PID is the process ID of the running process
and time is the current UTC time), after the
lockfile is locked.
-t timeout: If the lockfile is locked by another process,
wait up to timeout seconds to obtain the lock
before printing an error message and exiting
with a nonzero status code.
-b blocksec: After obtaining the lock (and writing the PID,
if the -p option is specified), sleep blocksec
before running program.
-s: Write error messages to syslog before writing
them to STDERR (also works for non-root users)
History:
--------
Written 2004/02/16 by [email protected]. For the revision history
of this file, check the CVS log.
Notes:
------
1) This wrapper program doesn't affect the UID or GID of the
running process. To tweak these aspects of the process
(eg to limit permissions) consider the as_user wrapper
program or one of the standard wrappers in the daemontools
suite (eg envuidgid). Note that wrapper invocations can
be combined, but the order matters (particularly when
changing UID or GID).
2) The options supported by this wrapper program are parsed
directly from the "front" (left end) of the command line
and are *not* passed through to the specified program
(whose options are specified towards the "back" of the
command line), so there should be no danger of confusing
arguments to the wrapper with arguments to the "wrapped"
program. But since this wrapper is intended for use with
the 'rpm' program, I've avoided using any option letters
which are also used by rpm:
http://www.netadmintools.com/html/8rpm.man.html
*/
#define FATAL "setlock2: fatal: "
#define INFO "setlock2: "
/* This sets the maximum length of a PID string to be 17 digits,
which seems safe (though arbitrary). If a real PID string is
longer, then we'll write only the first 17 digits to the
indicated pidfile. */
#define MAX_PID 18
/* This sets the maximum length of a UTC time string to be 17
digits, which is safe (though arbitrary). Since Feb 19, 2004
~ 1077217000, ten decimal digits should take us from 1970 to
about 2285 (315 years), and the datestamps produced by this
program will start to be truncated in about 3 billion years. */
#define MAX_TIME 18
/* This sets the maximum delay to 60*60*24*7 seconds, or 1 week,
which seems safe (though arbitrary). If a user specifies a
longer time delay, then we flag an error and exit. */
#define MAX_DELAY 604800
// #define DEBUG
void usage() {
// there's no point sending the usage string to syslog...
strerr_die1x(100,"setlock2: usage: setlock2 [ -nNxXp:t:b:s ] lockfile program [ arg ... ]");
}
int flagndelay = 0; // if nonzero, don't wait for lock (from command line)
int flagx = 0; // if nonzero, don't print any errors (from command line)
/* The reverse() and itoa() routines defined below were borrowed
from p.64 of K&R _The C Programming Language_ Edition 2. */
/* Reverse a string in place (used by itoa). */
void reverse(char s[]) {
int c,i,j;
for(i=0,j=strlen(s)-1; i<j; i++,j--) {
c=s[i]; s[i]=s[j]; s[j]=c;
}
} // reverse
/* Convert a signed integer into a string, whose allocated
length is only guaranteed to be the passed limit (including
the terminal NULL). */
void itoa(int n, char s[], int m) {
int i, sign;
/* Since we need one slot for the terminal NULL character,
and (potentially) one slot for the leading minus sign,
and at least one slot for a digit, the passed limit must
be at least 3. */
if (m < 3)
strerr_die2sys(111,FATAL,"itoa called with m < 3 : ");
m -= 2; // leave room for terminal null and sign char
if ((sign = n) < 0) // record sign, and flip if necessary
n = -n;
i = 0;
do {
s[i++] = n%10 + '0'; // get next digit
} while (((n/=10) > 0)&&(m-- > 0)); // remove it
if (sign < 0)
s[i++] = '-';
s[i] = '\0';
reverse(s);
} // itoa
char pidstr[MAX_PID]; // PID as printable string
int pidval = 0; // PID as comparable number
char *timestr; // lock timeout (in seconds) as string
int timeval = 0; // lock timeout (in seconds) as number
char *pidfile; // place to write PID and time (from command line)
char *blockstr; // seconds to block, as string (from command line)
int blockval = 0; // seconds to block, as number
int flags = 0; // write messages to syslog if on (from command line)
const char *file; // lockfile (from command line)
/* The alarm() invocation (see below) causes this routine
to catch a timeout signal after we've been waiting for
'timeval' seconds for a lock on 'file'. */
void timed_out(void)
{
char msg[100];
if (flags)
syslog(LOG_INFO,"lock request for %s timed out after %s seconds",file,timestr);
sprintf(msg, "lock request for %s timed out after %s seconds : ",file,timestr);
strerr_die2sys(111,INFO,msg);
} // timed out
int main(int argc,const char *const *argv,const char *const *envp)
{
int opt; // for processing command line options
int fd, fd2; // file descriptors for lockfile and pidfile
long t; // current time as number (epoch sec)
char tbuf[MAX_TIME+1]; // current time as string
while ((opt = getopt(argc,argv,"nNxXp:t:b:s")) != opteof)
switch(opt) {
case 'n': flagndelay = 1; break;
case 'N': flagndelay = 0; break;
case 'x': flagx = 1; break;
case 'X': flagx = 0; break;
case 'p':
pidfile = strdup(optarg);
break;
case 't':
timestr = strdup(optarg);
timeval = atoi(timestr);
if (timeval > MAX_DELAY) {
fprintf(stderr,"lock timeout (%d) > 1 week (%d) - too long!\n",timeval,MAX_DELAY);
usage();
}
else if (timeval < 0) {
fprintf(stderr,"lock timeout (%d) must not be negative!\n",timeval);
usage();
}
break;
case 'b':
blockstr = strdup(optarg);
blockval = atoi(blockstr);
if (blockval > MAX_DELAY) {
fprintf(stderr,"blocking time (%d) > 1 week (%d) - too long!\n",blockval,MAX_DELAY);
usage();
}
else if (blockval < 0) {
fprintf(stderr,"blocking time (%d) must not be negative!\n",blockval);
usage();
}
break;
case 's': flags = 1; break;
default: usage();
}
argv += optind;
if (!*argv) {
printf("Got no command-line parameters (need at least 2)\n");
usage();
}
file = *argv++;
if (!*argv) {
printf("Got only one command-line parameter (need at least 2)\n");
usage();
}
#ifdef DEBUG
printf("Before opening lockfile %s ...\n",file); fflush(stdout);
#endif
fd = open_append(file);
if (fd == -1) {
if (flagx) _exit(0);
if (flags)
syslog(LOG_ERR,"unable to open_append %s",file);
strerr_die4sys(111,FATAL,"unable to open_append ",file,": ");
}
#ifdef DEBUG
printf("Before checking timeval %d ...\n",timeval); fflush(stdout);
#endif
if (timeval) { // legal timeout was specified
signal(SIGALRM,timed_out); // set signal handler
alarm(timeval); // set alarm
} // legal timeout was specified
#ifdef DEBUG
printf("Before locking lockfile %s ...\n",file); fflush(stdout);
#endif
if ((flagndelay ? lock_exnb : lock_ex)(fd) == -1) {
if (flagx) _exit(0);
if (flags)
syslog(LOG_ERR,"unable to lock %s",file);
strerr_die4sys(111,FATAL,"unable to lock ",file,": ");
}
if (timeval) // legal timeout was specified
alarm(0); // clear alarm
#ifdef DEBUG
printf("Before checking/writing pidfile %s ...\n",pidfile); fflush(stdout);
#endif
if (strlen(pidfile)) { // pidfile was specified
fd2 = open_trunc(pidfile);
if (fd2 == -1) {
if (flagx) _exit(0);
if (flags)
syslog(LOG_ERR,"unable to open_trunc %s",pidfile);
strerr_die4sys(111,FATAL,"unable to open_trunc ",pidfile,": ");
}
pidval = getpid();
itoa(pidval,pidstr,MAX_PID);
write(fd2,pidstr,strlen(pidstr)); // don't write the null char!
close(fd2);
fd2 = open_append(pidfile);
if (fd2 == -1) {
if (flagx) _exit(0);
if (flags)
syslog(LOG_ERR,"unable to open_append %s",pidfile);
strerr_die4sys(111,FATAL,"unable to open_append ",pidfile,": ");
}
t = (long)time(NULL);
tbuf[0] = ',';
itoa(t,tbuf+1,MAX_TIME);
write(fd2,tbuf,strlen(tbuf)); // don't write the null char!
close(fd2);
} // pidfile was specified
#ifdef DEBUG
printf("Before checking blockval %d ...\n",blockval); fflush(stdout);
#endif
if (blockval) { // blocking interval was specified (and legal)
#ifdef DEBUG
printf("setlock2: blocking for %s seconds before launching %s ...\n",blockstr,argv[0]);
#endif
sleep(blockval);
} // blocking interval was specified (and legal)
#ifdef DEBUG
printf("Before running child %s ...\n",argv[0]); fflush(stdout);
#endif
pathexec_run(*argv,argv,envp);
if (flags)
syslog(LOG_ERR,"unable to run %s",*argv);
strerr_die4sys(111,FATAL,"unable to run ",*argv,": ");
} // main