Re: unexpected stall when join()ing

Hello again,

I was wondering whether anybody may have any updates on this?  My problem
still persists, but only on every once in a while.  Does anybody else
experience this?

The 'perl -V' output from my computer is included below.  I've since tried
it on another machine, a debian 2.4.18, which also experiences the same
problems.

Any suggestions would be appreciated.

Regards,
James Taylor

perl -V output:



Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.18-5, archname=i686-linux-thread-multi
    uname='linux <hidden> 2.4.18-5 #1 mon jun 10 15:31:48 edt 2002 i686
unknown '
    config_args='-de -Dusethreads'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-I/usr/include/gdbm',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing
-I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.3 2.96-112)',
gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
    libc=/lib/libc-2.2.5.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.2.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
  Built under linux
  Compiled at Oct 30 2002 16:55:12
  @INC:
    /usr/local/lib/perl5/5.8.0/i686-linux-thread-multi
    /usr/local/lib/perl5/5.8.0
    /usr/local/lib/perl5/site_perl/5.8.0/i686-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.8.0
    /usr/local/lib/perl5/site_perl
    .






James Taylor wrote:

> Hello,

> (I sent this message on Friday to [email protected], but it doesn't
> seem to have made it there.  I'm now sending this through the perldiscuss
> website.  Apologies if this gets posted twice.)

> I have written a small webbot that uses threads.  Due to the memory leak
> reported by the "Scalars leaked" error message, I have coded the programme
> to allow a fixed number of threads to execute, then quit, then respawn
> itself.  (I just read that this memory leak may have been fixed in 5.8.1,
> so I may be using an outdated version.)   This way works fine, as the
> state of the bot is saved (to files) between runs and restarting the bot
> only takes a few seconds.  (Ideally, though, this bot would be running as
> a daemon-like process.)

> The problem I am experiencing occurrs during the final "clean-up" phase. 
> After having run a number of threads, my programme essentially just waits
> around for the remaining children to complete their job before it (the
> parent) quits.  During this time, and more specifically when few ( < 10 )
> children still remain, the still-running children (and occasionally the
> parent, too, ) stall.   I have written a minimalist programme that
> (sometimes) experiences the same behaviour and have included it below.  I
> am using both 'threads' and 'threads::shared' and don't know which of
> these (if any) is related to my problem.  Any help would be greatly
> appreciated.

> Quick English explanation of programme:
> Create a total of 1000 threads, but never create any new ones if 50 are
> already running.  (i.e. try to save some resources)  Each thread just
> sleeps a random (integer) number of seconds (0 to 9), and then returns
> with the number of seconds slept.   After the 1000 threads have been
> created, we wait for them to finish before the main programme exits. 
> Diagnostic messages are printed to screen.

> ################################################################

> #!/usr/bin/perl -w

> use strict;

> use threads;
> use threads::shared;

> $| = 1;

> sub func( $ );

> my @threads = ();
> my $num_threads = 0;
> # these numbers produce the error fairly frequently on my machine
> #   you may need to change this on yours
> my $total_threads = 1000;
> my $max_simultaneous_threads = 50;
> my @finished_threads : shared = ();
> my $thread_id;
> my $result;




> for my $i (0..$total_threads) {
>  {
>    # join any finished threads
>    lock(@finished_threads);
>    while (@finished_threads) {
>      $thread_id = shift @finished_threads;
>      $result = $threads[$thread_id]->join;
>      $num_threads--;
>      print "Thread $thread_id finished: $result.n";
>    }
>  }


>  while ($num_threads >= $max_simultaneous_threads) {
>    # don't continue until we join at least some threads
>    {
>      lock(@finished_threads);
>      while (@finished_threads) {
>    $thread_id = shift @finished_threads;
>    $result = $threads[$thread_id]->join;
>    $num_threads--;
>    print "Thread $thread_id finished: $result.n";
>      }
>    }
>    if ($num_threads >= $max_simultaneous_threads) {
>      print "Yielding ($num_threads)...n";
>      threads->yield;
>    }
>  }

>  # ok, create a new thread now
>  $threads[$i] = threads->new(&func, $i);
>  $num_threads++;
> }


> while ($num_threads > 0) {
>  # just wait around until all threads join back
>  {
>    lock(@finished_threads);
>    while (@finished_threads) {
>      $thread_id = shift @finished_threads;
>      $result = $threads[$thread_id]->join;
>      $num_threads--;
>      print "Thread $thread_id finished: $result.n";
>    }
>  }
>  if ($num_threads > 0) {
>    print "Yielding at end ($num_threads)...n";
>    threads->yield;
>  }
> }


> # this subroutine is what is called as a new thread each time
> sub func( $ ) {
>  my ($num) = @_;
>  my $return = int(rand(10));
>  sleep($return);

>  {
>    lock(@finished_threads);
>    push @finished_threads, $num;
>    return $return;
>  }
> }

> ###########################################################

> Thank you for any assistance you may provide.

> Cheers,
> James
> P.S. I used this "online" email address in order to protect myself from
> spam.  If desired, I can reply personnally to postings from my "real"
> email address.