Re: Index Helper for beagle++ hangs occasionally

Hi dBera,

Looks like you hit it.

Am Samstag, den 08.12.2007, 10:02 -0500 schrieb D Bera:
> Hi Max,
> 
> > We are using external programms to do some of the metadata extraction
> > for us. The communication works via stdin and stdout - i.e. we start the
> > programm from within the IndexHelper and then read and parse its stdout.
> > Now from time to time IndexHelper will stop reading even though more
> > lines are available on stdin. It sits in a
> > 	while((line = StreamReader.ReadLine()) != null)
> > loop and does nothing.
> > The whole thing happens during the flush. All flushed filenames are
> > handed to the external filter via its stdIn and then the metadata is
> > read from its stdout.
> 
> So you mean there is an external-filter like scenario here ? In
> IndexHelper, it calls this external application, which reads its input
> and outputs some data on its stdout and then IndexHelper reads that
> output data (via a pipe ? setting stdout of the child external
> application ? reads stdout of the external application as our
> external-filter does it - just by reading the stdout of the external
> application ?).
Right now i am creating a new Process and setting it's StartInfo with
RedirectStandartOut = true;
RedirectStandartIn = true;

Then i use 
StreamReader ProcOut = p.StandartOut;
StreamWriter ProcIn = p.StandartIn;
with the Process.
I use these for all indexables in a flush so i don't have to restart the
process again and again. So i can't use 
ProcOut.ReadToEnd();
ProcOut.Close();

Instead i write a line with info about the next indexable and then read
ProcOut until it prints "EOM" - end of metadata. Write the next line and
so on.


> Make sure that the stderr of the external application is closed.
> Otherwise, the external application might be printing some information
> to its stderr, which will block stdout for a long time. Check our
> pdf/mplayer etc external app based filters.
I was trying to read the external stdout and print the external stderr
to my stderr for debugging purposes. Now i moved that to writing stderr
of the external process to a seperate logfile instead. Seems to fix the
issue. 

> > Looks like BEAGLE_EXERCISE_THE_DOG was not set but i still wonder why it
> > would sleep for 20 minutes especialy since the machiene was not under
> > heavy load.
> 
> If its all happening in IndexHelper then EXERCISE_THE_DOG will not
> affect anything. EXERCISE_THE_DOG controls beagled (the scheduler) and
> not indexhelper.
I thought it would keep the IndexHelper from beeing niced to a certain
level - however i was able to reproduce the problem with
EXERCISE_THE_DOG set. 


Thanks a lot... Tried to find the reason for a good amount of time.
 Max