Re: Very long operations with FastCGI

Thanks for the great advice.  Looks like I'm going to perform a fork, and
have the child process regularly update status information for the browser
to view on a different web page.

I had tried this before, unsuccessfully... the browser would simply hang.
Doing some research, it seems when forking through perl, your child and
parent process need explit exit() calls when they are done to prevent them
from hanging around.

In case anybody's interested...

my $pid = fork();
if ( not defined $pid )
{ #fork call failed
    &main::show_error( 'Unable to perform fork to run import' );
    return();
}
elsif ( $pid == 0 )
{ # child process
    #
    # */ LONG RUNNING PROCESS /*
    #   regularly update status information to the database
    #

    # * exit() is REQUIRED at the end of this block if
    #   you don't want hung processes
    exit(0);
}

# parent process

#
# show 'click here to view status screen' message
#  which redirects to page that reads the status information
#  from the database
#

#
# finish with html templates
#

# force an exit when done with the parent process, so it doesn't
# hang waiting for the child process to return
exit(0);



On 8/11/06, Jose Adriano Baltieri <jabaltie-a/[email protected]> wrote:
>
> Mitch Jackson wrote:
>
> > I have a long-running operation to spawn from a perl script within the
> > FastCGI environment.  I would like the script to cause the user's
> > browser to redirect to a 'view operation status' page, and continue to
> > do the processing of the operation in the background.  I see two
> > obvious ways to do this.  First using the HTTP header 'redirect:
> > [url]' and the other by writing a page with a meta refresh tag.
> >
> > Once I have written a redirect command to the user's browser, how do I
> > signal to the server and the browser that i'm done sending input, so
> > the server will flush the buffer and deliver the output, and the
> > browser will acknowledge this session has completed and do it's thing,
> > instead of waiting for more output from the web server.
>
> Mr Jackson :
>
> You have had good ideas about how to do it.
>
>
> I'll tell you the way we do it today. You may pick some ideas from it
> too.... I bet you will find it very complicated but, we think it is
> sophisticated. It takes a good time to implement but when it's done ,
> that is, when the standard building blocks are ready, you have a very
> good standard of batch processing (I think)...
>
> So, here's how it is :
>
> Whenever the user has a batch processing, he/she has a page with two
> links : start processing and get last processing results.
>
> Why do we split it like that ? Because usually these batch processing
> take too long to be generated. And we like to offer he/she a link for
> it. So, he/she may start processing and logout. Processing goes on, it's
> on the background (as a matter of fact it can be even on a mainframe).
> He/She may come back later and check processing results, to see if it's
> done or not.
>
> When he/she starts processing, he/she enters initially on a page where
> parameters for the processing are read. When this page is sent (after
> parameter checking, of course), processing DOES start. Then, this page
> becomes a process monitoring page. It show the parameters submitted for
> the processing and the processing status (say how many records have been
> read and generated for instance). This processing status page is
> self-refreshing. Also, user has the option to discontinue (kill) the
> process too, if the programmer offered that.
>
> When the processing is OK, this pages offer a link to the results and do
> not refreshes itself anymore.
>
> That other initial link (last processing results) points to the same
> place of this later one.
>
> So, user may start processing and left browser opened up to the time
> when it ends or, he/she may start it and go away. Later on, he/she may
> logon again to see how are the things.
>
> Another very important thing : usually these processing generate huge
> amount of data. DO NOT open it up on a single browser window ! You may
> flood the client memory !
>
> We split the result into several pages. So he/she can FF REW across the
> results where he/she has say 50 records per page (on a html table).
>
> But we also offer a *FTP* link for the entire results. These entire
> results are available as TXT files or CSV files. CSV is an easy way to
> generate simple plain text files that can be directly opened on Excel.
>
> This is the general idea.
>
> More specific issues :
>
> In our case, the batch processing is done on a Unisys mainframe. So, the
> batch processing runs in there. The script is communicating on-the-fly
> with that batch processing.
>
> When mainframe generates the output file, it ftp it out to the machine
> where the script is. This machine will make copies of the plain txt file
> that came from the mainframe transforming it into those CSV files
> (adding the separating commas) and also, transforming (stuffing) it into
> DBF files. Cause DBF files will be used for the on-line viewing of the
> pages.
>
> Each user has a folder where these files of his own are generated. Each
> user folder has also several subfolders, one for each type of processing
> (each user may have N different batch processing types). So each type
> has its own folder, to preserve it's processing results.
>
> We clean up these folders for files older than 7 days. So user has the
> results available up to 7 days after end processing.
>
>
> Hope to have offered some help to you !
>
> See ya !
>
>
> --
> Obrigado,
>
> ------------------------------------------------------------------------------
>                 Jose Adriano Baltieri - Analista de Sistemas
>                 DTI - CENTRO - UNIMEP - Universidade Metodista de
> Piracicaba
>                 PIRACICABA - SP - Brasil - Fone : (19) 3124-1858
>
> ------------------------------------------------------------------------------
>
> ___________________________________
> fastcgi-developers mailing list
> http://fastcgi.com/fastcgi-developers/
>

___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/