Re: Very long operations with FastCGI

Mitch Jackson wrote:

> I have a long-running operation to spawn from a perl script within the 
> FastCGI environment.  I would like the script to cause the user's 
> browser to redirect to a 'view operation status' page, and continue to 
> do the processing of the operation in the background.  I see two 
> obvious ways to do this.  First using the HTTP header 'redirect: 
> [url]' and the other by writing a page with a meta refresh tag.
>
> Once I have written a redirect command to the user's browser, how do I 
> signal to the server and the browser that i'm done sending input, so 
> the server will flush the buffer and deliver the output, and the 
> browser will acknowledge this session has completed and do it's thing, 
> instead of waiting for more output from the web server.

Mr Jackson :

You have had good ideas about how to do it.


I'll tell you the way we do it today. You may pick some ideas from it 
too.... I bet you will find it very complicated but, we think it is 
sophisticated. It takes a good time to implement but when it's done , 
that is, when the standard building blocks are ready, you have a very 
good standard of batch processing (I think)...

So, here's how it is :

Whenever the user has a batch processing, he/she has a page with two 
links : start processing and get last processing results.

Why do we split it like that ? Because usually these batch processing 
take too long to be generated. And we like to offer he/she a link for 
it. So, he/she may start processing and logout. Processing goes on, it's 
on the background (as a matter of fact it can be even on a mainframe). 
He/She may come back later and check processing results, to see if it's 
done or not.

When he/she starts processing, he/she enters initially on a page where 
parameters for the processing are read. When this page is sent (after 
parameter checking, of course), processing DOES start. Then, this page 
becomes a process monitoring page. It show the parameters submitted for 
the processing and the processing status (say how many records have been 
read and generated for instance). This processing status page is 
self-refreshing. Also, user has the option to discontinue (kill) the 
process too, if the programmer offered that.

When the processing is OK, this pages offer a link to the results and do 
not refreshes itself anymore.

That other initial link (last processing results) points to the same 
place of this later one.

So, user may start processing and left browser opened up to the time 
when it ends or, he/she may start it and go away. Later on, he/she may 
logon again to see how are the things.

Another very important thing : usually these processing generate huge 
amount of data. DO NOT open it up on a single browser window ! You may 
flood the client memory !

We split the result into several pages. So he/she can FF REW across the 
results where he/she has say 50 records per page (on a html table).

But we also offer a *FTP* link for the entire results. These entire 
results are available as TXT files or CSV files. CSV is an easy way to 
generate simple plain text files that can be directly opened on Excel.

This is the general idea.

More specific issues :

In our case, the batch processing is done on a Unisys mainframe. So, the 
batch processing runs in there. The script is communicating on-the-fly 
with that batch processing.

When mainframe generates the output file, it ftp it out to the machine 
where the script is. This machine will make copies of the plain txt file 
that came from the mainframe transforming it into those CSV files 
(adding the separating commas) and also, transforming (stuffing) it into 
DBF files. Cause DBF files will be used for the on-line viewing of the 
pages.

Each user has a folder where these files of his own are generated. Each 
user folder has also several subfolders, one for each type of processing 
(each user may have N different batch processing types). So each type 
has its own folder, to preserve it's processing results.

We clean up these folders for files older than 7 days. So user has the 
results available up to 7 days after end processing.


Hope to have offered some help to you !

See ya !


-- 
Obrigado,
------------------------------------------------------------------------------
                Jose Adriano Baltieri - Analista de Sistemas
                DTI - CENTRO - UNIMEP - Universidade Metodista de Piracicaba
                PIRACICABA - SP - Brasil - Fone : (19) 3124-1858
------------------------------------------------------------------------------

___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/