Re: How does POE speed up operation in reality?

Hello,

On Sun, May 3, 2009 at 1:27 AM, Rocco Caputo <[email protected]> wrote:
> On May 2, 2009, at 05:37, Roy M. wrote:
> Time constraint = 1 hour.
> 50 servers * 300 files = 15,000 files.
> 15,000 files * 5 minutes = 75,000 minutes.
>

sorry for the misleading:

I mean to download 300 files from a single ftp server is less than a
few minutes, e.g. 5 minutes for safe. (files are just very small, most
are a few KB, e.g. <10KB)

so currently I already have a sequential Net::FTP perl script which
can handle the job for ONE ftp server every hours very nicely.


The problem arise when I add more ftp servers to handle by the script,
e.g. 50 ftp servers x 5 minutes = 250 mins =  4 hours (worst case)

total bandwidth needed for all ftp sessions = 10KB x 300 files x 50 = 146MB

as you can see, bandwidth (giga), disk, cpu should not be a problem to
handle 146MB of data in 1 hour of time.


so I am thinking 2 solutions:

1. Use thread (e.g. 8 concurrent process as my CPU is quad core SMP)
to parallel the worker to ftp servers.
2. Investigate other solutions such as POE and that's why I am asking here:

So far the POE::FTP components only allow me to connect to one ftp
server at a time, how can I get the same/similar speed as 8 concurrent
process?).

In fact, this is a quite common multiple-producer and
multiple-consumer pattern in muilt-threading theory. But so far the
POE example only allow me to do thing in a event-driven way, but not a
way to handle this  pattern, or something I have missed?



Thanks.