Re: Most time spent on POE/Loop/Select.pm?

Ryan Chan wrote:
> I am profiling my POE program which fetch HTML pages from remote site.
>
> The result show that 80% of time are spent on this file "POE/Loop/Select.pm"
>
> Is it normal and expected behavior?
>   
Hello,

    Without knowing what kind of profiling you are doing ( Devel::DProf? 
NYTProf? ) and the code you are writing, it's hard to answer your 
question. However, you are in luck as I'm the author of 
POE::Devel::Benchmarker and has been benchmarking POE on and off over 
the years. What you are seeing is very normal - let me explain a little 
why it's happening :)

    Your app fetches HTML pages, right? So you are presumably using a 
HTTP client - and it's making a lot of network requests, right? What's 
happening here is your app spends a lot of time waiting for network 
input before it can process them. All this waiting takes up wallclock 
time, and that's what your profiler is seeing. The 80% number you see 
does not mean the loop is crunching away at the CPU! ( obviously, a 
busy-wait loop is a BAD idea, ha! )

    If you wanted to reduce the time spent waiting, then you need to 
increase the number of parallel HTTP requests. That way, your app spends 
more of it's time processing the HTML instead of waiting for network 
input. There's plenty of methods to achieve that, and you can ask on 
this list or browse the CPAN modules to do it.

    Tip: The POE developers have spent years optimizing POE the best 
they can. You can safely ignore the core POE modules from your profiling 
( POE::Loop::Select, POE::Kernel, etc ) and focus on your own code when 
profiling. Also, there are numerous Loop adapters ( POE::Loop::IO_Poll, 
POE::XS::Loop::EPoll, etc ) that could take advantage of your platform 
or workload to reduce the overhead. As always, benchmark/test your code 
against the various loops to see if one of them is best for your 
workload. As taken from the POE::Kernel pod: "By default POE uses its 
select() based loop to drive its event system. This is perhaps the least 
efficient loop, but it is also the most portable. POE optimizes for 
correctness above all."

    Have fun coding POE apps!

~Apocalypse