Re: Regarding my old mmap patch

On 9/5/06, Charlie Reitzel <[email protected]> wrote:
> At 04:03 AM 9/6/2006 +0200, Bjoern Hoehrmann wrote:
> >I certainly do use tidy regularily on > 1MB files.
>
> I would think of 1MB as large wrt tidy.  One might describe anything a user
> has to wait more that 1 second for might qualify as "large".  Ergonomic
> studies show that wait times > 1.5 seconds let your mind wander and people
> notice the response time.  You want to allow time for other work besides tidy.
>
> At 07:07 PM 9/5/2006 -0700, Cory Nelson wrote:
> >The performance boost given by mmapping will be there regardless of file size.
>
> OK, but this raises the larger question of how does tidy fit into your
> processing flow?  Is it a bottleneck (or even a runner up)?  Are live
> people waiting on the response for large files?  I guess I continue to be
> surprised that Tidy is anyone's performance bottleneck.  I know certain
> bulk web scraping projects (thinking of Mark Weaver) might  have this
> issue, but these are few and far between (or so I thought).
>
> I'm not trying to be a pain.  I just want to justify the complexity.
>

Maximum performance is certainly un_need_ed for most, but a 40%
increase with such little code is nothing to sneeze at.  When 40% of
the *cpu* usage (forgetting i/o usage) of the process of
parsing+tidying+pretty printing is dedicated only to the file i/o of
parsing, I think a patch like this becomes even more significant.

An example: I know this is a minority situation, but, a past contract
of mine involved getting malformed xml from a certain company, about
20mb of it every couple minutes.  I had a script running it through
Tidy before processing it, and Tidy took up roughly 1/3rd of the time
of the operation.  Other processes (one of which was already I/O
bound) were getting their time cut into by Tidy, and halving that time
would have been very welcome!

>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Tidy-develop mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tidy-develop
>


-- 
Cory Nelson
http://www.int64.org

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642