Re: Regarding my old mmap patch

Thanks for sharing your experience.  Now that you mention it, I recently 
talked to someone who had to do a lot of parsing work because an online 
trading partner was submitting malformed XML.  Sad, but true.  Anyway, I'm 
convinced.

Can you guys figure out a combined patch so most platforms can benefit?

Arnaud, are you on board?


At 10:57 PM 9/5/2006 -0700, Cory Nelson wrote:
>On 9/5/06, Charlie Reitzel <[email protected]> wrote:
> > At 04:03 AM 9/6/2006 +0200, Bjoern Hoehrmann wrote:
> > >I certainly do use tidy regularily on > 1MB files.
> >
> > I would think of 1MB as large wrt tidy.  One might describe anything a user
> > has to wait more that 1 second for might qualify as "large".  Ergonomic
> > studies show that wait times > 1.5 seconds let your mind wander and people
> > notice the response time.  You want to allow time for other work 
> besides tidy.
> >
> > At 07:07 PM 9/5/2006 -0700, Cory Nelson wrote:
> > >The performance boost given by mmapping will be there regardless of 
> file size.
> >
> > OK, but this raises the larger question of how does tidy fit into your
> > processing flow?  Is it a bottleneck (or even a runner up)?  Are live
> > people waiting on the response for large files?  I guess I continue to be
> > surprised that Tidy is anyone's performance bottleneck.  I know certain
> > bulk web scraping projects (thinking of Mark Weaver) might  have this
> > issue, but these are few and far between (or so I thought).
> >
> > I'm not trying to be a pain.  I just want to justify the complexity.
> >
>
>Maximum performance is certainly un_need_ed for most, but a 40%
>increase with such little code is nothing to sneeze at.  When 40% of
>the *cpu* usage (forgetting i/o usage) of the process of
>parsing+tidying+pretty printing is dedicated only to the file i/o of
>parsing, I think a patch like this becomes even more significant.

Nit pick: no I/O is actually saved.  Tidy still needs read the entire file 
from disk.  Using a Memory Mapped file is essentially a neat hack to reuse 
the OS implementation of asynchronous I/O (which it uses internally for 
binary images).  "Neat hack" is, of course, a compliment.

>An example: I know this is a minority situation, but, a past contract
>of mine involved getting malformed xml from a certain company, about
>20mb of it every couple minutes.  I had a script running it through
>Tidy before processing it, and Tidy took up roughly 1/3rd of the time
>of the operation.  Other processes (one of which was already I/O
>bound) were getting their time cut into by Tidy, and halving that time
>would have been very welcome!

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642