Re: cache effectiveness testing

On 1/18/2012 3:26 PM, Jason Duell wrote:
> It's great to be getting these numbers--thanks!
>
> Just to be clear--so "Run1" is from a new cache/profile.  Is "Run2" 
> using a populated cache, or from an existing cache you've cleared?  
> I'm guessing the former from the test description page.  In that case 
> it's definitely worth noting that no-cache is about as fast (or 
> faster) then the cache-hit case for many pages (the bigger the 
> page--ex the home pages--the more likely caching seems to help).   
> We'd want to break down the page content to see what's up with that 
> (loading non-cachable or small items that need validation is probably 
> always going to be slower for a disk cache, given the lookup overhead, 
> so that's excusable.  Otherwise we have perf work to do).
>
> I'm sure you've sunk lots of time into this already, but it'd be great 
> to see Chrome numbers--they seem to be outperforming us the most in 
> the tomshardware Browser contests.
I think this test data highlights significant deficiencies in our cache 
performance that we should address asap. Chrome data would be amusing to 
look at, but I don't think it's important to get to fix our cache in the 
near term.
>
> Jason
>
>
> On 01/17/2012 04:50 PM, Anthony Hughes wrote:
>> Thanks for the additional details.
>>
>> "Firefox, no cache" was tacked on at the end. We ran the original 4 
>> testruns in order, then ran the "no cache" scenario on a new profile 
>> after the fact.
>>
>> I hope that answers your question. Let me know if not.
>>
>>
>> ----- Original Message -----
>> From: "Brian Smith"<[email protected]>
>> To: "Taras Glek"<[email protected]>
>> Cc: "Johnny Stenback"<[email protected]>, "Boris 
>> Zbarsky"<[email protected]>, "Lawrence Mandel"<[email protected]>, 
>> "Anthony Hughes"<[email protected]>, 
>> [email protected]
>> Sent: Tuesday, January 17, 2012 4:43:09 PM
>> Subject: Re: cache effectiveness testing
>>
>> Taras Glek wrote:
>>> On 1/17/2012 2:55 PM, Taras Glek wrote:
>>>>> The testplan is described at
>>>>> https://wiki.mozilla.org/Performance/Snappy/Testing:BFCache_Sprint#Reference_Test_Case 
>>>>>
>> Thank you for this.
>>
>> I would suggest the following changes to the test methodology for 
>> future tests:
>>
>> 1. Document how the disk cache is cleared.
>> 2. Have the user clear the disk cache and then reboot the machine and 
>> then run the test.
>> 3. Document which anti-virus is installed and what it's configuration 
>> is. In particular, is it set to scan when files are written? (I think 
>> that the baseline should be Microsoft Security Essentials in its 
>> default configuration, which should have such scanning enabled by 
>> default.)
>> 4. Reboot between switches in browsers. (maybe not too important)
>> 5. Add Chrome and/or IE.
>>
>> It would be great to have comparisons where MSE is enabled in its 
>> default configuration and where MSE is disabled.
>>
>>>> No-disk-cache followup appears to confirm that writing cache
>>>> absolutely annihilates pageload perf.
>>>>
>>>> http://people.mozilla.com/~tglek/snappy/cache2.html
>> Are the "Firefox, no-cache" numbers comparable to the first column 
>> (Run 1, Firefox) or the third column (Run 2, Firefox)? That is, were 
>> the "no-cache" numbers generated with an empty cache or with the 
>> cache in the state it would be in after Run 2?
>>
>> Here is the test procedure as far as I understand it:
>>
>>      Run the test for all three sites in Firefox
>>          * This is "Firefox, Run 1"
>>      Quit Firefox and start Opera
>>      Run the test for all three sites in Opera
>>          * This is "Opera, Run 1"
>>      Quit Opera and start Firefox
>>      Run the test for all three sites in Firefox
>>          * This is "Firefox, Run 2"
>>      Quit Firefox and start Opera
>>      Run the test for all three sites in Opera
>>      Quit Opera
>>
>> Thanks again,
>> Brian
>> _______________________________________________
>> dev-tech-network mailing list
>> [email protected]
>> https://lists.mozilla.org/listinfo/dev-tech-network
>