Re: -warmup
Ahmet Göksu <[email protected]>
| Newsgroups | gmane.comp.fonts.freetype.devel |
|---|---|
| Message-ID | <457dddf3-dc22-4581-8a5e-3c9fd7ff6b6c@Spark> |
> What happens if you use, say, `-c 100000', just running the > `Get_Char_Index` test? Are the percental timing differences then > still that large? Actually Get_Char_Index, on the three pages I have sent in the prev. mail, is higher than 6% only 4 times out of 15 total. (which is seem on other tests as well). > Why do you think so? Please explain your reasoning. Just remember > that backup processes (like cleaning up the hard disk, running some > cron jobs, etc.) can pop up anytime, thus influencing the result. > Such spontaneous events have to be eliminated. yes, right, didn't thought of the spontaneous events. about outliers, i splitted every tests into chuncks that is sized 100. Made IQR calculations and calculated average time on valid chunks. you can find the result in the attachment also pushed to gitlab. also, since statistics and benchmarking are a sciences their self, i am a bit struggling while approaching the problem as well as feels like out of the gsoc project scope. I would like to share this with your indulgence. yet, of course I will move in accordance with your instructions. Best, Goksu goksu.in On 18 Aug 2023 00:02 +0300, Werner LEMBERG <[email protected]>, wrote: > > > I have added the total table that you suggested. > > Thanks. > > > I think Get_Char_Index is not the problem, the results varies all > > the time. > > As far as I can see, there is a direct relationship between the total > cumulated time of a test and the timing variation: The smaller the > cumulated time, the larger the variation. > > What happens if you use, say, `-c 100000', just running the > `Get_Char_Index` test? Are the percental timing differences then > still that large? > > > Should I proceed to detect outliers? Since we do not get the same > > error rate consistently, I think we will not find the target we > > expected by outliers. > > Why do you think so? Please explain your reasoning. Just remember > that backup processes (like cleaning up the hard disk, running some > cron jobs, etc.) can pop up anytime, thus influencing the result. > Such spontaneous events have to be eliminated. > > Have you actually tried something along the method I suggested? > > > Werner
benchmark.html
(text/html, 17 KB)
<html>
<head>
<style>
table {
}
th, td {
padding: 3px;
text-align: center;
}
th {
background-color: #ccc;
color: black;
}
.warning{
color: red;
}
.col1 {
background-color: #eee;
}
.highlight {
background-color: #0a0;
}
</style>
</head>
<body>
<h1>Freetype Benchmark Results</h1>
<h2 class="warning">Warning: Baseline and Benchmark have the same commit ID!</h2>
<h2>Info</h2>
<table border="1">
<tr><th>Info</th><th>Baseline</th><th>Benchmark</th></tr>
<tr><td class="col1">Parameters</td><td>-c 550 -w 50</td><td>-c 550 -w 50</td></tr>
<tr><td class="col1">Commit ID</td><td><a href="https://gitlab.freedesktop.org/freetype/freetype/-/commit/3553148174ceb9c01eec1ca660aa77d5f0bd0b0e">35531481</a></td><td><a href="https://gitlab.freedesktop.org/freetype/freetype/-/commit/3553148174ceb9c01eec1ca660aa77d5f0bd0b0e">35531481</a></td></tr>
<tr><td class="col1">Commit Date</td><td>2023-08-18 02:04:38 +0300</td><td>2023-08-18 02:04:38 +0300</td></tr>
<tr><td class="col1">Branch</td><td>GSoC-2023-Ahmet</td><td>GSoC-2023-Ahmet</td></tr>
</table><br/><p>* Average time for all iterations. Smaller values are better.</p><p>** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.</p><h2>Total Results</h2>
<table border="1">
<tr><th>Test</th><th>N</th><th>Baseline (µs)</th> <th>Benchmark (µs)</th><th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>50000</td> <td class="">402930</td> <td class="highlight">402170</td><td>0.2</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>50000</td> <td class="highlight">327190</td> <td class="">357270</td><td>-9.2</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>50000</td> <td class="highlight">1388</td> <td class="">1554</td><td>-12.0</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>50000</td> <td class="">1534</td> <td class="highlight">1398</td><td>8.9</td></tr>
<tr><td class="col1">Render</td><td>50000</td> <td class="highlight">199593</td> <td class="">213029</td><td>-6.7</td></tr>
<tr><td class="col1">Get_Glyph</td><td>50000</td> <td class="highlight">85044</td> <td class="">88477</td><td>-4.0</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>47000</td> <td class="highlight">1316</td> <td class="">1365</td><td>-3.8</td></tr>
<tr><td class="col1">Iterate CMap</td><td>500</td> <td class="highlight">1023</td> <td class="">1039</td><td>-1.6</td></tr>
<tr><td class="col1">New_Face</td><td>500</td> <td class="highlight">25827</td> <td class="">28613</td><td>-10.8</td></tr>
<tr><td class="col1">Embolden</td><td>50000</td> <td class="highlight">120237</td> <td class="">125197</td><td>-4.1</td></tr>
<tr><td class="col1">Stroke</td><td>50000</td> <td class="highlight">1549165</td> <td class="">1561868</td><td>-0.8</td></tr>
<tr><td class="col1">Get_BBox</td><td>50000</td> <td class="highlight">53948</td> <td class="">54843</td><td>-1.7</td></tr>
<tr><td class="col1">Get_CBox</td><td>50000</td> <td class="highlight">38993</td> <td class="">39678</td><td>-1.8</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>50000</td> <td class="highlight">61266</td> <td class="">63766</td><td>-4.1</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">598000</td> <td class="col1">2869453</td><td class="col1">2940267</td> <td class="col1">-2.5</td></tr>
</table><br/>
<h3>Results for Roboto_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Roboto_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Roboto_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>12000</td> <td class="lowlight">69078</td><td class="highlight">67847</td><td>1.8</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>12000</td> <td class="highlight">56817</td><td class="lowlight">64449</td><td>-13.4</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>12000</td> <td class="highlight">311</td><td class="lowlight">358</td><td>-15.4</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>12000</td> <td class="highlight">290</td><td class="lowlight">340</td><td>-17.4</td></tr>
<tr><td class="col1">Render</td><td>12000</td> <td class="highlight">48146</td><td class="lowlight">51436</td><td>-6.8</td></tr>
<tr><td class="col1">Get_Glyph</td><td>12000</td> <td class="highlight">19760</td><td class="lowlight">20782</td><td>-5.2</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>9400</td> <td class="highlight">284</td><td class="lowlight">306</td><td>-7.9</td></tr>
<tr><td class="col1">Iterate CMap</td><td>100</td> <td class="lowlight">240</td><td class="highlight">232</td><td>3.5</td></tr>
<tr><td class="col1">New_Face</td><td>100</td> <td class="highlight">4514</td><td class="lowlight">6317</td><td>-40.0</td></tr>
<tr><td class="col1">Embolden</td><td>12000</td> <td class="highlight">24713</td><td class="lowlight">28423</td><td>-15.0</td></tr>
<tr><td class="col1">Stroke</td><td>12000</td> <td class="lowlight">363268</td><td class="highlight">352657</td><td>2.9</td></tr>
<tr><td class="col1">Get_BBox</td><td>12000</td> <td class="lowlight">11871</td><td class="highlight">11488</td><td>3.2</td></tr>
<tr><td class="col1">Get_CBox</td><td>12000</td> <td class="lowlight">9476</td><td class="highlight">8818</td><td>6.9</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>12000</td> <td class="lowlight">11584</td><td class="highlight">11097</td><td>4.2</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">283200</td> <td class="col1">620350</td><td class="col1">624551</td><td class="col1">0.7</td></tr>
</table><br/>
<h3>Results for Arial_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Arial_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Arial_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>9500</td> <td class="highlight">92134</td><td class="lowlight">94812</td><td>-2.9</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>9500</td> <td class="highlight">77163</td><td class="lowlight">90204</td><td>-16.9</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>9500</td> <td class="highlight">285</td><td class="lowlight">370</td><td>-29.9</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>9500</td> <td class="lowlight">333</td><td class="highlight">291</td><td>12.8</td></tr>
<tr><td class="col1">Render</td><td>9500</td> <td class="highlight">40990</td><td class="lowlight">43614</td><td>-6.4</td></tr>
<tr><td class="col1">Get_Glyph</td><td>9500</td> <td class="highlight">17531</td><td class="lowlight">17640</td><td>-0.6</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>9400</td> <td class="lowlight">262</td><td class="highlight">249</td><td>4.8</td></tr>
<tr><td class="col1">Iterate CMap</td><td>100</td> <td class="lowlight">199</td><td class="highlight">189</td><td>5.1</td></tr>
<tr><td class="col1">New_Face</td><td>100</td> <td class="highlight">5261</td><td class="lowlight">5432</td><td>-3.2</td></tr>
<tr><td class="col1">Embolden</td><td>9500</td> <td class="lowlight">22123</td><td class="highlight">21703</td><td>1.9</td></tr>
<tr><td class="col1">Stroke</td><td>9500</td> <td class="highlight">294112</td><td class="lowlight">295591</td><td>-0.5</td></tr>
<tr><td class="col1">Get_BBox</td><td>9500</td> <td class="highlight">11297</td><td class="lowlight">11569</td><td>-2.4</td></tr>
<tr><td class="col1">Get_CBox</td><td>9500</td> <td class="highlight">7673</td><td class="lowlight">8519</td><td>-11.0</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>9500</td> <td class="highlight">13120</td><td class="lowlight">14388</td><td>-9.7</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">228200</td> <td class="col1">582485</td><td class="col1">604572</td><td class="col1">3.8</td></tr>
</table><br/>
<h3>Results for TimesNewRoman_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/TimesNewRoman_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/TimesNewRoman_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>9500</td> <td class="lowlight">112074</td><td class="highlight">108096</td><td>3.5</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>9500</td> <td class="lowlight">84173</td><td class="highlight">83690</td><td>0.6</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>9500</td> <td class="highlight">258</td><td class="lowlight">260</td><td>-0.9</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>9500</td> <td class="lowlight">409</td><td class="highlight">241</td><td>41.0</td></tr>
<tr><td class="col1">Render</td><td>9500</td> <td class="highlight">39939</td><td class="lowlight">40323</td><td>-1.0</td></tr>
<tr><td class="col1">Get_Glyph</td><td>9500</td> <td class="highlight">15357</td><td class="lowlight">15654</td><td>-1.9</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>9400</td> <td class="highlight">254</td><td class="lowlight">259</td><td>-2.1</td></tr>
<tr><td class="col1">Iterate CMap</td><td>100</td> <td class="lowlight">183</td><td class="highlight">183</td><td>0.1</td></tr>
<tr><td class="col1">New_Face</td><td>100</td> <td class="lowlight">5982</td><td class="highlight">5418</td><td>9.4</td></tr>
<tr><td class="col1">Embolden</td><td>9500</td> <td class="highlight">28093</td><td class="lowlight">28294</td><td>-0.7</td></tr>
<tr><td class="col1">Stroke</td><td>9500</td> <td class="lowlight">375318</td><td class="highlight">373881</td><td>0.4</td></tr>
<tr><td class="col1">Get_BBox</td><td>9500</td> <td class="highlight">11941</td><td class="lowlight">12127</td><td>-1.6</td></tr>
<tr><td class="col1">Get_CBox</td><td>9500</td> <td class="highlight">7505</td><td class="lowlight">7736</td><td>-3.1</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>9500</td> <td class="lowlight">14238</td><td class="highlight">13969</td><td>1.9</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">228200</td> <td class="col1">695723</td><td class="col1">690131</td><td class="col1">-0.8</td></tr>
</table><br/>
<h3>Results for Tahoma_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Tahoma_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Tahoma_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>9500</td> <td class="highlight">67346</td><td class="lowlight">67455</td><td>-0.2</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>9500</td> <td class="lowlight">58555</td><td class="highlight">58316</td><td>0.4</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>9500</td> <td class="highlight">255</td><td class="lowlight">258</td><td>-1.2</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>9500</td> <td class="lowlight">244</td><td class="highlight">238</td><td>2.3</td></tr>
<tr><td class="col1">Render</td><td>9500</td> <td class="lowlight">35180</td><td class="highlight">34455</td><td>2.1</td></tr>
<tr><td class="col1">Get_Glyph</td><td>9500</td> <td class="lowlight">15612</td><td class="highlight">15512</td><td>0.6</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>9400</td> <td class="highlight">251</td><td class="lowlight">252</td><td>-0.3</td></tr>
<tr><td class="col1">Iterate CMap</td><td>100</td> <td class="lowlight">200</td><td class="highlight">194</td><td>2.6</td></tr>
<tr><td class="col1">New_Face</td><td>100</td> <td class="highlight">4938</td><td class="lowlight">5048</td><td>-2.2</td></tr>
<tr><td class="col1">Embolden</td><td>9500</td> <td class="lowlight">22558</td><td class="highlight">22027</td><td>2.4</td></tr>
<tr><td class="col1">Stroke</td><td>9500</td> <td class="lowlight">254696</td><td class="highlight">254466</td><td>0.1</td></tr>
<tr><td class="col1">Get_BBox</td><td>9500</td> <td class="highlight">9621</td><td class="lowlight">9704</td><td>-0.9</td></tr>
<tr><td class="col1">Get_CBox</td><td>9500</td> <td class="lowlight">7220</td><td class="highlight">6952</td><td>3.7</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>9500</td> <td class="lowlight">11067</td><td class="highlight">10561</td><td>4.6</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">228200</td> <td class="col1">487743</td><td class="col1">485439</td><td class="col1">-0.5</td></tr>
</table><br/>
<h3>Results for Verdana_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Verdana_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Verdana_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>9500</td> <td class="highlight">62299</td><td class="lowlight">63960</td><td>-2.7</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>9500</td> <td class="highlight">50481</td><td class="lowlight">60612</td><td>-20.1</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>9500</td> <td class="highlight">279</td><td class="lowlight">307</td><td>-10.2</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>9500</td> <td class="highlight">259</td><td class="lowlight">288</td><td>-11.2</td></tr>
<tr><td class="col1">Render</td><td>9500</td> <td class="highlight">35338</td><td class="lowlight">43200</td><td>-22.2</td></tr>
<tr><td class="col1">Get_Glyph</td><td>9500</td> <td class="highlight">16784</td><td class="lowlight">18888</td><td>-12.5</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>9400</td> <td class="highlight">265</td><td class="lowlight">299</td><td>-12.7</td></tr>
<tr><td class="col1">Iterate CMap</td><td>100</td> <td class="highlight">200</td><td class="lowlight">241</td><td>-20.0</td></tr>
<tr><td class="col1">New_Face</td><td>100</td> <td class="highlight">5133</td><td class="lowlight">6398</td><td>-24.7</td></tr>
<tr><td class="col1">Embolden</td><td>9500</td> <td class="highlight">22749</td><td class="lowlight">24750</td><td>-8.8</td></tr>
<tr><td class="col1">Stroke</td><td>9500</td> <td class="highlight">261771</td><td class="lowlight">285272</td><td>-9.0</td></tr>
<tr><td class="col1">Get_BBox</td><td>9500</td> <td class="highlight">9218</td><td class="lowlight">9954</td><td>-8.0</td></tr>
<tr><td class="col1">Get_CBox</td><td>9500</td> <td class="highlight">7119</td><td class="lowlight">7653</td><td>-7.5</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>9500</td> <td class="highlight">11258</td><td class="lowlight">13751</td><td>-22.1</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">228200</td> <td class="col1">483152</td><td class="col1">535574</td><td class="col1">10.8</td></tr>
</table><br/>
<center>Freetype Benchmark</center>
</body>
</html>