Re: -warmup
Ahmet Göksu <[email protected]>
| Newsgroups | gmane.comp.fonts.freetype.devel |
|---|---|
| Message-ID | <99188e53-a250-4890-8f62-5638a4b25b31@Spark> |
Hi! I changed code to warmup with number of iterations. > What exactly means 'Baseline (ms)'? Is the shown number the time > for one loop? For all loops together? Please clarify and mention > this on the HTML page. Clarified that the times are milliseconds for the cumulative time for all iterations. > There seems to be a fundamental math problem in calculating the > percentage numbers. For example, looking at the 'TOTAL' field, the > percental difference between 2.788 and 2.740 is not -6.1% but -1.7%! it was average of the all percentages but you are right. I have changed it percentage of total time changes. > Looking at the 'Load_Advances (Unscaled)' row, I think that 100% > difference between 0.001 and 0.002 doesn't make any sense. How do > you compute the percentage? Is this based on the cumulative time of > all loops? If so, and you really get such small numbers, there must > be some fine-tuning for high-speed tests (for example, increasing N > for this particular test by a factor of 10, say) to get meaningful > timing values. it was cumulative time in milliseconds but converted it microseconds as how it was and it seem got better. If any fine-tuning needed since now, i will. Looking for reply. Best, Goksu goksu.in On 3 Aug 2023 19:50 +0300, Werner LEMBERG <[email protected]>, wrote: > > > It is warming up as the given number of seconds with -w flag before > > every benchmark test. > > > > There are still differences like 100%.. Also, 1 sec warmup means > > (test count)*(font count) 70 secs for the results. > > Mhmm, I'm not sure whether a warmup *time span* makes sense. I would > rather have thought that every test would get a certain number of > warmup *loops*. For example, '--warmup 100' means that for a value of > N=50000, the first 100 loops of each test are not taken into account > for timing so that effects of the various processor and memory caches, > the operating system's memory page swapping, etc., etc., doesn't have > too much influence. This should be just a very small fraction of > time, not 70s. > > > I am thinking of what else can be done and waiting for your test. > > Just looking at your most recent HTML page I see some peculiarities. > > * What exactly means 'Baseline (ms)'? Is the shown number the time > for one loop? For all loops together? Please clarify and mention > this on the HTML page. > > * There seems to be a fundamental math problem in calculating the > percentage numbers. For example, looking at the 'TOTAL' field, the > percental difference between 2.788 and 2.740 is not -6.1% but -1.7%! > What am I missing? > > * Looking at the 'Load_Advances (Unscaled)' row, I think that 100% > difference between 0.001 and 0.002 doesn't make any sense. How do > you compute the percentage? Is this based on the cumulative time of > all loops? If so, and you really get such small numbers, there must > be some fine-tuning for high-speed tests (for example, increasing N > for this particular test by a factor of 10, say) to get meaningful > timing values. > > > Werner
benchmark.html
(text/html, 14.8 KB)
<html>
<head>
<style>
table {
}
th, td {
padding: 3px;
text-align: center;
}
th {
background-color: #ccc;
color: black;
}
.warning{
color: red;
}
.col1 {
background-color: #eee;
}
.highlight {
background-color: #0a0;
}
</style>
</head>
<body>
<h1>Freetype Benchmark Results</h1>
<h2 class="warning">Warning: Baseline and Benchmark have the same commit ID!</h2>
<h2>Info</h2>
<table border="1">
<tr><th>Info</th><th>Baseline</th><th>Benchmark</th></tr>
<tr><td class="col1">Parameters</td><td>-c 1000 -w 100</td><td>-c 1000 -w 100</td></tr>
<tr><td class="col1">Commit ID</td><td><a href="https://gitlab.freedesktop.org/freetype/freetype/-/commit/d7371720b0e4947b07a8c551071ba6410b623468">d7371720</a></td><td><a href="https://gitlab.freedesktop.org/freetype/freetype/-/commit/d7371720b0e4947b07a8c551071ba6410b623468">d7371720</a></td></tr>
<tr><td class="col1">Commit Date</td><td>2023-08-03 19:08:57 +0300</td><td>2023-08-03 19:08:57 +0300</td></tr>
<tr><td class="col1">Branch</td><td>GSoC-2023-Ahmet</td><td>GSoC-2023-Ahmet</td></tr>
</table><br/>* Cumulative time for iterations which is better in smaller values<br/>
<h2>Results for Roboto_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Roboto_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Roboto_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>120000</td> <td class="highlight">544769</td><td class="lowlight">548050</td><td>-0.6</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>120000</td> <td class="highlight">472392</td><td class="lowlight">483467</td><td>-2.3</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>120000</td> <td class="lowlight">2818</td><td class="highlight">2804</td><td>0.5</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>120000</td> <td class="highlight">2774</td><td class="lowlight">2875</td><td>-3.6</td></tr>
<tr><td class="col1">Render</td><td>120000</td> <td class="highlight">407268</td><td class="lowlight">425227</td><td>-4.4</td></tr>
<tr><td class="col1">Get_Glyph</td><td>120000</td> <td class="highlight">160786</td><td class="lowlight">166644</td><td>-3.6</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="lowlight">2728</td><td class="highlight">2318</td><td>15.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="lowlight">1771</td><td class="highlight">1718</td><td>3.0</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="lowlight">39404</td><td class="highlight">39015</td><td>1.0</td></tr>
<tr><td class="col1">Embolden</td><td>120000</td> <td class="lowlight">214085</td><td class="highlight">209987</td><td>1.9</td></tr>
<tr><td class="col1">Stroke</td><td>68160 | 65880</td> <td class="lowlight">1622171</td><td class="highlight">1618429</td><td>0.2</td></tr>
<tr><td class="col1">Get_BBox</td><td>120000</td> <td class="highlight">101134</td><td class="lowlight">101693</td><td>-0.6</td></tr>
<tr><td class="col1">Get_CBox</td><td>120000</td> <td class="lowlight">81055</td><td class="highlight">79277</td><td>2.2</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>120000</td> <td class="highlight">97837</td><td class="lowlight">100719</td><td>-2.9</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">2726040</td> <td class="col1">3750992</td><td class="col1">3782223</td><td class="col1">0.8</td></tr>
</table><br/>
<h2>Results for Arial_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Arial_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Arial_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="highlight">696891</td><td class="lowlight">751976</td><td>-7.9</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="highlight">614680</td><td class="lowlight">740438</td><td>-20.5</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="highlight">2292</td><td class="lowlight">2519</td><td>-9.9</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="highlight">2168</td><td class="lowlight">2516</td><td>-16.1</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="lowlight">333506</td><td class="highlight">325103</td><td>2.5</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="lowlight">142596</td><td class="highlight">137808</td><td>3.4</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="highlight">2289</td><td class="lowlight">3078</td><td>-34.5</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="lowlight">1875</td><td class="highlight">1751</td><td>6.6</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="lowlight">49544</td><td class="highlight">46965</td><td>5.2</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="lowlight">192603</td><td class="highlight">187243</td><td>2.8</td></tr>
<tr><td class="col1">Stroke</td><td>61750 | 60610</td> <td class="lowlight">1480048</td><td class="highlight">1471704</td><td>0.6</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="lowlight">92079</td><td class="highlight">88674</td><td>3.7</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="highlight">66369</td><td class="lowlight">66422</td><td>-0.1</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="lowlight">106669</td><td class="highlight">100515</td><td>5.8</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">2214360</td> <td class="col1">3783609</td><td class="col1">3926712</td><td class="col1">3.8</td></tr>
</table><br/>
<h2>Results for TimesNewRoman_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/TimesNewRoman_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/TimesNewRoman_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="highlight">849280</td><td class="lowlight">869354</td><td>-2.4</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="lowlight">782796</td><td class="highlight">759969</td><td>2.9</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="lowlight">2326</td><td class="highlight">2315</td><td>0.5</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="highlight">2147</td><td class="lowlight">2181</td><td>-1.6</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="highlight">355258</td><td class="lowlight">372212</td><td>-4.8</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="highlight">142597</td><td class="lowlight">145345</td><td>-1.9</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="lowlight">2515</td><td class="highlight">2298</td><td>8.6</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="lowlight">1757</td><td class="highlight">1724</td><td>1.9</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="lowlight">49602</td><td class="highlight">49108</td><td>1.0</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="lowlight">256686</td><td class="highlight">255087</td><td>0.6</td></tr>
<tr><td class="col1">Stroke</td><td>49305 | 48640</td> <td class="lowlight">1544194</td><td class="highlight">1540442</td><td>0.2</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="highlight">105618</td><td class="lowlight">108203</td><td>-2.4</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="highlight">67453</td><td class="lowlight">70368</td><td>-4.3</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="lowlight">126134</td><td class="highlight">124325</td><td>1.4</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">2189945</td> <td class="col1">4288363</td><td class="col1">4302931</td><td class="col1">0.3</td></tr>
</table><br/>
<h2>Results for Tahoma_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Tahoma_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Tahoma_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="highlight">525068</td><td class="lowlight">575170</td><td>-9.5</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="highlight">447570</td><td class="lowlight">455850</td><td>-1.8</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="lowlight">2323</td><td class="highlight">2268</td><td>2.4</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="lowlight">2239</td><td class="highlight">2169</td><td>3.1</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="highlight">311839</td><td class="lowlight">312159</td><td>-0.1</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="highlight">135936</td><td class="lowlight">137695</td><td>-1.3</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="highlight">2265</td><td class="lowlight">2347</td><td>-3.6</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="highlight">1721</td><td class="lowlight">1736</td><td>-0.9</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="highlight">44266</td><td class="lowlight">45632</td><td>-3.1</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="lowlight">202235</td><td class="highlight">196470</td><td>2.9</td></tr>
<tr><td class="col1">Stroke</td><td>69160 | 67450</td> <td class="lowlight">1566809</td><td class="highlight">1555649</td><td>0.7</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="lowlight">85721</td><td class="highlight">85337</td><td>0.4</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="lowlight">63373</td><td class="highlight">62761</td><td>1.0</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="lowlight">96532</td><td class="highlight">94596</td><td>2.0</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">2228610</td> <td class="col1">3487897</td><td class="col1">3529839</td><td class="col1">1.2</td></tr>
</table><br/>
<h2>Results for Verdana_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>N</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/baseline/Verdana_subset.txt">Baseline</a> (µs)</th> <th>* <a href="/Users/gks/Desktop/freetype/objs/benchmark/Verdana_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="highlight">511496</td><td class="lowlight">544501</td><td>-6.5</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="highlight">417777</td><td class="lowlight">433294</td><td>-3.7</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="lowlight">2352</td><td class="highlight">2244</td><td>4.6</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="highlight">2148</td><td class="lowlight">2194</td><td>-2.1</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="lowlight">316300</td><td class="highlight">312850</td><td>1.1</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="lowlight">142782</td><td class="highlight">135812</td><td>4.9</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="highlight">2274</td><td class="lowlight">2411</td><td>-6.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="highlight">1724</td><td class="lowlight">1743</td><td>-1.1</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="highlight">42758</td><td class="lowlight">43630</td><td>-2.0</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="lowlight">191345</td><td class="highlight">186361</td><td>2.6</td></tr>
<tr><td class="col1">Stroke</td><td>69160 | 70585</td> <td class="highlight">1567084</td><td class="lowlight">1568601</td><td>-0.1</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="lowlight">86768</td><td class="highlight">83590</td><td>3.7</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="highlight">63951</td><td class="lowlight">64501</td><td>-0.9</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="lowlight">95327</td><td class="highlight">92563</td><td>2.9</td></tr>
<tr><td class="col1">TOTAL</td><td class="col1">2231745</td> <td class="col1">3444086</td><td class="col1">3474295</td><td class="col1">0.9</td></tr>
</table><br/>
<center>Freetype Benchmark</center>
</body>
</html>