Re: ftbench: last updates
Ahmet Göksu <[email protected]>
| Newsgroups | gmane.comp.fonts.freetype.devel |
|---|---|
| Message-ID | <15610c00-150c-4741-8b44-51a3e4b31146@Spark> |
Hi, I have created a new branch. Here you can see it: https://gitlab.freedesktop.org/freetype/freetype/-/tree/gsoc-2023-ahmet-final -Made documentation and comment line (will continue). -trailing whitespaces cleared -more verbose commit messages -formatted the code. In this version of the code, i just didnt splitted iterations into chunks, got same results with it. Best, Goksu goksu.in On 18 Sep 2023 10:34 +0300, Werner LEMBERG <[email protected]>, wrote: > > Hello Ahmet, > > > > -I have changed the * and the sentence > > Thanks. Unfortunately, I was unclear that '*x*' in my e-mail is meant > as Markdown syntax and not to be taken verbatim. In other words, 'x' > should be typeset in italics, similar to a mathematical variable. > Sorry for that, and please fix. > > Something different: Please don't use `\r` in messages for newlines. > Use `\n` instead, which is standard (file `testing.mk`). > > > > I already changed the working way of the timing. I only start the > > > benchmark at beginning and stop at the end. > > > > i mean, it times chunks, not single iteration. timer starts at the > > beginning of the chunk and stop at the end (then divide the results > > size of a chunk). because of it does not time single iteration, it > > is already a bulk test. > > Ok. > > > > BTW, I suggest that you add another sentence, explaining *why* > > > there are two values at all. > > > > actually, i didnt get the reason well but it may differ even with > > same flags. i need help in this case. > > Alexei, please have a look. It seems that only(?) the 'Stroke' test > is affected... > > > the code seems producing more accurate results after splitting the > > results into chunks. are results seem satisfactory in your machine? > > Alas, they aren't. Running your code on my GNU/Linux machine, I get > percental differences up to 14%. > > BTW, I've also just compiled `valgrind` from its git repository (to be > sure that it exactly fits my 'libc' library version) and run > > ``` > valgrind --trace-children=yes make baseline > valgrind --trace-children=yes make benchmark > ``` > > expecting to get identical results, because `valgrind` uses a CPU > emulator. However, to my great surprise, they are *not* identical, > and sometimes the values differ even enormously. Has anybody an idea > why this happens? AFAICS from valgrind's source files, the > `clock_gettime` function *is* caught. > > Note that with valgrind the creation of the webpage fails; I get > > ``` > Traceback (most recent call last): > File "/home/wl/ahmet/tohtml.py", line 310, in <module> > main() > File "/home/wl/ahmet/tohtml.py", line 67, in main > generate_total_results_table(html_file, BASELINE_DIR, BENCHMARK_DIR) > File "/home/wl/ahmet/tohtml.py", line 192, in generate_total_results_table > diff = ((baseline - benchmark) / baseline) * 100 > ZeroDivisionError: float division by zero > ``` > > Attached are the two directories from valgrind runs. Comparing this > with non-valgrind values I think there is a fundamental interaction > problem with the `bench` program: It seems that `bench` controls the > number of loops *in addition* to what you specify via the command > line. Unfortunately, I don't have time right now to check this by > myself. > > Too bad that you didn't try `valgrind` by yourself earlier, as I've > asked you repeatedly to do ... > > > Also, While there are less than 10 days for final evaluation, there > > are points that are not completed: > > > > * meson > > * cmake > > * documentation > > > > because of our focus a bit changed, didnt worked on them > > much. Should I complete them all? Is there a priority? > > Please forget meson and cmake, this is not important at this stage. > What I now ask is to create a new branch, with *all* of your changes > recreated as logical and (relatively small) incremental commits, > rebased to 'master'. Ideally every commit is compilable. If not > (which should be an exception, and there should be a good explanation > for that), please add a remark in the commit message. > > * Try to adjust the formatting to be as near to the remaining code of > FreeType. > > * The commit messages should also be in FreeType style, which is much* > *more verbose than what you currently do. > > * Avoid trailing whitespace – whatever editor you use, *please* change > the settings so that this gets automatically removed. Right now, I > see zillions of trailing whitespace in file `tohtml.py`. > > * Hin-Tak is right: Documentation is of paramount importance so that > you (or someone else) can continue the work. > > Have a look at previous GSoC students' (remote) branches to see how > such a recreated branch should look like, for example > `GSoC-2019-nikhil` or `GSoC-2017-ewaldhew`. > > > Werner
benchmark.html
(text/html, 15.5 KB)
<html>
<head>
<style>
table {
table-layout: fixed;
}
th, td {
padding: 3px;
text-align: center;
}
th {
background-color: #ccc;
color: black;
}
.warning{
color: red;
}
.col1 {
background-color: #eee;
}
.highlight {
background-color: #0a0;
}
</style>
</head>
<body>
<h1>Freetype Benchmark Results</h1>
<h2 class="warning">Warning: Baseline and Benchmark have the same commit ID!</h2>
<h2>Info</h2>
<table border="1">
<tr><th>Info</th><th>Baseline</th><th>Benchmark</th></tr>
<tr><td class="col1">Parameters</td><td>-c 1000 -w 100</td><td>-c 1000 -w 100</td></tr>
<tr><td class="col1">Commit ID</td><td><a href="https://gitlab.freedesktop.org/freetype/freetype/-/commit/e61380fe6fdf5363f695b1ee153fafb85a9069e1">e61380fe</a></td><td><a href="https://gitlab.freedesktop.org/freetype/freetype/-/commit/e61380fe6fdf5363f695b1ee153fafb85a9069e1">e61380fe</a></td></tr>
<tr><td class="col1">Commit Date</td><td>2023-09-21 15:34:38 +0300</td><td>2023-09-21 15:34:38 +0300</td></tr>
<tr><td class="col1">Branch</td><td>gsoc-2023-ahmet-final</td><td>gsoc-2023-ahmet-final</td></tr>
</table><br/><p>* Average time for single iteration. Smaller values are better.</p><p>* If a value in the 'Iterations' column is given as '<i>x | y</i>', values <i>x</i> and <i>y</i> give the number of iterations in the baseline and the benchmark test, respectively.</p><h2>Total Results</h2>
<table border="1">
<tr><th>Test</th><th>Iterations</th><th>* Baseline (µs)</th> <th>* Benchmark (µs)</th><th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>100000</td> <td class="">7.2</td> <td class="highlight">7.0</td><td>2.4</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>100000</td> <td class="">6.4</td> <td class="highlight">6.2</td><td>1.7</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>100000</td> <td class="highlight">0.0</td> <td class="">0.0</td><td>-4.8</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>100000</td> <td class="highlight">0.0</td> <td class="">0.0</td><td>-0.7</td></tr>
<tr><td class="col1">Render</td><td>100000</td> <td class="">9.7</td> <td class="highlight">9.3</td><td>4.1</td></tr>
<tr><td class="col1">Get_Glyph</td><td>100000</td> <td class="">7.0</td> <td class="highlight">6.8</td><td>3.2</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="">0.0</td> <td class="highlight">0.0</td><td>3.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="">2.1</td> <td class="highlight">2.0</td><td>4.3</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="">51.8</td> <td class="highlight">49.1</td><td>5.3</td></tr>
<tr><td class="col1">Embolden</td><td>100000</td> <td class="">7.8</td> <td class="highlight">7.6</td><td>3.4</td></tr>
<tr><td class="col1">Stroke</td><td>56642 | 57701</td> <td class="">36.3</td> <td class="highlight">35.6</td><td>1.9</td></tr>
<tr><td class="col1">Get_BBox</td><td>100000</td> <td class="">5.8</td> <td class="highlight">5.7</td><td>1.3</td></tr>
<tr><td class="col1">Get_CBox</td><td>100000</td> <td class="">7.0</td> <td class="highlight">6.8</td><td>1.6</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>100000</td> <td class="">1.1</td> <td class="highlight">1.1</td><td>0.6</td></tr>
<tr><td class="col1">Total duration for all tests:</td><td class="col1" colspan="4">81 s</td></table>
<h3>Results for Roboto_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>Iterations</th> <th>* <a href="./baseline/Roboto_subset.txt">Baseline</a> (µs)</th> <th>* <a href="./benchmark/Roboto_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>120000</td> <td class="">4.9</td><td class="highlight">4.9</td><td>0.6</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>120000</td> <td class="">4.2</td><td class="highlight">4.2</td><td>0.6</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>120000</td> <td class="highlight">0.0</td><td class="">0.0</td><td>-21.4</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>120000</td> <td class="highlight">0.0</td><td class="">0.0</td><td>-7.7</td></tr>
<tr><td class="col1">Render</td><td>120000</td> <td class="">7.6</td><td class="highlight">7.4</td><td>2.9</td></tr>
<tr><td class="col1">Get_Glyph</td><td>120000</td> <td class="highlight">4.9</td><td class="">4.9</td><td>-0.2</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="">2.0</td><td class="highlight">2.0</td><td>0.3</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="">43.5</td><td class="highlight">42.8</td><td>1.5</td></tr>
<tr><td class="col1">Embolden</td><td>120000</td> <td class="">5.3</td><td class="highlight">5.3</td><td>0.6</td></tr>
<tr><td class="col1">Stroke</td><td>60720 | 60600</td> <td class="highlight">33.0</td><td class="">33.0</td><td>-0.1</td></tr>
<tr><td class="col1">Get_BBox</td><td>120000</td> <td class="highlight">1.2</td><td class="">1.2</td><td>-2.2</td></tr>
<tr><td class="col1">Get_CBox</td><td>120000</td> <td class="">5.0</td><td class="highlight">4.9</td><td>1.3</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>120000</td> <td class="">0.9</td><td class="highlight">0.9</td><td>1.2</td></tr>
<tr><td class="col1">Total duration for the font:</td><td class="col1" colspan="4">14 s</td></table>
<h3>Results for Arial_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>Iterations</th> <th>* <a href="./baseline/Arial_subset.txt">Baseline</a> (µs)</th> <th>* <a href="./benchmark/Arial_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="">9.7</td><td class="highlight">8.4</td><td>12.9</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="">8.7</td><td class="highlight">7.9</td><td>9.1</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="">11.9</td><td class="highlight">10.6</td><td>10.7</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="">9.2</td><td class="highlight">8.1</td><td>12.1</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="">0.0</td><td class="highlight">0.0</td><td>10.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="">2.3</td><td class="highlight">2.0</td><td>9.8</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="">62.2</td><td class="highlight">52.2</td><td>16.1</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="">10.1</td><td class="highlight">8.6</td><td>15.1</td></tr>
<tr><td class="col1">Stroke</td><td>51585 | 55670</td> <td class="">38.8</td><td class="highlight">36.0</td><td>7.4</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="">7.9</td><td class="highlight">7.5</td><td>4.3</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="">8.6</td><td class="highlight">8.0</td><td>6.3</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="">1.2</td><td class="highlight">1.2</td><td>0.8</td></tr>
<tr><td class="col1">Total duration for the font:</td><td class="col1" colspan="4">19 s</td></table>
<h3>Results for TimesNewRoman_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>Iterations</th> <th>* <a href="./baseline/TimesNewRoman_subset.txt">Baseline</a> (µs)</th> <th>* <a href="./benchmark/TimesNewRoman_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="highlight">9.9</td><td class="">10.0</td><td>-1.0</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="">9.0</td><td class="highlight">8.9</td><td>0.6</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="highlight">0.0</td><td class="">0.0</td><td>-3.6</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="highlight">0.0</td><td class="">0.0</td><td>-7.7</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="">12.9</td><td class="highlight">12.6</td><td>2.2</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="highlight">9.8</td><td class="">10.0</td><td>-1.3</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="">2.0</td><td class="highlight">2.0</td><td>1.6</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="">54.6</td><td class="highlight">54.0</td><td>1.1</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="highlight">11.0</td><td class="">11.1</td><td>-0.4</td></tr>
<tr><td class="col1">Stroke</td><td>41895 | 41990</td> <td class="">47.8</td><td class="highlight">47.7</td><td>0.2</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="">9.6</td><td class="highlight">9.6</td><td>0.2</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="highlight">9.9</td><td class="">10.0</td><td>-1.2</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="highlight">1.4</td><td class="">1.5</td><td>-1.1</td></tr>
<tr><td class="col1">Total duration for the font:</td><td class="col1" colspan="4">20 s</td></table>
<h3>Results for Tahoma_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>Iterations</th> <th>* <a href="./baseline/Tahoma_subset.txt">Baseline</a> (µs)</th> <th>* <a href="./benchmark/Tahoma_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="highlight">6.0</td><td class="">6.4</td><td>-6.8</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="highlight">5.1</td><td class="">5.3</td><td>-2.8</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="">8.3</td><td class="highlight">8.1</td><td>1.7</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="">6.0</td><td class="highlight">5.8</td><td>2.1</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="">0.0</td><td class="highlight">0.0</td><td>3.7</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="">2.0</td><td class="highlight">2.0</td><td>2.8</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="">51.9</td><td class="highlight">49.5</td><td>4.5</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="">6.7</td><td class="highlight">6.6</td><td>1.8</td></tr>
<tr><td class="col1">Stroke</td><td>62985 | 63935</td> <td class="">31.8</td><td class="highlight">31.3</td><td>1.5</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="">5.4</td><td class="highlight">5.3</td><td>0.4</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="">5.9</td><td class="highlight">5.8</td><td>1.2</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="">1.1</td><td class="highlight">1.1</td><td>1.6</td></tr>
<tr><td class="col1">Total duration for the font:</td><td class="col1" colspan="4">14 s</td></table>
<h3>Results for Verdana_subset.ttf</h2>
<table border="1">
<tr><th>Test</th><th>Iterations</th> <th>* <a href="./baseline/Verdana_subset.txt">Baseline</a> (µs)</th> <th>* <a href="./benchmark/Verdana_subset.txt">Benchmark</a> (µs)</th> <th>Difference (%)</th></tr>
<tr><td class="col1">Load</td><td>95000</td> <td class="">5.4</td><td class="highlight">5.4</td><td>1.6</td></tr>
<tr><td class="col1">Load_Advances (Normal)</td><td>95000</td> <td class="highlight">4.7</td><td class="">4.9</td><td>-4.4</td></tr>
<tr><td class="col1">Load_Advances (Fast)</td><td>95000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Load_Advances (Unscaled)</td><td>95000</td> <td class="">0.0</td><td class="highlight">0.0</td><td>9.7</td></tr>
<tr><td class="col1">Render</td><td>95000</td> <td class="">7.8</td><td class="highlight">7.8</td><td>1.1</td></tr>
<tr><td class="col1">Get_Glyph</td><td>95000</td> <td class="">5.4</td><td class="highlight">5.4</td><td>0.4</td></tr>
<tr><td class="col1">Get_Char_Index</td><td>94000</td> <td class="highlight">0.0</td><td class="highlight">0.0</td><td>0.0</td></tr>
<tr><td class="col1">Iterate CMap</td><td>1000</td> <td class="">2.1</td><td class="highlight">2.0</td><td>6.6</td></tr>
<tr><td class="col1">New_Face</td><td>1000</td> <td class="">47.0</td><td class="highlight">46.9</td><td>0.3</td></tr>
<tr><td class="col1">Embolden</td><td>95000</td> <td class="highlight">6.0</td><td class="">6.3</td><td>-4.8</td></tr>
<tr><td class="col1">Stroke</td><td>66025 | 66310</td> <td class="">30.3</td><td class="highlight">30.2</td><td>0.4</td></tr>
<tr><td class="col1">Get_BBox</td><td>95000</td> <td class="">4.9</td><td class="highlight">4.9</td><td>0.3</td></tr>
<tr><td class="col1">Get_CBox</td><td>95000</td> <td class="highlight">5.4</td><td class="">5.4</td><td>-0.2</td></tr>
<tr><td class="col1">New_Face & load glyph(s)</td><td>95000</td> <td class="">1.1</td><td class="highlight">1.1</td><td>0.9</td></tr>
<tr><td class="col1">Total duration for the font:</td><td class="col1" colspan="4">14 s</td></table>
<center>Freetype Benchmark</center>
</body>
</html>