Re: 30% faster with a batch size of 63 instead of 64!

--===============8810509996320759099==
Content-Type: multipart/alternative; boundary="0000000000005017be05a53f1b77"

--0000000000005017be05a53f1b77
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Nice job! And the graph makes it super clear how the edge effects work.

s.

On Sat, May 9, 2020, 2:19 PM R=C3=A9mi Coulom <[email protected]> wrote=
:

> Hi,
>
> I am probably not the only one who made this mistake: it is usually very
> bad to use a power of 2 for the batch size!
>
> Relevant documentation by NVIDIA:
>
> https://docs.nvidia.com/deeplearning/performance/dl-performance-convoluti=
onal/index.html#quant-effects
>
> The documentation is not extremely clear, so I figured out the formula:
> N=3Dint((n*(1<<14)*SM)/(H*W*C))
>
> SM is the number of multiprocessors (80 for V100 or Titan V, 68 for RTX
> 2080 Ti).
> n is an integer (usually n=3D1 is slightly worse than n>1).
>
> So the efficient batch size is 63 for 9x9 Go on a V100 with 256-channel
> layers. 53 on the RTX 2080 Ti.
>
> There is my tweet with an empirical plot:
> https://twitter.com/Remi_Coulom/status/1259188988646129665
>
> I created a new CGOS account to play with this improvement. Probably not =
a
> huge different in strength, but it is good to get such an improvement so
> easily.
>
> R=C3=A9mi
> _______________________________________________
> Computer-go mailing list
> [email protected]
> http://computer-go.org/mailman/listinfo/computer-go
>

--0000000000005017be05a53f1b77
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto">Nice job! And the graph makes it super clear how the edge=
 effects work.<div dir=3D"auto"><br></div><div dir=3D"auto">s.</div></div><=
br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sat,=
 May 9, 2020, 2:19 PM R=C3=A9mi Coulom &lt;<a href=3D"mailto:remi.coulom@gm=
ail.com">[email protected]</a>&gt; wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex"><div dir=3D"ltr"><div>Hi,</div><div><br></div><div>I am probably=
 not the only one who made this mistake: it is usually very bad to use a po=
wer of 2 for the batch size!</div><div><br></div><div>Relevant documentatio=
n by NVIDIA:</div><div><a href=3D"https://docs.nvidia.com/deeplearning/perf=
ormance/dl-performance-convolutional/index.html#quant-effects" target=3D"_b=
lank" rel=3D"noreferrer">https://docs.nvidia.com/deeplearning/performance/d=
l-performance-convolutional/index.html#quant-effects</a></div><div><br></di=
v><div>The documentation is not extremely clear, so I figured out the formu=
la:</div><div>
<span>N=3Dint((n*(1&lt;&lt;14)*SM)/(H*W*C))</span> <br></div><div><br></div=
><div>SM is the number of multiprocessors (80 for V100 or Titan V, 68 for R=
TX 2080 Ti).<br></div><div>n is an integer (usually n=3D1 is slightly worse=
 than n&gt;1).</div><div><br></div><div>So the efficient batch size is 63 f=
or 9x9 Go on a V100 with 256-channel layers. 53 on the RTX 2080 Ti.</div><d=
iv><br></div><div>
<div>There is my tweet with an empirical plot:</div><div><a href=3D"https:/=
/twitter.com/Remi_Coulom/status/1259188988646129665" target=3D"_blank" rel=
=3D"noreferrer">https://twitter.com/Remi_Coulom/status/1259188988646129665<=
/a></div><div><br></div>

</div><div>I created a new CGOS account to play with this improvement. Prob=
ably not a huge different in strength, but it is good to get such an improv=
ement so easily.</div><div><br></div><div>R=C3=A9mi<br></div></div>
_______________________________________________<br>
Computer-go mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank" rel=3D"nor=
eferrer">[email protected]</a><br>
<a href=3D"http://computer-go.org/mailman/listinfo/computer-go" rel=3D"nore=
ferrer noreferrer" target=3D"_blank">http://computer-go.org/mailman/listinf=
o/computer-go</a><br>
</blockquote></div>

--0000000000005017be05a53f1b77--

--===============8810509996320759099==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Computer-go mailing list
[email protected]
http://computer-go.org/mailman/listinfo/computer-go

--===============8810509996320759099==--