30% faster with a batch size of 63 instead of 64!

--===============7653077639911930978==
Content-Type: multipart/alternative; boundary="000000000000bc758305a53be553"

--000000000000bc758305a53be553
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi,

I am probably not the only one who made this mistake: it is usually very
bad to use a power of 2 for the batch size!

Relevant documentation by NVIDIA:
https://docs.nvidia.com/deeplearning/performance/dl-performance-convolution=
al/index.html#quant-effects

The documentation is not extremely clear, so I figured out the formula:
N=3Dint((n*(1<<14)*SM)/(H*W*C))

SM is the number of multiprocessors (80 for V100 or Titan V, 68 for RTX
2080 Ti).
n is an integer (usually n=3D1 is slightly worse than n>1).

So the efficient batch size is 63 for 9x9 Go on a V100 with 256-channel
layers. 53 on the RTX 2080 Ti.

There is my tweet with an empirical plot:
https://twitter.com/Remi_Coulom/status/1259188988646129665

I created a new CGOS account to play with this improvement. Probably not a
huge different in strength, but it is good to get such an improvement so
easily.

R=C3=A9mi

--000000000000bc758305a53be553
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,</div><div><br></div><div>I am probably not the on=
ly one who made this mistake: it is usually very bad to use a power of 2 fo=
r the batch size!</div><div><br></div><div>Relevant documentation by NVIDIA=
:</div><div><a href=3D"https://docs.nvidia.com/deeplearning/performance/dl-=
performance-convolutional/index.html#quant-effects">https://docs.nvidia.com=
/deeplearning/performance/dl-performance-convolutional/index.html#quant-eff=
ects</a></div><div><br></div><div>The documentation is not extremely clear,=
 so I figured out the formula:</div><div>
<span class=3D"gmail-css-901oao gmail-css-16my406 gmail-r-1qd0xha gmail-r-a=
d9z0x gmail-r-bcqeeo gmail-r-qvutc0">N=3Dint((n*(1&lt;&lt;14)*SM)/(H*W*C))<=
/span> <br></div><div><br></div><div>SM is the number of multiprocessors (8=
0 for V100 or Titan V, 68 for RTX 2080 Ti).<br></div><div>n is an integer (=
usually n=3D1 is slightly worse than n&gt;1).</div><div><br></div><div>So t=
he efficient batch size is 63 for 9x9 Go on a V100 with 256-channel layers.=
 53 on the RTX 2080 Ti.</div><div><br></div><div>
<div>There is my tweet with an empirical plot:</div><div><a href=3D"https:/=
/twitter.com/Remi_Coulom/status/1259188988646129665">https://twitter.com/Re=
mi_Coulom/status/1259188988646129665</a></div><div><br></div>

</div><div>I created a new CGOS account to play with this improvement. Prob=
ably not a huge different in strength, but it is good to get such an improv=
ement so easily.</div><div><br></div><div>R=C3=A9mi<br></div></div>

--000000000000bc758305a53be553--

--===============7653077639911930978==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Computer-go mailing list
[email protected]
http://computer-go.org/mailman/listinfo/computer-go

--===============7653077639911930978==--