Re: Monte-Carlo Tree Search as Regularized Policy Optimization

SSB1c2VkIHN0b2NoYXN0aWMgc2FtcGxpbmcgYXQgaW50ZXJuYWwgbm9kZXMsIGJlY2F1c2Ugb2Yg
dGhpcy4KPiBEdXJpbmcgdGhlIGZvcndhcmQgc2ltdWxhdGlvbiBwaGFzZSBvZiBTRUFSQ0gsIHRo
ZSBhY3Rpb24gYXQgZWFjaCBub2RlIHggaXMgc2VsZWN0ZWQgYnkgc2FtcGxpbmcgYSDiiLwgz4DC
ryjCt3x4KS4KPiBBcyBhIHJlc3VsdCwgdGhlIGZ1bGwgaW1hZ2luYXJ5IHRyYWplY3RvcnkgaXMg
Z2VuZXJhdGVkIGNvbnNpc3RlbnRseSBhY2NvcmRpbmcgdG8gcG9saWN5IM+Awq8uCgo+IEluIHRo
aXMgc2VjdGlvbiwgd2UgZXN0YWJsaXNoIG91ciBtYWluIGNsYWltIG5hbWVseSB0aGF0IEFscGhh
WmVyb+KAmXMgYWN0aW9uIHNlbGVjdGlvbiBjcml0ZXJpYSBjYW4gYmUgaW50ZXJwcmV0ZWQgYXMg
YXBwcm94aW1hdGluZyB0aGUgc29sdXRpb24gdG8gYSByZWd1bGFyaXplZCBwb2xpY3ktb3B0aW1p
emF0aW9uIG9iamVjdGl2ZS4KCkkgdGhpbmsgdGhleSBzYXkgVUNUIGFuZCBQVUNUIGlzIGFwcHJv
eGltYXRpb24gb2YgZGlyZWN0IM+Awq8gc2FtcGxpbmcsCmJ1dCBJIGhhdmVuJ3QgdW5kZXJzdG9v
ZCBzZWN0aW9uIDMgd2VsbC4KCjIwMjDlubQ35pyIMjDml6Uo5pyIKSAyOjUxIERhbmllbCA8ZHNo
YXd1bEBnbWFpbC5jb20+Ogo+Cj4gQEtlbnN1a2UgSSBzdXBwb3NlIGFsbCB0aGUgcHJvcG9zZWQg
YWxnb3JpdGhtcyBBQ1QsIFNFQVJDSCBhbmQgTEVBUk4gYXJlIG1lYW50IHRvIGJlIHVzZWQgZHVy
aW5nIHRyYWluaW5nLCBubz8KPiBJIHRoaW5rIEkgdW5kZXJzdGFuZCBBQ1QgYW5kIExFQVJOIGJ1
dCBJIGFtIG5vdCBzdXJlIGFib3V0IFNFQVJDSCBmb3Igd2hpY2ggdGhleSBzYXkgdGhpczoKPgo+
ID4gRHVyaW5nIHNlYXJjaCwgd2UgcHJvcG9zZSB0byBzdG9jaGFzdGljYWxseSBzYW1wbGUgYWN0
aW9ucyBhY2NvcmRpbmcgdG8gz4DCryBpbnN0ZWFkIG9mIHRoZSBkZXRlcm1pbmlzdGljIGFjdGlv
biBzZWxlY3Rpb24gcnVsZSBvZiBFcS4gMS4KPgo+IFRoaXMgc291bmRzIG11Y2ggbGlrZSB0aGUg
cmFuZG9tIHNlbGVjdGlvbiBkb25lIGF0IHRoZSByb290IHdpdGggdGVtcGVyYXR1cmUsIGJ1dCB0
aGlzIHRpbWUgYXBwbGllZCBhdCBpbnRlcm5hbCBub2Rlcy4KPiBEb2VzIGl0IG1lYW4gdGhlIHBV
Q1QgZm9ybXVsYSBpcyBub3QgdXNlZD8gV2h5IGRvZXMgdGhlIHNlbGVjdGlvbiBoYXZlIHRvIGJl
IHN0b2NoYXN0aWMgbm93Pwo+IE9uIHNlbGVjdGlvbiwgeW91IGNvbXB1dGUgz4BfYmFyIGV2ZXJ5
IHRpbWUgZnJvbSAocSwgz4BfdGhldGEsIG5fdmlzaXRzKSBzbyBJIHN1cHBvc2Ugz4BfYmFyIGhh
cyBldmVyeXRoaW5nIGl0IG5lZWRzIHRvIGJhbGFuY2UgZXhwbG9yYXRpb24gYW5kIGV4cGxvaXRh
dGlvbi4KPgo+Cj4gT24gU3VuLCBKdWwgMTksIDIwMjAgYXQgODoxMCBBTSBEYXZpZCBXdSA8bGln
aHR2ZWN0b3JAZ21haWwuY29tPiB3cm90ZToKPj4KPj4gSSBpbWFnaW5lIHRoYXQgYXQgbG93IHZp
c2l0cyBhdCBsZWFzdCwgIkFDVCIgYmVoYXZlcyBzaW1pbGFybHkgdG8gTGVlbGEgWmVybydzICJM
Q0IiIG1vdmUgc2VsZWN0aW9uLCB3aGljaCBhbHNvIGhhcyB0aGUgZWZmZWN0IG9mIHNvbWV0aW1l
cyBzZWxlY3RpbmcgYSBtb3ZlIHRoYXQgaXMgbm90IHRoZSBtYXgtdmlzaXRzIG1vdmUsIGlmIGl0
cyB2YWx1ZSBlc3RpbWF0ZSBoYXMgcmVjZW50bHkgYmVlbiBmb3VuZCB0byBiZSBzdWZmaWNpZW50
bHkgbGFyZ2VyIHRvIGJhbGFuY2UgdGhlIGZhY3QgdGhhdCBpdCBpcyBsb3dlciBwcmlvciBhbmQg
bG93ZXIgdmlzaXRzIChhdCBsZWFzdCwgdHlwaWNhbGx5LCB0aGlzIGlzIHdoeSB0aGUgbW92ZSB3
b3VsZG4ndCBoYXZlIGJlZW4gdGhlIG1heCB2aXNpdHMgbW92ZSBpbiB0aGUgZmlyc3QgcGxhY2Up
LiBJdCBhbHNvIHNjYWxlcyBpbiBhbiBpbnRlcmVzdGluZyB3YXkgd2l0aCBlbXBpcmljYWwgb2Jz
ZXJ2ZWQgcGxheW91dC1ieS1wbGF5b3V0IHZhcmlhbmNlIG9mIG1vdmVzLCBidXQgSSB0aGluayBi
eSBmYXIgdGhlIGltcG9ydGFudCBwYXJ0IGlzIHRoYXQgaXQgY2FuIHVzZSBzdWZmaWNpZW50bHkg
Y29uZmlkZW50IGhpZ2ggdmFsdWUgdG8gb3ZlcnJpZGUgbWF4LXZpc2l0cy4KPj4KPj4gVGhlIGdh
aW4gZnJvbSAiTENCIiBpbiBtYXRjaCBwbGF5IEkgcmVjYWxsIGlzIG9uIHRoZSB2ZXJ5IHZlcnkg
cm91Z2ggb3JkZXIgb2YgMTAwIEVsbywgYWx0aG91Z2ggaXQgY291bGQgYmUgbGVzcyBvciBtb3Jl
IGRlcGVuZGluZyBvbiBtYXRjaCBjb25kaXRpb25zIGFuZCB3aGF0IG5ldXJhbCBuZXQgaXMgdXNl
ZCBhbmQgb3RoZXIgdGhpbmdzLiBTbyBmb3IgTFogYXQgbGVhc3QsICJBQ1QiLWxpa2UgYmVoYXZp
b3IgYXQgbG93IHZpc2l0cyBpcyBub3QgbmV3Lgo+Pgo+Pgo+PiBPbiBTdW4sIEp1bCAxOSwgMjAy
MCBhdCA1OjM5IEFNIEtlbnN1a2UgTWF0c3V6YWtpIDxrbnNrLm10emtAZ21haWwuY29tPiB3cm90
ZToKPj4+Cj4+PiBIaSwKPj4+Cj4+PiBJIGNvdWxkbid0IGltcHJvdmUgbGVlbGEgemVybydzIHN0
cmVuZ3RoIGJ5IGltcGxlbWVudGluZyBTRUFSQ0ggYW5kIEFDVC4KPj4+IGh0dHBzOi8vZ2l0aHVi
LmNvbS96YWtraS9sZWVsYS16ZXJvL2NvbW1pdHMvcmVndWxhcml6ZWRfcG9saWN5Cj4+Pgo+Pj4g
MjAyMOW5tDfmnIgxN+aXpSjph5EpIDI6NDcgUsOpbWkgQ291bG9tIDxyZW1pLmNvdWxvbUBnbWFp
bC5jb20+Ogo+Pj4gPgo+Pj4gPiBUaGlzIGxvb2tzIHZlcnkgaW50ZXJlc3RpbmcuCj4+PiA+Cj4+
PiA+IEZyb20gYSBxdWljayBnbGFuY2UsIGl0IHNlZW1zIHRoZSBpbXByb3ZlbWVudCBpcyBtYWlu
bHkgd2hlbiB0aGUgbnVtYmVyIG9mIHBsYXlvdXRzIGlzIHNtYWxsLiBBbHNvIHRoZXkgZG9uJ3Qg
dGVzdCBvbiB0aGUgZ2FtZSBvZiBHby4gSGFzIGFueWJvZHkgdHJpZWQgaXQ/Cj4+PiA+Cj4+PiA+
IEkgd2lsbCB0YWtlIGEgZGVlcGVyIGxvb2sgbGF0ZXIuCj4+PiA+Cj4+PiA+IE9uIFRodSwgSnVs
IDE2LCAyMDIwIGF0IDk6NDkgQU0gUmF5IFRheWVrIDxydGF5ZWtAY2EucnIuY29tPiB3cm90ZToK
Pj4+ID4+Cj4+PiA+PiBodHRwczovL29sZC5yZWRkaXQuY29tL3IvTWFjaGluZUxlYXJuaW5nL2Nv
bW1lbnRzL2hyem9vaC9yX21vbnRlY2FybG9fdHJlZV9zZWFyY2hfYXNfcmVndWxhcml6ZWRfcG9s
aWN5Lwo+Pj4gPj4KPj4+ID4+Cj4+PiA+PiAtLQo+Pj4gPj4gSG9uZXN0eSBpcyBhIHZlcnkgZXhw
ZW5zaXZlIGdpZnQuIFNvLCBkb24ndCBleHBlY3QgaXQgZnJvbSBjaGVhcCBwZW9wbGUgLSBXYXJy
ZW4gQnVmZmV0dO+7vwo+Pj4gPj4gaHR0cDovL3RheWVrLmNvbS8KPj4+ID4+Cj4+PiA+PiBfX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+Pj4gPj4gQ29tcHV0
ZXItZ28gbWFpbGluZyBsaXN0Cj4+PiA+PiBDb21wdXRlci1nb0Bjb21wdXRlci1nby5vcmcKPj4+
ID4+IGh0dHA6Ly9jb21wdXRlci1nby5vcmcvbWFpbG1hbi9saXN0aW5mby9jb21wdXRlci1nbwo+
Pj4gPgo+Pj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
Xwo+Pj4gPiBDb21wdXRlci1nbyBtYWlsaW5nIGxpc3QKPj4+ID4gQ29tcHV0ZXItZ29AY29tcHV0
ZXItZ28ub3JnCj4+PiA+IGh0dHA6Ly9jb21wdXRlci1nby5vcmcvbWFpbG1hbi9saXN0aW5mby9j
b21wdXRlci1nbwo+Pj4KPj4+Cj4+Pgo+Pj4gLS0KPj4+IEtlbnN1a2UgTWF0c3V6YWtpCj4+PiBf
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+Pj4gQ29tcHV0
ZXItZ28gbWFpbGluZyBsaXN0Cj4+PiBDb21wdXRlci1nb0Bjb21wdXRlci1nby5vcmcKPj4+IGh0
dHA6Ly9jb21wdXRlci1nby5vcmcvbWFpbG1hbi9saXN0aW5mby9jb21wdXRlci1nbwo+Pgo+PiBf
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+PiBDb21wdXRl
ci1nbyBtYWlsaW5nIGxpc3QKPj4gQ29tcHV0ZXItZ29AY29tcHV0ZXItZ28ub3JnCj4+IGh0dHA6
Ly9jb21wdXRlci1nby5vcmcvbWFpbG1hbi9saXN0aW5mby9jb21wdXRlci1nbwo+Cj4gX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBDb21wdXRlci1nbyBt
YWlsaW5nIGxpc3QKPiBDb21wdXRlci1nb0Bjb21wdXRlci1nby5vcmcKPiBodHRwOi8vY29tcHV0
ZXItZ28ub3JnL21haWxtYW4vbGlzdGluZm8vY29tcHV0ZXItZ28KCgoKLS0gCktlbnN1a2UgTWF0
c3V6YWtpCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkNv
bXB1dGVyLWdvIG1haWxpbmcgbGlzdApDb21wdXRlci1nb0Bjb21wdXRlci1nby5vcmcKaHR0cDov
L2NvbXB1dGVyLWdvLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2NvbXB1dGVyLWdvCg==