Re: [IPython-User] ipyparallel: Can't pickly memoryview objects
Fernando Perez <[email protected]> Mon, 20 Mar 2017 11:32:41 -0700
| Newsgroups | gmane.comp.python.ipython.devel,gmane.comp.python.ipython.user |
|---|---|
| Message-ID | <CAHAreOrMgE0zBs9uwvHOf9Z6PmHXBYYdaX9Ysg_HfQ+MTqEd_g@mail.gmail.com> |
--===============3168798036537355222==
Content-Type: multipart/alternative; boundary=94eb2c12cad080f38c054b2dc2ac
--94eb2c12cad080f38c054b2dc2ac
Content-Type: text/plain; charset=UTF-8
Hi Florian,
sorry, this list is mostly deprecated, with all IPython discussions now
happening on a single list:
https://mail.scipy.org/mailman/listinfo/ipython-dev
The issue is the closure over xs in your lambda; pickle doesn't know how to
serialize closures. You can make it work if you enable the cloudpickle
serializer instead:
dview.use_cloudpickle()
as indicated here:
http://ipyparallel.readthedocs.io/en/latest/details.html#closures
An interesting tidbit is that if you move the code in `main` out to the
top-level of the script and manually push `xs` as well, the code works fine
even without the cloudpickle change. That's because names that require a
lookup into globals() don't require the creation of a closure object, as
explained in the link above.
Cheers,
f
On Thu, Mar 16, 2017 at 2:07 AM, Florian Lindner <[email protected]>
wrote:
> Hello,
>
> I have this small sample program:
>
> import ipyparallel as ipp, numpy as np
>
> class X:
> def __init__(self, x, xs):
> self.A = np.full((10, 10), x)
> np.fill_diagonal(self.A, x*10)
>
> def norm(self):
> return np.linalg.norm(self.A)
>
>
> def main():
> rc = ipp.Client()
> lview = rc.load_balanced_view()
> dview = rc[:]
> dview.push({"X" : X})
> dview.execute("import numpy as np")
>
> xs = np.linspace(1, 100, 10)
> sqxs = []
>
> sqxs = lview.map(lambda x: X(x, xs), xs)
> sqxs.wait()
>
> for i in sqxs:
> print(i.norm())
>
> if __name__ == "__main__":
> main()
>
>
> trying it to run gives TypeError: can't pickle memoryview objects.
> Probably because I provide the xs array to X.
>
> How can I work around this situation?
>
> Thanks,
> Florian
>
> Complete traceback:
>
> Traceback (most recent call last):
> File "ipython_parallel.py", line 30, in <module>
> main()
> File "ipython_parallel.py", line 23, in main
> sqxs = lview.map(lambda x: X(x, xs), xs)
> File "<decorator-gen-136>", line 2, in map
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 50, in sync_results
> ret = f(self, *args, **kwargs)
> File "<decorator-gen-135>", line 2, in map
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 35, in save_ids
> ret = f(self, *args, **kwargs)
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 1109, in map
> return pf.map(*sequences)
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py",
> line 285, in map
> return self(*sequences, __ipp_mapping=True)
> File "<decorator-gen-118>", line 2, in __call__
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py",
> line 76, in sync_view_results
> return f(self, *args, **kwargs)
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py",
> line 259, in __call__
> ar = view.apply(f, *args)
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 211, in apply
> return self._really_apply(f, args, kwargs)
> File "<decorator-gen-134>", line 2, in _really_apply
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 47, in sync_results
> return f(self, *args, **kwargs)
> File "<decorator-gen-133>", line 2, in _really_apply
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 35, in save_ids
> ret = f(self, *args, **kwargs)
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 1037, in _really_apply
> metadata=metadata)
> File "/usr/lib/python3.6/site-packages/ipyparallel/client/client.py",
> line 1395, in send_apply_request
> item_threshold=self.session.item_threshold,
> File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py",
> line 166, in pack_apply_message
> serialize_object(arg, buffer_threshold, item_threshold) for arg in
> args))
> File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py",
> line 166, in <genexpr>
> serialize_object(arg, buffer_threshold, item_threshold) for arg in
> args))
> File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py",
> line 112, in serialize_object
> buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))
> TypeError: can't pickle memoryview objects
>
> _______________________________________________
> IPython-User mailing list
> [email protected]
> https://mail.scipy.org/mailman/listinfo/ipython-user
>
--
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
--94eb2c12cad080f38c054b2dc2ac
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Florian,<div><br></div><div>sorry, this list is mostly =
deprecated, with all IPython discussions now happening on a single list:</d=
iv><div><br></div><div><a href=3D"https://mail.scipy.org/mailman/listinfo/i=
python-dev">https://mail.scipy.org/mailman/listinfo/ipython-dev</a><br></di=
v><div><br></div><div>The issue is the closure over xs in your lambda; pick=
le doesn't know how to serialize closures.=C2=A0 You can make it work i=
f you enable the cloudpickle serializer instead:</div><div><br></div><div><=
div>=C2=A0 =C2=A0 dview.use_cloudpickle()</div></div><div><br></div><div>as=
indicated here:=C2=A0<a href=3D"http://ipyparallel.readthedocs.io/en/lates=
t/details.html#closures">http://ipyparallel.readthedocs.io/en/latest/detail=
s.html#closures</a></div><div><br></div><div>An interesting tidbit is that =
if you move the code in `main` out to the top-level of the script and manua=
lly push `xs` as well, the code works fine even without the cloudpickle cha=
nge.=C2=A0 That's because names that require a lookup into globals() do=
n't require the creation of a closure object, as explained in the link =
above.</div><div><br></div><div>Cheers,</div><div><br></div><div>f</div><di=
v><br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"=
>On Thu, Mar 16, 2017 at 2:07 AM, Florian Lindner <span dir=3D"ltr"><<a =
href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</=
a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0=
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I have this small sample program:<br>
<br>
import ipyparallel as ipp, numpy as np<br>
<br>
class X:<br>
=C2=A0 =C2=A0 def __init__(self, x, xs):<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 self.A =3D np.full((10, 10), x)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 np.fill_diagonal(self.A, x*10)<br>
<br>
=C2=A0 =C2=A0 def norm(self):<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 return np.linalg.norm(self.A)<br>
<br>
<br>
def main():<br>
=C2=A0 =C2=A0 rc =3D ipp.Client()<br>
=C2=A0 =C2=A0 lview =3D rc.load_balanced_view()<br>
=C2=A0 =C2=A0 dview =3D rc[:]<br>
=C2=A0 =C2=A0 dview.push({"X" : X})<br>
=C2=A0 =C2=A0 dview.execute("import numpy as np")<br>
<br>
=C2=A0 =C2=A0 xs =3D np.linspace(1, 100, 10)<br>
=C2=A0 =C2=A0 sqxs =3D []<br>
<br>
=C2=A0 =C2=A0 sqxs =3D lview.map(lambda x: X(x, xs), xs)<br>
=C2=A0 =C2=A0 sqxs.wait()<br>
<br>
=C2=A0 =C2=A0 for i in sqxs:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 print(i.norm())<br>
<br>
if __name__ =3D=3D "__main__":<br>
=C2=A0 =C2=A0 main()<br>
<br>
<br>
trying it to run gives TypeError: can't pickle memoryview objects. Prob=
ably because I provide the xs array to X.<br>
<br>
How can I work around this situation?<br>
<br>
Thanks,<br>
Florian<br>
<br>
Complete traceback:<br>
<br>
Traceback (most recent call last):<br>
=C2=A0 File "ipython_parallel.py", line 30, in <module><br>
=C2=A0 =C2=A0 main()<br>
=C2=A0 File "ipython_parallel.py", line 23, in main<br>
=C2=A0 =C2=A0 sqxs =3D lview.map(lambda x: X(x, xs), xs)<br>
=C2=A0 File "<decorator-gen-136>", line 2, in map<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 50, in sync_results<br>
=C2=A0 =C2=A0 ret =3D f(self, *args, **kwargs)<br>
=C2=A0 File "<decorator-gen-135>", line 2, in map<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 35, in save_ids<br>
=C2=A0 =C2=A0 ret =3D f(self, *args, **kwargs)<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 1109, in map<br>
=C2=A0 =C2=A0 return pf.map(*sequences)<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>remotefunction.py", line 285, in map<br>
=C2=A0 =C2=A0 return self(*sequences, __ipp_mapping=3DTrue)<br>
=C2=A0 File "<decorator-gen-118>", line 2, in __call__<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>remotefunction.py", line 76, in sync_view_results<br>
=C2=A0 =C2=A0 return f(self, *args, **kwargs)<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>remotefunction.py", line 259, in __call__<br>
=C2=A0 =C2=A0 ar =3D view.apply(f, *args)<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 211, in apply<br>
=C2=A0 =C2=A0 return self._really_apply(f, args, kwargs)<br>
=C2=A0 File "<decorator-gen-134>", line 2, in _really_apply=
<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 47, in sync_results<br>
=C2=A0 =C2=A0 return f(self, *args, **kwargs)<br>
=C2=A0 File "<decorator-gen-133>", line 2, in _really_apply=
<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 35, in save_ids<br>
=C2=A0 =C2=A0 ret =3D f(self, *args, **kwargs)<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>view.py", line 1037, in _really_apply<br>
=C2=A0 =C2=A0 metadata=3Dmetadata)<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/client/=
<wbr>client.py", line 1395, in send_apply_request<br>
=C2=A0 =C2=A0 item_threshold=3Dself.session.<wbr>item_threshold,<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/<wbr>se=
rialize/serialize.py", line 166, in pack_apply_message<br>
=C2=A0 =C2=A0 serialize_object(arg, buffer_threshold, item_threshold) for a=
rg in args))<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/<wbr>se=
rialize/serialize.py", line 166, in <genexpr><br>
=C2=A0 =C2=A0 serialize_object(arg, buffer_threshold, item_threshold) for a=
rg in args))<br>
=C2=A0 File "/usr/lib/python3.6/site-<wbr>packages/ipyparallel/<wbr>se=
rialize/serialize.py", line 112, in serialize_object<br>
=C2=A0 =C2=A0 buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))<br>
TypeError: can't pickle memoryview objects<br>
<br>
______________________________<wbr>_________________<br>
IPython-User mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]</a><br>
<a href=3D"https://mail.scipy.org/mailman/listinfo/ipython-user" rel=3D"nor=
eferrer" target=3D"_blank">https://mail.scipy.org/<wbr>mailman/listinfo/ipy=
thon-user</a><br>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature">Fernando Perez (@fp=
erez_org; <a href=3D"http://fperez.org" target=3D"_blank">http://fperez.org=
</a>)<br>fperez.net-at-gmail: mailing lists only (I ignore this when swampe=
d!)<br>fernando.perez-at-berkeley: contact me here for any direct mail<br><=
/div>
</div>
--94eb2c12cad080f38c054b2dc2ac--
--===============3168798036537355222==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
IPython-dev mailing list
[email protected]
https://mail.scipy.org/mailman/listinfo/ipython-dev
--===============3168798036537355222==--