Re: Speeding up Python

--===============0886493251347316969==
Content-Type: multipart/alternative; boundary="000000000000b2668005857e7a5e"

--000000000000b2668005857e7a5e
Content-Type: text/plain; charset="UTF-8"

Ah, that's copying from one imath array to another, and that's not what I'm
trying to accomplish.
I'm trying to get the data out of the imath array objects into another form
(in this case, numpy)
I know the imathnumpy module exists. However, the code only handles 2 of
the array types IIRC.
It also requires the compiled module to be linked with numpy, and numpy
isn't always available like in Windows Maya, or if I wanted to use the
built-in python array.array, or my own custom module.

~T.Fox



On Mon, Apr 1, 2019 at 1:00 PM Nick Rasmussen <[email protected]> wrote:

>
> The native imath python arrays are pretty fast, I think your version is
> triggering an element by element python copy of data into the numpy array.
> Here's the timings with the native (fast) array copy and the python only
> iteration on the array types.  Here's the script:
>
> import imath
> import time
>
> count = 1000000
>
> ai = imath.V3dArray(imath.V3d(0.25, 0.3, 0.75), count)
> ao = imath.V3dArray(count)
> start = time.time()
> ao[:] = ai
> end = time.time()
>
> print 'Copying %d elements: %0.06f seconds' % (count, end-start)
>
> start = time.time()
>
> for i in range(count):
>     ao[i] = ai[i]
>
> end = time.time()
>
> print 'Python iteration %d elements: %0.06f seconds' % (count, end-start)
>
> And the output:
>
> > python2 foo.py
> Copying 1000000 elements: 0.003245 seconds
> Python iteration 1000000 elements: 0.994242 seconds
>
> Does that match what you're seeing?  That's also without enabling the
> multithreading that's built into the array operations.
>
> -nick
>
> On Mon, Apr 1, 2019 at 12:46 PM Tyler Fox <[email protected]> wrote:
>
>> Getting data in and out of imath in python is *SLOW*. And yes, Python is
>> a slow language, but that doesn't mean we can't kick it up a notch.
>> Could I get some feedback on my proposals to speed things up?
>>
>> https://github.com/openexr/openexr/pull/373
>> First is a PR with a *very* simple change that just exposes the memory
>> address of the imath object.
>> With a few lines of python code, you can read the c-data and re-interpret
>> it in whatever way you want.
>>
>> https://github.com/openexr/openexr/issues/367
>> This is an issue where I described a more complex (and maybe more
>> pythonic?) way of exposing the underlying data.
>>
>> Either of these proposals could remove the need for imathnumpy... Or
>> imathnumpy could be replaced with a pure-python module that handles the
>> more error prone memory mapping.
>>
>> Thoughts?
>> ~T.Fox
>> _______________________________________________
>> Openexr-devel mailing list
>> [email protected]
>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>
>

--000000000000b2668005857e7a5e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Ah, that&#39;s copying from one imath array to another, an=
d that&#39;s not what I&#39;m trying to accomplish.<div>I&#39;m trying to g=
et the data out of the imath array objects into another form (in this case,=
 numpy)<div><div>I know the imathnumpy module exists. However, the code onl=
y handles 2 of the array types IIRC.</div><div>It also requires the compile=
d module to be linked with numpy, and numpy isn&#39;t always available like=
 in Windows Maya, or if I wanted to use the built-in python array.array, or=
 my own custom module.</div><div><br></div><div>~T.Fox</div><div><br><div><=
br></div></div></div></div></div><br><div class=3D"gmail_quote"><div dir=3D=
"ltr" class=3D"gmail_attr">On Mon, Apr 1, 2019 at 1:00 PM Nick Rasmussen &l=
t;<a href=3D"mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div dir=3D"=
ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div><br></div><div>The native imath=
 python arrays are pretty fast, I think your version is triggering an eleme=
nt by element python copy of data into the numpy array.=C2=A0 Here&#39;s th=
e timings with the native (fast) array copy and the python only iteration o=
n the array types.=C2=A0 Here&#39;s the script:</div><div><br></div><div><d=
iv>i<font face=3D"monospace, monospace">mport imath</font></div><div><font =
face=3D"monospace, monospace">import time</font></div><div><font face=3D"mo=
nospace, monospace"><br></font></div><div><font face=3D"monospace, monospac=
e">count =3D 1000000</font></div><div><font face=3D"monospace, monospace"><=
br></font></div><div><font face=3D"monospace, monospace">ai =3D imath.V3dAr=
ray(imath.V3d(0.25, 0.3, 0.75), count)</font></div><div><font face=3D"monos=
pace, monospace">ao =3D imath.V3dArray(count)</font></div><div><font face=
=3D"monospace, monospace">start =3D time.time()</font></div><div><font face=
=3D"monospace, monospace">ao[:] =3D ai</font></div><div><font face=3D"monos=
pace, monospace">end =3D time.time()</font></div><div><font face=3D"monospa=
ce, monospace"><br></font></div><div><font face=3D"monospace, monospace">pr=
int &#39;Copying %d elements: %0.06f seconds&#39; % (count, end-start)</fon=
t></div><div><font face=3D"monospace, monospace"><br></font></div><div><fon=
t face=3D"monospace, monospace">start =3D time.time()</font></div><div><fon=
t face=3D"monospace, monospace"><br></font></div><div><font face=3D"monospa=
ce, monospace">for i in range(count):</font></div><div><font face=3D"monosp=
ace, monospace">=C2=A0 =C2=A0 ao[i] =3D ai[i]</font></div><div><font face=
=3D"monospace, monospace"><br></font></div><div><font face=3D"monospace, mo=
nospace">end =3D time.time()</font></div><div><font face=3D"monospace, mono=
space"><br></font></div><div><font face=3D"monospace, monospace">print &#39=
;Python iteration %d elements: %0.06f seconds&#39; % (count, end-start)</fo=
nt></div></div><div><font face=3D"monospace, monospace"><br></font></div>An=
d the output:<div><font face=3D"monospace, monospace"><br></font></div><div=
><div><div><font face=3D"monospace, monospace">&gt; python2 foo.py</font></=
div><div><font face=3D"monospace, monospace">Copying 1000000 elements: 0.00=
3245 seconds</font></div><div><font face=3D"monospace, monospace">Python it=
eration 1000000 elements: 0.994242 seconds</font></div></div></div><div><br=
></div><div>Does that match what you&#39;re seeing?=C2=A0 That&#39;s also w=
ithout enabling the multithreading that&#39;s built into the array operatio=
ns.</div><div><br></div><div>-nick</div></div></div></div></div><br><div cl=
ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Apr 1, 20=
19 at 12:46 PM Tyler Fox &lt;<a href=3D"mailto:[email protected]" target=3D"_b=
lank">[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quo=
te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204=
);padding-left:1ex"><div dir=3D"ltr"><div>Getting data in and out of imath =
in python is *SLOW*. And yes, Python is a slow language, but that doesn&#39=
;t mean we can&#39;t kick it up a notch.</div><div>Could I get some feedbac=
k on my proposals to speed things up?</div><div><br></div><a href=3D"https:=
//github.com/openexr/openexr/pull/373" target=3D"_blank">https://github.com=
/openexr/openexr/pull/373</a><div>First is a PR with a *very* simple change=
 that just exposes the memory address of the imath object.</div><div>With a=
 few lines of python code, you can read the c-data and re-interpret it in w=
hatever way you want.</div><div><br></div><div><div><div><a href=3D"https:/=
/github.com/openexr/openexr/issues/367" target=3D"_blank">https://github.co=
m/openexr/openexr/issues/367</a>=C2=A0<br></div></div></div><div>This is an=
 issue where I described a more complex (and maybe more pythonic?) way of e=
xposing the underlying data.</div><div><br></div><div>Either of these propo=
sals could remove the need for imathnumpy... Or imathnumpy could be replace=
d with a pure-python module that handles the more error prone memory mappin=
g.</div><div><br></div><div>Thoughts?</div><div>~T.Fox</div></div>
_______________________________________________<br>
Openexr-devel mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Openexr-devel=
@nongnu.org</a><br>
<a href=3D"https://lists.nongnu.org/mailman/listinfo/openexr-devel" rel=3D"=
noreferrer" target=3D"_blank">https://lists.nongnu.org/mailman/listinfo/ope=
nexr-devel</a><br>
</blockquote></div>
</blockquote></div>

--000000000000b2668005857e7a5e--


--===============0886493251347316969==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Openexr-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/openexr-devel

--===============0886493251347316969==--