[bug #68035] [Statistics] pdist2 memory requirement and speed

Please use the bug tracker to post updates to a bug report.  The mailing list is intended as a read-only notification stream.  Info posted to this mailing list address won't appear in the tracker database where it is most useful.

Follow-up Comment #5, bug #68035 (group octave):

With release 1.8.1, *pdist2* supports both the "proper" euclidean and the fast
euclidean distance methods. The change in the original implementation (as
mentioned by Carnë) was for MATLAB compatibility reasons. Hence, Dmitri's
recently proposed patch was integrated as the "fasteuclidean" option.
Moreover, with 1.8.1 release, *pdist2* also supports a more memory efficient
blocked approach for large input arrays, which was addded with this PR
https://github.com/gnu-octave/statistics/pull/371

Running the large array example on my system (relatively old hardware), I get

>> pkg load statistics
>> A = rand(4754,1024);
>> B = rand(6800,1024);
>> tic; D = pdist2(A,B,'euclidean'); toc
Elapsed time is 575.896 seconds.
>> tic; D = pdist2(A,B,'fasteuclidean'); toc
Elapsed time is 3.3965 seconds.


The proper "euclidean" is considerably slower, but it runs without any memory
issues. With smaller input arrays, I get

>> A = rand(1024,1024);
>> B = rand(1024,1024);
>> tic; D = pdist2(A,B,'euclidean'); toc
Elapsed time is 13.9862 seconds.
>> tic; D = pdist2(A,B,'fasteuclidean'); toc
Elapsed time is 0.15058 seconds.


I did the original change to avoid differences in computed distances which
propagated to the output of other statistical functions. IMHO, the current
implementation (after the contributions by Dmitri and Aman) provides the best
of all worlds.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?68035>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQk97aszIMMAvLLwm6qLAuaBUf3TgUCaY2BigAKCRCqLAuaBUf3
TiXTAP9rTnGWhyoaErYpzG5y2bS79wVHhEfCPpEwQwi6nquJ2QD9EqEk7xkZ/Tvt
CGCa4CLKC5qaLTJERX9VaS9/ie6ZygU=
=ecXS
-----END PGP SIGNATURE-----