GPU, neural networks and APU

On 08/03/2016 07:22 PM, Karen Shaeffer wrote:

> For those interested, here is an informative paper:
>
> http://ai.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgCatanzaro_icml2013.pdf

If anyone want to have a more procedural and general idea:

http://neuralnetworksanddeeplearning.com/chap2.html

(you've to enable js to see the formulas properly)

Doing neural networks means lot of matrix multiplications.
GPU are quite good at linear algebra.

Up to my understanding as they say in the paper they had 2 major problems:

1) their matrices were sparse (matrix with most elements equal to 0)
2) they had to move data around

1) is a problem because algorithms to multiply sparse matrix are not as 
efficient. 1) is somehow another effect of the cost of moving a lot of 
data around.

"Compacting" a sparse matrix is expensive in terms of computation, 
getting it in the GPU RAM is expensive in terms of moving it to the GPU 
RAM + wasting a lot of computations that results in 0 submatrix.

Then they had to solve the problem of moving data between different GPU 
on different nodes and they smartly used a library made by someone else.

It's definitively not my field of expertise, but while GPU are awfully 
fast in specialized repetitive operation on large dataset feeding them 
with enough data is tricky.
That's why stuff like APU are pretty interesting.

Even in the paper CPU/GPU ratio was 1. so having a GPU inside your CPU 
sounds nice.

The paper was published in 2013. I bet many things have changed.

-- 
Ivan Sergio Borgonovo
http://www.webthatworks.it http://www.borgonovo.net