matrix-ref is slow

My informal experiments seems to indicate that matrix-ref is much
slower than aref.  In particular (not that this is suprising, since
matrix-ref seems to be doing much more):

* (setf r (make-real-matrix 1 100))
#<REAL-MATRIX  1 x 100
    0.0000      0.0000      0.0000      0.0000     ...  0.0000     >
* (time (dotimes (i 100000000) (setf (matrix-ref r (random 100)) (random 1.0d0))))
Evaluation took:
  201.49 seconds of real time
  164.01 seconds of user run time
  30.88 seconds of system run time
  [Run times include 4.38 seconds GC run time]
  0 page faults and
  12799208008 bytes consed.
NIL

vs:

* (setf r (make-array 100 :element-type 'double-float))
#(0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0
  0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0 0.0d0)
* (time (dotimes (i 100000000) (setf (aref r (random 100)) (random 1.0d0))))
Evaluation took:
  37.98 seconds of real time
  32.61 seconds of user run time
  3.75 seconds of system run time
  [Run times include 0.7 seconds GC run time]
  0 page faults and
  1599880616 bytes consed.
NIL

This implies to me that if I need to do a LOT of manipulating the
entries of vectors, I'm better off moving them to an array, doing the
manipulation there, then moving them back into matrix form to do
matrix operations.  Is this essentially correct, or am I missing
something?  I guess the alternate approach is to extend matlisp myself
to provide "unsafe" accessors that assume more --- I'm willing to tell
it I'm using a real matrix to avoid the generic function dispatch, and
I'm willing to just run an aref directly on the store component
(although the store isn't currently exported by matlisp).

Cheers,

rif