RE: [PHP] Re: refernces, arrays, and why does it take up so much memory?

> -----Original Message-----
> From: Jim Giner [mailto:[email protected]]
> Sent: Monday, September 02, 2013 8:14 PM
> To: [email protected]
> Subject: [PHP] Re: refernces, arrays, and why does it take up so much
> memory?
> 
> On 9/2/2013 9:30 PM, Daevid Vincent wrote:
> > I'm confused on how a reference works I think.
> >
> > I have a DB result set in an array I'm looping over. All I simply want
to
> do
> > is make the array key the "id" of the result set row.
> >
> > This is the basic gist of it:
> >
> >         private function _normalize_result_set()
> >         {
> >                foreach($this->tmp_results as $k => $v)
> >                {
> >                       $id = $v['id'];
> >                       $new_tmp_results[$id] =& $v; //2013-08-29 [dv]
using
> a
> > reference here cuts the memory usage in half!
> >                       unset($this->tmp_results[$k]);
> >
> >                       /*
> >                       if ($i++ % 1000 == 0)
> >                       {
> >                             gc_enable(); // Enable Garbage Collector
> >                             var_dump(gc_enabled()); // true
> >                             var_dump(gc_collect_cycles()); // # of
> elements
> > cleaned up
> >                             gc_disable(); // Disable Garbage Collector
> >                       }
> >                       */
> >                }
> >                $this->tmp_results = $new_tmp_results;
> >                //var_dump($this->tmp_results); exit;
> >                unset($new_tmp_results);
> >         }
> >
> > Without using the =& reference, my data works great:
> > $new_tmp_results[$id] = $v;
> >
> > array (size=79552)
> >    6904 =>
> >      array (size=4)
> >        'id' => string '6904' (length=4)
> >        'studio_id' => string '5' (length=1)
> >        'genres' => string '34|' (length=3)
> >    6905 =>
> >      array (size=4)
> >        'id' => string '6905' (length=4)
> >        'studio_id' => string '5' (length=1)
> >        'genres' => string '6|37|' (length=5)
> >
> > However it takes a stupid amount of memory for some unknown reason.
> > MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES
> > MEMORY PEAK USAGE: 216,530,944 BYTES
> >
> > When using the reference the memory drastically goes down to what I'd
> EXPECT
> > it to be (and actually the problem I'm trying to solve).
> > MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES
> > MEMORY PEAK USAGE: 82,051,072 BYTES
> >
> > However my array is all kinds of wrong:
> >
> > array (size=79552)
> >    6904 => &
> >      array (size=4)
> >        'id' => string '86260' (length=5)
> >        'studio_id' => string '210' (length=3)
> >        'genres' => string '8|9|10|29|58|' (length=13)
> >    6905 => &
> >      array (size=4)
> >        'id' => string '86260' (length=5)
> >        'studio_id' => string '210' (length=3)
> >        'genres' => string '8|9|10|29|58|' (length=13)
> >
> > Notice that they're all the same values, although the keys seem right. I
> > don't understand why that happens because
> > foreach($this->tmp_results as $k => $v)
> > Should be changing $v each iteration I'd think.
> >
> > Honestly, I am baffled as to why those unsets() make no difference. All
I
> > can think is that the garbage collector doesn't run. But then I had also
> > tried to force gc() and that still made no difference. *sigh*
> >
> > I had some other cockamamie idea where I'd use the same tmp_results
array
> in
> > a tricky way to avoid a  second array. The concept being I'd add 1
million
> > to the ['id'] (which we want as the new array key), then unset the
> existing
> > sequential key, then when all done, loop through and shift all the keys
by
> 1
> > million thereby they'd be the right index ID. So add one and unset one
> > immediately after. Clever right? 'cept it too made no difference on
> memory.
> > Same thing is happening as above where the gc() isn't running or
something
> > is holding all that memory until the end. *sigh*
> >
> > Then I tried a different way using array_combine() and noticed something
> > very disturbing.
> > http://www.php.net/manual/en/function.array-combine.php
> >
> >
> >         private function _normalize_result_set()
> >         {
> >                if (!$this->tmp_results || count($this->tmp_results) < 1)
> > return;
> >
> >                $D_start_mem_usage = memory_get_usage();
> >                foreach($this->tmp_results as $k => $v)
> >                {
> >                       $id = $v['id'];
> >                       $tmp_keys[] = $id;
> >
> >                       if ($v['genres'])
> >                       {
> >                              $g = explode('|', $v['genres']);
> >                             $this->tmp_results[$k]['g'] = $g; //this
> causes a
> > massive spike in memory usage
> >                       }
> >                }
> >                //var_dump($tmp_keys, $this->tmp_results); exit;
> >                echo "\nMEMORY USED BEFORE array_combine:
> > ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
> > (".number_format(memory_get_peak_usage(true)).")<br>\n";
> >                $this->tmp_results = array_combine($tmp_keys,
> > $this->tmp_results);
> >                echo "\nMEMORY USED FOR array_combine:
> > ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
> > (".number_format(memory_get_peak_usage(true)).")<br>\n";
> >                var_dump($tmp_keys, $this->tmp_results); exit;
> >         }
> >
> > Just the simple act of adding that 'g' variable element to the array
> causes
> > a massive change in memory usage. WHAT THE F!?
> >
> > MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264)
> > MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264)
> >
> > And taking out the
> > $this->tmp_results[$k]['g'] = $g;
> >
> > Results in
> > MEMORY USED BEFORE array_combine: 8,050,456 PEAK: (78,118,912)
> > MEMORY USED FOR array_combine: 8,050,376 PEAK: (86,507,520)
> >
> > Just as a wild guess, I also added 'g' to my SQL so that PHP would
already
> > have a placeholder variable there in tmp_results, but that made no
> > difference. And still used up nearly double the memory as above.
> > SELECT DISTINCT `id`, sag.`genres`, 'g' FROM.
> >
> >
> Are you sure that the data is what you expect?  I've never used an
> object to hold the results of a query, but I'm picturing that your
> foreach may not be working at all unless an object as a result is
> totally different than the type of query results I always process.
> 
> You're taking each object of the query results and assigning the key and
> value to vars.  But aren't query results 'keyless'?  So what are you
> actually assigning to $k and $v?
> 
> If I'm wrong and using an object is quite different, forget I said
> anything.  :)

Positive the data is right. It all works. It's just the memory usage is the
issue.

I'm not assigning an object. I'm assigning an array to a property (array) IN
an object ($this). 

The query itself is using mysql_fetch_array() and MYSQL_ASSOC
http://php.net/manual/en/function.mysql-fetch-array.php

Results are certainly NOT keyless as you wouldn't be able to use them
otherwise.

They are a sequential multi-dimensional array starting at [0] ... [x] with
column names as the sub-array hash keys and column values as the
corresponding hash value.