Re: refernces, arrays, and why does it take up so much memory?

On 9/2/2013 9:30 PM, Daevid Vincent wrote:
> I'm confused on how a reference works I think.
>
> I have a DB result set in an array I'm looping over. All I simply want to do
> is make the array key the "id" of the result set row.
>
> This is the basic gist of it:
>
>         private function _normalize_result_set()
>         {
>                foreach($this->tmp_results as $k => $v)
>                {
>                       $id = $v['id'];
>                       $new_tmp_results[$id] =& $v; //2013-08-29 [dv] using a
> reference here cuts the memory usage in half!
>                       unset($this->tmp_results[$k]);
>
>                       /*
>                       if ($i++ % 1000 == 0)
>                       {
>                             gc_enable(); // Enable Garbage Collector
>                             var_dump(gc_enabled()); // true
>                             var_dump(gc_collect_cycles()); // # of elements
> cleaned up
>                             gc_disable(); // Disable Garbage Collector
>                       }
>                       */
>                }
>                $this->tmp_results = $new_tmp_results;
>                //var_dump($this->tmp_results); exit;
>                unset($new_tmp_results);
>         }
>
> Without using the =& reference, my data works great:
> $new_tmp_results[$id] = $v;
>
> array (size=79552)
>    6904 =>
>      array (size=4)
>        'id' => string '6904' (length=4)
>        'studio_id' => string '5' (length=1)
>        'genres' => string '34|' (length=3)
>    6905 =>
>      array (size=4)
>        'id' => string '6905' (length=4)
>        'studio_id' => string '5' (length=1)
>        'genres' => string '6|37|' (length=5)
>
> However it takes a stupid amount of memory for some unknown reason.
> MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES
> MEMORY PEAK USAGE: 216,530,944 BYTES
>
> When using the reference the memory drastically goes down to what I'd EXPECT
> it to be (and actually the problem I'm trying to solve).
> MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES
> MEMORY PEAK USAGE: 82,051,072 BYTES
>
> However my array is all kinds of wrong:
>
> array (size=79552)
>    6904 => &
>      array (size=4)
>        'id' => string '86260' (length=5)
>        'studio_id' => string '210' (length=3)
>        'genres' => string '8|9|10|29|58|' (length=13)
>    6905 => &
>      array (size=4)
>        'id' => string '86260' (length=5)
>        'studio_id' => string '210' (length=3)
>        'genres' => string '8|9|10|29|58|' (length=13)
>
> Notice that they're all the same values, although the keys seem right. I
> don't understand why that happens because
> foreach($this->tmp_results as $k => $v)
> Should be changing $v each iteration I'd think.
>
> Honestly, I am baffled as to why those unsets() make no difference. All I
> can think is that the garbage collector doesn't run. But then I had also
> tried to force gc() and that still made no difference. *sigh*
>
> I had some other cockamamie idea where I'd use the same tmp_results array in
> a tricky way to avoid a  second array. The concept being I'd add 1 million
> to the ['id'] (which we want as the new array key), then unset the existing
> sequential key, then when all done, loop through and shift all the keys by 1
> million thereby they'd be the right index ID. So add one and unset one
> immediately after. Clever right? 'cept it too made no difference on memory.
> Same thing is happening as above where the gc() isn't running or something
> is holding all that memory until the end. *sigh*
>
> Then I tried a different way using array_combine() and noticed something
> very disturbing.
> http://www.php.net/manual/en/function.array-combine.php
>
>
>         private function _normalize_result_set()
>         {
>                if (!$this->tmp_results || count($this->tmp_results) < 1)
> return;
>
>                $D_start_mem_usage = memory_get_usage();
>                foreach($this->tmp_results as $k => $v)
>                {
>                       $id = $v['id'];
>                       $tmp_keys[] = $id;
>
>                       if ($v['genres'])
>                       {
>                              $g = explode('|', $v['genres']);
>                             $this->tmp_results[$k]['g'] = $g; //this causes a
> massive spike in memory usage
>                       }
>                }
>                //var_dump($tmp_keys, $this->tmp_results); exit;
>                echo "\nMEMORY USED BEFORE array_combine:
> ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
> (".number_format(memory_get_peak_usage(true)).")<br>\n";
>                $this->tmp_results = array_combine($tmp_keys,
> $this->tmp_results);
>                echo "\nMEMORY USED FOR array_combine:
> ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
> (".number_format(memory_get_peak_usage(true)).")<br>\n";
>                var_dump($tmp_keys, $this->tmp_results); exit;
>         }
>
> Just the simple act of adding that 'g' variable element to the array causes
> a massive change in memory usage. WHAT THE F!?
>
> MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264)
> MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264)
>
> And taking out the
> $this->tmp_results[$k]['g'] = $g;
>
> Results in
> MEMORY USED BEFORE array_combine: 8,050,456 PEAK: (78,118,912)
> MEMORY USED FOR array_combine: 8,050,376 PEAK: (86,507,520)
>
> Just as a wild guess, I also added 'g' to my SQL so that PHP would already
> have a placeholder variable there in tmp_results, but that made no
> difference. And still used up nearly double the memory as above.
> SELECT DISTINCT `id`, sag.`genres`, 'g' FROM.
>
>
Are you sure that the data is what you expect?  I've never used an 
object to hold the results of a query, but I'm picturing that your 
foreach may not be working at all unless an object as a result is 
totally different than the type of query results I always process.

You're taking each object of the query results and assigning the key and 
value to vars.  But aren't query results 'keyless'?  So what are you 
actually assigning to $k and $v?

If I'm wrong and using an object is quite different, forget I said 
anything.  :)