Re: [PHP] Re: refernces, arrays, and why does it take up so much memory?

On 9/3/2013 1:09 AM, Daevid Vincent wrote:
>
>
>> -----Original Message-----
>> From: Jim Giner [mailto:[email protected]]
>> Sent: Monday, September 02, 2013 8:14 PM
>> To: [email protected]
>> Subject: [PHP] Re: refernces, arrays, and why does it take up so much
>> memory?
>>
>> On 9/2/2013 9:30 PM, Daevid Vincent wrote:
>>> I'm confused on how a reference works I think.
>>>
>>> I have a DB result set in an array I'm looping over. All I simply want
> to
>> do
>>> is make the array key the "id" of the result set row.
>>>
>>> This is the basic gist of it:
>>>
>>>          private function _normalize_result_set()
>>>          {
>>>                 foreach($this->tmp_results as $k => $v)
>>>                 {
>>>                        $id = $v['id'];
>>>                        $new_tmp_results[$id] =& $v; //2013-08-29 [dv]
> using
>> a
>>> reference here cuts the memory usage in half!
>>>                        unset($this->tmp_results[$k]);
>>>
>>>                        /*
>>>                        if ($i++ % 1000 == 0)
>>>                        {
>>>                              gc_enable(); // Enable Garbage Collector
>>>                              var_dump(gc_enabled()); // true
>>>                              var_dump(gc_collect_cycles()); // # of
>> elements
>>> cleaned up
>>>                              gc_disable(); // Disable Garbage Collector
>>>                        }
>>>                        */
>>>                 }
>>>                 $this->tmp_results = $new_tmp_results;
>>>                 //var_dump($this->tmp_results); exit;
>>>                 unset($new_tmp_results);
>>>          }
>>>
>>> Without using the =& reference, my data works great:
>>> $new_tmp_results[$id] = $v;
>>>
>>> array (size=79552)
>>>     6904 =>
>>>       array (size=4)
>>>         'id' => string '6904' (length=4)
>>>         'studio_id' => string '5' (length=1)
>>>         'genres' => string '34|' (length=3)
>>>     6905 =>
>>>       array (size=4)
>>>         'id' => string '6905' (length=4)
>>>         'studio_id' => string '5' (length=1)
>>>         'genres' => string '6|37|' (length=5)
>>>
>>> However it takes a stupid amount of memory for some unknown reason.
>>> MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES
>>> MEMORY PEAK USAGE: 216,530,944 BYTES
>>>
>>> When using the reference the memory drastically goes down to what I'd
>> EXPECT
>>> it to be (and actually the problem I'm trying to solve).
>>> MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES
>>> MEMORY PEAK USAGE: 82,051,072 BYTES
>>>
>>> However my array is all kinds of wrong:
>>>
>>> array (size=79552)
>>>     6904 => &
>>>       array (size=4)
>>>         'id' => string '86260' (length=5)
>>>         'studio_id' => string '210' (length=3)
>>>         'genres' => string '8|9|10|29|58|' (length=13)
>>>     6905 => &
>>>       array (size=4)
>>>         'id' => string '86260' (length=5)
>>>         'studio_id' => string '210' (length=3)
>>>         'genres' => string '8|9|10|29|58|' (length=13)
>>>
>>> Notice that they're all the same values, although the keys seem right. I
>>> don't understand why that happens because
>>> foreach($this->tmp_results as $k => $v)
>>> Should be changing $v each iteration I'd think.
>>>
>>> Honestly, I am baffled as to why those unsets() make no difference. All
> I
>>> can think is that the garbage collector doesn't run. But then I had also
>>> tried to force gc() and that still made no difference. *sigh*
>>>
>>> I had some other cockamamie idea where I'd use the same tmp_results
> array
>> in
>>> a tricky way to avoid a  second array. The concept being I'd add 1
> million
>>> to the ['id'] (which we want as the new array key), then unset the
>> existing
>>> sequential key, then when all done, loop through and shift all the keys
> by
>> 1
>>> million thereby they'd be the right index ID. So add one and unset one
>>> immediately after. Clever right? 'cept it too made no difference on
>> memory.
>>> Same thing is happening as above where the gc() isn't running or
> something
>>> is holding all that memory until the end. *sigh*
>>>
>>> Then I tried a different way using array_combine() and noticed something
>>> very disturbing.
>>> http://www.php.net/manual/en/function.array-combine.php
>>>
>>>
>>>          private function _normalize_result_set()
>>>          {
>>>                 if (!$this->tmp_results || count($this->tmp_results) < 1)
>>> return;
>>>
>>>                 $D_start_mem_usage = memory_get_usage();
>>>                 foreach($this->tmp_results as $k => $v)
>>>                 {
>>>                        $id = $v['id'];
>>>                        $tmp_keys[] = $id;
>>>
>>>                        if ($v['genres'])
>>>                        {
>>>                               $g = explode('|', $v['genres']);
>>>                              $this->tmp_results[$k]['g'] = $g; //this
>> causes a
>>> massive spike in memory usage
>>>                        }
>>>                 }
>>>                 //var_dump($tmp_keys, $this->tmp_results); exit;
>>>                 echo "\nMEMORY USED BEFORE array_combine:
>>> ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
>>> (".number_format(memory_get_peak_usage(true)).")<br>\n";
>>>                 $this->tmp_results = array_combine($tmp_keys,
>>> $this->tmp_results);
>>>                 echo "\nMEMORY USED FOR array_combine:
>>> ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK:
>>> (".number_format(memory_get_peak_usage(true)).")<br>\n";
>>>                 var_dump($tmp_keys, $this->tmp_results); exit;
>>>          }
>>>
>>> Just the simple act of adding that 'g' variable element to the array
>> causes
>>> a massive change in memory usage. WHAT THE F!?
>>>
>>> MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264)
>>> MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264)
>>>
>>> And taking out the
>>> $this->tmp_results[$k]['g'] = $g;
>>>
>>> Results in
>>> MEMORY USED BEFORE array_combine: 8,050,456 PEAK: (78,118,912)
>>> MEMORY USED FOR array_combine: 8,050,376 PEAK: (86,507,520)
>>>
>>> Just as a wild guess, I also added 'g' to my SQL so that PHP would
> already
>>> have a placeholder variable there in tmp_results, but that made no
>>> difference. And still used up nearly double the memory as above.
>>> SELECT DISTINCT `id`, sag.`genres`, 'g' FROM.
>>>
>>>
>> Are you sure that the data is what you expect?  I've never used an
>> object to hold the results of a query, but I'm picturing that your
>> foreach may not be working at all unless an object as a result is
>> totally different than the type of query results I always process.
>>
>> You're taking each object of the query results and assigning the key and
>> value to vars.  But aren't query results 'keyless'?  So what are you
>> actually assigning to $k and $v?
>>
>> If I'm wrong and using an object is quite different, forget I said
>> anything.  :)
>
> Positive the data is right. It all works. It's just the memory usage is the
> issue.
>
> I'm not assigning an object. I'm assigning an array to a property (array) IN
> an object ($this).
>
> The query itself is using mysql_fetch_array() and MYSQL_ASSOC
> http://php.net/manual/en/function.mysql-fetch-array.php
>
> Results are certainly NOT keyless as you wouldn't be able to use them
> otherwise.
>
> They are a sequential multi-dimensional array starting at [0] ... [x] with
> column names as the sub-array hash keys and column values as the
> corresponding hash value.
>
Hard to know that from your code sample.

Please ignore my my post.  :)

Although - why not just use the query results instead of first putting 
them into an array only to put some of them (?) into a second array?