Re: DateTime performance

On Thursday 03 May 2012 02:10:04 you wrote:
> On 2012.5.1 3:29 PM, Philipp K. Janert wrote:
> > However, when working through a files with a few
> > tens of millions of records, DateTime turns into a
> > REAL drag on performance.
> > 
> > Is this expected behavior? And are there access
> > patterns that I can use to mitigate this effect?
> > (I tried to supply a time_zone explicitly, but that
> > does not seem to improve things significantly.)
> 
> Unfortunately due to the way DateTime is architected it does a lot of
> precalculation upon object instantiation which is usually not used.  So
> yes, it is expected in that sense.

Ok.

> 
> If all you need is date objects with a sensible interface, try
> DateTimeX::Lite.  It claims to replicate a good chunk of the DateTime
> interface in a fraction of the memory.

I'll check it out, thanks.

> 
> Given how much time it takes to make a DateTime object, and your scale of
> tens of millions of records, you could cache DateTime objects for each
> timestamp and use clone() to get a new instance.

I considered that, but in reality, most of my timestamps
are actually different. (There are about 30M seconds in
a year, so I won't have much duplication, looking at 10-50M
records spread over a year...)

> 
>     sub get_datetime {
>         my $timestamp = shift;
> 
>         state $cache = {};
> 
>         if( defined $cache->{$timestamp} ) {
>             return $cache->{$timestamp}->clone;
>         }
>         else {
>             $cache->{$timestamp} =
> make_datetime_from_timestamp($timestamp); return $cache->{$timestamp};
>         }
>     }