Re: Processing large data files.

On 04/06/2007 03:27 PM, Bill Stephenson wrote:
> Perl crashes when using the included script for large data files. How 
> can I prevent this from happening?
> 

What do you mean by crashes? Does it abort with an error message placed 
in the web-server log file? If so, that's not a crash; it's just an 
error abort of the script.


> Kindest Regards,
> 
> -- 
> Bill Stephenson
> 
> 
> Sample data source is a Delimited text file with the following fields:
> 
> FMT:STORE:TIKEY:TITLE:QTY:DC:COMPANY:ZONE
> 
> DV:043712:20882903:DVD-ALL THE KING'S M('06/WS/2D:8.00000:ECDC:Hlyw:4
> DV:043712:20883071:DVD-ALTERED (2006/WS):8.00000:ECDC:Hlyw:4
> DV:043712:20883070:DVD-AMERICAN PIE 5-NAKE(UR/WS:32.00000:ECDC:Hlyw:4
> DV:043712:20883253:DVD-BROOKLYN LOBSTER (WS):2.00000:ECDC:Hlyw:4
> DV:043704:20882903:DVD-ALL THE KING'S M('06/WS/2D:16.00000:WCDC:Hlyw:4
> DV:043704:20883071:DVD-ALTERED (2006/WS):8.00000:WCDC:Hlyw:4
> DV:043693:20882903:DVD-ALL THE KING'S M('06/WS/2D:24.00000:ECDC:Hlyw:4
> DV:043693:20883071:DVD-ALTERED (2006/WS):8.00000:ECDC:Hlyw:4
> DV:043693:20883070:DVD-AMERICAN PIE 5-NAKE(UR/WS:24.00000:ECDC:Hlyw:4
> DV:043693:20883092:DVD-ZOMBIE NATION (2006/WS):4.00000:ECDC:Hlyw:4
> DV:043670:20882903:DVD-ALL THE KING'S M('06/WS/2D:8.00000:ECDC:Hlyw:4
> 
> <code>
> 
> #!/usr/bin/perl
> 
> use strict;
> use warnings;
> 
> # set-up our variable....
> 
> my $source = "dataSource.txt";
> my $destination = "testOUTPUT.txt";
> 
> my $fmt = "";
> my $storeno = "";
> my $co = "";
> my $dc = "";
> my $zone = "";
> my $store = "";
> my $tikey = "";
> my $title = "";
> my $qty = "";
> my $text1 = "";
> my $text2 = "";
> my $output = "";
> 
> my @stores = "";

This creates an array with one element, an empty string. Usually people 
just want to create an empty array which can be created like so:

my @stores = ();

> 
> 
> # start the work...
> 
> open my $IN, $source or die "Can't read source file $source: $!\n";
> 
> # put all the stores into an array (@stores)...
> my @lines = <$IN>;
> 
> foreach my $line(@lines) {
> 

This reads all of the lines into memory at once. Whoa! You almost 
certainly want to use a while loop to read the lines one-by-one:

my @lines;
while (my $line = <$IN>) {

Or even better, use Tie::File to treat the lines in the file as an 
array. That would almost certainly suit you better since your code uses 
an array of lines that you search through more than once.

tie my @lines, 'Tie::File', $source
   or die "Can't read file $source: $!\n";

Read the manual pages for Tie::File and perltie:
Start->Run->"perldoc Tie::File"
Start->Run->"perldoc perltie"


>     ($fmt, $store, $tikey, $title, $qty, $dc, $co, $zone) = 
> split(/:/,$line);
>        
>     push @stores, $store;
>     
> }
> 
> 
> # remove duplicate stores from the list...
> my %seen = ();
> my @uniq = ();
> 
> foreach my $item (@stores) {
> 
>     unless ($seen{$item}) {
>         # if we get here, we have not seen it before
>         $seen{$item} = 1;
>         push(@uniq, $item);
>     }
> }
> 
> # for each store in our @uniq store list we look for
> # lines in our file that have a matching store
> 
> foreach my $uniqStore (@uniq) {
> 
>     foreach my $line(@lines) {
>     
>         ($fmt, $store, $tikey, $title, $qty, $dc, $co, $zone) = 
> split(/:/,$line);
>     
> # if they match we create our entry for the destination file....
>         if ($store eq $uniqStore) {
>        
>             $text1= "Street Date $source \n". "Company $co \n". 
> "Distribution Center $dc \n". "Zone $zone \n". "Store Number $store \n\n";
>        
>             $text2 .= "$tikey \t $title \t $qty\n";
>            
>             }
>     
>      }
>     
> # then add it to the final $output.
>     if ($text1 ne "") {
>     $output .= 
> "$text1$text2\n####################\n####################\n\n\f\n";
>     $text2 = "";
>     }
> }
> 
> # and finally print all out
> 
> # print $output; #for debugging
> 
> open(my $DATA, "> $destination")
>     or die (" Error 1.3 Couldn't file:: $destination for writing: \n");
> 
> print $DATA ($output);
> 
> close $DATA;
> 
> 

Once you start using Tie::File, your program will be half its current 
size and much faster.