Re: Insert Query with place holders Vs SQLLDR

[email protected] (Parag Kalra)
Newsgroups perl.dbi.users
Message-ID <[email protected]>
Ok here is the thing.

This afternoon I coded few scripts and created a test setup to validate the
benchmarking results of the various tools and following are the results:

Environment Settings
Operating System: Windoze XP SP3  (sorry couldn't test it Nix as it is a
prod box)
Database: Oracle 10g 10.2.0.1.0
Perl: 5.10.1
DBD-Oracle: 1.21
Total records used for testing: 1048576

Time of insertion taken by sqlldr in conventional mode -                35
seconds
Time of insertion taken by sqldr in direct mode -
06 seconds
Time of insertion taken by DBD::Oracle using the array interface - 46
seconds

PFA the Perl script used for testing. Let me if any of you interested in
viewing the sqlldr control scripts.

Cheers,
Parag


On Sun, May 2, 2010 at 12:27 PM, Jeffrey Seger <[email protected]> wrote:

> My experience is that DBD::Oracle using the array interface is
> comparable performance-wise to SQLLDR in conventional path mode.
> However, SQLLDR can also be called in direct path mode, which is even
> faster, but you should really know what you are doing before using
> that.  It has implications beyond the performance aspect that you need
> to be aware of.
>
> If you want/need/understand using direct path, then SQLLDR is the way
> to go.  If you want better integration with the rest of your program
> then DBD::Oracle is the way to go.
>
> On Sun, May 2, 2010 at 4:20 AM, Parag Kalra <[email protected]> wrote:
> > Hi All,
> >
> > I want to know which one of these algorithm would be the most optimized
> > solution to insert large number of records and Why:
> >
> > 1. Preparing an Insert query once and executing it with place holders for
> > the entire set of records
> > 2. Executing Oracle's sqlldr command and uploading the same set of
> records
> >
> > Cheers,
> > Parag
> >
>
>
>
> --
> "Champions do not become champions when they win the event, but in the
> hours, weeks, months and years they spend preparing for it. The
> victorious performance itself is merely the demonstration of their
> championship character." -T. Alan Armstrong
>
> "The Ow that can be expressed is not the true Ow." - Ao Tzu
>
bind_param_array_file.pl (application/octet-stream, 2.3 KB)
#!/usr/bin/perl
#===============================================================================
#
#         FILE:  bind_param_array_file.pl
#
#        USAGE:  perl bind_param_array_file.pl
#
#  DESCRIPTION:  This script demonstrates how to use bind_param_array in DBI
#                when the data is being read from the file
#                It is a replacement of sqlldr used in conventional path mode
#       AUTHOR:  Parag Kalra, [email protected]
#      VERSION:  1.0
#      CREATED:  02-MAY-2010
# LAST CHANGED:  02-MAY-2010
# CHANGES MADE:  NONE
#===============================================================================

#Required modules
use strict;
use warnings;
use DBI;

# Declaring variables
my $uname="sa";
my $passwd="root123";
my $dbname="orcl";
my $row_count=0;
my (@product_code, @qty, @price, @order_date);

# Connecting to database
my $dbh = DBI->connect("dbi:Oracle:$dbname", $uname, $passwd,{
      PrintError => 0,
      RaiseError => 0
  } ) or die "Could not connect to the database\n";

# Storing the data into bind arrays
while(<>){
    chomp;
	($product_code[$row_count],$qty[$row_count],$price[$row_count],$order_date[$row_count])=split(/\t/);
	$row_count = $row_count+1;
}

my $drop_sql = "drop table PERL_SALES"; # Query to drop existing table
my $create_sql = "create table PERL_SALES (PRODUCT_CODE varchar(255), QTY NUMBER, PRICE FLOAT, ORDER_DATE date)"; # Query to create table
my $sth1 = $dbh->do(qq{ $drop_sql  }) ; #Droping table
my $sth2 = $dbh->do(qq{ $create_sql}) ; #Creating table

# Query to insert fetched records into SALES
my $INS = qq(insert into PERL_SALES(PRODUCT_CODE, QTY, PRICE, ORDER_DATE) values(?,?,?,?));
my $ins = $dbh->prepare($INS); #Preparing the query to insert records into child test table

# Using bind_param_array to execute large records using less inserts
$ins->bind_param_array( 1, \@product_code );
$ins->bind_param_array( 2, \@qty );
$ins->bind_param_array( 3, \@price );
$ins->bind_param_array( 4, \@order_date );
$ins->execute_array( { ArrayTupleStatus => \my @tuple_status } );

my $record_count = $row_count + 1;
print "\nTotal records processed: $record_count\n";

# This calculates the time of execution of code.
END {warn "\nTime of execution - ", time - $^T, " second(s)\n"}
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.