Re: Insert Query with place holders Vs SQLLDR
[email protected] (Parag Kalra)
| Newsgroups | perl.dbi.users |
|---|---|
| Message-ID | <[email protected]> |
Ok here is the thing. This afternoon I coded few scripts and created a test setup to validate the benchmarking results of the various tools and following are the results: Environment Settings Operating System: Windoze XP SP3 (sorry couldn't test it Nix as it is a prod box) Database: Oracle 10g 10.2.0.1.0 Perl: 5.10.1 DBD-Oracle: 1.21 Total records used for testing: 1048576 Time of insertion taken by sqlldr in conventional mode - 35 seconds Time of insertion taken by sqldr in direct mode - 06 seconds Time of insertion taken by DBD::Oracle using the array interface - 46 seconds PFA the Perl script used for testing. Let me if any of you interested in viewing the sqlldr control scripts. Cheers, Parag On Sun, May 2, 2010 at 12:27 PM, Jeffrey Seger <[email protected]> wrote: > My experience is that DBD::Oracle using the array interface is > comparable performance-wise to SQLLDR in conventional path mode. > However, SQLLDR can also be called in direct path mode, which is even > faster, but you should really know what you are doing before using > that. It has implications beyond the performance aspect that you need > to be aware of. > > If you want/need/understand using direct path, then SQLLDR is the way > to go. If you want better integration with the rest of your program > then DBD::Oracle is the way to go. > > On Sun, May 2, 2010 at 4:20 AM, Parag Kalra <[email protected]> wrote: > > Hi All, > > > > I want to know which one of these algorithm would be the most optimized > > solution to insert large number of records and Why: > > > > 1. Preparing an Insert query once and executing it with place holders for > > the entire set of records > > 2. Executing Oracle's sqlldr command and uploading the same set of > records > > > > Cheers, > > Parag > > > > > > -- > "Champions do not become champions when they win the event, but in the > hours, weeks, months and years they spend preparing for it. The > victorious performance itself is merely the demonstration of their > championship character." -T. Alan Armstrong > > "The Ow that can be expressed is not the true Ow." - Ao Tzu >
bind_param_array_file.pl
(application/octet-stream, 2.3 KB)
#!/usr/bin/perl #=============================================================================== # # FILE: bind_param_array_file.pl # # USAGE: perl bind_param_array_file.pl # # DESCRIPTION: This script demonstrates how to use bind_param_array in DBI # when the data is being read from the file # It is a replacement of sqlldr used in conventional path mode # AUTHOR: Parag Kalra, [email protected] # VERSION: 1.0 # CREATED: 02-MAY-2010 # LAST CHANGED: 02-MAY-2010 # CHANGES MADE: NONE #=============================================================================== #Required modules use strict; use warnings; use DBI; # Declaring variables my $uname="sa"; my $passwd="root123"; my $dbname="orcl"; my $row_count=0; my (@product_code, @qty, @price, @order_date); # Connecting to database my $dbh = DBI->connect("dbi:Oracle:$dbname", $uname, $passwd,{ PrintError => 0, RaiseError => 0 } ) or die "Could not connect to the database\n"; # Storing the data into bind arrays while(<>){ chomp; ($product_code[$row_count],$qty[$row_count],$price[$row_count],$order_date[$row_count])=split(/\t/); $row_count = $row_count+1; } my $drop_sql = "drop table PERL_SALES"; # Query to drop existing table my $create_sql = "create table PERL_SALES (PRODUCT_CODE varchar(255), QTY NUMBER, PRICE FLOAT, ORDER_DATE date)"; # Query to create table my $sth1 = $dbh->do(qq{ $drop_sql }) ; #Droping table my $sth2 = $dbh->do(qq{ $create_sql}) ; #Creating table # Query to insert fetched records into SALES my $INS = qq(insert into PERL_SALES(PRODUCT_CODE, QTY, PRICE, ORDER_DATE) values(?,?,?,?)); my $ins = $dbh->prepare($INS); #Preparing the query to insert records into child test table # Using bind_param_array to execute large records using less inserts $ins->bind_param_array( 1, \@product_code ); $ins->bind_param_array( 2, \@qty ); $ins->bind_param_array( 3, \@price ); $ins->bind_param_array( 4, \@order_date ); $ins->execute_array( { ArrayTupleStatus => \my @tuple_status } ); my $record_count = $row_count + 1; print "\nTotal records processed: $record_count\n"; # This calculates the time of execution of code. END {warn "\nTime of execution - ", time - $^T, " second(s)\n"}