Re: XS Module Access to libpq handle

Hi Greg

First, I have to apologize for sending this and then not following up. I 
actually sent it initially using google groups, since accessing NNTP 
from behind the company firewall isn't easily possible. Strangely my 
post didn't seem to turn up when the list was viewed via google groups. 
I had assumed my post didn't make it onto the group, and in fact I sent 
another one just yesterday. Then when I checked with a real NNTP client 
I found my old post! So please ignore the post from yesterday, if it 
shows up at all.

 >
 > What exactly is the slow part? DBD::Pg uses XS extensively as well,
 > so it's good to know about real world limitations where the Perl
 > bumps up against the C. Maybe we can fix that part instead of having
 > to tie yet another language into the mix.
 >

The application has grown over the years, to the point where there are 
now a number of installations, each handling 100's of thousands of files 
a day, parsing their content, processing and storing in the database. 
The database size is in the 10's of TB, and can be processing many GB 
per day. There are a number of transformations and operations performed 
on the data before putting it in the database, and as it has grown we 
have found that a pure perl solution has not been able to keep up.

I doubt you could do much to DBD::Pg to help with this.

 > My first instinct is to make the version a requirement somehow.
 > Not knowing your system, I'm guessing that's problematic.

Yes, we are doing that at the moment, using RPM dependancies. However it 
makes it difficult to accomodate additional host OS variants, and there 
is also the danger that an admin at one of the sites might install 
another version of DBD::Pg from source.

 > At first blush, I don't see any reason to deny that. It's just
 > hooking to the underlying plumbing, there is no escalation of
 > privs or anything. And we could certainly keep it as a
 > super-advanced, here-be-dragons kind of attribute.

OK, I will try some experiments to see if I can come up with a workable 
solution.

In the other post I attempted to make yesterday I suggested three 
possible methods:

  1. Allow access to the PGconn ptr from perl as an attribute or via a 
method
  2. Provide a C accessor function which can be called externally from 
another XS module
  3. Include a header file with DBD::Pg which defines access macros. Of 
course I didn't think that through too well as it really does not solve 
the problem, so perhaps forget that.

My gut feel is to go with option #2, since the raw PG connection is not 
likely to be that useful from Perl and it allows us to have our module 
accept a SV* dbi handle as an argument in a transparent way.

Regards
Mike