Re: Hello and a comment - speed

On Tue, Sep 13, 2005 at 11:54:44AM -0700, Henri Asseily wrote:
> Hello to the list,
> 
> I read the small archive, and regarding the first steps and goals I'd  
> like to propose that some type of speed consideration be inserted in there.

(So the last 12 years of my talking about speed isn't enough ;-)

Seriously though, I was surprised when I looked again at the goals to
see that some mention of performance wasn't included, even for DBI v1.

> While Jochen does think that we can be somewhat lenient with speed,  
> when I look to the future I see some of us (certainly myself) going  
> further into extremely high performance systems. Those systems are  
> characterized by efficient in-memory databases for very fast lookups  
> (full-fledged RDBMS or just centralized hash tables) and high-speed  
> interconnects (infiniband & co.) that reduce communication latency.
> If the DBI starts to lose some of its stellar performance, it will be  
> less suited to become the de-facto standard for communication with  
> these systems, and we'll have to build custom APIs (which I currently  
> do) as opposed to simpler drivers. I'm not saying that the DBI should  
> be a general access API for anything remotely resembling databases,  
> but I'd like to put this on the table for discussion.
> Does anyone think that it makes sense to extend the DBI goals to  
> encompass more esoteric database/storage systems? If so, then speed  
> will be a prime concern on the fringes.

I think the speed of the DBI is very important even for non-esoteric
data sources. If the DBI is inefficient then everything built on it
is inefficient.

While CPU speeds and L1 and L2 cache speeds have increased dramatically
(and significantly faster than typical disk i/o speeds), the amount of
data being processed has also increased dramatically.

The *key* optimization for DBI v1 was that the same row array, with the
same scalar values within the array elements, is reused for each row.
That means the driver just has to call sv_setiv() (or sv_setpv() etc)
for each field and after the first row those calls are updating the
*existing pre-typed and sized SVs* very cheaply.

With Parrot the same kinds of optimizations are possible and I have no
doubt they'll be used - along with anything else we can think of.

However, having said all that, I also want to try to push for greater
formality and correctness (whatever you want those to mean).
So I'm also keen to explore use of whatever functionality Perl6 has to
offer by way of type-safety and parameter checking etc etc.

I don't know what those will be yet, or what run-time cost they'll have.
Anything with significant run-time cost, especially for fetch loops,
will be avoided or just off by default.

Tim.