Re: New DBD::File feature - can we do more with it?

On 06/29/10 19:58, Darren Duncan wrote:
> Jens Rehsack wrote:
>> because of a bogus implementation for PRECISION, NULLABLE etc.
>> attributes in DBD::CSV (forced by limitations of SQL::Statement)
>> we have a new attribute for tables in DBD::File meta data:
>> 'table_defs'. This is filled when a 'CREATE TABLE ...' is executed
>> and copies the $stmt->{table_defs} structure (containing column
>> name and some more information - what ever could be specified
>> using (ANSI) SQL to create tables).
>>
>> Could it makes sense to have a DBD::File supported way to store
>> and load this meta-data (serialized, of course)?
>>
>> I would really like to do this - this would bring us a big step
>> in the right direction.
>>
>> DBD::DBM could store it in it's meta-data (instead of saving column
>> names it could safe the entire table_defs structure), but what should
>> DBD::CSV do?
>
> You have several possible options and the short answer is "let the user
> tell you".

This doesn't answer the real question:

What's the best way to provide an implementation and how far should the 
DBD::File (part of DBI, remember) should be done?

> For backwards or sideways compatibility, don't store any extra metadata
> by default; the only metadata is what can be gleaned from the normal CSV
> file itself.

Fair.

> Then, if the user tells you to via extra DBI config args, then retrieve
> or store extra metadata in accordance with those args. For example, the
> extra args may give you a second file name in which the meta data is /
> is to be stored. Or the extra args may indicate that one of the first
> rows in the CSV file contains metadata rather than normal data.
>
> For example, if all of your metadata is column-specific, the CSV file
> could contain 2 heading rows instead of the usual one (or none)? The
> first heading row would be the name of the column as usual. The new
> second heading row would be the encoded type/constraint/etc definition
> that says how to interpret the data of the column, and then the
> third-plus columns are the data.

These are both interesting points - but I'm unsure if FreezeThaw can
be used for serialization for the 2nd option - JSON looks like a better
candidate, but is not in core.

> But the point is, let the user tell you how to interpret the particular
> CSV files they throw at you, and so DBD::CSV is then more flexible and
> compatible.

User can only choose provided choices ;)
What is really interesting to me is to discuss the different choices,
how they impact the different modules DBD::File, DBD::CSV, DBD::DBM,
DBD::PO, later DBD::AnyData and what might come, too.

Currently - especially for DBD::AnyData, DBI::DBD::SqlEngine is limited.
The $m->{table_defs} support belongs to there - as well as some other
table meta data handling.

Maybe a kind of table API extension (optionally, of course) would be
a nice idea to discuss, too.

Jens