Re: Skip rows (number of rows?), column type in library(csv)

On 10/29/2013 09:56 AM, Boris Vassilev wrote:
> Hello Jan (and others),
>
> finally getting to the suggested additions to library(csv), but having
> looked at the existing functionality and implementation, I need more
> feedback.
>
> If the user wants to provide a term that describes the rows, as the
> option `term`, in the form:
>
> --
> term=row(col1_spec, col2_spec, ..., coln_spec)
> --
> (my original proposal)
>
> then how should relevant existing options be handled?
>
> - `match_arity`
>    * if `match_arity` is false, ignore the `term` option *for the
> offending rows only*

I'm tempted to have the term option imply match_arity(true).  If you
have an insane CSV file with inconsistent rows, you'll have to massage
the data anyway.

> - `arity`:
>    * if provided, and nonneg, and `match_arity` is true, then it must be
> equal to the number of arguments of the provided term; domain_error
> otherwise?

Same story, term implies arity(Arity), where Arity is the arity of the
term.  You can check for these options if term is specified and throw
an error if they are incosistent.  Not really sure what error one should
throw then.  Possibly a domain_error?  There is (AFAIK) no ISO error
that states that two options/arguments are incompatible (i.e., neither
is wrong, but the combination is).

>    * if a variable, and `match_arity` is true, unify with the arity of
> all rows
>
> Small aside: I know that at the moment I can't say:
>
> --
> csv_read_file(Filename, Rows, [arity(A)]).
> --
>
> but I don't understand why not.

Would make sense to make that work.  Just debug :-)

> - `convert`
>    * have a `default` column specification that then falls back to the
> action of `convert` (if `convert` is true, try to make a number out of
> the value)?
>
> I have to admit, I did not exactly understand Jan's suggestion how to
> deal with the option `term`. If I understood correctly, add a multi-file
> predicate `csv:convert_field/3` to library(csv) that has definitions for
> `integer`, `float`, `number`, `atom`, and `default` (see above for
> `default`'s meaning):
>
> --
> convert_field(integer, Field, Int) :- ...
> convert_field(float, Field, Float) :- ...
> ...
> --
>
> but allow the user to add clauses to `convert_field/3` in the calling
> module, with the first argument an atom, and use these atoms in the
> description of the `term` option to `csv_read_file/3`.

I don't really recall this, but there are roughly two options: use a
multifile definition OR call the converter in the module from which
you call the csv predicate.  The multifile option is probably easier
to implement and I think that that will suffice.

>
> Thank you for the feedback,
> Boris
> P.S. Someone else would have written the code in the time it took me to
> type in this email.

	Hope this helps

		--- Jan

>
>
> On Mon, Oct 7, 2013 at 4:44 PM, Jan Wielemaker <[email protected]
> <mailto:[email protected]>> wrote:
>
>     On 09/24/2013 10:05 AM, Boris Vassilev wrote:
>
>         Hello,
>
>         I am using the (highly useful) library(csv) and I had to do two
>         tricks, in order to:
>         1. Skip the header;
>         2. Control (and validate) the datatype of the columns.
>
>         It is easiest to explain with the code:
>
>         load_csv.pl <http://load_csv.pl>
>         ==
>         :- module(load_csv,
>               [load_csv_file/4
>               ]).
>
>         :- use_module(library(csv)).
>
>         load_csv_file(File, Functor, Skip, ColumnTypes) :-
>               FirstLine is Skip + 1,
>               forall(
>                   load_table_row(File, Functor, ColumnTypes, Line, Row),
>                   (   Line >= FirstLine
>                   ->  assertz(Row)
>                   ;   true
>                   )
>               ).
>
>         load_table_row(File, Functor, ColumnTypes, Line, Row) :-
>               csv_read_file_row(
>                   File,
>                   R,
>                   [convert(false),line(Line)]
>               ),
>               R =.. [row|RawColumns],
>               maplist(convert_field, ColumnTypes, RawColumns, Columns),
>               Row =.. [Functor|Columns].
>
>         convert_field(atom, Field, Field).
>         convert_field(number, Field, Number) :-
>               atom_number(Field, Number).
>         % add additional types as needed
>         ==
>
>         Is there a way to achieve this using the available options that
>         I didn't see?
>
>         Does anyone thinks it would be useful to add options to the
>         library for
>         A. Skipping N lines at the beginning;
>         B. Loading at most N lines;
>         C. Pass a list of datatypes for validation and convertion of the
>         data fields?
>
>         In other words, adding options
>         A. 'skip(+Integer)' with a default 0;
>         B. 'nrows(+Integer)' with a default -1 meaning all (or maybe
>         'infinite'?)
>
>
>     Yes.  Maybe call them offset(+Skip) and limit(+Count) (using the
>     common database jargon?)  I'd use =infinite= rather than -1.
>
>
>         C. 'col_types(+List:atom)' which could be for example 'atom',
>         'number', 'integer', 'float', etc. There is a 'convert(+Boolean)'
>         option at the moment but it only gets in the way if the column has
>         values like "00", "01", ..., "A0", ..., "QZ", etc.
>
>
>     Yes.  Only, this is calling convert_field/3.  This should properly
>     call in the calling module (making the CSV predicates meta-predicates).
>     I'd call the call-back cvs_convert_field/3 and maybe add an option
>     to specify it.  An alternative might be to call cvs:convert_field/3,
>     providing a sensible default implementation which first calls a
>     multifile hook.
>
>
>         If there is any interest and we agree on the interface I can
>         submit a
>         proposed implementation.
>
>
>     If properly implemented and documented, I'm happy to apply the patch.
>
>              Thanks --- Jan
>
>
>
>         Cheers,
>         Boris
>         _________________________________________________
>         SWI-Prolog mailing list
>         [email protected].__de
>         <mailto:[email protected]>
>         https://lists.iai.uni-bonn.de/__mailman/listinfo.cgi/swi-__prolog <https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog>
>
>
>