RE: Parser-sig digest, Vol 1 #17 - 1 msg

Fair enough, the question in more direct terms...

Today's "ETL" (extract/transform/load) tools have gotten to be rather =
complex and expensive affairs, creating a gap between simple file =
processing (which can be handled by custom "one-off" programs) and =
complex file processing (denormalization/renormalization, reorganization =
into star schemas, distributed/parallel processing, etc.).  I am looking =
for tools that would fill this gap, ie. tools/libraries/whatever that =
support reuse and configurability, avoiding hard-coded data parsing, are =
moderately fast, but not exotic.

My vision is that there is a library which makes the parsing of data =
files declarative instead of procedural. One would declare the =
types/sizes/delimiters of fixed/variable length attributes within =
records (sometimes binary, sometimes text), creating Dict's or other =
structures facilitating the easy coding of transformation functions. The =
reverse would also be available, ie. given a structure with processed =
data and a declaration of the output format, this library would render =
the data structure into the format.

Another vision (and the reason I decided to post to this list) is that, =
instead of a library, there is a parser generator that would create the =
appropriate read/write statements from a configuration file (or data =
parsing language, if you will). In this way, the parser generator is =
writing most of the I/O code for me, again freeing me to concentrate on =
transformation logic. The generated language could be Python, or perhaps =
C which is subsequently wrapped by SWIG, or whatever; the key is not to =
have to code the procedures but rather to code the config file, which =
would be alot easier to validate and maintain.

This problem is undoubtedly solved every day when using Python to =
integrate disparate systems; I'm wondering if anyone has abstracted this =
process so the tedium of coding the read/write/conversion logic is =
minimized. I think of this problem as analogous to pickling/unpickling, =
however the format is externally specified, not internal to Python. I =
need to manipulate data in EBCDIC as well as ASCII, big-endian as well =
as little-endian, and stay close to the legacy formats.

Hope this explains my=20

-----Original Message-----
From: [email protected]
[mailto:[email protected]]
Sent: Saturday, December 14, 2002 10:00 AM
To: [email protected]
Subject: Parser-sig digest, Vol 1 #17 - 1 msg


Send Parser-sig mailing list submissions to
	[email protected]

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.python.org/mailman/listinfo/parser-sig
or, via email, send a message with subject or body 'help' to
	[email protected]

You can reach the person managing the list at
	[email protected]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Parser-sig digest..."


Today's Topics:

   1. Re: Rapid binary data stream parsing (Ilpo =
=3D?iso-8859-1?q?Nyyss=3DF6nen?=3D)

--__--__--

Message: 1
To: [email protected]
Subject: Re: [Parser-sig] Rapid binary data stream parsing
From: [email protected] (Ilpo =3D?iso-8859-1?q?Nyyss=3DF6nen?=3D)
Organization: ypy - it's just a name
Date: Fri, 13 Dec 2002 20:36:45 +0200

"Wagner, Paul" <[email protected]> writes:

> I joined this list in the hopes that my question is "in scope" for
> this list; please refer me elsewhere if appropriate. My issue is with
> the generation of parsers that rapidly parse/reformat data, not
> programs; is this an issue that can be discussed here?

1) This list is almost too quiet, yes.
2) That is parsing.
3) Python has some part in the problem or the solution?
4) Normally it is better to ask directly the actual question.

And no, I am not any authority, but I think that I will be the fastest
to answer. :)

--=3D20
Ilpo Nyyss=3DF6nen # biny # /* :-) */



--__--__--

_______________________________________________
Parser-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/parser-sig


End of Parser-sig Digest