Re: DataTable efficiency

As Rico Mariani would say: measure, measure, measure ...

If you need memory efficiency, I would recommend you try to change
your algorithm to make it fully streaming. If that is impossible, then
you can measure the use of DataTable vs the CachedCsvReader class
included in my library and compare. I did not do that measurement
myself, but I would assume CachedCsvReader is more efficient since it
is much more simple (ArrayList of string arrays). On the other hand,
you will lose speed as you need to convert your data from string
everytime you access them. Of course, you can roll your own class
based on mine and store converted values instead of strings (ie
ArrayList of object arrays).

Btw, you may want to try DataTable.Select(...) instead of using
relations. It may or may not be faster, depending on your usage.

S=E9bastien

On 11/27/07, Alan Baljeu <[email protected]> wrote:
> One of the problems I have with ADO.net, is Microsoft forgot to mention e=
xactly how
> efficient their code is.  This doesn't matter for a lot of things, but wi=
th database stuff
> and lots of data, people would like to know.  If I load data into a strea=
m, and then feed
> that stream to a datatable, how long does that take, or what steps does t=
he system follow?
> If I add the two tables and then create a DataRelation, is creating the r=
elation fast or
> slow?  What datastructure is built?  How are the tables indexed?  Is ther=
e some mysterious
> overhead due to the tremendous generality of the system?
>
> Is your CSV reader more efficient because it bypasses ODBC, or is there m=
ore gains here?
> Etc.  It's almost enough to make me want to roll my own, so I know and ca=
n change things if
> I'm not satisfied.
>
>
> > -----Original Message-----
> > From: Discussion relating to the specifics of the C# and
> > Managed C++ languages [mailto:[email protected]]
> > On Behalf Of S=E9bastien Lorion
> > Sent: Friday, November 23, 2007 2:19 AM
> > To: [email protected]
> > Subject: Re: [DOTNET-CX] DataTable efficiency
> >
> > Sorry, just realized that the DataTable.Load method is
> > available only in .NET 2.0.... In that case, you will need to
> > derive a class from DbDataAdapter in order to expose the
> > protected method DbDataAdapter.Fill(DataTable, IDataReader).
> >
> > S=E9bastien
> >
> > On 11/23/07, S=E9bastien Lorion <[email protected]> wrote:
> > > You can create the DataTable before hand and then use the
> > Load method
> > > (the CsvReader implement IDataReader).
> > >
> > > S=E9bastien
> > >
> > > On 11/23/07, Alan Baljeu <[email protected]> wrote:
> > > > I like it in concept, but... Currently I'm using the ODBC
> > driver to
> > > > read these files.  It uses schema.ini to convert
> > specified columns
> > > > to bool or int.  Your code just gives me strings.  Have you any
> > > > suggestion on how I might adapt the CSV reader to parse columns
> > > > according to a schema?  Initially I imagine putting this
> > stuff into DataTables because anything else will require much
> > code conversion.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> This list is hosted by DevelopMentor(R)  http://www.develop.com
>
> View archives and manage your subscription(s) at http://discuss.develop.c=
om
>


--=20
S=E9bastien
www.sebastienlorion.com

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This list is hosted by DevelopMentor=AE  http://www.develop.com

View archives and manage your subscription(s) at http://discuss.develop.com