Re: DataTable efficiency

If your data contains lots of duplicate strings and is going to hang
around in memory for a while, it may be worthwhile to implement custom
string interning as you read the data.=20

By this I *don't* mean the string.Intern method as this will leave your
strings hanging around forever!=20

Rather for each record you read from your CSV file, look through each
field that is a string and then put it through a routine which looks for
a duplicate string already read and replace with that existing reference
instead. This can dramatically reduce memory-pressure for a given set of
data.
There is some code you could crib from this article,
http://www.codeproject.com/dotnet/FastSerializer.asp, look in the
FastSerializer.cs class for "UniqueStringList" and then add this
override:
  public string Add(string value)
        {
            int bucketIndex =3D getBucketIndex(value);
            int index =3D buckets[bucketIndex] - 1;
            if (index !=3D -1)
            {
                return stringList[index];
            }
            stringList[stringListIndex++] =3D value;
            buckets[bucketIndex] =3D stringListIndex;
            if (stringListIndex > loadLimit)
            {
                this.Expand();
            }
            return value;
        }

Just replace each string value you read with the one returned from this
method.

Not sure if Sebastien already has this feature in his excellent CSV
reader project but it would be easy to add it for the situations where
you know the data is going to stay around.

Cheers
Simon




-----Original Message-----
From: Discussion relating to the specifics of the C# and Managed C++
languages [mailto:[email protected]] On Behalf Of Alan
Baljeu
Sent: 21 November 2007 18:38
To: [email protected]
Subject: [DOTNET-CX] DataTable efficiency

 My application uses a largish readonly database (CSV files,
 actually) consisting of several tables.  These tables are  accessed
somewhat frequently, and I want to make this as  efficient as possible.
I'm currently using plain DataTable
 objects produced by OdbcAdapters and queries.     These
 tables are cached so I'm just searching in-memory tables  after that.
My queries look like "SELECT * FROM [{0}]" and  occasionally "SELECT *
FROM [{0}] where [{1}] =3D '{2}'".
 I'm thinking of splitting the latter into just  table.Select("colname =
=3D
value").

 I have a nagging suspicion that the DataTable structure isn't
super-efficient, or that certain ways of using it are  effcient while
others aren't.

 Is this so?  Are there certain things I should avoid?  Do you
recommend a certain style for efficient access?  Is there a  better
approach that DataTable?

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This list is hosted by DevelopMentor(r)  http://www.develop.com

View archives and manage your subscription(s) at
http://discuss.develop.com

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This list is hosted by DevelopMentor=AE  http://www.develop.com

View archives and manage your subscription(s) at http://discuss.develop.com