RE: Data Structure for use vs storage

Brian Halbert spake:

[snip description of data format]

> So, in the end, I guess my question is how important is the format of data storage, 

Well, use whatever mechanism is most natural for the problem you're trying to solve

> and where do rules-of-thumb put decisions like hash table vs a-list? 1227 variables 
> seems like a lot for an a-list, (and maybe > for a hash table?), but totally normal 
> for an array, with data being easy to use in a  better/more natural data storage,

_Personally_, I'd stuff all this into a database, and do my querying via SQL.
That _is_ what DBs are good at. 

> but harder to implement (parsing text lines). The data is pretty static (it might update
> once every year or two based on the new American Community Survey, but not for each query).

But you're only parsing each time you load up the data into core, yes? 
Speed isn't terribly important, unless I'm misunderstanding the load process?

> Should I even worry about this until what I try doesn't work well enough? Is it premature optimization?

Sure... but for how long are you actually going to remember that "variable 625 is the count of 
households earning less than $10k per year" ?

Me, I just wrote it down and I've already forgotten. 

I'd be shoving that into a DB with lots of juicy joins and foreign keys, so you can do tasty things
like joining on arbitrary income values, rather than having to recall (in code) to use array
index 625 if income< 10,000 and index 626 otherwise, and index ____ for _____. 

SQLite is perfect for this, I would think - no database involved, just a single file.
And if you want blazing speed, SQLite can work on RAM-only databases.

You can write a FLI lisp wrapper for SQLite in a day. Or you can use one of the millions of open
source ones on the 'net. Or you can use LW-PRO and use CLSQL. 

Adam


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
[email protected]
http://www.lispworks.com/support/lisp-hug.html