Data Structure for use vs storage

This may not be a well formed question; I'm also not a very proficient
coder compared to what I see y'all discussing, but trying my best here.

I'm trying to automate creating and printing an income distribution summary
for a chosen zip code, based off  flattened census data containing
household counts for every zip code (by household income, household size,
and householder age; example below), and I'm not sure how best to hold and
display this data. The data as given to me in a flat list doesn't really
mirror the 'natural' data structure (a 3d array).

*Data Description*
In short, I have data from the US Census that is a large data set (large
for humans, not necessarily machines) in the form of several CSV files. For
each Zip code, I have a list of 1227 possible unique variables, each
holding a household count. Not all Zip Codes will have all variables (eg,
only non-zero values are listed), but the variable name is accurate. The
household count variables are a key (1 to 1227) overlaying ordinal
categories:  age of householder (9 buckets), household size (10 buckets),
and household income (13 buckets). This data is therefore 'naturally' a 3d
array (4d if you add the zip code), with 9x10x13 = 1170 variables, plus 100
odd summary variables along various dimensions, e.g. variable 1 refers to
the total households (for all ages, sizes, and incomes), variable 2 is all
owner households, variable 3 is all renter households (of any size),
variable 4 is all single person owner households, etc. This list is in a
single row per variable format, though I can create a pivot table to offer
it instead as columns.

eg, the number of owner households with 2 persons (HH Size bucket 2),
earning 50 to 60k a year (HH income bucket 7), with householder age 75 to
84 (HHolder age bucket 8), would be most naturally addressed something like
(aref *zipcode-census-table* 2 7 8) returning a household count of 381,374
(this is the total count of US households meeting those criteria, as it
happens). BUT, I have not been given those bucket values explicitly (though
I could infer them), I just have a flattened list of created variable
names. The csv file looks like the below, but note that the description for
any given row doesn't specify all relevant underlying categories, only the
'innermost', you have to track the state in some way to determine the
relevant inputs.

```
GEOCODE, VARIABLE, VALUE, DESCRIPTION
Zip55555, variable=1, value=125736352, description="All Households"
Zip55555, variable=2, value=81497760, description="All Owner Households"
Zip55555, variable=3, value=2391479, description="Less than $10,000"
Zip55555, variable=4, value=1515595, description="1 Person Owner HH"
Zip55555, variable=5, value=34883, description="Householder 15 to 24 years
old"
... ;other age categories
Zip55555, variable=13, value=147626, description="Householder 85 years and
older"
Zip55555, variable=14, value=549079, description="2-Person Owner HH"
Zip55555, variable=15, value=15729, description=Householder 15 to 24 years
old"
... ;etc, looping over internal variables, getting to the richest (200k+),
then largest (5+ renter HH), then oldest households:
Zip55555, variable=1227, value=1281, description="Householder 85 years and
older"
Zip55556, variable=1, value=xxxxxxx, description="All Households"
etc.
```
*Use Case*
What I want to serve (eventually as a web app) is human readable tables in
response to a chosen zip code, taking slices along two dimensions (income
by size), with a separate 2d slice showing for each age group. The variable
numbers above were created in a different order (age x size x income rather
than income x size x age), iterating *something* like the below, but I only
see the result.

```
(loop for income from 1 to 13
      with i = 4
      (loop for hh-size from 1 to 10 ;hh size was actually done in two
passes
            (loop for age from 1 to 10  ;a little squirelly. Some summary
variables inserted
             do (+ i 1)
             do (format "Zip X, Variable=~a, Value=~a" i (get-census-value
age size income))))
```

So, I could read in the data, in essence, as a nested hashtable (or
alists), with each key (zip code) having as a value a hash map with 1277
associated variables, and I could then print the output tables by iterating
in some way over the known variable names needed (suggestive pseudo-code):

```
;;; Create the data
(defparameter *zipcode-hash* (make-hash-table))

;;; You would need to initialize a new inner hash when you reached a new
zip code, etc.
(defun read-csv-file (file-name)
    (with-open-file (stream file-name :direction :input)
        loop for line = (read-line stream nil :eof)
             for zip#, var#, value = (parse-line) ;pseudo code here
             do (setf (gethash variable# (gethash zip# *zipcode-hash*))
value)))))

;;; use the data
(defun get-values (zip var-list)
    (loop for var in var-list
          collect (gethash var (gethash zip *zipcode-hash*))))

;;; having now read the data into memory, it could be printed out as a
table using format

(format t "~{~a~^, ~}" (get-values 55555 '(5, 15, 25, 35, 45, 618, 628,
638, 648, 658)))
(format t "~{~a~^, ~}" (get-values 55555 '(56, 66, 76, 86, 96, 669, 679,
689, 699, 709)))
(... 115 similar format rows ...)
```

And since I know the needed variable numbers, and they follow rules (eg,
the variable naming relationship between rows and columns is steady, with
each subsequent row being a number 51 larger than the variable 'above'
it.), I could create an outer loop around the format code rather than
writing all 117 (9x13) rows of the table explicitly, or since it's static I
could put that in a text file and operate on each text row, etc. In effect,
this is how I do it now, just in excel -- SUMIF(ZIP=x, Variable=y, values).
BUT it relies basically on having 1227 'magic numbers' for printing. I just
have to know that variable 625 is the count of households (1) earning less
than $10,000 a year, (2) with householder ages 75 to 84, and (3) that
householder being a single renter.

That is -- I'm storing/accessing the data as a flat list, and turning it
into a table when I want to look at it.

Alternately, I could instead parse the data (perhaps reading the
descriptions as I go to track the category values while the age changes,
and then store the data in its 'natural' form as an array.

```
(defparameter *zip-array* (make-array '(9 10 13) :initial-contents 0))

(defun parse-file-into-array (file)
           (with file ...
                    (setf (aref *zip-array* (Parse-row-for-buckets)))))
;;;This is tricky in practice, would have to get the indexes going in the
right direction.

;;; But now printing is easy, maybe, since the storage structure matches
the data, just iterate over the thing to print the tables

(format t "~{~{~{~a~^, ~}~}~}" *zip-array*)
```

So, in the end, I guess my question is how important is the format of data
storage, and where do rules-of-thumb put decisions like hash table vs
a-list? 1227 variables seems like a lot for an a-list, (and maybe for a
hash table?), but totally normal for an array, with data being easy to use
in a  better/more natural data storage, but harder to implement (parsing
text lines). The data is pretty static (it might update once every year or
two based on the new American Community Survey, but not for each query).

Should I even worry about this until what I try doesn't work well enough?
Is it premature optimization?