Re: Startup performance

On 10/13/20 1:50 AM, Martin H=F6ller wrote:
> Hi!
> =

> Am 12. Okt. 2020 schrieb Robert Krawitz:
> =

>> It really is the startup that's the most visible aspect of all of this. =
 Again, that either means
>> pretty fundamental changes in the storage backend, a faster XML parser, =
or maybe a way of bringing
>> up the UI while images are still loading (and since you're often going t=
o want to look at your most
>> recent photos, not even very helpul).  If we stored photos in the databa=
se in _reverse_ order of
>> time, though, that might be workable (I'd love to have instant access to=
 my most recent basketball
>> game while the rest of the file is being loaded).
> =

> Just an idea: what about "partitioning" the database. I mean we could
> have a single XML file per year, per decade, per 10k images or whatever.
> That way, loading the more recent images first as Robert suggested would
> be easier.

One of the goals of KPA is for the database to be human readable and editab=
le.  Splitting up the
image file in that way makes the structure more complicated to move/copy an=
d edit.

The decision of a human-readable database certainly makes parallelizing par=
sing more difficult
Human-readable (with no tools beyond a pager or text editor) files have to =
be line-delimited plain
text.  In this case, since the file metadata can be arbitrarily long (many =
categories and members),
we can't require a fixed line length, which is problematic for editing anyw=
ay.  So we've set
ourselves a bit of a challenge here.

Breaking the file up would allow for parallel parsing, certainly.

Maybe a way to do it is to store the data in an RDBMS, preferably with the =
raw storage format
matching the underlying data format so we don't have to do a lot of convers=
ions on the way in, and
provide a tool to convert from/to XML format.  This would also allow for ba=
ckups in the current
manner.  Saving is at present about twice as fast as startup, not including=
 time to make a backup
(which is a little more than 1/3 of the time to otherwise save the database=
) and is less
objectionable other than autosave.

Back to day job...