Re: Anyone depending on having per directory SConsign files

Indeed a rewrite of the sconsign persistence is planned.
The fact that a no-op build will still write the sconsign and burn 5-10
seconds doing so on a large build is not optimal.

JSON and sqlite are (in my mind) the leading contenders and not some
smarter pickling logic.
But, as always, it's open for  (thoughtful) discussion.

-Bill
SCons Project Co-Manager

On Thu, Nov 26, 2020 at 7:19 AM Mats Wichmann <[email protected]> wrote:

> On 11/26/20 12:53 AM, Steve Hill (Wireless) via Scons-users wrote:
> > Hi Bill,
> >
> > Yes, we do use this – but the use-case is a little complex…
> >
> > It started when we were still running with a 32-bit version of Python.
> > Since the code to write back the SConsign assembles it in memory first,
> > and our SConsign was many hundreds of MB in size, it was sometimes
> > causing an out-of-memory exception.
> >
> > Hence, what I did was to create a patched version of the DirFile class
> > that wrote back a file per directory but named with a hash of the
> > directory name and stored in a single sub-directory below the
> > SConstruct, and patched SCons with that class. I then used
> > SConsignFile(None) to invoke the new behaviour.
> >
> > This solved the out-of-memory issue and, even when we moved to a 64-bit
> > version of Python, it was a performance improvement since, if a dev was
> > only building a small proportion of the overall codebase (a common
> > use-case – just running the unit-test suite for a single source file
> > when TDDing, for instance), writing back a few relatively small files
> > was often much quicker than writing back a several hundred MB file.
>
> That's not so wild.  There's been an intent for a while (not backed up
> by any pull request or experimental branch so far) to switch to a more
> database-like model for the signature stuff, where records can be
> read/written individually, rather than a huge dump of everything that's
> previously all been kept in memory.   Hundreds of MB of data to pickle
> in one go is a pretty scary thought even with the improvements that have
> happened to pickle performance in Python....
>
> _______________________________________________
> Scons-users mailing list
> [email protected]
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users