CSIG and caching for subst in Parts
Jason Kenny <[email protected]> Tue, 5 May 2020 17:30:52 +0000
| Newsgroups | gmane.comp.programming.tools.scons.devel |
|---|---|
| Message-ID | <BYAPR05MB5736B06A740164019EFF8C3E84A70@BYAPR05MB5736.namprd05.prod.outlook.com> |
--===============0445391509163552187==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_BYAPR05MB5736B06A740164019EFF8C3E84A70BYAPR05MB5736namp_"
--_000_BYAPR05MB5736B06A740164019EFF8C3E84A70BYAPR05MB5736namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hello,
As request by bdbaddog I am adding to the mailing list.
I am adding in my next drop of Parts some work to allow subst to work faste=
r. It is restricted at the moment to Parts Mapper objects at this time unti=
l it is better understood how this could be expanded better in a more gener=
al way. The core of the fix is to make a cache key that allows me to store=
a result given that I know at this point it is not going to change. This i=
s a little more complex in that with Parts I have builders that deal with w=
hat I call dynamic scanners, or scanners that add build actions not known u=
ntil during the build stage ( ie not after the read stage of all build file=
s).
The current code for this caching is here https://bitbucket.org/sconsparts/=
parts/src/7a186e7a9c56ff117c8a32f603d2e048704fcbc6/src/parts/mappers.py#lin=
es-140:171
It is based on the use of mapping what environment is used for generating t=
he value. To make this work I had to add a get_csig() call to the environme=
nt. This allows me to know that $FOO_NAME is "joe" in this environment and =
"bill" in a different one.
The current code for this at this time is here:
https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f603d2e048=
704fcbc6/src/parts/overrides/env_csig.py
I am still refactoring this at the moment as I had some older code that nee=
ds to be merged in with this ( ie ongoing cleanup I have to do). This code =
is here https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f6=
03d2e048704fcbc6/src/parts/common.py#lines-152:181
I plan to make a get_content() function in parts.core.util that will be use=
d instead as a reusable function that can deal given objects as needed.
This sig work is also part of me cleaning up and resetting code that is tur=
ned off in Parts that allows for smart loading and caching of results to sp=
eed up the incremental build. That is a different subject from here which i=
s about what I am putting together for making a better reproducible sig for=
the environment and how this can be used to improve subst() logic.
the early number for a no-op check that everything is up-to-date build I ha=
ve with 315 parts ( think of this as Sconscripts) when from about 20 minute=
s to about 4. however as I tweak with an updated get_content function I exp=
ect this to probably go up a little. I also have better times for the full =
build but did not get any formal numbers to share as of yet
Jason
--_000_BYAPR05MB5736B06A740164019EFF8C3E84A70BYAPR05MB5736namp_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hello,</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
As request by bdbaddog I am adding to the mailing list.</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I am adding in my next drop of Parts some work to allow subst to work faste=
r. It is restricted at the moment to Parts Mapper objects at this time unti=
l it is better understood how this could be expanded better in a more gener=
al way. The core of the fix is
to make a cache key that allows me to store a result given that I know at =
this point it is not going to change. This is a little more complex in that=
with Parts I have builders that deal with what I call dynamic scanners, or=
scanners that add build actions
not known until during the build stage ( ie not after the read stage of al=
l build files). </div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The current code for this caching is here <a href=3D"https://bitbucket=
.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f603d2e048704fcbc6/src/part=
s/mappers.py#lines-140:171" style=3D"" id=3D"LPlnk491877">https://bitbucket=
.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f603d2e048704fcbc6/src/part=
s/mappers.py#lines-140:171</a></div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
It is based on the use of mapping what environment is used for generating t=
he value. To make this work I had to add a get_csig() call to the environme=
nt. This allows me to know that $FOO_NAME is "joe" in this enviro=
nment and "bill" in a different one.</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The current code for this at this time is here: </div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<a href=3D"https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a3=
2f603d2e048704fcbc6/src/parts/overrides/env_csig.py" id=3D"LPlnk360194" sty=
le=3D"">https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f6=
03d2e048704fcbc6/src/parts/overrides/env_csig.py</a><br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I am still refactoring this at the moment as I had some older code that nee=
ds to be merged in with this ( ie ongoing cleanup I have to do). This code =
is here <a href=3D"https://bitbucket.org/sconsparts/parts/src/7a186e7a=
9c56ff117c8a32f603d2e048704fcbc6/src/parts/common.py#lines-152:181" style=
=3D"" id=3D"LPlnk685507">https://bitbucket.org/sconsparts/parts/src/7a186e7=
a9c56ff117c8a32f603d2e048704fcbc6/src/parts/common.py#lines-152:181</a></di=
v>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I plan to make a get_content() function in parts.core.util that will be use=
d instead as a reusable function that can deal given objects as needed.</di=
v>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
This sig work is also part of me cleaning up and resetting code that is tur=
ned off in Parts that allows for smart loading and caching of results to sp=
eed up the incremental build. That is a different subject from here which i=
s about what I am putting together
for making a better reproducible sig for the environment and how this can =
be used to improve subst() logic. </div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
the early number for a no-op check that everything is up-to-date build I ha=
ve with 315 parts ( think of this as Sconscripts) when from about 20 minute=
s to about 4. however as I tweak with an updated get_content function I exp=
ect this to probably go up a little.
I also have better times for the full build but did not get any formal num=
bers to share as of yet</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Jason</div>
</body>
</html>
--_000_BYAPR05MB5736B06A740164019EFF8C3E84A70BYAPR05MB5736namp_--
--===============0445391509163552187==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev
--===============0445391509163552187==--