Re: [PATCH] make: add valid-for and valid-max target TTL support
Yair Lenga <[email protected]> Tue, 17 Feb 2026 21:16:35 -0500
| Newsgroups | gmane.comp.gnu.make.bugs |
|---|---|
| Message-ID | <CAK3_KpPMRh0tQJv11USgSZz=jFXwFL-6TerPXb__Rk5rbQnmbQ@mail.gmail.com> |
--000000000000a51511064b0fc7b8
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi Britton,
Thanks for taking the time to read my proposal. Good point about
documentation, should have included more use case. See below for some
additional example. If the proposal will be accepted - I look into creating
a documentation pages for "info make" for various use cases:
Is it usable only for leaves ?
- Short answer: It can be used also on non-leaves to limit rebuild, or
limit reuse of targets. In it's simplest form - it can be used on leaves
that (including leaf that depend on "PHONY" targets).
Below are the main use cases where new features will help, all
involve speeding up process when there is "expensive" operation, which can
be skipped MOST of the time, and it's OK to be slightly behind. I'm
implementing all of those cases with combination of macros/shell commands.
1. Cache (original example):
Assuming the build need a remote resource that take lot of time to
download/process, and does not change frequently, Adding "valid-max" make
it possible to declare - It's OK with this target if it was created in the
last day - no need to reload.
cache.json: private valid-max=3D1d
cache.json:
curl -o $@ https://example/api
2. Reduce excessive updates.
In some cases, the build process depend on files can be updated multiple
time. This could be a CI process that produces new artifacts on every
commit. when running heavy tests, it might not be practical to execute the
full test every time the remote artifact is recreated. Adding "valid-max=3D=
"
on non-leaf node, limit the execution of the tests to no more than once
every 24 hours (1d).
tests.txt: private valid-for=3D1d
tests.txt: /path/to/remote/artifact.tar
tar xvf $<
run_tests
touch $@
3. Expiration of targets, when dependencies are incomplete.
In some cases, it's not possible to describe the full set of dependencies
(e.g., when it's dynamic), or when dependencies are "noisy". For example -
I had a case where a job had a dependency on remote artifacts and local
artifacts, and we wanted to force a policy that require the job to run (and
re-fetch) the remote artifacts, even if none of the local artifacts got
updated. Adding valid-max indicate that the job should be re-make if it's
older than 1 day, even of leave dependency was updated.
job.txt: private valid-max=3D1d
job.txt: /path/to/other/artifact.tgz
tar xvf $<
curl .... | tar xvf -
run_job.sh
touch job.txt
I hope that those examples show how the new functionality make it easier to
address common use cases when using make for CI/data integration processes
- where the simple DAG rule may result in non-practical results. I'm sure
that there are other potential usages for building practical processes for
testing, deployment, database integration - and other systems that do not
provide exact modification timestamp, or when it's not practical to use
simple DAG rules.
Regards,
Yair
On Tue, Feb 17, 2026 at 11:17=E2=80=AFAM Britton Kerin <britton.kerin@gmail=
.com>
wrote:
>
>
> On Mon, Feb 16, 2026, 1:46=E2=80=AFPM Yair Lenga <[email protected]> w=
rote:
>
>> Hello,
>>
>> This patch introduces two optional target-specific variables,
>> 'valid-for' and 'valid-max', that allow controlling the freshness
>> and expiration of existing targets based on their modification time.
>>
>> Motivation
>> ----------
>>
>> GNU make treats an existing target with no prerequisites as
>> permanently up to date. In practice, many build workflows generate
>> cache or artifact files that should be reused while fresh but
>> periodically regenerated even without explicit dependencies. Today
>> this: requires introducing artificial phony prerequisites or external
>> timestamp logic.
>>
>> The new variables provide a declarative mechanism for this:
>>
>> valid-for =3D DURATION
>> If the target exists and its age is less than or equal to the
>> specified duration, make treats it as up to date and skips
>> dependency checking.
>>
>> valid-max =3D DURATION
>> If the target exists and its age exceeds the specified duration,
>> make forces a rebuild even if the target has no prerequisites.
>>
>> Durations are specified as either an integer number of seconds or a
>> sequence of <number><unit> segments without spaces, where unit is one
>> of 's', 'm', 'h', 'd', or 'w' (e.g. "300", "5m", "1d5h").
>>
>> When both variables are present, 'valid-for' takes precedence for
>> fresh targets, while 'valid-max' forces rebuild of expired targets.
>>
>
> I'm not sure I entirely understand this and think it could probably be
> better expressed for documentation at least. By fresh I assume you mean
> otherwise up-to-date?
>
>
>> Implementation
>> --------------
>>
>> The logic is implemented in update_file_1() via a small helper that
>> parses and evaluates the TTL variables. Invalid values are ignored
>> and reported under --debug=3Dv. Successful use of the TTL is reported
>> under --debug=3Db.
>>
>> This approach preserves existing make semantics while enabling
>> expiration of leaf targets and optional dependency skipping for
>> recent artifacts.
>>
>
> Is it usable only for leaves? I guess these variables are not inherited
> by dependent targets (as other target-local variables are) since that wou=
ld
> presumably add a bunch of undesired edges to the DAG?
>
>
>> -------
>> Example 1: Periodic reloading of remote resources that change very
>> frequently.
>>
>> cache.json: private valid-max=3D1d
>> cache.json:
>> curl -o $@ https://example/api
>>
>> The file will be regenerated once per day even without prerequisites.Thi=
s
>> can be useful when a test job is running every few minutes, but loading =
of
>> remote resources is time consuming.
>>
>> Other use cases include periodically refreshing generated version
>> stamps, expiring downloaded resources in reproducible builds, and
>> limiting reuse of expensive intermediate artifacts in CI or code
>> generation workflows.
>>
>> ---
>>
>> Patch attached. Comments welcome.
>>
>> Best regards,
>> Yair Lenga
>>
>
--000000000000a51511064b0fc7b8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Britton,<br><div><br></div><div>Thanks for taking the t=
ime to read my proposal. Good point about documentation, should have includ=
ed more use case. See below for some additional example. If the proposal wi=
ll be accepted - I look into creating a documentation pages for "info =
make" for various use cases:</div><div><br></div><div>Is it usable onl=
y for leaves ?</div><div>- Short answer: It can be used also on non-leaves =
to limit rebuild, or limit reuse of targets. In it's simplest form - it=
can be used on leaves that (including leaf that depend on "PHONY"=
; targets).</div><div><br></div><div>Below are the main use cases where new=
features will help, all involve=C2=A0speeding up process when there is &qu=
ot;expensive" operation, which can be skipped MOST of the time, and it=
's OK to be slightly behind. I'm implementing all of those cases wi=
th combination of macros/shell commands.</div><div><br></div><div>1. Cache =
(original example):</div><div>Assuming the build need a remote resource tha=
t take lot of time to download/process, and does not change frequently, Add=
ing "valid-max" make it possible to declare - It's OK with th=
is target if it was created in the last day - no need to reload.</div><div>=
<br></div><div>cache.json: private valid-max=3D1d<br>cache.json:<br>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 curl -o $@=C2=A0<a href=3D"https://example/api"=
rel=3D"noreferrer" target=3D"_blank">https://example/api</a><br></div><div=
><br></div><div>2. Reduce excessive updates.</div><div>In some cases, the b=
uild process depend on files can be updated multiple time. This could be a =
CI process that produces new artifacts on every commit. when running heavy =
tests, it might not be practical to execute the full test every time the re=
mote artifact is recreated. Adding "valid-max=3D" on non-leaf nod=
e, limit the execution of the tests to no more than once every 24 hours (1d=
).</div><div><br></div><div>tests.txt: private valid-for=3D1d</div><div>tes=
ts.txt: /path/to/remote/artifact.tar<br></div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 tar xvf $<<br></div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r=
un_tests</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 touch $@</div><div><b=
r></div><div>3. Expiration of targets, when dependencies are incomplete.</d=
iv><div>In some cases, it's not possible to describe the full set of de=
pendencies (e.g., when it's dynamic), or when dependencies are "no=
isy". For example - I had a case where a job had a dependency on remot=
e artifacts and local artifacts, and we wanted to force a policy that requi=
re the job to run (and re-fetch) the remote artifacts, even if none of the =
local artifacts got updated. Adding valid-max indicate that the job should =
be re-make if it's older than 1 day, even of leave dependency was updat=
ed.</div><div><br></div><div>job.txt: private valid-max=3D1d</div><div>job.=
txt: /path/to/other/artifact.tgz</div><div>=C2=A0 =C2=A0 =C2=A0 tar xvf $&l=
t;</div><div>=C2=A0 =C2=A0 =C2=A0 curl .... | tar xvf -</div><div>=C2=A0 =
=C2=A0 =C2=A0 run_job.sh</div><div>=C2=A0 =C2=A0 =C2=A0 touch job.txt</div>=
<div><br></div><div>I hope that those examples show how the new functionali=
ty make it easier to address common use cases when using make for CI/data i=
ntegration processes - where the simple DAG rule may result in non-practica=
l results. I'm sure that there are other potential usages for building =
practical processes for testing, deployment, database integration - and oth=
er systems that do not provide exact modification timestamp, or when it'=
;s not practical to use simple DAG rules.</div><div><br></div><div>Regards,=
</div><div><br>Yair</div></div><br><div class=3D"gmail_quote"><div dir=3D"l=
tr" class=3D"gmail_attr">On Tue, Feb 17, 2026 at 11:17=E2=80=AFAM Britton K=
erin <<a href=3D"mailto:[email protected]" target=3D"_blank">britt=
[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quote=
" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);=
padding-left:1ex"><div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"=
><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Feb 16, 2026, 1:46=E2=80=AFP=
M Yair Lenga <<a href=3D"mailto:[email protected]" target=3D"_blank">=
[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quo=
te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204=
);padding-left:1ex"><div dir=3D"ltr">Hello,<br><br>This patch introduces tw=
o optional target-specific variables,<br>'valid-for' and 'valid=
-max', that allow controlling the freshness<br>and expiration of existi=
ng targets based on their modification time.<br><br>Motivation<br>---------=
-<br><br>GNU make treats an existing target with no prerequisites as<br>per=
manently up to date.=C2=A0 In practice, many build workflows generate<br>ca=
che or artifact files that should be reused while fresh but<br>periodically=
regenerated even without explicit dependencies.=C2=A0 Today<br>this:=C2=A0=
requires introducing artificial phony prerequisites or external<br>timesta=
mp logic.<br><br>The new variables provide a declarative mechanism for this=
:<br><br>=C2=A0 valid-for =3D DURATION<br>=C2=A0 =C2=A0 =C2=A0 If the targe=
t exists and its age is less than or equal to the<br>=C2=A0 =C2=A0 =C2=A0 s=
pecified duration, make treats it as up to date and skips<br>=C2=A0 =C2=A0 =
=C2=A0 dependency checking.<br><br>=C2=A0 valid-max =3D DURATION<br>=C2=A0 =
=C2=A0 =C2=A0 If the target exists and its age exceeds the specified durati=
on,<br>=C2=A0 =C2=A0 =C2=A0 make forces a rebuild even if the target has no=
prerequisites.<br><br>Durations are specified as either an integer number =
of seconds or a<br>sequence of <number><unit> segments without =
spaces, where unit is one<br>of 's', 'm', 'h', '=
;d', or 'w' (e.g. "300", "5m", "1d5h&q=
uot;).<br><br>When both variables are present, 'valid-for' takes pr=
ecedence for<br>fresh targets, while 'valid-max' forces rebuild of =
expired targets.<br></div></blockquote></div></div><div dir=3D"auto"><br></=
div><div dir=3D"auto">I'm not sure I entirely understand this and think=
it could probably be better expressed for documentation at least.=C2=A0 By=
fresh I assume you mean otherwise up-to-date?</div><div dir=3D"auto"><br><=
/div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=3D"gmai=
l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20=
4,204);padding-left:1ex"><div dir=3D"ltr"><br>Implementation<br>-----------=
---<br><br>The logic is implemented in update_file_1() via a small helper t=
hat<br>parses and evaluates the TTL variables.=C2=A0 Invalid values are ign=
ored<br>and reported under --debug=3Dv.=C2=A0 Successful use of the TTL is =
reported<br>under --debug=3Db.<br><br>This approach preserves existing make=
semantics while enabling<br>expiration of leaf targets and optional depend=
ency skipping for<br>recent artifacts.<br></div></blockquote></div></div><d=
iv dir=3D"auto"><br></div><div dir=3D"auto">Is it usable only for leaves?=
=C2=A0 I guess these variables are not inherited by dependent targets (as o=
ther target-local variables are) since that would presumably add a bunch of=
undesired edges to the DAG?</div><div dir=3D"auto"><br></div><div dir=3D"a=
uto"><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"=
margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef=
t:1ex"><div dir=3D"ltr"><br>-------<br>Example 1: Periodic reloading of rem=
ote resources that change very frequently.<br><br>=C2=A0 cache.json: privat=
e valid-max=3D1d<br>=C2=A0 cache.json:<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 curl -o $@ <a href=3D"https://example/api" rel=3D"noreferrer" target=3D=
"_blank">https://example/api</a><br><br>The file will be regenerated once p=
er day even without prerequisites.This can be useful when a test job is run=
ning every few minutes, but loading of remote resources is time consuming.<=
div><br></div><div>Other use cases include periodically refreshing generate=
d version<br>stamps, expiring downloaded resources in reproducible builds, =
and<br>limiting reuse of expensive intermediate artifacts in CI or code<br>=
generation workflows.</div><div><br></div><div>---</div><div><br></div><div=
>Patch attached. Comments welcome.</div><div><br></div><div>Best regards,</=
div><div>Yair Lenga</div></div>
</blockquote></div></div></div>
</blockquote></div>
--000000000000a51511064b0fc7b8--