Re: parallelization of ./configure compiler test processes

On Thu, Mar 30, 2023 at 9:58 PM Zack Weinberg <[email protected]> wrote:

We can already say with high confidence that the four checks done by

AC_CHECK_HEADERS([a.h b.h c.h d.h])

can be run in parallel, for instance.

[…]

This is getting into things that I'm not sure can be done in M4 at

It is not the first email I write about this: for whatever thing can be
done in parallel I had written an M4 framework
<https://github.com/madmurphy/not-autotools/blob/master/m4/not-parallel-configure.m4>
to make life simpler. And so the code above in parallel would become
(assuming that we want to check for the headers stddef.h, stdint.h, stdio.h
and stdlib.h):

m4_include([not-parallel-configure.m4])

# Launch AC_CHECK_HEADERS() in parallel...

NC_THREAD_NEW([headers_thread],
	[AC_CHECK_HEADERS([stddef.h stdint.h stdio.h stdlib.h])])

# Do other stuff in the meanwhile...
# ...

# Wait until the AC_CHECK_HEADERS thread has returned...
NC_JOIN_THREADS([headers_thread])

AS_IF([test "x${ac_cv_header_stddef_h}" = 'xyes'],
	[AC_MSG_NOTICE([We have stddef.h])],
	[AC_MSG_NOTICE([We don't have stddef.h])])

A complete configure.ac example is available at
madmurphy/not-autotools/examples/not-parallel-configure/configure.ac
<https://github.com/madmurphy/not-autotools/blob/master/examples/not-parallel-configure/configure.ac>
.

--madmurphy

On Thu, Mar 30, 2023 at 9:58 PM Zack Weinberg <[email protected]> wrote:

> On Wed, Mar 29, 2023, at 7:05 PM, Thomas Jahns wrote:
> > I spent some time thinking about improvements to autoconf configure
> > scripts (while waiting for builds to proceed). In my view, it is
> > currently still easier to seek small efficiency gains that, in sum,
> > could still improve run-time substantially than parallelizing the
> > whole would be, because there is so much often untapped potential:
>
> I do generally agree with this, and I'd also like to remind everyone
> that Autoconf has very limited developer resources relative to how
> difficult the task of parallelizing _all_ of a configure script will be.
>
> There _are_ some bits of "low hanging fruit" here.  If parallelism
> within a configure script is the specific thing what you're interested
> in working on, I'd recommend that you start with the macros that check
> for a list of things.  We can already say with high confidence that the
> four checks done by
>
>     AC_CHECK_HEADERS([a.h b.h c.h d.h])
>
> can be run in parallel, for instance.
>
> > Regarding parallelization for autoconf in particular, I think
> > autoconf could very much benefit from having first more explicit
> > effects of each macro, i.e. which variables end up being set, which
> > file will be appended to etc. To my knowledge this is mostly well
> > documented for the human reader, but not programmatically available
> > in the M4 phase at all.
>
> This is getting into things that I'm not sure can be done in M4 at
> all...
>
> zw
>
>