Re: RPi 4 build time

On 2021-May-22, at 13:12, Mark Millard <marklmi at yahoo.com> wrote:

> On 2021-May-22, at 09:01, tech-lists <tech-lists at zyxst.net> wrote:
> 
>> On Fri, May 21, 2021 at 03:51:35PM -0700, Mark Millard via freebsd-arm wrote:
>> 
>>> So, if I read this right, you are reporting 4.5 hrs
>>> for a "hot ccache" result, which I had mentioned as
>>> one of the things leading to large variations in
>>> reported build times.
>> 
>> Hi,
>> 
>> not sure what you mean by "hot cache"
> 
> The first time ccache is used it has no prior results
> to use to avoid compiles/links: an empty cache (a form
> of "cold" cache). Another form of "cold" cache could
> result from changing compiler options that would change
> the code generated for (nearly) every file produced so
> that the cache becomes ineffective.
> 
> "hot" refers to having a significant amount of
> "effective/used cache content" that makes a notable
> difference in the build times. I'm not that impressed
> with the terminology but it is was I've seen used the
> most frequently for ccache. So I used it.
> 
>> - I always use devel/ccache-static
>> as have tended to build from source throughout my time of using freebsd.
>> It provides tremendous speedups and generally i'll disable it only if a
>> problem arises and am debugging it, or crossing a version boundary like
>> from stable to current. What I'm saying is I don't know when ccache was
>> last used for building anything.
> 
> I'm confused how you can know it "provides tremendous
> speedups" while simultaneously not knowing "when ccache
> was last used for building anything". It sounds like you
> think the 4.5 hr build might have not have been from
> having a notable speed up from ccache?
> 
> Remember that when comparing to my "from scratch"
> build times: in my build everything was compiled
> and linked, no prior build materials around to be
> reused. So I'm reporting a context where I know
> how to interpret the result and I'm presenting
> enough history to establish a repeatable context.
> 
>> 1. rpi4 here is clocked to 2.0GHz
>> 2. ccache is in use and /var/cache/ccache has *not* been previously cleared
>> (i'll clear it for next test)
>> 
>> 3. make cleanworld cleandir clean has been run on /usr/src
>> 4. sources are at 246839
>> 
>> 5. this rpi4 has the following properties for its disk:
>> [i] root-on-zfs
>> [ii] boot-to-usb3
>> [iii] 4k sectorsize forced
>> [iv] encrypted swapspace
>> [v] entire filesystem encryption
> 
> FYI: My build-experiment boot media are never
> encrypted for the file system or swap/paging
> space. Another thing I'd not thought to comment
> on. As I've reported, my UFS based and ZFS based
> experiments get only minor variations in
> build times (variations of minutes for from-
> scratch builds that take hours).
> 
>> /etc/src.conf is
>> https://cloud.zyxst.net/~john/FreeBSD/rpi4-main/src.conf
>> 
>> make -j10 cleanworld started on Sat May 22 15:41:58 BST 2021
>> make -j10 cleanworld completed on Sat May 22 15:43:23 BST 2021
>> 
>> make -j10 cleandir started on Sat May 22 15:43:23 BST 2021
>> make -j10 cleandir completed on Sat May 22 15:43:50 BST 2021
>> 
>> make -j10 clean started on Sat May 22 15:43:50 BST 2021
>> make -j10 clean completed on Sat May 22 15:44:11 BST 2021
>> 
>> make -j6 buildworld started on Sat May 22 15:44:11 BST 2021
>> make -j6 buildworld completed on Sat May 22 16:20:48 BST 2021
> 
> So between 36 min and 37 min to rebuild the same version
> with the same build options and compiler/link command lines
> (near[?] maximal effective-ccache content that leads to
> near[?] maximal avoidance of rebuild activity).
> 
> Cool.
> 
> For META_MODE builds, seeing how long it takes to go through
> and discover that little or nothing needs to be rebuild would
> be the build times for 2nd build from doing back-to-back builds
> (not even an install to the live system between). The META_MODE
> use would then prevent most rebuild activity. I've not done
> such a timing in a long time and it does not approximate any
> normal build time for my typical rebuild patterns. So I do not
> normally time that.
> 
> I'm not claiming META_MODE is similarly effective to ccache.
> In fact, I know of issues where META_MODE rebuilds files that
> ccache would avoid rebuilding the same file: for example,
> doing an install of a build to the live system between the
> rebuilds has side effects that lead META_MODE to rebuild far
> more things.
> 
>> make -j6 buildkernel started on Sat May 22 16:20:48 BST 2021
>> make -j6 buildkernel completed on Sat May 22 16:49:18 BST 2021
> 
> So between 28 min and 29 min to rebuild the same version with
> the same build options and compiler/link command lines
> (near[?] maximal effective-ccache content).
> 
> Total between 64 min and 66 min overall for buildworld buildkernel
> for the near[?] maximal effective-ccache content and needing all
> the files.
> 
> Good to know. Thanks.
> 

I happen to have ended up with an opportunity to do
(no cleanout of old results after the first rebuild,
no installationa of any of the builds):

rebuild world
reboot
rebuild world

The 2nd rebuild of world got:

World built in 354 seconds, ncpu: 4, make -j4

So a little under 6 minutes via META_MODE. META_MODE
does end up causing some rebuild activity, just not
much. Much of it is re-linking.

I did another "rebuild world" without a new reboot and
got:

World built in 293 seconds, ncpu: 4, make -j4

So, somewhat under 5 minutes for more context cached
in RAM.


A similar sequence for a debug build instead of non-debug
build (building machine running non-debug) got:

World built in 526 seconds, ncpu: 4, make -j4

So, somewhat under 9 minutes.

Then (no reboot between):

World built in 296 seconds, ncpu: 4, make -j4

So, somewhat under 5 minutes again.


In general these figures are approximations of the low
bound on a buildworld that is a (near) no-op but is
not frequently approached in my normal activity. But
it is rare for me to update the source tree again
and rebuild after only a few source commits after
what was originally rebuilt. For such, sub-half hour
rebuilds can certainly occur via META_MODE use.

The context happened to be the ZFS based one in all
cases. Still no ccache use.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)