Re: [PATCH v3 0/4] Introduce fdt_overlay_merge() to allow merge of overlay blobs

[...]
> > 
> > Our repositores are following android structure based on android-treble
> > where vendor subsystems needs to be modular for ease of OTA upgrade and
> > development.
> > 
> > As a result of which each feature/techpack (viz. audio, video etc.) have
> > its own independent repositories where it can build its modules and
> > overlay DT (soc & board).
> > Kernel is separate and provides only the core images and base DT (soc &
> > board).
> 
> The build system should serve the needs of the project, not the other
> way around.  If the feature repos can emit a dtbo, why can't they emit
> a dts as well?
> 
> > So, dtbo is the option to build it independently and at last we have
> > options to either "overlay all dtbo at boot-time" or "merge it at build-time and
> > overlay final dtbo at boot-time".
> > 
> > This build-time merge is done outside techpack build system.
> > But if we want to have dts from each techpack, then techpack-build system needs
> > to communicate somehow to know which dts to include with which base dts.
> 
> Why is that a harder problem than knowing which dtbo?  The dtbo must
> have been built from some sort of dts, no?
> 
> > This is not possible at build-time between techpacks.
> 
> The build system is not immutable - you're proposing changes to libfdt
> and/or dtc; why not changes to the build system?
> 

Hi David,

As each techpack operates in a sandboxed environment with its own build
structure, all dependencies are managed within that sandbox, so the
techpack owner only needs to focus on their own module. Since each
techpack produces a compiled DTBO, it is self-contained and can be
merged with other DTBOs without additional coordination.

If we were to use DTS files instead, every techpack owner would need to
be involved in resolving cross-dependencies, which is not scalable in
our modular setup.

> > > > > > target (besides improving the overall size of DT blobs flashed on target, Android
> > > > > > Treble also requires separation of soc and board DT bits). Bootloader will pick
> > > > > > one of the board overlay blobs and merge it with soc.dtb, before booting kernel
> > > > > > which is presented a unified DT blob (soc + board overlay).


[...]
> > > 
> > > I really don't know what you mean by that.
> > > 
> > > A few other things bother me about the situation, but maybe I'm
> > > misunderstanding.
> > > 
> > > 1) You imply you need many various of the soc.dtb as well as the
> > > board.dtbo.  How does that come to be the case?  Isn't there a fixed
> > > set of SoCs with known features?  Remember that device trees should -
> > > as much as is possible - describe just the hardware, not how it's to
> > > be configured or used.
> > > 
> > > 2) To a certain extent the same concern applies to boards.  What's
> > > controlling when the extra features are needed?  Are extre pieces
> > > physically connected on?  Is it controlled by on-board switches?
> > > Something else?
> > > 
> > 
> > We do have a fixed set of SoCs, but there can be multiple boards using same SoC.
> > socX-featureZ.dtbo & boardY-featureZ.dtbo is describing one of the
> > feature (viz. audio, video etc.) for each soc & board respectively.
> 
> But why does the set of features on a SoC vary?  Generally, by its
> nature, a SoC has the components it has, they're not variable.
> 

SoC features are fixed, but different boards may utilize different
subsets of those features which are also fixed. To support multiple
board combinations efficiently, we select DTs feature-wise, allowing
reuse of SoC/feature overlays across boards.

> > > 3) What exactly is costing the additional time when applying may
> > > .dtbos at boot time.  Combining many together at build time will
> > > obviously result in a larger dtbo with more fragments that will itself
> > > take longer to apply.  I can certainly believe it's still faster
> > > overall, but it's not obvious to me why,  Understanding that will
> > > allow us all to reason better about what's a good approach here.
> > 
> > In our setup, we have 7 overlays (boardY-featureZ.dtbo) for 1 of the board
> > (soc overlay (socX-featureZ.dtbo) are different so excluding that here).
> > So, if we do overlay-merge at build-time we are saving ~60% boot-time spent on
> > overlaying the board dtbo's.
> 

I realized there was an error in my earlier boot-time improvement
estimate due to some unintended factors in measurement. The actual
improvement is around 30%, not 60%. Apologies for the confusion.

> Well, sure,  but I'm trying to understand *why* doing it all at once
> is so much faster.  I know the code for overlay application, and
> there's not an obvious large cost per-dtbo - I'd expect the time to be
> dominated by things that are per-fragment or (roughly) proportional
> to the total size of the applied dtbos.  That leaves two possibilities
> that I can see:
> 
> 1) the later dtbos are frequently overwriting properties in the
>    earlier dtbos, resulting in a final dtbo substantially smaller than
>    the sum of the original dtbos.  This seems implausible to me given
>    the structure you've described - I'd expect each dtbo to be
>    altering different parts of the tree.
> 
> 2) There *is*, contrary to my intuition, a substantial once off cost
>    for each fdt_overlay_apply() call, independent of the dtbo's size.
>    If that's the case, we should understand why and see if it can be
>    mitigated.  That would help all use cases, not just your one.
> 
> In short, adding a substantial new feature to dtc/libfdt seems
> premature without better understanding the cause of the poor
> performance you're seeing.

Thanks, Alex. I appreciate your thoughtful analysis and the effort to
understand the performance aspects of overlay application.

That said, I think we might be diverging from the core intent of this
patch series. The goal here isn’t to optimize or fix the current overlay
mechanism, but to introduce a new capability that’s currently missing:
the ability to merge two DTBOs directly.

Let’s set aside how our build system produces multiple DTBOs. Imagine a
scenario where we have one DTBO from our side and another from an OEM
and we don’t have the source for the OEM overlay. If we want to test
how the OEM overlay behaves in combination with ours, there’s no
straightforward way to do that today. This is where DTBO-to-DTBO merging
becomes useful.

So instead of focusing on optimizing fdt_overlay_apply(), this patch
series introduces support for a currently missing use case: merging
multiple DTBOs into a single overlay at build time.


-- 
Regards,
Wasim