Re: Debian-based product build system

Hi Baurzhan,

I tried to bitbake with Isar. This is very interesting approach for me.
I would like to think how could we collaborate.

I am a member who is creating the meta-debian layer for Poky build system.
(https://github.com/meta-debian/meta-debian)
The layer focuses to make a customized filesystem image and kernel with
Poky (bitbake) build system. The layer includes bitbake recipes for
cross-compiling by using the Debian's source codes.

The Isar currently focus to create root file system by pre-compiled binary
packages. Isar is also able to compile own program while creating rootfs
images by using user-level emulation (e.g. qemu-arm-static) with native
toolchain.

Each approach have pros and cons.
To make a customized Linux environment, I think there are many requirement
with different aspects. And some of requirements may conflict and difficult
to ensure at same time. There are some example.

1. Need to customize compile options
   1.1 add/delete functionality
   1.2 add/delete package dependency
   1.3 Change optimization option
2. Want to use compiled binary
   2.1 Use Debian original binary packages
   2.2 Create a cache storage for compiled binary packages
3. Reproduce a set of root filesystem and kernel whenever we want

For me, I am happy with 1 and 3 by using meta-debian layer recipes. But sometimes,
I would like to use binary packages to re-create or to make modification on existing
images. In that case, binary package cache may help a lot and Isar's approach is very
useful. But I am still not sure my understanding is correct or not.
Does Isar have ability to use other pre-compiled binary package store?

Best regards,
Yoshi

On 2016/03/31 2:03, Baurzhan Ismagulov wrote:
> Hello,
>
> we're developing an integration system for building Debian-based products, isar
> [1]. It is essentially a set of commands for building product packages and
> installing them into a Debian root filesystem to produce a complete working
> image. It aims at supporting the whole product development lifecycle:
>
> 1. Create a base system:
>     * Define a list of binary Debian packages as a base system.
>     * Define base system customizations, such as inittab, fstab, etc.
>
> 2. Use a clean environment (chroot / VM) for building the following components:
>     * Product's modified Debian packages.
>     * Product's own driver, library, and application packages.
>
> 3. Support multiple diverse targets in one project:
>     * Debug / release builds.
>     * Images for different products.
>     * Images for different hardware platforms.
>
> 4. Good support for typical project use cases, such as:
>     * Create complete, bootable images automatically, with one command.
>     * Do that reproducibly, for any older version.
>
>
> = Prior Art =
>
> The task is trivial enough that everyone develops his own scripts for that [2].
> After maintaining such scripts since 2004, we'd like to have something better.
> Reasons:
>
> * The scripts have become a mess and are difficult to maintain.
>
> * They don't scale well for the following use cases:
>
>    * Maintaining several products (sharing or not sharing components) leads to
>      exploding complexity.
>
>    * The same applies for several levels of development (upstream software,
>      Debian packages, hardware vendors' BSP components, software vendors'
>      libraries, product's own components).
>
> * They don't offer structure, and developers happen to solve every next problem
>    in a different way.
>
> * "Every tool is tailored for the developer's use case and personal taste" [3].
>
> A build system should Just Work(tm), be easy to use, offer structure, and be
> flexible. The best would be to combine individual efforts to avoid solving the
> same problems many times and constantly improve a single toolset that is
> suitable for various use cases.
>
> The closest to that vision is ELBE [4]. It solves the right problem and has
> many features OOTB -- very important if you want to focus on your end product
> and not develop another build system. However, its current implementation
> doesn't scale well for our use cases, and we expect maintainability issues in
> the projects using it.
>
>
> = How Isar Works =
>
> Isar follows the one job, one tool approach. Integration is achieved by using
> BitBake, a build tool similar to make. It is the tool behind the source-based
> Yocto embedded Linux distribution. With BitBake, one can define files with
> build targets (recipes like coreutils, busybox), their dependencies (other
> recipes like libc), and rules how to build them (shell or python statements).
> All recipes together are collectively called "metadata" as opposed to the
> source code itself, which may contain configure, Makefile.am, debian/, etc.
>
> Isar is implemented as a set of BitBake recipes. There is one recipe per source
> package. Build target is the Debian package. Since download ("fetch") and build
> rules ("tasks") in Debian are uniform thanks to apt and dpkg, they are defined
> once in a BitBake class and don't have to be duplicated in every recipe. The
> source code of individual packages may be a part of an isar-based project, or
> it may be in a different (public or private) repo. The recipes check out the
> software packages from their repos and call dpkg-buildpackage on them. The
> packages are later installed into the root filesystem using multistrap. There
> are separate recipes for creating the build chroot and the final rootfs.
>
> Conceptually, the whole build system can be seen as a result of splitting one
> monolithic script into chunks within a predefined structure. Recipes, in turn,
> are grouped in directories ("layers") by distro, vendor, product, or target
> platform.
>
> The structure is achieved as follows:
>
> * On file level, by BitBake recipes. Different packages are built in different
>    recipes. Recipe execution order results from dependencies specified in the
>    recipes.
>
> * On directory level, layers:
>
>    * meta: Base system recipes (e.g., coreutils).
>
>    * meta-VENDOR-bsp: Hardware vendor's recipes (e.g., drivers, libs).
>
>    * meta-PRODUCT: Product recipes and customizations (e.g., product apps and
>      libs, partitioning, splash screen).
>
>    Layer directories may be a part of a project repo, or be external to it. In
>    this way, anyone can fork an upstream repo, add his own layers (possibly
>    extending or overriding the core recipes), and pass that to his downstream
>    without having to fork and modify the core recipes (thus reducing maintenance
>    effort). The desired layer mix is chosen in the build config file by the end
>    developer.
>
> * Different target platforms are chosen by specifying the MACHINE variable
>    in the BitBake configuration file in the local build directory.
>
> * Component sharing between products is achieved by dependencies specified in
>    BitBake recipes. If several products share 90% of components, their top-level
>    recipes depend on the recipe of a common base system, similar to Debian
>    metapackages.
>
>    Component variation (kernel, U-Boot) is achieved by the Provides: mechanism,
>    like in Debian.
>
>    Depending on the extent of differences, product variation may be achieved by
>    different build targets or co-existing meta-PRODUCT1, meta-PRODUCT2 layers.
>
> With added structure, flexibility of "raw" scripts is also preserved: BitBake
> recipes contain procedural shell or Python code. New tasks and their order can
> be defined in any recipe; the development is not blocked if the build system
> doesn't provide an appropriate hook.
>
> An important issue is readable SCM history. With isar, editing one big build
> script or mixing changes to code and blobs (e.g., tarballs) can be avoided.
> Patches tend to be very local and very readable.
>
>
> = Workflow =
>
> The workflow is similar to that of Yocto.
>
> To create a project based on isar, its repository is cloned to become the
> project repository. Then, recipes for project-specific packages are added.
> Debian package recipes are almost one-liners with no procedural code necessary
> by default.
>
> To build the images, a build directory must be created and configured. This is
> done with a helper script: isar-init-build-env <dir>. The config files in
> <dir>/conf/ can be customized.
>
> System images are built with one command: bitbake <image name>. A default image
> recipe is supplied in the isar repo. Packages are built natively using a chroot
> with qemu (cross-building is also possible). Root filesystem customizations go
> into multistrap scripts. Repeatability is achieved by versioning apt repos and
> referencing the tags in project recipes.
>
>
> = Advantages of Isar =
>
> * Modularity, flexibility and scalability through using BitBake.
>
> * Proven-in-use structure and workflows of the Yocto project.
>
> * Re-use of familiar, mature tools like BitBake, multistrap, dpkg, apt, qemu.
>
> * Re-use of pre-built binary Debian packages rather than building them from
>    scratch every time.
>
>
> = What's next? =
>
> Siemens is using a predecessor of this system and would like to develop it
> further. What we are looking for is a discussion. Do you see a need for uniting
> forces to develop such a system as a community? Would you use it for your next
> project? Do you think this is the optimal approach? Feedback and questions
> welcome. We'd like to combine efforts under a shared vision rather than having
> everyone developing his own tool.
>
>
> References:
>
> 1. Proof-of-concept implementation: https://github.com/ilbers/isar
> 2. http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/
> 3. http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/24
> 4. https://lists.debian.org/debian-embedded/2016/01/msg00000.html
>
>
> With kind regards,
>