Re: Feedback from bigger installations - installer improvment potential

Sorry I missed, too, your detailed email ;-)

On 2026-03-20 07:29, Marc Haber wrote:
> Hi,
>=20
> =3D me =3D
>=20
> I'm Marc. I have not been present much on debian-boot in the last=20
> decades, but I have been around for a while. I do remember Debian when=20
> d-i was not yet availebl. You might know me as the person taking care=20
> of adduser and sudo.
>=20
> In my other life, I am not only the can-opener for four lovely cats,=20
> but also a freelance IT consultant in Germany. I do quite some work in=20
> larger Linux environments, and customers using Debian get a (small)=20
> discount. My largest (sadly, ex-) customer is running a mid-size=20
> five-digit number of Debian installations, the majority of those on=20
> bare metal.
>=20
> =3D Why this mail =3D
>=20
> First let me thank you to provide d-i. Debian would not be here without=
=20
> you guys providing the way to get Debian on systems and running. But I=20
> am kind of concerned about recent developments in Debian, for example=20
> debian-live growing more and more into "the method to install Debian on=
=20
> Laptop and desktop" recommended in the press, social media and user=20
> groups, leaving d-i as the method to install servers, moving into the=20
> role of a second-class citizen.
>=20
> Whenever I talk with people running mid-size to large numbers of Debian=
=20
> installations, people roll their eyes when one talks about the=20
> Installer, in particular its way to partition disks. I must admit that=20
> I am not partman's best friend either.

A few years back there was an effort to introduce Growlight as a Partman=20
replacement. I even began to experiment by preliminary adding reiser4/5=20
support to Growlight code at the time. Please see old thread:
< https://lists.debian.org/debian-devel/2021/09/msg00401.html >

Additionally, I even created a small video clip taken from an Debian=20
native reiser5 installation into a VirtualBox slice instance which /root=20
was formatted with reiser5 filesystem and showcasing a test Growlight=20
invocation:
<=20
https://metztli.blog/media/blogs/calli/Bullseye-SFRN5/xonecuiltzin-5.13.1=
9-reizer4-sfrn-5.1.3.mp4?mtime=3D1636642043=20
 >

Yet, I do not know what happen to the project as it went silent.=20
Probably the developer of Growlight might be interested in working with=20
you on such an ambitious project[?!]

> Most of those installation don't use d-i to install Debian on their=20
> systems (rolling their own installer or using a golden image to clone=20
> new systems from), and I have seen sites migrate away from Debian=20
> because Anaconda is so much better automatable.
>=20
> The big installation mentioned above is the only larger installation I=20
> have ever been working with who actually uses d-i: They boot d-i from=20
> the network, using a preseed-file that is generated from their CMDB and=
=20
> after installation hand over the system to puppet, using a=20
> 400-or-so-character late_command that made my eyes water. In this=20
> installation, the inflexibility of disk partitioning has been constant=20
> gripe of the other workgroups that end up using the systems installed=20
> by this method. They all want more flexibility. I haven't worked with=20
> them in eight years so I don't know whether they still use this.
>=20
> Myself, I developed my own Debian installation method that involves=20
> booting a live linux=C2=B9 and then running debootstrap and and a numbe=
r of=20
> idempotent scripts. That was before d-i was invented and I just never=20
> got around to migrate to d-i beause my method was still working=20
> reasonably well. Nowadays, I use an ansible playbook to install Debian=20
> (which allows me to reuse code from the normal configuration management=
=20
> that takes care of my Debian systems during their lifecycle). I am=20
> therefore not very familiar with current d-i mechanisms. I do=20
> occasionally use d-i to install Debian on notebooks and desktop=20
> computers that don't run in a datacenter and in installations where=20
> only a handful of Debian systems are in use (or when the customer=20
> explicitly doesn't want automation).
>=20
> =C2=B9 you might remember Linuxcare's Bootable Business Card from the e=
arly=20
> 2000 years, and there is a reason why the first grml-small release had=20
> the codename "Zugschlus" back in - i think - 2005.
>=20
> Whenever I use d-i manually, partman shows up as my nemesis. I don't=20
> think that I ever was able to get partman non-trivial partitioning=20
> right on the first try.
>=20
> I would like to give you input to improve this. Sadly, I neither do=20
> have time nor the knowledge to be of actual help, but I'd like to offer=
=20
> my input, wishlist reports to help you improve and I am also willing to=
=20
> write docs and specs. I might be able to contribute code, but please=20
> don't expect my code to be directly useable. I am way too bad a=20
> programmer for that. But I am willing to test.
>=20
> I'd like to provide input about the following topics.
>=20
> =3D Find preseeding data, make it easy to place preseeding data =3D
>=20
> Preseeding usually means network boot or rolling your own .iso. As far=20
> as I know, if you do neither, you have to give a (network) path to the=20
> preseed file on the boot prompt otherwise. That doesn't scale, and it=20
> is even not nice for just installing a single system. I am wondering=20
> whether you have considered more default places to have the installer=20
> look for preseed files. For example, I'd like to have the following=20
> default places for preseed files in addition to the places where the=20
> installer already looks for:
>  * (any EFI partition on any hdd)/d-i/preseed.cfg
>    * that is likely to already exist or can trivially be pre-created
>  * /d-i/preseed.cfg on the (virtual) cdrom on the second CD-ROM drive
>    * having a second CD-ROM drive is trivially achievable in     =20
> virtualization
>  * /d-i/preseed.cfg on any filesystem carrying a certain label or   =20
> partlabel.
>    * that can be on the target disk as well, or on an USB key, ...
>  * /d-i/preseed.cfg on a second filesystem on the installer .iso
>    * not sure whether this will actually work with an (emulated) cd-rom=
=20
>      drive, but live linuxes use this mechanism on usb boot media to   =
=20
>   allow customization of system boot with an unchanged .iso
> The file names searched for could be mutated using a firmware asset=20
> tag, MAC address prefixes of different length (like PXE clients already=
=20
> do), ISO release numbers, etc bla foo.
>=20
> The installer then could put up a dialog "preseed file foo found on=20
> (path), do you want to use it?" by default. Of course, to facilitate a=20
> fully automated install, there must be a (preseedable) option to=20
> silence this.
>=20
> Gold Plating: If no preseed file was found, the (expert) installer=20
> could put up a dialog "No preseed file found, do you want to enter the=20
> path to one" to allow the user to add a preseed file after missing the=20
> timeout boot prompt (that could also be used as an indicator that the=20
> search for a preseed file failed).
>=20
> =3D verification of preseed files =3D
>=20
> There is the option to add a checksum to the preseed file option. The=20
> fact that md5 is ths only option offered here suggests that this part=20
> hasnt been touched in quite a while. Have there been thoughts of=20
> introducing signed preseed files, putting the public part(s) of key(s)=20
> in there and just processing preseed files that have the right=20
> signature?
>=20
> =3D Kickstart, anyone? =3D
>=20
> Other installers after a manual install leave a file in the installed=20
> system that can directly be used to preseed another installation, which=
=20
> then will run identically to the one that was just finished, just=20
> unattended. Searching for this on the Web suggests that this was left=20
> out of d-i intentionally, but I didn't find the rationale for that. I=20
> one was in charge of a classroom setting with 15 identical machines,=20
> and I would have been really nice be able to tell a tutor "Just do the=20
> install manually on machine 1, and I can then identically repeat the=20
> installation on machines 2-15".
>=20
> The method outlined in the installation manual B.3 is a bit arcane, I=20
> am not very happy about that. There could be a framework to automate=20
> this, probably removing those items that "should not be preseeded".
>=20
> Also the recommendation to use an editor in a running installation to=20
> check for possible options sounds like an excuse. Nobody should be=20
> forced to manually read debconf template files.
>=20
> There could be a framework that reads the templates when the installer=20
> is built and as a corollary generates a nice web page or an .md=20
> document giving the templates and the possible values. Or, the example=20
> preseed file (like=20
> https://www.debian.org/releases/trixie/example-preseed.txt) could be=20
> automatically augmented with the valid values.
>=20
> =3D Which preseed setting corresponds to which question in d-i =3D
>=20
> In addition, I have found it quite hard to see the connection between=20
> the questions that the installer asked and the respective configuration=
=20
> being put into the debconf database. If the Installer would write its=20
> own preseed file instead of referring people to debconf-get-selections=20
> --installer, it would be possible to put the title of the respective=20
> installer dialog in a comment, making it way easier to find out which=20
> answer to the installer dialogs has resulted in this particular preseed=
=20
> option being generated.
>=20
> =3D Interactive Preseed Preparation =3D
>=20
> I would love to have a possibility to run through a virtual installer=20
> inside a running Debian system in a window, and get a preseed file that=
=20
> contains my answers. I could live with running an actual installation=20
> in a VM and having some side-channel into that VM to interactively and=20
> directly see what is being written to the installer's debconf database=20
> without having to change virtual console hundreds of times. Maybe=20
> another virtual serial port that I could attach to would be a good=20
> method.
>=20
> =3D syslog =3D
>=20
> Can the Installer be made to optionally log to syslog once the network=20
> is up? Maybe I could give a syslog=3Dsyslogserver.example.com on the=20
> kernel command line? In an ideal world, dmesg and everything that is=20
> already logged at this time would be (optionally) dumped to the syslog=20
> server at the beginning, and a meaningful host name would be given in=20
> the syslog messages as well.
>=20
> =3D Make early_command easier =3D
>=20
> Would it make sense to automatically search for an=20
> (preseed|partman)/early_command in the same way than it looks for the=20
> preseed file? That might be an easier way to influence installation and=
=20
> partitioning without having to do a full preseed? I guess that it would=
=20
> be easier for many people to just drop a shell script in some=20
> pre-defined place than having to grok the preseeding file format.
>=20
> When this is addressed, a similar process could be used for the=20
> late_command that many installations use to hand over the system to=20
> puppet, ansible or relatives.
>=20
> =3D Partitioning =3D
>=20
> This is a huge issue in Linux installation, always. I have yet to see a=
=20
> Linux installation tool for any distribution that does this as flexibly=
=20
> as big installations want. Most installations that I am aware of that=20
> have rolled their own installation procedure did that because of the=20
> inflexibility of the distribution-provided partitioning tools. This is=20
> probably caused by the fact that partitioning naturally happens before=20
> anything is installed and that a partitioning process therefore is=20
> spartanic at best.
>=20
> I don't expect Debian to be the first distribution to provide a perfect=
=20
> partitioning tool. But I would like to have some methods to=20
> bypass/replace partman while still being able to use Debian Installer=20
> proper.
>=20
> I think there is too much documentation out there that explains how to=20
> use partman. Therefore, partman as it is should stay. While part of me=20
> is all for a rewrite (see above), I would also be happy with=20
> alternative ways to partition for a Debian installation, while still=20
> making it possible to make use of the Installer.
>=20
> =3D=3D early_command =3D=3D
>=20
> I could for example imagine having an early_command (or even a partman=20
> early_command) that does the partitioning in the way I want it.=20
> Probably the easiest way I can imaging would be making partman and the=20
> rest of the Installer assume that if something is mounted on /target at=
=20
> the point when partman gets invoked, that is already the way the system=
=20
> should be installed. That way, partman could (preseedably) ask "do you=20
> want to use /target as already mounted" and if yes, just advance in d-i=
=20
> wihout even doing anything.
>=20
> =3D=3D pre-partition =3D=3D
>=20
> Just in case one is too lazy to write a proper early_command (which=20
> probably must run in busybox with a rather limited userland), the=20
> system could first be booted to a rescue system with more capabilities=20
> to do the partitioning and building the file systems, with a much=20
> simpler early_command that only needs to mount the prepared filesystems=
=20
> once the installer is booted.
>=20
> =3D=3D fstab import =3D=3D
>=20
> Developing the use cases outlined above further leads to the idea of=20
> the preparation system leaving an /etc/fstab file around that the=20
> installer could directly use to mount /target and as /etc/fstab for the=
=20
> installed system. That would completely eliminate the need for an=20
> early_command if partitioning is done before the actual installer is=20
> booted.
>=20
> =3D=3D YAML Import =3D=3D
>=20
> Partman could be extended in a form that allows it to read YAML, which=20
> could be  * pasted into a free-text field of the installer
>  * read from the network
>  * searched for in a similar way like preseed and (early|late)_command
> That YAML input could be pre-crafted or generated to allow the=20
> partitioning to be exactly like people want it, without having to learn=
=20
> partman's preseed partitioning syntax, and without having to live with=20
> partman-auto's limitations.
>=20
> After having written this, I feel that I begin to be a fan of the=20
> early_command approach.
>=20
> That's what I have for you today, I hope that we can discussion the=20
> options and what I missed. Once the discussion is finished, I intend to=
=20
> file wishlist bug requests against the appropraite parts of d-i to=20
> suggest implementation of our results, if feasible.
>=20
> I will join #debian-boot later today and will be available there in the=
=20
> next weeks.
>=20
> Greetings
> Marc

(I am cc'ing the Growlight developer and hope he does not mind.)

--=20
Best Professional Regards.

--
Jose R R
http://metztli.it
-------------------------------------------------------------------------=
--------------------
Download Metztli Reiser4: Debian Trixie w/ Linux 5.17.15-2 AMD64
-------------------------------------------------------------------------=
--------------------
feats ZSTD compression https://sf.net/projects/metztli-reiser4/
-------------------------------------------------------------------------=
------------------
Official current Reiser4 resources: https://reiser4.wiki.kernel.org/