Effects of new nbcon console interface and boot times

Hello all,

With the recent addition of the new "nbcon" interface to the Linux
console, there is now support for threaded console writes, handled by
a dedicated kthread. While the motivation behind the nbcon interface
was to support bringing real-time preemption into the mainline kernel,
I have been interested in investigating what impact nbcon has on boot
times. I thought this may be of interest to some of you. Some key
points that I have found:

- `register_console()` registers a newly initialised serial port as a
console for printk output. At which point, the printk buffer is
flushed out to the serial device. Even with an nbcon console, this
initiall flushing is handled by `console_flush_all()` in a synchronous
manner, and this is the cause of delays during system boot up. I have
also confirmed this as register_console can take > 10 seconds to
execute at lower baudrates when there are many printk messages to
flush.

- The nbcon printer kthread does not take over the task of writing to
consoles until after `register_console()` has set up the console.

- Depending on how late in the boot process the serial port drivers
are actually initialised, there can be a lot of messages already in
the printk buffer that need to be flushed. I found that with a
Raspberry Pi 3B+, for example, due to dependencies on clocks and other
system hardware, the serial drivers were actually initialised as part
of `deferred_initcall` very late on in the boot process.

I believe it should be possible for nbcon-enabled consoles to have the
initial printk flushing be handled by the nbcon kthread, rather than
by `console_flush_all()` as is currently the case. Subsequent calls to
printk after `register_console()` already appear to execute faster due
to the actual work being offloaded to the kthread. This would, in
theory, significantly reduce boot times when using a serial console,
without having to resort to adjusting the loglevel and losing the
ability to debug boot issues.

If you would like to read more about this investigation, I have
written a blog post that you can find here:
https://www.thegoodpenguin.co.uk/blog/the-impact-of-non-blocking-consoles-nbcon-on-linux-boot-time/

I would be very interested to know if this is something that anyone
has picked up and is looking into?

Looking forward to hearing your thoughts on this.

Thanks,
Michael