Re: Moving firmware to userland

Hi,

> On 26 Jul 2017, at 04:38, Sepherosa Ziehau <[email protected]> =
wrote:
>=20
> Looks good to me.  I'd leave some spare fields in the in-image record
> head (zero filled), for future extension of the in-image record head.

...and/or add a record version field to the in-image record head

> On Sun, Jul 23, 2017 at 6:00 PM, Jan Sucan <[email protected]> =
wrote:
>> Hello,
>>=20
>> this is the proposal for the new firmware file format.
>>=20
>> ---- Firmware file format ----
>>=20
>> Data structure used is a singly linked list. Operations on it have a
>> linear time complexity (worst-case), but because there is only a few
>> records it is sufficient.
>>=20
>> Conditions for ordering of the records:
>> - checksum record is the first
>> - firmware image data is the last
>>=20
>> Checksum and image data are the only mandatory records.
>>=20
>> Start of the linked list is at the end of the file (and it continues
>> "backwards") so the file can be resized more easily when modifying
>> records (firmware attributes).
>>=20
>> Each record has an ID (1 byte) which describes the meaning of it and
>> its allowed values. ID is followed by the length of attribute data in
>> bytes (4 bytes, little-endian). The next record starts immediately
>> after data of the current record.
>>=20
>> Each record is read-write for a user except for the two read-only
>> records:
>> - checksum
>> - firmware image data
>>=20
>> ---- Records ----
>>=20
>> - Checksum: string (checksum will cover everything after this record)
>> - License text: string (file path or complete text of the license, no
>> license ack is needed if not defined)
>> - Firmware version: integer number
>> - Firmware image endianness: boolean (little-endian, big-endian)
>> - Firmware image word size: 1,2,4 or 8 (used for endianness)
>> - Firmware image data: array
>>=20
>> ---- Operations with firmware files ----
>>=20
>> - Initialize raw firmware file: create linked list structure and
>> checksum and firmware image data records
>> - Uninitialize firmware file: remove linked list structure and keep
>> only the original raw firmware data
>> - Check firmware using the checksum
>> - Set record's value: only for read-write records (this also creates
>> record which is not defined yet)
>> - Get record's value: for read-only and read-write records
>> - Remove record: only for read-write records
>>=20
>>=20
>> jan
>>=20
>>=20
>>=20
>> On Thu, 20 Jul 2017, Sepherosa Ziehau wrote:
>>=20
>>> Hi,
>>>=20
>>> On Wed, Jul 19, 2017 at 4:27 PM, Jan Sucan <[email protected]> =
wrote:
>>>>=20
>>>> That's good and interesting idea. Attaching the config to the image
>>>> could unify the way in which config info is saved for both built-in
>>>> and non-built-in images, and also unify a code for parsing the =
config
>>>> info in the kernel firmware subsystem. Possibilities for firmware =
file
>>>> formats:
>>>>=20
>>>> - uuencoded
>>>> Pros: Config can be added, modified and removed by a text editor
>>>> Cons: .uu files are about 35% larger, need to implement uudecoding =
in
>>>>       the kernel, config correctness checked at firmware loading =
time
>>>>=20
>>>> - binary
>>>> Pros: firmware data size, user-interface of the new tool can be
>>>>       better suited for given purpose than a text editor (which is =
too
>>>>       general), config correctness checked at config modifying time
>>>> Cons: need to create new tool for handling attached info (something
>>>>       like kenv(1) for firmware)
>>>>=20
>>>> For binary firmware it will be necessary to also add some checksum
>>>> data in case when config is not attached and firmware data is also
>>>> valid config. Checksum could be computed only from firmware data or =
it
>>>> can cover the config data too.
>>>>=20
>>>> I think, that the second file format would be better. I can try to
>>>> propose a format for the attached config and post it here for a
>>>> review.
>>>=20
>>>=20
>>> Yeah, sure, go ahead :)
>>>=20
>>> Thank you,
>>> sephe
>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>> On Tue, 18 Jul 2017, Sepherosa Ziehau wrote:
>>>>=20
>>>>> I'd choose plain text config per-image.  BTW, is it possible to =
attach
>>>>> the config at the beginning/end of the image?  Assuming we can =
force
>>>>> some image file creation rules.
>>>>>=20
>>>>> On Tue, Jul 11, 2017 at 8:20 PM, J=C3=A1n Su=C4=8Dan =
<[email protected]> wrote:
>>>>>>=20
>>>>>>=20
>>>>>> Hello,
>>>>>>=20
>>>>>> I would like to introduce a few ideas about the new firmware =
subsystem.
>>>>>> I
>>>>>> assume that it should not require adding a new system tools or
>>>>>> modifying
>>>>>> boot process.
>>>>>>=20
>>>>>> Simplification is the first. It would be good to remove =
parent-child
>>>>>> relationship and corresponding functionality. It would =
significantly
>>>>>> simplify firmware handling. Its only practical use is when there =
are
>>>>>> multiple images in one loadable kernel module. The module can be
>>>>>> unloaded
>>>>>> when all children are not in use. Usage of the children images is
>>>>>> tracked
>>>>>> through the counter for the parent image. If images will not be =
placed
>>>>>> inside loadable kernel modules, the parent-child mechanism won't =
have
>>>>>> any
>>>>>> practical meaning. I think, currently the mechanism is not used
>>>>>> anywhere
>>>>>> in
>>>>>> the DragonFly system and if it was, it could be easily replaced =
by
>>>>>> putting
>>>>>> every child image in its own module without modifying drivers.
>>>>>>=20
>>>>>> There are two use cases according to who will provide firmware =
images
>>>>>> to
>>>>>> the
>>>>>> system:
>>>>>>=20
>>>>>> - developers of DragonFly BSD (they can modify and rebuild the =
system)
>>>>>> - third-parties (they should not be required to modify and =
rebuild the
>>>>>> system)
>>>>>>=20
>>>>>> Providing a new non-built-in firmware should not require:
>>>>>>=20
>>>>>> a) system rebuild
>>>>>> b) system reboot
>>>>>> c) loading a kernel module
>>>>>>=20
>>>>>> All firmware images needs to have some information attached (at =
least,
>>>>>> if
>>>>>> ack with a license is needed) which should be
>>>>>>=20
>>>>>> d) stored persistently.
>>>>>>=20
>>>>>> The question is where to save the information for non-built-in =
images.
>>>>>> For
>>>>>> built-in images the info is saved in the kernel. Some =
possibilities
>>>>>> are:
>>>>>>=20
>>>>>> - A plain text configuration file per image.
>>>>>> This satisfies requirements a), b), c) and d). Disadvantage is
>>>>>> parsing
>>>>>> the
>>>>>> text content.
>>>>>>=20
>>>>>> - Kernel environment variables.
>>>>>> The problem with this is that they are initialized:
>>>>>>   - by some kernel code (this violates a), b), c) )
>>>>>>   - manually from userland by kenv command or kenv() libc =
function
>>>>>> (violates d) )
>>>>>>   - in the loader.conf (violates b) since the content is =
processed at
>>>>>> boot
>>>>>> time)
>>>>>> Advantage is that the parsing is done by the kernel.
>>>>>>=20
>>>>>> There would be two firmware sources: kernel and filesystem. In =
case of
>>>>>> the
>>>>>> same image names, user could have a choice by setting a kernel
>>>>>> environment
>>>>>> variable, firmware from which source has higher priority and will =
be
>>>>>> provided to consumer.
>>>>>>=20
>>>>>>=20
>>>>>> I will be happy for a comments and ideas
>>>>>> jan
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> --
>>>>> Tomorrow Will Never Die
>>>>>=20
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Tomorrow Will Never Die
>>>=20
>>=20
>=20
>=20
>=20
> --=20
> Tomorrow Will Never Die
>=20



--
Bob Bishop
[email protected]