Re: Configuration, component and repository versioning (was Re: Upgrade issues. VM won't start.)
Eugeniy Kozhanov <[email protected]> Thu, 15 May 2025 12:57:53 +0200
| Newsgroups | gmane.comp.micro-kernel.l4.devel |
|---|---|
| Message-ID | <CAMWQVGPN8wWPU=Ak6uTH9PTHBUAFVXG+=hha+Mcb7YgYKH_STg@mail.gmail.com> |
--===============3781801507653026285== Content-Type: multipart/alternative; boundary="00000000000059aa9606352a8a4a" --00000000000059aa9606352a8a4a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi everyone, > I want, would rather that you have a fully-blessed-and-tested release all rolled up in a tar (or zip) file for download it seems like a suitable solution for one-off cases. However, I believe it may lead to significant overhead when maintaining the final product, as it would require manually tracking the history of each package and compiling change logs. In my opinion, it's sometimes worth putting in a bit more effort at the beginning to ensure a smoother path forward later on. :) I=E2=80=99m just an outside observer, but I=E2=80=99d like to share my prop= osed solution in the hope that it might help address some of the challenges you're facing. 1) using a project manifest format (github.com/kernkonzept/manifest) aligns well with the repo tool (gerrit.googlesource.com/git-repo), which is used by AOSP and several other large projects consisting of 100+ sub-projects, each with its own commit history, etc. In practice, we could explore applying this tool for version management in our current project as well. repo also has broader documentation and more usage examples available online, which could be beneficial. 2) with minimal changes, the manifest could also be used to create local versions of repositories. For example, the following command can be used to initialize a local mirror based on a manifest (my rpersonal example): repo init --mirror -b repotool_support -u https://github.com/EugeniyKozhanov/l4re_manifest.git repo sync -j9 3) additionally, you can create mirrors for each release and use them to clone and rebuild the entire project tree with a single command. This approach is almost like using .tar archives, but with the advantage of preserving full commit histories and enabling centralized synchronization across all projects. 4) you can also centrally tag your set of subprojects in your local mirror. This makes it easy to mark consistent states across all components of your system and later reference or rebuild them as needed, with full traceability: repo forall -c 'git tag -a v2025.05 -m "Release tag for May 2025"' repo forall -c 'git push origin v2025.05' *P.S.* It's possible that *Hammer* is also capable of doing all of this. But I must admit =E2=80=94 even after trying to understand its philosophy, = I found it very difficult to use without extensive documentation, examples, or community support. I even had to recall some long-forgotten Perl skills, which I was really hoping to avoid. I worry that many newcomers might be discouraged from even starting with it if they can=E2=80=99t easily underst= and how it works or install the missing dependencies on their systems ( https://github.com/EugeniyKozhanov/ham/pull/1/files). Thanks! Best regards, Am Do., 15. Mai 2025 um 00:34 Uhr schrieb Richard Clark < [email protected]>: > Marcus, > > I, for one, rather than having to learn yet-another-tool that doesn't > quite do what I want, would rather that > you have a fully-blessed-and-tested release all rolled up in a tar (or > zip) file for download. You could even have > a git repo just for the tar files. One caveat, being that it must contain > ALL the repos and possible little pieces > that could possibly go with it. One thing I hate about git is sub-repos > that get lost or are not pulled down and > cause severe headaches trying to match versions. Like lwip, or > virtio_switch, or any of the other couple dozen > packages that don't get pulled down with the new ham build process. I'd > like to be able to go to one place, > use a command I already know, and get a fully functional, fully populated= , > "blessed" version. You do have your > old downloads site with a tgz file, but even that doesn't contain all the > packages any more. This is the frustrating part. > I need a way to get a snapshot of everything that could possibly go > together in a release whether I need it or not, because > some day I will need it and then the version I need will no longer be > available. So, not a "demo" or "example" > version, but a full-source-everything-including-the-kitchen-sink version. > > > Thank you for considering our opinions! > > > Richard > > > -----Original Message----- > From: Marcus H=C3=A4hnel <[email protected]> > Sent: Wednesday, May 14, 2025 10:53 AM > To: Paul Boddie <[email protected]>; [email protected] > Subject: Re: Configuration, component and repository versioning (was Re: > Upgrade issues. VM won't start.) > > On Mon, 2025-05-12 at 22:37 +0200, Paul Boddie wrote: > > On Monday, 12 May 2025 15:09:52 CEST Marcus H=C3=A4hnel wrote: > > > > > > On Mon, 2025-05-05 at 11:25 +0000, Richard Clark wrote: > > > > > > > But that brings me to a bigger question. > > > > How do I fetch only Long-Term-Support or Fully-Tested-and-Blessed > > > > versions? Github is woefully lacking in proper version support. > > > > I can't send random untested code to my customers. > > > > > > All code we push to Github went through our internal QA process, > > > running compile checks for all our supported architectures as well > > > as an extensive test suite on different platforms and > > > configurations. So from that point of view I would say you can > > > consider all code pushed to Github as =E2=80=9CFully-Tested-and-Bless= ed=E2=80=9D. > > > > I don't have a strong opinion about quality assurance for people > > wanting to provide solutions for paying customers, but I have > > personally wondered how I might successfully and conveniently > > reproduce repository configurations when creating new L4Re development > environments. > > > > For example, if I decide to work on support for a new board, I might > > want to replicate the L4Re configuration I have been using for another > > board. Starting from scratch, it was possible to use the ham tool, but > > despite it apparently maintaining version details for the different > > repositories, it wasn't particularly clear how one might preserve or > > export that metadata for further use. > > > > I now see that there is another tool involved: > > > > https://l4re.org/getting_started/bob.html > > > > Although that doesn't seem to replace the ham tool: > > > > https://l4re.org/getting_started/make.html > > > > Naturally, one might say that this is the point at which anyone > > serious-enough about using L4Re would get in touch with Kernkonzept > > and start talking business, but such a lack of clarity tends to > > suggest that either there aren't particularly adequate solutions for > > such fundamental needs or that any adequate solutions that may exist > > aren't for people merely investigating or evaluating the technology. > > > > Again, it isn't my concern if there's a business decision involved > > that everybody feels comfortable with, and if there's a steady stream > > of interested customers that seems to justify such a decision, but I > > could easily see potential users going elsewhere if the answer to > > simple questions is "talk to us". Even in reasonably large > > organisations, hitting an approval barrier that "talk to us" or > > "register your interest" represents can be a strong disincentive, > especially if other solutions exist. > > > > I accept that my opinion isn't important, however, since my own > > activities are confined to my own interests and driven by a general > > belief that L4Re represents a reasonable foundation for certain kinds > > of systems. That there isn't exactly much of a public community around > > L4Re could also be regarded as a disincentive for potential adopters, > which is unfortunate. > > > > Paul > > Hi Paul, > > thank you very much for your thoughtful message =E2=80=94 it's really app= reciated. > > First of all: your opinion absolutely matters. The input from users like > you, who engage deeply and share candid feedback, helps us make L4Re > better. While we are aware that L4Re is still quite niche and the communi= ty > small, we want to support it as best we can and would love to see it grow= . > > I think you're raising a different, but equally important, point compared > to what Richard was asking. My original response focused on the idea of > providing a fully tested release for a specific combination of software > configuration and hardware, which is understandably difficult for us to > maintain as part of the open-source offering given our limited resources. > > Your concern =E2=80=94 about being able to reproduce a known working stat= e of your > development environment =E2=80=94 is much more fundamental. You're right:= this > should be straightforward, and if it's not, then it's something we want t= o > improve. Ease of use and accessibility are important to us, and sometimes > we=E2=80=99re just too close to the system to see where friction arises = =E2=80=94 so thank > you again for pointing this out. > > To clarify one key point: we don=E2=80=99t deliberately withhold features= or > usability improvements from the open-source version of L4Re to push peopl= e > into commercial contact. That=E2=80=99s not our business philosophy. In f= act, that > would go directly against our goal of getting L4Re into more hands and > making it easier to work with. > > Some of the convenience features =E2=80=94 like release tagging =E2=80=94= do exist in our > customer repositories, but it=E2=80=99s more of a workflow habit than a c= onscious > decision to exclude them from GitHub. No one had brought up the need for > that kind of reproducibility in the open repo so far =E2=80=94 and now th= at you > have, let=E2=80=99s fix it. > > Would something like weekly tags on GitHub help you? For example, a tag > like `l4re-2025-05-14` that you could use with `ham checkout` to reproduc= e > that specific state? > > Also, ham already supports pinned revisions in the manifest (`revision` > attribute in `project` tags), so you can share a complete and reproducibl= e > state that way as well. But I agree that this could be made more convenie= nt. > > One possible improvement could be a `ham create-pinned-manifest` > sub-command that generates such a manifest from your current state. That= =E2=80=99s > not trivial =E2=80=94 it would require resolving different remotes and ch= ecking > that all commits are actually reachable in one of the remotes =E2=80=94 b= ut it=E2=80=99s > definitely doable. If you're interested, feel free to open a proposal or > even an issue on the ham GitHub repository =E2=80=94 we=E2=80=99d love to= hear your > thoughts or collaborate on a solution. > > Would any of these ideas help in your workflow? Do you have something els= e > in mind? We're always happy to improve L4Re together with the people who > use it. > > Best regards, > > - Marcus H=C3=A4hnel > Principal Engineering Lead > > > -- > +++ Register now for our workshop =E2=80=9CGet to know L4Re in 3 days=E2= =80=9D on > +++ October 28=E2=80=9330. Learn to design and deploy secure system > architectures for your product with L4Re: > https://www.kernkonzept.com/workshop-getting-started-with-l4re/ +++ > > --- > > Kernkonzept GmbH > Sitz: Dresden > HRB 31129 > Gesch=C3=A4ftsf=C3=BChrer: Dr.-Ing. Michael Hohmuth > > > > > _______________________________________________ > l4-hackers mailing list -- [email protected] To unsubscribe > send an email to [email protected] > _______________________________________________ > l4-hackers mailing list -- [email protected] > To unsubscribe send an email to [email protected] > --00000000000059aa9606352a8a4a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi everyone,<br><br>>=C2=A0=C2=A0I want, would rather t= hat<br>you have a fully-blessed-and-tested release all rolled up in a tar (= or zip) file for download<br><div><br></div><div>it seems like a suitable s= olution for one-off cases. However, I believe it may lead to significant ov= erhead when maintaining the final product, as it would require manually tra= cking the history of each package and compiling change logs.<br><br>In my o= pinion, it's sometimes worth putting in a bit more effort at the beginn= ing to ensure a smoother path forward later on. :)<br><br>I=E2=80=99m just = an outside observer, but I=E2=80=99d like to share my proposed solution in = the hope that it might help address some of the challenges you're facin= g.<br><br>1) using a project manifest format (<a href=3D"http://github.com/= kernkonzept/manifest">github.com/kernkonzept/manifest</a>) aligns well with= the repo tool (<a href=3D"http://gerrit.googlesource.com/git-repo">gerrit.= googlesource.com/git-repo</a>), which is used by AOSP and several other lar= ge projects consisting of 100+ sub-projects, each with its own commit histo= ry, etc. In practice, we could explore applying this tool for version manag= ement in our current project as well. repo also has broader documentation a= nd more usage examples available online, which could be beneficial.</div><d= iv><br></div><div>2) with minimal changes, the manifest could also be used = to create local versions of repositories. For example, the following comman= d can be used to initialize a local mirror based on a manifest (my rpersona= l example):<br>repo init --mirror -b repotool_support -u <a href=3D"https:/= /github.com/EugeniyKozhanov/l4re_manifest.git">https://github.com/EugeniyKo= zhanov/l4re_manifest.git</a></div><div>repo sync -j9</div><div><br></div><d= iv>3) additionally, you can create mirrors for each release and use them to= clone and rebuild the entire project tree with a single command. This appr= oach is almost like using <code>.tar</code> archives, but with the advantag= e of preserving full commit histories and enabling centralized synchronizat= ion across all projects.</div><div><br></div><div>4) you can also centrally= tag your set of subprojects in your local mirror. This makes it easy to ma= rk consistent states across all components of your system and later referen= ce or rebuild them as needed, with full traceability:</div><div>repo forall= -c 'git tag -a v2025.05 -m "Release tag for May 2025"'<b= r>repo forall -c 'git push origin v2025.05'</div><div><br></div><di= v><strong>P.S.</strong> It's possible that <strong>Hammer</strong> is a= lso capable of doing all of this. But I must admit =E2=80=94 even after try= ing to understand its philosophy, I found it very difficult to use without = extensive documentation, examples, or community support. I even had to reca= ll some long-forgotten Perl skills, which I was really hoping to avoid. I w= orry that many newcomers might be discouraged from even starting with it if= they can=E2=80=99t easily understand how it works or install the missing d= ependencies on their systems (<a href=3D"https://github.com/EugeniyKozhanov= /ham/pull/1/files">https://github.com/EugeniyKozhanov/ham/pull/1/files</a>)= .<br><br>Thanks!<br><br>Best regards,</div></div><br><div class=3D"gmail_qu= ote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">Am Do., 15= . Mai 2025 um 00:34=C2=A0Uhr schrieb Richard Clark <<a href=3D"mailto:ri= [email protected]">[email protected]</a>>:= <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Marcus,<br> <br> I, for one, rather than having to learn yet-another-tool that doesn't q= uite do what I want, would rather that<br> you have a fully-blessed-and-tested release all rolled up in a tar (or zip)= file for download. You could even have<br> a git repo just for the tar files. One caveat, being that it must contain A= LL the repos and possible little pieces<br> that could possibly go with it. One thing I hate about git is sub-repos tha= t get lost or are not pulled down and<br> cause severe headaches trying to match versions. Like lwip, or virtio_switc= h, or any of the other couple dozen<br> packages that don't get pulled down with the new ham build process. I&#= 39;d like to be able to go to one place,<br> use a command I already know, and get a fully functional, fully populated, = "blessed" version. You do have your<br> old downloads site with a tgz file, but even that doesn't contain all t= he packages any more. This is the frustrating part.<br> I need a way to get a snapshot of everything that could possibly go togethe= r in a release whether I need it or not, because<br> some day I will need it and then the version I need will no longer be avail= able. So, not a "demo" or "example"<br> version, but a full-source-everything-including-the-kitchen-sink version.<b= r> <br> <br> Thank you for considering our opinions!<br> <br> <br> Richard<br> <br> <br> -----Original Message-----<br> From: Marcus H=C3=A4hnel <<a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a>> <br> Sent: Wednesday, May 14, 2025 10:53 AM<br> To: Paul Boddie <<a href=3D"mailto:[email protected]" target=3D"_blank"= >[email protected]</a>>; <a href=3D"mailto:[email protected]= .de" target=3D"_blank">[email protected]</a><br> Subject: Re: Configuration, component and repository versioning (was Re: Up= grade issues. VM won't start.)<br> <br> On Mon, 2025-05-12 at 22:37 +0200, Paul Boddie wrote:<br> > On Monday, 12 May 2025 15:09:52 CEST Marcus H=C3=A4hnel wrote:<br> > > <br> > > On Mon, 2025-05-05 at 11:25 +0000, Richard Clark wrote:<br> > > <br> > > > But that brings me to a bigger question.<br> > > > How do I fetch only Long-Term-Support or Fully-Tested-and-Bl= essed <br> > > > versions? Github is woefully lacking in proper version suppo= rt.<br> > > > I can't send random untested code to my customers.<br> > > <br> > > All code we push to Github went through our internal QA process, = <br> > > running compile checks for all our supported architectures as wel= l <br> > > as an extensive test suite on different platforms and <br> > > configurations. So from that point of view I would say you can <b= r> > > consider all code pushed to Github as =E2=80=9CFully-Tested-and-B= lessed=E2=80=9D.<br> > <br> > I don't have a strong opinion about quality assurance for people <= br> > wanting to provide solutions for paying customers, but I have <br> > personally wondered how I might successfully and conveniently <br> > reproduce repository configurations when creating new L4Re development= environments.<br> > <br> > For example, if I decide to work on support for a new board, I might <= br> > want to replicate the L4Re configuration I have been using for another= <br> > board. Starting from scratch, it was possible to use the ham tool, but= <br> > despite it apparently maintaining version details for the different <b= r> > repositories, it wasn't particularly clear how one might preserve = or <br> > export that metadata for further use.<br> > <br> > I now see that there is another tool involved:<br> > <br> > <a href=3D"https://l4re.org/getting_started/bob.html" rel=3D"noreferre= r" target=3D"_blank">https://l4re.org/getting_started/bob.html</a><br> > <br> > Although that doesn't seem to replace the ham tool:<br> > <br> > <a href=3D"https://l4re.org/getting_started/make.html" rel=3D"noreferr= er" target=3D"_blank">https://l4re.org/getting_started/make.html</a><br> > <br> > Naturally, one might say that this is the point at which anyone <br> > serious-enough about using L4Re would get in touch with Kernkonzept <b= r> > and start talking business, but such a lack of clarity tends to <br> > suggest that either there aren't particularly adequate solutions f= or <br> > such fundamental needs or that any adequate solutions that may exist <= br> > aren't for people merely investigating or evaluating the technolog= y.<br> > <br> > Again, it isn't my concern if there's a business decision invo= lved <br> > that everybody feels comfortable with, and if there's a steady str= eam <br> > of interested customers that seems to justify such a decision, but I <= br> > could easily see potential users going elsewhere if the answer to <br> > simple questions is "talk to us". Even in reasonably large <= br> > organisations, hitting an approval barrier that "talk to us"= or <br> > "register your interest" represents can be a strong disincen= tive, especially if other solutions exist.<br> > <br> > I accept that my opinion isn't important, however, since my own <b= r> > activities are confined to my own interests and driven by a general <b= r> > belief that L4Re represents a reasonable foundation for certain kinds = <br> > of systems. That there isn't exactly much of a public community ar= ound <br> > L4Re could also be regarded as a disincentive for potential adopters, = which is unfortunate.<br> > <br> > Paul<br> <br> Hi Paul,<br> <br> thank you very much for your thoughtful message =E2=80=94 it's really a= ppreciated.<br> <br> First of all: your opinion absolutely matters. The input from users like yo= u, who engage deeply and share candid feedback, helps us make L4Re better. = While we are aware that L4Re is still quite niche and the community small, = we want to support it as best we can and would love to see it grow.<br> <br> I think you're raising a different, but equally important, point compar= ed to what Richard was asking. My original response focused on the idea of = providing a fully tested release for a specific combination of software con= figuration and hardware, which is understandably difficult for us to mainta= in as part of the open-source offering given our limited resources.<br> <br> Your concern =E2=80=94 about being able to reproduce a known working state = of your development environment =E2=80=94 is much more fundamental. You'= ;re right: this should be straightforward, and if it's not, then it'= ;s something we want to improve. Ease of use and accessibility are importan= t to us, and sometimes we=E2=80=99re just too close to the system to see wh= ere friction arises =E2=80=94 so thank you again for pointing this out.<br> <br> To clarify one key point: we don=E2=80=99t deliberately withhold features o= r usability improvements from the open-source version of L4Re to push peopl= e into commercial contact. That=E2=80=99s not our business philosophy. In f= act, that would go directly against our goal of getting L4Re into more hand= s and making it easier to work with.<br> <br> Some of the convenience features =E2=80=94 like release tagging =E2=80=94 d= o exist in our customer repositories, but it=E2=80=99s more of a workflow h= abit than a conscious decision to exclude them from GitHub. No one had brou= ght up the need for that kind of reproducibility in the open repo so far = =E2=80=94 and now that you have, let=E2=80=99s fix it.<br> <br> Would something like weekly tags on GitHub help you? For example, a tag lik= e `l4re-2025-05-14` that you could use with `ham checkout` to reproduce tha= t specific state?<br> <br> Also, ham already supports pinned revisions in the manifest (`revision` att= ribute in `project` tags), so you can share a complete and reproducible sta= te that way as well. But I agree that this could be made more convenient.<b= r> <br> One possible improvement could be a `ham create-pinned-manifest` sub-comman= d that generates such a manifest from your current state. That=E2=80=99s no= t trivial =E2=80=94 it would require resolving different remotes and checki= ng that all commits are actually reachable in one of the remotes =E2=80=94 = but it=E2=80=99s definitely doable. If you're interested, feel free to = open a proposal or even an issue on the ham GitHub repository =E2=80=94 we= =E2=80=99d love to hear your thoughts or collaborate on a solution.<br> <br> Would any of these ideas help in your workflow? Do you have something else = in mind? We're always happy to improve L4Re together with the people wh= o use it.<br> <br> Best regards,<br> <br> - Marcus H=C3=A4hnel<br> =C2=A0 Principal Engineering Lead<br> <br> <br> -- <br> +++ Register now for our workshop =E2=80=9CGet to know L4Re in 3 days=E2=80= =9D on <br> +++ October 28=E2=80=9330. Learn to design and deploy secure system<br> architectures for your product with L4Re: <a href=3D"https://www.kernkonzep= t.com/workshop-getting-started-with-l4re/" rel=3D"noreferrer" target=3D"_bl= ank">https://www.kernkonzept.com/workshop-getting-started-with-l4re/</a> ++= +<br> <br> ---<br> <br> Kernkonzept GmbH<br> Sitz: Dresden<br> HRB 31129<br> Gesch=C3=A4ftsf=C3=BChrer: Dr.-Ing. Michael Hohmuth<br> <br> <br> <br> <br> _______________________________________________<br> l4-hackers mailing list -- <a href=3D"mailto:[email protected]= e" target=3D"_blank">[email protected]</a> To unsubscribe sen= d an email to <a href=3D"mailto:[email protected]" targ= et=3D"_blank">[email protected]</a><br> _______________________________________________<br> l4-hackers mailing list -- <a href=3D"mailto:[email protected]= e" target=3D"_blank">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= u-dresden.de" target=3D"_blank">[email protected]</a><b= r> </blockquote></div> --00000000000059aa9606352a8a4a-- --===============3781801507653026285== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ l4-hackers mailing list -- [email protected] To unsubscribe send an email to [email protected] --===============3781801507653026285==--