Re: The new Buildbot: what’s happ ening, what needs to happen

Hi guys,

Thank you both for your question about the worker requirement, it is a
great question to ask.

To me, the biggest argument for such a requirement is that the bus
factor on these ports is really low, usually just one person. Making
sure we always have a worker image helps to reduce the risk of dead
ports by ensuring that at least the build process for all of the
“official” (i.e. in our tree) ports is documented, reproducible, and
(strongly preferably) updatable, so binaries can be created by
automation or any other team member instead of relying on a single
person. There are at least four examples I can point to that indicate a
real need for this kind of strict requirement for any of our official ports:

1. Code for a new “official” 3DS port was added to our tree in 2016, and
then we never actually released for this platform because the porter
disappeared before any release could be made;
2. When the usual Windows porter was lost around the 1.9 release time,
there was apparently a non-trivial amount of time & effort needed to get
a working build system up and running again, and then I essentially had
to re-do this work yet again to update Buildbot, which just wastes the
core team’s limited time;
3. Porters do stuff which is totally undocumented, like the Nero images
which are currently being created for the Dreamcast port. If the porter
disappears, it’s unclear if anybody is going to be able to reproduce
this. I was only even able to create a Dreamcast worker because digitall
sent me some scripts that they’d used to create a cross-compiler for
this platform in the past;
3. We *still* don’t have any binaries available of ScummVM 2.0 for
Ubuntu/Debian, iOS, PSP, or Wii, even though these are our top platforms
after Windows, macOS, and Android. We wouldn’t even have an Android
package if I hadn’t already done the Buildbot work, and it was unclear
for a time whether we would have a Windows package ready to go for 2.0
either.

If we don’t have a policy like this, these exact same issues will recur
eventually with every newly accepted port. It does no good for us to say
we “officially” support some platform when we can’t actually make a
release binary for it, and it is damaging to the ScummVM brand and to
our users when we can’t even get our most popular ports’ binaries ready
by release day. This change won’t fix all the release process issues
(this is probably impossible), but it does at least get us to the point
where we don’t ever have to worry if we’re even going to be able to
actually *release* something at release time.

A secondary reason for such a requirement is build security. There’s a
small but sadly non-zero risk that bad code could be introduced by a
porter. Since we are giving these binaries our stamp of approval by
calling them “official”, and we have no good way for the core team to
actually vet them, we can partially mitigate the risk of a bad actor
introducing something ugly by making the process more transparent and
robust against subversion. In the current global environment I feel that
it is important to try to reduce these kinds of secondary risks where
possible, and having a transparent and reproducible build system which
fixes the above issues will also help to ensure the integrity of our
releases.

I understand that there are some potential downsides to this proposal, I
just can’t think of another more reasonable approach that won’t leave us
right back where we are today. If you can, let me know of the
alternative. We could grandfather some of the manual ports, but that
still doesn’t really solve the bus factor problem. Also, it’s important
to acknowledge that this policy change would only move unbuildable ports
outside our “official” realm, so we do not have a direct hand in
managing them or maintaining their code/tracking their bugs in our bug
tracker any longer; it won’t prevent them from existing, or even from
being linked on our downloads page, or from being pulled back in again
in the future if someone makes the effort later on to create a
functioning worker image.

With regards to the specific technical questions: For anything that
can’t be cross-compiled from Linux for some reason, there is always the
nuclear option of running qemu in a worker. With regards to macOS, I
don’t think there is any particular reason why those releases need to
come in a dmg any more than they need to be delivered as pkg/mpkg
installers, though there is a project called libdmg-hfsplus which can
also create dmgs.

Best,

On 2017-12-18 16:54, Thierry Crozat wrote:
> Hi Colin,
>
> I was about to write almost the same thing.
>
> This looks like a great idea overall, and thank you for all your work
> on this.
>
> I am however not sure having a buildbot worker for all ports should be
> a requirement. Can you give your reasoning for this suggestion?
> I am worried this would be a death sentence for a few ports (you
> mentioned difficulties with Samsung TV and OS/2, and I have also had
> difficulties getting mac PPC compilation to work with anything
> remotely modern). There is probably not many users for those, but I am
> a bit uncomfortable with killing them if we have porters willing to
> build those manually. I am a bit less worried for ports on more modern
> platforms as cross-compilation on Linux seems to be usually well
> supported. Packaging might be an issue though (I don't know of any way
> to create a dmg image on Linux for example, which might force us to
> use zip instead for macOS packages).
>
> Thierry
>
>> On 18 Dec 2017, at 22:21, Eugene Sandulenko <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> On 17 December 2017 at 15:36, Willem Jan Palenstijn <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>>     On Sat, Dec 16, 2017 at 09:08:15PM -0600, Colin Snover wrote:
>>     > Team,
>>     >
>>     > One of my goals for the next release of ScummVM is to automate the
>>     > process of building and generating release packages. My hope is
>>     that
>>     > this will allow us to have a faster release cycle, more consistent
>>     > release quality, and free up manpower to focus on the many parts of
>>     > ScummVM which cannot be so easily automated (like adding more
>>     engines! :)).
>>     >
>>     > The first two major parts of this project are nearly complete: the
>>     > upgrade of Buildbot, and the introduction of infrastructure
>>     management
>>     > using Ansible.
>>
>>     Hi Colin,
>>
>>     Let me start by saying: Thank you! This is great work, and will
>>     really help
>>     keep ScummVM portable and maintainable. I'm very enthusiastic
>>     about these
>>     changes.
>>
>>
>> In short, I also like the idea, and I see no problem with granting
>> you the needed permissions.
>>
>> The only thing which I would like to discuss, is the requirements for
>> every port to have a buildbot instance. I have no real data, but I
>> would be not surprised that in some cases some toolchains will not be
>> easy to install on Linux environment. And of course, I could be wrong
>> in this regard.
>>
>>
>> Eugene 
>> _______________________________________________
>> Scummvm-devel mailing list
>> [email protected] <mailto:[email protected]>
>> http://lists.scummvm.org/listinfo/scummvm-devel
>
>
>
> _______________________________________________
> Scummvm-devel mailing list
> [email protected]
> http://lists.scummvm.org/listinfo/scummvm-devel


-- 
Colin Snover
https://zetafleet.com

_______________________________________________
Scummvm-devel mailing list
[email protected]
http://lists.scummvm.org/listinfo/scummvm-devel