Re: Hashing out the scope of work

On 4/24/24 12:45 PM, Konstantin Ryabitsev wrote:
> Hello, all:
> 
> This is the Scope of Work section of the larger Statement of Work document.
> Please send any comments, adjustments, additions, or clarifications you would
> like to see.

Thanks for writing this up.

I've read through the entire thread and I will comment on a few items here that summarize
the discussions to date.

May we please consider (1), (3), (4), (5) and (7) agreed upon?

With (2) and (6) needing some review, notes below.

> This SOW focuses on the migration primarily, and the tentative time frame is
> through mid-2025.
> 
> Regards,
> Konstantin
> 
> ---
> When choosing solutions and deploying services, the guiding principle is to
> remain compliant with the [GNU Ethical Repository Criteria][1], Grade B. This
> mandates exclusive use of Free Software and promotes open platforms over
> proprietary hosting environments.
> 
> 1. Mailing lists for decentralized patch-based development
> 
>   a. LF will set up mailing list hosting under the lists.coretoolcain.dev
>      domain, using the following technology stack:
> 
>     - postfix
>     - mlmmj
>     - public-inbox

No contention here, we can consider (1) resolved and part of the SOW.

We should be able to import all public-inbox archives from inbox.sourceware.org.

> 
>   b. For scalability and IP reputation reasons, LF will reuse its current
>      mailing list platform already deployed for Linux kernel development
>      (subspace.kernel.org).

Agreed.

>   c. Individual glibc mailing lists will be created on request and do not
>      require to be enumerated in this SOW.

Agreed.

>   d. Mailing list archives will be available either via lore.kernel.org, or a
>      dedicated coretoolchain.dev domain name, depending on project preferences.

Agreed.

> 2. Bug tracking software (Bugzilla)
> 
>   a. LF will set up a bug tracking system using [Bugzilla Harmony][2]. This
>      will be a new, dedicated installation in the bugzilla.coretoolchain.dev
>      domain.
>   b. LF will work with the CTI leadership to properly configure the software
>      for use with the glibc project.
>   c. LF will work with the glibc community to offer the tooling to integrate
>      git repositories, mailing lists, and the bugtracker (e.g. using the
>      [bugspray][3] project).

Comments from Joseph include:

- Bug submission form customization
- HTML template customization
- Logic changes
- New upstream version suitably configurable
- Local fields
- FOSS spam protection for comments
- REST API enabled
- High numbered bugs e.g. 100000+
- Account creation moderation

My position is that if we can't configure vanilla bugzilla to do what we want
then we as a *community* needs to contribute upstream until it does and then ask
LF IT to update to a newly released version.

This is exactly the process we did, "Upstream First", for Patchwork development
of the new Cover letter event features. DJ Delorie worked upstream with the
developers, we got a new version out, and upgraded to the new version (and
found bugs and fixed them upstream).

Everything listed above needs to come in via upstream IMO.

That doesn't mean we halt the migration, but we configure bugzilla to match
as required as possible, and continue to work upstream to enable as required.

The local fields:
	cf_gcchost
	cf_gcctarget
	cf_gccbuild
	cf_reconfirmed_on
... are all configured in the UX, so I don't think we have any problem there.

Comments from Konstantin include:

- Complexity and cost of migrating data.
- Suggestion to reduce cost via hybrid public-inbox + new bugs setup.

The most contentious things is the move of the data.

May I suggest an alternative to the migration cost problem?

(1) Split the trackers e.g. one tracker per project.
    - Better security and isolation.
    - Scale up or scale down.
    - Decouples configuration management for projects.

(2) Import *all* of the older bugzilla database...
    - Dump and reload with no filtering.
    (2.1) Then use the UX to delete unwanted products.
          - Ensures database referential consistency.

(3) Bump the bump numbering up to 100,000.

In this case (1) is just machine cost and has benefits.
Likewise (2) is the standard operating procedures for migrating bugzilla.
And (3) should be straight forward.

I want this to be lowest cost while still providing value.

Having distinct trackers per project also facilitates future export/import.

> 3. Git repository hosting (Gitolite)
> 
>   a. LF will set up the necessary glibc repositories reusing the existing
>      gitolite.coretoolchain.dev service.
>   b. LF will work with the CTI leadership to migrate existing repositories and
>      grant the necessary permissions.
>   c. LF will work with the glibc project community to analyze, port, and adapt
>      the existing set of [AdaCore][4] post-commit hooks to ensure that they
>      are functioning after the migration, or are replaced with suitable
>      alternatives if existing implementations cannot be used directly due to
>      security considerations or duplicating functionality provided in
>      Gitolite.
>   d. LF will set up mirroring and replication using [grokmirror][5] to offer
>      multiple redundant sites for git repository access.

No contention here, we can consider (3) resolved and part of the SOW.

> 4. Documentation repository and website (Sphinx)
> 
>   a. LF will work with the glibc community to help migrate the wiki site from
>      [MoinMoin Wiki][6] to a set of restructured-text documents (e.g. using
>      [pandoc][7]).
>   b. LF will provide a publishing framework to automatically build and deploy
>      documentation to the dedicated docs website (exact domain name to be
>      established).

No contention here, we can consider (4) resolved and part of the SOW.

> 5. Static website (Sphinx)
> 
>   a. LF will provide hosting for a separate static website for glibc (exact
>      domain name to be established).
>   b. LF will provide a mechanism to publish release tarballs and offer
>      downloads via the mirrors.kernel.org system of worldwide mirrors (already
>      provided via mirrors.kernel.org/gnu/glibc/).

No contention here, we can consider (5) resolved and part of the SOW.
 
> 6. Patch tracking services (Patchwork)
> 
>   a. LF will deploy the latest version of Patchwork patch tracking software
>      under patchwork.coretoolchain.dev.
>   b. LF will set up automation and integration services between mailing lists,
>      patchwork, and git repositories (using [git-patchwork-bot][8]).
>   c. LF will work with the glibc community to set up projects and access as
>      appropriate.

Comments from Joseph include:

- Migrate existing patchwork data.

Comments from Konstantin include:

- Replay mailing list data to recreate patchwork instance.

Comments from Siddhesh include:

- Replaying mailing list data would allow us to test if patches apply again.
  - Note: We already have a patch retry bot running e.g. redhat-pt-bot/TryBot-still_applies

May I suggest an alternative to the migration cost problem?

(1) Split the trackers e.g. one patchwork instance per project.
    - Better security and isolation.
    - Scale up or scale down.
    - Decouples configuration management for projects.

(2) Import *all* of the Django database.
    - Dump and reload with no filtering with manage.py.
    (2.1) Then use the UX to delete unwanted products.
          - Ensures Django referential consistency.

This is again a straight forward solution of importing via Django's
standard process.

Then someone with Django site administration deletes all the other
projects which deletes all the unwanted data correctly and using the
right rules for the project.

Having distinct patchwork instances per project also facilitates
future export/import.

Eventually with all the instances we will want a FOSS SSO solution,
but that is a *definately* a future work item for the CTI TAC.

> 7. Video conferencing (BigBlueButton)
> 
>   a. LF will provide video conferencing to the project using [BigBlueButton][9]
>      (BBB), either with a 3rd-party provider or as a fully self-hosted service.
>   b. LF will work with the glibc project to set up conference rooms and access
>      as required.

Comments from Ian Kelling

- BBB is using non-free mongodb backend.

Comments from Konstantin

- Suggest Jitsi.

Comments from Siddhesh

- We used FSF's Jitsi instance and struggled.

  - Carlos' Notes: We used Jitsi for 18 meetings and had a very difficult
    time with community member access. We have used BBB to host 73 meetings
    without a glitch.

Comments from Carlos

- Compliance can't be instantaneous.
- The CTI TAC needs to review BBB vs. Jitsi.

For now I think BBB is working and we should continue using it while we
resolve what to do with the MongoDB issue. This question needs to go to
the CTI TAC for review.

I would like to consider (6) resolved and part of the SOW.

Eventually the CTI TAC may decide to change services and that would be
a change to the SOW with an associated cost change (up or down we don't
know until evaluated).

> [1]: https://www.gnu.org/software/repo-criteria.en.html
> [2]: https://github.com/bugzilla/harmony
> [3]: https://git.kernel.org/pub/scm/utils/bugspray/bugspray.git
> [4]: https://github.com/AdaCore/git-hooks
> [5]: https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git
> [6]: https://sourceware.org/glibc/wiki/HomePage
> [7]: https://pandoc.org/
> [8]: https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/git-patchwork-bot.py
> 

-- 
Cheers,
Carlos.