Re: dune and opam issues: a summary of my fuzzy understanding

Hi all,

Let me first give my answers to the questions you asked:
>    - Why are control files sometimes in the above repo and sometimes in
>      the project repos?


On the one hand, the opam file should be published in opam repository so 
that opam, as a package manager, can be used to install unison easily, 
i.e., by simply typing "opam install unison". Users of opam need not 
explicitly give the URL of the project repository: opam will lookup in 
opam-repository the opam file and use it to get all the information it 
needs. Also, the opam-repository repo contains one opam file for each 
release, so that a user can explicitly ask to install e.g., unison 
2.53.0 via "opam install unison.2.53.0", just like we would do with a 
package manager from a Linux distribution. Thanks to this repository, 
the command "opam upgrade" allows to upgrade the version of unison, but 
only when a new release is published (a regular user doesn't want to 
upgrade unison at every commit, because the content of the repository 
between two releases can be bogus/unstable).

On the other hand, the project repository contains an opam file, so that 
its contents can be kept in sync when details specific to the software 
changes (e.g., which build system, configuration options, dependencies). 
This file is also useful when people want to install a development 
version of the software, e.g., for testing an unreleased feature. If 
they want to do that, they need to issue special commands to opam to 
tell it that this package should be installed from the development 
repository. When doing that, opam will stop tracking the version of 
unison used (and instead use a generic "dev" version string which tells 
nothing), and "opam upgrade" will simply upgrade to the most recent 
content of the repository.

When publishing a release, the release manager needs to copy the opam 
file from the development repository to opam-repository, and add a "url" 
section specifying where to find the sources of that particular release.

>
>    - Which location is preferred and why?

Both locations are useful, but for different purposes:
- opam-repository is useful for end-users wanting to use released 
versions of unison, without invoking complex command to tell opam where 
to find the code
- the dev repo is useful for easier maintenance, and to let advanced 
users install and test development versions

>    - Does the file have to be at top-level in the project, if it's there
>      (there are N packaging systems and it's not really ok for each to
>      demand stuff in the top-level directory)?

If this file is used to install development versions, then opam 
*expects* it to be there. I imagine there might be options to tell opam 
where the opam file is, but this is non-standard and should be avoided.

Of course, if the file is kept in the repo only for maintenance 
purposes, then it does not need to be at the root of the project. But 
maintaining an opam file in an exotic location is going to be 
cumbersome, because one will not be able to test it by installing the 
development version.
>    - When there are multiple files for multiple build flavors, can one
>      build them without colliding?   Or is this really a dune issue?

I'm not sure I understand your question: opam is really a "package 
manager", so the question is not really *building* the software in 
multiple flavors, but rather *installing* it.

It is possible to have several opam files at the root of the same 
repository. They will correspond to different packages using the same 
source tree, for example because the software has been split into 
several packages (a library may be distributed using several packages 
representing different features set o the library...). But if these 
packages are installed simultaneously, then it's just like with "make 
install": they need to install a different set of files. Otherwise, they 
are conflicting, and this should be part of a "conflict" clause of the 
opam files.

In the case of unison, we have several options:
1- have two packages "unison-cli" and "unison-gui", both installing the 
executable "unison", and declare the two packages in conflict. The user 
will have to choose between the two packages when typing "opam". The 
problem is perhaps that there is no "canonical" package for unison.
2- same as 1-, but with two different names for the executable. The user 
can install both packages, but once installed, she will have to type two 
different commands depending on whether she wants to use the version 
with the GUI or not: this is not the same behavior as with "make 
install" and thus I don't really like this solution.
3- use only one package, but with an optional dependency on lablgtk. The 
package will be built with two different configuration depending on 
whether lablgtk is installed, and an upgrade of lablgtk will 
automatically trigger a rebuild of unison. The disadvantage is that a 
user willing to install unison with the gui will type "opam install 
unison", but will not have the gui installed because lablgtk will not be 
installed by default; this can be confusing.
4- same as 3-, but with a pseudo-package "unison-gui" (i.e., without any 
code), with a dependency on both "unison" and "lablgtk", so that 
installing "unison-gui" will make sure the GUI is enabled. This is 
somewhat less confusing than 3-, because a user can install an actual 
package that says "I want unison with its GUI".

>    - Is the definition of projects necessary to use dune?
Yes, but it can be minimal. It need not duplicate all the information of 
the opam file if we decide that we don't want to generate the opam file 
with dune.
>    - Why are there magic numbers in dune files, e.g.
>      "-w -3-6-9-10-26-27-32-34-35-38-39-50-52".
By default, dune enables more warnings than the default OCaml compiler 
enables, and make those warnings errors. So we need to disable the 
warnings the are triggered by the code of OCaml, because they will be 
reported as errors.

These "magic numbers" are simply identifiers of warnings.
>    - What are the arguments for and against the two styles
>      (dune-generated vs hand-maintained)?

Of course, generating opam files automatically avoids duplicating 
information between dune and opam. So, if dune is the primary build 
system of a project, and if all the information that one wants to place 
in the opam file can be generated by dune, then this is the way to go.

*However, opam can be instructed to use a different build system. There 
is no fundamental reason to use dune in the opam file. The opam file can 
very well tell opam to use the standard Makefile for both compilation 
and installation.* And I would argue that if the main unison developers 
are not using dune, then dune should not be used in the opam file, 
because if the dune build system is bogus (because not properly 
configured...), then the unison developers will not test it, and 
therefore they will not notice the bug. I don't see why the validity of 
the opam file should rely on the minority of developers using dune.
>    - If opam files are in the opam repository, and also generated, how does
>      that work?

Each time someone changes the content of the dune-project file, she 
should regenerate the opam file(s), and commit both at the same time.

It is a good idea to setup a CI job that checks that the commited opam 
file corresponds to the one that would be generated by dune.

>    - If files are generated, then mostly-obviously they shouldn't be
>      checked in.  Do people think they should be?  If they are, are we
>      sure they will be bit-for-bit identical when regenerated on
>      different systems?
They should be, because the opam file needs to be present if one wants 
to use opam to install a development version of unison (as explained 
above). Yes, they should be bit-for-bit identical (otherwise, I consider 
this as a dune bug).


> * way forward
>
> At this point, my inclination is:
>
>    - Reset opam maintainer to `unison-hackers at lists.seas.upenn.edu  <https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers>`
>      because the current maintainer has not answered mail, and we don't
>      really have another plan.  This is saying that the file is
>      maintained by the project, just like every other file in the repo.
Agreed.
>    - Apply the lablgtk3 and dune 2.7 changes from #671, basically
>      cherrypicking the parts I understand and agree with, and writing a
>      commit message.
Agreed.
>    - Consider turning on generation, and removing unison.opam from the
>      repo, pending feedback, probably via a PR that does only that.
>
>    - See if there are things to be done other than the above 3.

I would rather suggest to change the opam file so that it uses the 
Makefile. If make is the primary build system, then the dune files 
should be maintained by their users.  As a result, a user of the main 
build system can maintain the opam files without understanding dune, 
which is mainly orthogonal.

My 2 cents,
best regards,
-- 
JH Jourdan

_______________________________________________
Unison-hackers mailing list
[email protected]
https://LISTS.SEAS.UPENN.EDU/mailman/listinfo/unison-hackers