Re: Subversion, SCons, and reorganizationalizing

On Tue, 2003-01-21 at 19:37, Micah Dowty wrote:
> Hi Everybody,
> 
> I just wanted to give an outline of the source tree reorganization I'm planning after
> the 0.44 release. This is necessary in my opinion for several reasons:
> 
>  - The current build system is hard to port to non-linux platforms
>  - A lot of the code in CVS is arranged in a less than logical way
>  - The configuration system is hard to cross-compile with for people who
>    aren't familiar with all the GNUisms involved
>  - The configuration system doesn't handle dependencies
>  - The configuration system only configures pgserver
> 
> The reorganization I have planned would involve two major changes: moving the source
> tree to a Subversion server outside of sourceforge, and switching the build system,
> probably to something based on SCons. This subversion server has about 40GB of redundant
> storage, a 256kbps internet connection, and I'll be able to make offsite backups
> regularly. Releases will still be done through sourceforge.
> 
> The big concepts I want to emphasize with the new system are portability, and
> configuration of picogui as a complete system. The directory tree will look something
> like this:
> 
> ------8<------
>  - picogui
>     - README, COPYING, INSTALL...
>     - build (Intermediate files in compilation)
>        - x86-dev
>        - arm-zaurus
>        - ppc-dev
>          ...
>     - bin (final compiled binaries)
>        - theme
>        - x86-dev
>        - arm-zaurus
>        - ppc-dev
>          ...
>     - config (files for the configuration system)
>     - fonts
>     - src (clean source tree)
>        - pgserver
>        - appserver
>        - lib
>           - c
>              - client
>              - media
>           - python
>              - PicoGUI
>           - cpp
>             ...
>        - tools
>           - themec
>           - themedc
>           - future app packaging tools
>        - apps
>           - Only relatively usable apps (pgedit, pterm, scribble, working games...)
>        - devapps
>           - Apps that are only useful for developers
>        - themes
>           - Relatively usable themes, in their own directories
>        - devthemes
>           - Themes only interesting to developers
> ------>8------
> 
> This new source tree layout emphasizes PicoGUI as a complete system. It removes a lot
> of the confusing things about the old tree:
> 
>   - the cli_* naming convention for libraries will be gone, since it can be confusing.
>     The new source tree uses directories to organize libraries by language, and the
>     library should be called the "C client" rather than "cli_c"
> 
>   - Clean up the awful mess in themetools/examples. Themes will separated into generally
>     useful and developers-only, and each theme's source will be in a separate directory.
>     Since svn allows moving source trees easily, themes could be very simply migrated
>     between the categories.
> 
>   - Separate applications into mostly usable, and for developers only
> 
> The new configuration system will still be similar to the current one, but with a few
> key differences:
> 
>   - It will configure the entire source tree, not just pgserver. Every component will
>     be conditionally compiled depending on this configuration
> 
>   - Dependencies will be managed for the whole system. Outside libraries (libpng, zlib,
>     SDL, etc.) will be represented as targets in the config system, as will 'tasks'
>     that select common options for different situations. The operating environment
>     (OS and available libraries) for the target platform will be automatically detected.
> 
>   - Cross compilation can be managed by selecting a different target platform in the
>     config system. It would still support specifying the compiler and paths manually,
>     but this would remove the pain from cross-compiling to common platforms.
> 
>   - Once picogui has app packaging, it will integrate with the config system to provide
>     automatic installation of the completed picogui system into a distro.
> 
>   - There will be many frontends. If you're only selecting a task or two, you could use
>     the command line. There could be graphical configuration frontends for ncurses, gtk,
>     or PicoGUI. You could even have frontends that would glue this system to other
>     config systems, like the ones OpenZaurus or Smartdata use.
> 
>   - The config system will make it clear which options you selected manually, and which
>     are implicitly selected by a dependency. This will make it easy to avoid using
>     compile-time options that aren't necessary, so you can make the smallest binary
>     possible.
> 
> For example, you could start with a blank configuration, then add a "OpenGL game UI"
> task, which would include all the widgets, drivers, and vidlibs you would need for that.
> Or you could include a "Zaurus" task that would build the drivers the Zaurus needs, plus
> a "PDA environment" task that would build common applications. It would automatically
> resolve what vidlibs those drivers need, and what libraries those applications need.
> 
> --Micah


It is a good idea to rearrang a bit the whole project. A few remarks:

      * Some projects people live from (I do not speak only about us...)
        rely on the current structure, so we must first meet a consensus
        on the transition deadline. For instance, switching from CVS to
        Subversion is not a blocker, but would have certainly an impact
        on our schedules.
      * We also have a dependency system (reason why we bother everybody
        with our mdl files ;-) ) which is one layer above autoconf and
        co. Our dependency system deals with a lot of various components
        (kernel, drivers, libs, 3rd party software, etc. etc.) and
        pgserver and some pg apps are among. Your proposed dependency
        system should be an alternative parallel system. I advise to
        build it also on top of autoconf, so that one still can access
        the autoconf level, like before (which may be just sufficient
        for some people).
      * We can give a lot of hints about difficult
        ^H^H^H^H^H^H^H^H^Hinteresting points we faced when dealing with
        dependency issues. We for instance separated the platform from
        the product aspects.
      * The new dependency/platform/product management system should
        certainly not only deal with dependencies, but hide some
        platform-specific options related to cross-compilation. One
        would have to bother about these options only when introducing a
        new platform. All this above autoconf !
      * I find very good to be able to quickly build a (nearly) complete
        system, e.g. for several embedded platforms, or for demo
        purpose. We should however still be able to just pick necessary
        pieces separately.
      * You said:
        > - The current build system is hard to port to non-linux platforms
        Do you mean target platforms or build platforms ?

Regards,

  Pascal


-- 
Pascal Bauermeister
Head of Software Development

SMARTDATA
PSE-B / EPFL
CH-1015 Lausanne

http://www.smartdata.ch
mailto:[email protected]
Phone: +41 (21) 693 84 98
fax:   +41 (21) 693 84 91
mobile:+41 (79) 456 82 05


-------------------------------------------------------
This SF.net email is sponsored by: Scholarships for Techies!
Can't afford IT training? All 2003 ictp students receive scholarships.
Get hands-on training in Microsoft, Cisco, Sun, Linux/UNIX, and more.
www.ictp.com/training/sourceforge.asp