Re: API specification and library hell still not solved.

[This is in reply to the digest, so it may be a bit delayed - apologies if
I'm repeating anything]

> Two annoyances, and these are actually not specific for Linux. The first is
> IMHO a Unix or POSIX problem, the second is even more universal.
> 
> _The first:_
> 
> The API (headers) of Linux (kernel and *nix userland) is in C, and C is
> kind of problematic to convert to an arbitrary format (e.g. a different
> language).. 

This is because the APIs are not generic APIs. They're C APIs. At a push,
they could be C++ APIs, but unless you're working a system call level in C++
(which for most applications you shouldn't be), it won't be. If you use
Java, you have a Java runtime library. If you use FORTRAN, you have a
FORTRAN runtime library. Same applies to Smalltalk, Perl, COBOL, Pascal,
Oberon, Scheme, and everything else.

> Besides being in C (which is hard enough already), they are all different,
> and use different macros, constructs, defines etc, which pretty much means a
> custom parser for each set of headers (package), and even that isn't fully
> automatic. (and handconversions are costly from a time perspective, and
> therefore will nearly permanently be out of date)

They're different to what? If you really want to "convert" the headers,
you're probably going at it from the wrong angle. Look at the Single UNIX
Specification - which is the document glibc aims for conformance with, and
provide your declarations of what's there in whichever language you please.
The SuS really *doesn't* change that often, and when it does, it doesn't
break compatibility with earlier revisions (excepting deprecation of APIs).
While your headers might be 'out of date', they won't suddenly be unusable.

If you prefer to align your declarations with Linux kernel-land, then most
of the same applies. Though the kernel definitions change more frequently
than the SuS, the kernel doesn't break binary compatibility, as a matter of
policy -- one of the reasons why libc vendors are encouraged to completely
ignore the kernel C headers and use their own with matching definitions of
userspace-visible structures and macros.

> A nice illustration of this problem is the mere existance of "Configure" the
> way it exists now. It doesn't process the headers directly to query them,
> a very complex compiling/testing/analysis process is done to get some info
> about headers. 

Not at all. The job of a configure script is to determine the differences
between platforms - bought about by varying levels of (in)compatibility between
deceptively similar UNIX systems. (For example, some systems have a
<alloca.h>, some don't). The tests configure perform are actually very
simple indeed. The two most common are:

- Tests for headers - i.e., does the given header file exist and is useable?

  To do this, configure simply creates a small C file with an appropriate
  #include line, and passes it through the C preprocessor. If it fails to
  preprocess, the header is determined to be unavailable.
  
- Tests for functions and libraries

  To do this, configure creates a small C program which calls a named
  functions, and then attempts to compile and link (with the given libraries).
  If this fails, the function and/or libraries are unavailable.

> The Microsoft angle (since this group will probably do a lot of comparing
> with Windows): slightly better. Microsoft provides about a third in IDL, and
> the remaining 2/3's are coherent, and as a while much more suitable for
> automatic conversion.

Actually, the Microsoft angle is apalling. It offers no alignment with any
standards whatsoever in its SDK. The variant of IDL which it uses is utterly
non-standard - the only tools capable of parsing it are those written
specifically for Microsoft IDL tools - even DCE-RPC's tools (Microsoft's
RPC, and hence IDL, implementation was based upon DCE RPC) will choke on
most of Microsoft's IDL.

> Though this is only a gradual difference (still need custom parser, and
> still manual postprocessing), it is a factor 10-20 better (in lines/hour
> header translations).
> 
> Solution: Define the headers at a abstract, higher level, provide generators (and
> their skeletons) to generate the plain C headers from that abstract level.

Solution: Keep your custom language's runtime library separate from the
system C library. Each language is and should be independent, and each is
capable of making system calls should it need to.

The POSIX/SuS APIs are designed as an extension, specifically, of ISO *C*.
Trying to import the APIs, en masse, into another language is nothing short
of a kludge. If you want to use C APIs from another language, you should
probably be using C in the first place.

> - Easy parsability makes checking for API breakage between versions easier,
>   since analysing the interface is easier. 

Most "API breakage" tends to stem from the C Library evolving *towards* the
standards, and leaving behind old and incompatible versions without the use
of specific macros. These changes are usually well-advertised, along with
the appropriate #defines that need to be supplied to use the older APIs.
Typically, the older definitions come from trying to support non-POSIX APIs
in tandem with the POSIX ones (e.g., BSD-specific API support).

> _the second_ shared lib hell.(=dll hell)
> 
> The library system of Linux (*BSD too, I don't know how universal this is)
> is different from that of Windows. One can have multiple versions in
> paralel, at least _IF_ the binary links to the correct library. (and not to
> some symlink),

The linker is smart enough to resolve the symlink, in my experience. You
link against a specific *major* version of a library.

> However, while security wise it might be better to disallow shared libs (not
> dynloaded) in app directories, this kills some flexibility compared to
> Windows too, since Windows allows libraries to be moved/copied to the
> application directory if version conflicts arise.

LD_LIBRARY_PATH is the solution to this. You'll notice if you install
Mozilla that all of its shared libraries are placed into the application
directory.

> I'd say the overall situation is about the same.
> Both systems (windows and linux) rely too much on properly versioned
> libraries. A small error takes down the entire system. (a minor upgrade that
> is incompatible etc)
> 
> Solution: 
> - binaries must _always_ have full version nr of required library.

No, that would prevent bugfixes (which rightfully bump the minor version
number) from being applied.

> - I must be able to assign any library version to any binary.

Why should you even have to *care*?

> - there must be a system global db which lists version compabilities. 
>      so if binary lists 2.1.76, the db must list that 2.1.40-2.1.80
>      are compatible, and automatically assign the newest version of that
> 	library available in that range to the binary.
>     I'm still in dubio if this should be done only for libraries as a whole,
> 	or on a per symbol basis ( library version X is compatible with
> 	library version Y as long as you don't use for symbol Z)
> - There must be a database to register exceptions (specially for proprietary
> 	software).  E.g. if Heroes3 links to old versions libc and gtk, I
> 	must be able to list that heroes3 (as old binary only game) should
> 	get assigned some old libc/gtk versions only. (if file=heroes3 then
> 	assignlibc(x<=libc<=y) and assigngtk(z<=libgtk<=a)

Why?

The only thing that needs to be done is this:

1) Application *private* libraries live in the application directory.
2) Increasing the minor version number of a library will -never- break
compatibility.
3) Increasing the major version number is effectively a "branch". 

The only thing which doesn't happen normally is (1), where application
developers (often through ego) believe that everyone wants to link against
their new custom library. This is what caused Windows' DLL hell in the first
place.