Re: API specification and library hell still not solved.

> [This is in reply to the digest, so it may be a bit delayed - apologies if
> I'm repeating anything]

I'm also on the digest, same to you too :-)
 
> > Two annoyances, and these are actually not specific for Linux. The first is
> > IMHO a Unix or POSIX problem, the second is even more universal.
> > 
> > _The first:_
> > 
> > The API (headers) of Linux (kernel and *nix userland) is in C, and C is
> > kind of problematic to convert to an arbitrary format (e.g. a different
> > language).. 
> 
> This is because the APIs are not generic APIs. They're C APIs.

Cool. Don't care. Maybe I was mistaken. Where are the real API's then?

> At a push, they could be C++ APIs, but unless you're working a system call
> level in C++ (which for most applications you shouldn't be), it won't be.
> If you use Java, you have a Java runtime library. If you use FORTRAN, you
> have a FORTRAN runtime library. Same applies to Smalltalk, Perl, COBOL,
> Pascal, Oberon, Scheme, and everything else.

Yes. I'm actually maintainer of the Unix runtime of Free Pascal, and have
been so for several years. These experiences are pretty much what led me to
this msg. (and 10 years of BSD experience)

But those runtimes still have to connect to something OS dependant
somewhere, and whatever that is (syscalls/kernel, like in the FPC
case, or libc), you'll need headers. Which means header conversion
	
For our libc translation, see
http://www.freepascal.org/cgi-bin/viewcvs.cgi/fpc/packages/base/libc/
(+/- 800 kb)

Can you imagine checking that for each minor update of libc? 

Can you imagine what happens if you have 60-70 packages besides several
incarnations of libc, and say 20 platform-architecture combinations in use?

Enough header conversion to keep you busy till the sun goes out :-)

> > Besides being in C (which is hard enough already), they are all different,
> > and use different macros, constructs, defines etc, which pretty much means a
> > custom parser for each set of headers (package), and even that isn't fully
> > automatic. (and handconversions are costly from a time perspective, and
> > therefore will nearly permanently be out of date)
> 
> They're different to what? If you really want to "convert" the headers,
> you're probably going at it from the wrong angle. Look at the Single UNIX
> Specification - which is the document glibc aims for conformance with, and
> provide your declarations of what's there in whichever language you please.

Won't work. Those specifications are at the C level. It doesn't say anything
about the actual contents. More over it is not just the OS, but also pkgs,
BSD libc's etc.

Besides I'm talking to access to the entire system, so everything meant
for userland in /usr/include. Not just a subset governed by a standard.

> The SuS really *doesn't* change that often, and when it does, it doesn't
> break compatibility with earlier revisions (excepting deprecation of APIs).
> While your headers might be 'out of date', they won't suddenly be unusable.
> 
> If you prefer to align your declarations with Linux kernel-land, then most
> of the same applies. Though the kernel definitions change more frequently
> than the SuS, the kernel doesn't break binary compatibility, as a matter of
> policy -- one of the reasons why libc vendors are encouraged to completely
> ignore the kernel C headers and use their own with matching definitions of
> userspace-visible structures and macros.

This is pretty much what we do now. It's however simply too much
(and unnecessary) work.

> > A nice illustration of this problem is the mere existance of "Configure" the
> > way it exists now. It doesn't process the headers directly to query them,
> > a very complex compiling/testing/analysis process is done to get some info
> > about headers. 
> 
> Not at all. The job of a configure script is to determine the differences
> between platforms - bought about by varying levels of (in)compatibility between
> deceptively similar UNIX systems. (For example, some systems have a
> <alloca.h>, some don't). The tests configure perform are actually very
> simple indeed. The two most common are:

Sure. But why are the tests on _TYPING_, function existance etc done on
the binaries instead of headers?  
 
>   To do this, configure creates a small C program which calls a named
>   functions, and then attempts to compile and link (with the given libraries).
>   If this fails, the function and/or libraries are unavailable.

It also determines a lot of types. 
 
> > The Microsoft angle (since this group will probably do a lot of comparing
> > with Windows): slightly better. Microsoft provides about a third in IDL, and
> > the remaining 2/3's are coherent, and as a while much more suitable for
> > automatic conversion.
> 
> Actually, the Microsoft angle is apalling. It offers no alignment with any
> standards whatsoever in its SDK.

[strip M$ IDL version being non standard]

So? I'm not handing out cookies for standards compliance, I simply want
access to the OS with as little trouble as possible, as a user of that OS,
starting from what the OS vendor ( in the linux case GNU userland parts,
Linux kernel, and other packages, in Microsoft's case the Platform SDK, and
possibly other SDKs) provides.

Microsoft provides that better than Linux currently, from my point of view.
Microsoft has other evils, but those are not under discussion right now.
This is about improving *NIX, and the header situation could do with some
improvements.

> > Though this is only a gradual difference (still need custom parser, and
> > still manual postprocessing), it is a factor 10-20 better (in lines/hour
> > header translations).
> > 
> > Solution: Define the headers at a abstract, higher level, provide generators (and
> > their skeletons) to generate the plain C headers from that abstract level.
> 
> Solution: Keep your custom language's runtime library separate from the
> system C library. 

In 5 years working on the FPC compiler, you are the first one to agree with
me with that. The usual reaction is "Wow, you don't work via libc?"

However for FPC that situation is getting harder. Libc is starting to get
more and more functionality integrated, and FPC is simply more advanced, and
the avg programs needs to link to other C libs anyway. (see e.g.
lazarus.freepascal.org, our RAD as example)

The Unix userland library and the C language runtime are in practice not
separated.

I don't really think that is a big problem, if I could get easy
access to it (read: get decent, automatically convertable headers)

Also don't forget that under most *nixes, calls like readdir are solely handled in
libc, the kernel interface usually only provides getdents. The same with the
functions to handle passwords (which might integrate with yellow pages, and
shadow libraries) etc.

> Each language is and should be independent, and each is
> capable of making system calls should it need to.

Converting Unistd.h is not a problem, I could do that between dinner and
desert.

> The POSIX/SuS APIs are designed as an extension, specifically, of ISO *C*.
> Trying to import the APIs, en masse, into another language is nothing short
> of a kludge. If you want to use C APIs from another language, you should
> probably be using C in the first place.

That is ridiculous. Why should that be necessary? OSes have a history of
being written in C, yet zillions of other languages somehow interface to it.

I'm sorry, but that's simply Unix/C zealotry. I don't want to use the C api,
I simply have too, there is no other userland. If you want to split up libc
into libuser(land) and libc, you have my blessing.

Moreover it is not simply a question of C vs Pascal. While less severe, I'm
in the same situation if I want to use a C compiler that is not the same
(family as the) system C compiler. I might need to add directives, and might
not be able to process e.g. certain gcc extensions.

Separating of the abstract headers (e.g. in something IDLish), and a
translation of those into two different C header sets using the original
(for gcc) convertor, and one that is slightly patched for the other C
compiler solves that too.

> > - Easy parsability makes checking for API breakage between versions easier,
> >   since analysing the interface is easier. 
> 
> Most "API breakage" tends to stem from the C Library evolving *towards* the
> standards, and leaving behind old and incompatible versions without the use
> of specific macros.

Don't care. I simply need a working set, not a non working set based on
standards. I'll have to follow it no matter what.

> These changes are usually well-advertised,

I don't want to process those. I'm working on a compiler project that supports 10
OSes on three/four architectures, and the number is rapidly expanding.

I can't be on each kernel hacker, libc, userland tools etc maillist  for
each and every one of them? And then also follow a list for each
architecture-platform combination and the package (ports) system for all
distributions etc etc.

---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
 
> > _the second_ shared lib hell.(=dll hell)
 
> > The library system of Linux (*BSD too, I don't know how universal this is)
> > is different from that of Windows. One can have multiple versions in
> > paralel, at least _IF_ the binary links to the correct library. (and not to
> > some symlink),
> 
> The linker is smart enough to resolve the symlink, in my experience.
> You
> link against a specific *major* version of a library.

Yes. And assume if e.g. one minor fix in that major version series has an
incompability in retrospect. What do you do? Fix it manually.

> > However, while security wise it might be better to disallow shared libs (not
> > dynloaded) in app directories, this kills some flexibility compared to
> > Windows too, since Windows allows libraries to be moved/copied to the
> > application directory if version conflicts arise.
> 
> LD_LIBRARY_PATH is the solution to this. You'll notice if you install
> Mozilla that all of its shared libraries are placed into the application
> directory.

True. Forgot that Linux allows this.
 
> > I'd say the overall situation is about the same.
> > Both systems (windows and linux) rely too much on properly versioned
> > libraries. A small error takes down the entire system. (a minor upgrade that
> > is incompatible etc)
> > 
> > Solution: 
> > - binaries must _always_ have full version nr of required library.
> 
> No, that would prevent bugfixes (which rightfully bump the minor version
> number) from being applied.

That only means that it should have the version in addition. 

> > - I must be able to assign any library version to any binary.
> 
> Why should you even have to *care*?

To resolve and avoid library dependancy problems in a professional way.

By having a possibility to bypass if the primary system (library versioning)
goes wrong, like it did with Microsoft, and to do it so that it works in in
an automated and structural way.

> > - There must be a database to register exceptions (specially for proprietary
> > 	software).  E.g. if Heroes3 links to old versions libc and gtk, I
> > 	must be able to list that heroes3 (as old binary only game) should
> > 	get assigned some old libc/gtk versions only. (if file=heroes3 then
> > 	assignlibc(x<=libc<=y) and assigngtk(z<=libgtk<=a)
> 
> Why?
> 
> The only thing that needs to be done is this:
> 
> 1) Application *private* libraries live in the application directory.
> 2) Increasing the minor version number of a library will -never- break
> compatibility.

Yes. And mankind will tremendously benefit if all wars stop now.
Welcome to the real world :-)

This is _exactly_ what I want. I want a system that helps when that GOES
wrong.

> 3) Increasing the major version number is effectively a "branch". 
> 
> The only thing which doesn't happen normally is (1), where application
> developers (often through ego) believe that everyone wants to link against
> their new custom library. 

> This is what caused Windows' DLL hell in the first place.

And will on linux (and already does, on a much less limited scale), since
there is only a minor fundamental change (versions can be in paralel) I want
to avoid that.

The only other way (thinking as a former sysadmin) is to forbid using
packages not approved by the distro, and killing upgrading of distro's.

The second isn't that much of a problem, but the first is. It is too
limiting for most companies.