Re: Re: API specification and library hell still not solved.

On Wed, 2003-08-20 at 10:20, Marco van de Voort wrote:
> > > The API (headers) of Linux (kernel and *nix userland) is in C, and C is
> > > kind of problematic to convert to an arbitrary format (e.g. a different
> > > language).. 
> > 
> > This is because the APIs are not generic APIs. They're C APIs.
> 
> Cool. Don't care. Maybe I was mistaken. Where are the real API's then?

They are the real APIs. Unix was written in C. (Once it was finalised,
anyway, the first coding on the PDP-7 was in assembly).

If you don't like that, perhaps you should have paid more attention in C
class so you could port them to the language of your choice.

> > At a push, they could be C++ APIs, but unless you're working a system call
> > level in C++ (which for most applications you shouldn't be), it won't be.
> > If you use Java, you have a Java runtime library. If you use FORTRAN, you
> > have a FORTRAN runtime library. Same applies to Smalltalk, Perl, COBOL,
> > Pascal, Oberon, Scheme, and everything else.
> 
> Yes. I'm actually maintainer of the Unix runtime of Free Pascal, and have
> been so for several years. These experiences are pretty much what led me to
> this msg. (and 10 years of BSD experience)
> 
> But those runtimes still have to connect to something OS dependant
> somewhere, and whatever that is (syscalls/kernel, like in the FPC
> case, or libc), you'll need headers. Which means header conversion
> 	
> For our libc translation, see
> http://www.freepascal.org/cgi-bin/viewcvs.cgi/fpc/packages/base/libc/
> (+/- 800 kb)
>
> Can you imagine checking that for each minor update of libc? 

If you were writing a compiler and didn't verify each minor update of
libc, I personally wouldn't be remotely interested in using your
language.

This is called "work".

> Can you imagine what happens if you have 60-70 packages besides several
> incarnations of libc, and say 20 platform-architecture combinations in use?

No. I stick to one incarnation of libc per language.

> > > Besides being in C (which is hard enough already), they are all different,
> > > and use different macros, constructs, defines etc, which pretty much means a
> > > custom parser for each set of headers (package), and even that isn't fully
> > > automatic. (and handconversions are costly from a time perspective, and
> > > therefore will nearly permanently be out of date)
> > 
> > They're different to what? If you really want to "convert" the headers,
> > you're probably going at it from the wrong angle. Look at the Single UNIX
> > Specification - which is the document glibc aims for conformance with, and
> > provide your declarations of what's there in whichever language you please.
> 
> Won't work. Those specifications are at the C level. It doesn't say anything
> about the actual contents. More over it is not just the OS, but also pkgs,
> BSD libc's etc.

As the "C level" is virtually synonymous with machine level, that's
hardly a valid gripe.

> > The SuS really *doesn't* change that often, and when it does, it doesn't
> > break compatibility with earlier revisions (excepting deprecation of APIs).
> > While your headers might be 'out of date', they won't suddenly be unusable.
> > 
> > If you prefer to align your declarations with Linux kernel-land, then most
> > of the same applies. Though the kernel definitions change more frequently
> > than the SuS, the kernel doesn't break binary compatibility, as a matter of
> > policy -- one of the reasons why libc vendors are encouraged to completely
> > ignore the kernel C headers and use their own with matching definitions of
> > userspace-visible structures and macros.
> 
> This is pretty much what we do now. It's however simply too much
> (and unnecessary) work.

Ignoring the API which is in a state of flux and concentrating on the
static one is too much work?

> > > A nice illustration of this problem is the mere existance of "Configure" the
> > > way it exists now. It doesn't process the headers directly to query them,
> > > a very complex compiling/testing/analysis process is done to get some info
> > > about headers. 
> > 
> > Not at all. The job of a configure script is to determine the differences
> > between platforms - bought about by varying levels of (in)compatibility between
> > deceptively similar UNIX systems. (For example, some systems have a
> > <alloca.h>, some don't). The tests configure perform are actually very
> > simple indeed. The two most common are:
> 
> Sure. But why are the tests on _TYPING_, function existance etc done on
> the binaries instead of headers?  

Assuming you mean "Why are the header files tested and not the
binaries?", the answer is pretty simple. The two are supposed to match.
If they don't, you or your system administrator is not doing his job
properly.

> >   To do this, configure creates a small C program which calls a named
> >   functions, and then attempts to compile and link (with the given libraries).
> >   If this fails, the function and/or libraries are unavailable.
> 
> It also determines a lot of types. 

There are a lot of architectures.

> > > The Microsoft angle (since this group will probably do a lot of comparing
> > > with Windows): slightly better. Microsoft provides about a third in IDL, and
> > > the remaining 2/3's are coherent, and as a while much more suitable for
> > > automatic conversion.
> > 
> > Actually, the Microsoft angle is apalling. It offers no alignment with any
> > standards whatsoever in its SDK.
> 
> [strip M$ IDL version being non standard]
> 
> So? I'm not handing out cookies for standards compliance, I simply want
> access to the OS with as little trouble as possible, as a user of that OS,
> starting from what the OS vendor ( in the linux case GNU userland parts,
> Linux kernel, and other packages, in Microsoft's case the Platform SDK, and
> possibly other SDKs) provides.

Then perhaps you should use an OS that doesn't try and be so inclusive.
Microsoft let you code the Microsoft way, which provides the amount of
trouble you are willing to deal with, so perhaps you should be using
that platform.

> Microsoft provides that better than Linux currently, from my point of view.
> Microsoft has other evils, but those are not under discussion right now.
> This is about improving *NIX, and the header situation could do with some
> improvements.

I haven't had a single problem with it. I code in a number of languages,
including C, and they all deal with this in a perfectly acceptable way.

> > > Though this is only a gradual difference (still need custom parser, and
> > > still manual postprocessing), it is a factor 10-20 better (in lines/hour
> > > header translations).
> > > 
> > > Solution: Define the headers at a abstract, higher level, provide generators (and
> > > their skeletons) to generate the plain C headers from that abstract level.
> > 
> > Solution: Keep your custom language's runtime library separate from the
> > system C library. 
> 
> In 5 years working on the FPC compiler, you are the first one to agree with
> me with that. The usual reaction is "Wow, you don't work via libc?"

Make me the second.

> However for FPC that situation is getting harder. Libc is starting to get
> more and more functionality integrated, and FPC is simply more advanced, and
> the avg programs needs to link to other C libs anyway. (see e.g.
> lazarus.freepascal.org, our RAD as example)

Then your language should have the hooks necessary to do so. Perl does.

> The Unix userland library and the C language runtime are in practice not
> separated.
> 
> I don't really think that is a big problem, if I could get easy
> access to it (read: get decent, automatically convertable headers)
> 
> Also don't forget that under most *nixes, calls like readdir are solely handled in
> libc, the kernel interface usually only provides getdents. The same with the
> functions to handle passwords (which might integrate with yellow pages, and
> shadow libraries) etc.
> 
> > Each language is and should be independent, and each is
> > capable of making system calls should it need to.
> 
> Converting Unistd.h is not a problem, I could do that between dinner and
> desert.

1 down, X to go.

> > The POSIX/SuS APIs are designed as an extension, specifically, of ISO *C*.
> > Trying to import the APIs, en masse, into another language is nothing short
> > of a kludge. If you want to use C APIs from another language, you should
> > probably be using C in the first place.
> 
> That is ridiculous. Why should that be necessary? OSes have a history of
> being written in C, yet zillions of other languages somehow interface to it.
>
> I'm sorry, but that's simply Unix/C zealotry. I don't want to use the C api,
> I simply have too, there is no other userland. If you want to split up libc
> into libuser(land) and libc, you have my blessing.

If you don't want to use the C api, don't use Unix. The two have been
intertwined since the beginning and are likely to remain so for a
considerable time to come.

> Moreover it is not simply a question of C vs Pascal. While less severe, I'm
> in the same situation if I want to use a C compiler that is not the same
> (family as the) system C compiler. I might need to add directives, and might
> not be able to process e.g. certain gcc extensions.

If you want to use a different C compiler, you need different C
libraries. Somehow that sounds to me like logic.

> Separating of the abstract headers (e.g. in something IDLish), and a
> translation of those into two different C header sets using the original
> (for gcc) convertor, and one that is slightly patched for the other C
> compiler solves that too.
>
> > > - Easy parsability makes checking for API breakage between versions easier,
> > >   since analysing the interface is easier. 
> > 
> > Most "API breakage" tends to stem from the C Library evolving *towards* the
> > standards, and leaving behind old and incompatible versions without the use
> > of specific macros.
> 
> Don't care. I simply need a working set, not a non working set based on
> standards. I'll have to follow it no matter what.
> 
> > These changes are usually well-advertised,
> 
> I don't want to process those. I'm working on a compiler project that supports 10
> OSes on three/four architectures, and the number is rapidly expanding.
> 
> I can't be on each kernel hacker, libc, userland tools etc maillist  for
> each and every one of them? And then also follow a list for each
> architecture-platform combination and the package (ports) system for all
> distributions etc etc.

So you want to support 10 OSs and 4 architectures but you don't want to
know anything about them?

I'm glad I never wasted my time with Pascal.

[... snip library hell ...]

I'm sorry if this sounds flamewarish, but you give the impression of
wanting to code (wanting to code a compiler, no less) without bothering
to try and understand the system[s] you are coding for.

That, I'm afraid, sounds incredibly dumb.

Matthew

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQA/Q0rdTVZrwrzdcp4RApVDAJ9bgfK41oTgQE7VNrkxnjXVz3htMACffCXg
aUUQlT7DtkB9ntQq4i2H5+s=
=7nYP
-----END PGP SIGNATURE-----