Re: API specification and library hell still not solved.

On Tue, 19 Aug 2003, Marco van de Voort wrote:

>> _the second_ shared lib hell.(=dll hell)
>> 
>> The library system of Linux (*BSD too, I don't know how universal this is)
>> is different from that of Windows. One can have multiple versions in
>> paralel, at least _IF_ the binary links to the correct library. (and not to
>> some symlink),
>> 
>> However, while security wise it might be better to disallow shared libs (not
>> dynloaded) in app directories, this kills some flexibility compared to
>> Windows too, since Windows allows libraries to be moved/copied to the
>> application directory if version conflicts arise.

>No, you can do this under Linux too, just use something like:

>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.

>and . (the current working directory AT RUN TIME) will be searched for
>shared libraries after the existing library path. If you want the search
>order to be the CWD first:
 
>True, but it was only introduction. Let's move to the real issue.
>(mental note to self: test if FreeBSD allows this)
>~
>> I'd say the overall situation is about the same. Both systems (windows and
>> linux) rely too much on properly versioned libraries. A small error takes
>> down the entire system. (a minor upgrade that is incompatible etc)
>
> Correct me if I'm wrong, but as the major version number is encoded into a
> Linux shared library filename, at least applications can automatically
> request to be linked against at least /roughly/ the right version of the
> library

Yes, but I want 
- _precisely_ the right library
- Compability info (which versions are compatible) must be adjustable AFTER
  the binary is released. No blind faith, post-release configurability.
- not forfeit "static" linking of dynamic library. No changes to the way we work
	with libs now (so no dl_open and sneaky looking into a set of data
	in the app. Standarise and in the dynloader, leave nothing in apps
	of this kind)
- the binary to contain what exact versions it was built for, even if it
    links to "libc.so" instead of "glibc.so.5.1.23" or whatever.


>.Under windows, where most DLLs follow an 8.3 naming convention,
> with few encoding the version number into the name (CTL3D, MSVCRT and VBRUN
> being the only exceptions, IIRC). As a result clashes are more like to
> occur on Windows.

This is more or less what I said was the slight edge of Linux over Windows
in the introduction, specially if the moving libs locally thingy disappears.
However that is not the point, both systems don't solve anything
systematically

> Ultimately though, the decision comes between allowing Windows-style DLL
>hell, or Dependency Hell (where each application binary must be linked
>against /exactly/ the library it was compiled against). Linux is somewhere
>in the middle and still gets moaned at for "Dependency Hell".

Linux (or FreeBSD, what I use since 4.2BSD) _has_ a dependancy hell problem,
despite those few mitigating factors and it is only gradually better than
windows. The week spot in both systems is the same, reliance on correct
versioning of libraries, and not having automated ways in the dynlinker to
do something systematically AFTER binary and library release (see example
below)

If I release a minor patch that turns to be incompatible in retrospect, all
I can do is start mailing around that there has been a glitch, and hope that
version dies.

In the other case (with a dynamically updated library), distribution
maintainers can enter the data into the system, and the next update makes
sure that no binary that is build for a different versio than that links to
it (integration with the packaging system could then provide the good
library with a mere click)

Example:

I've a binary, and it links to libraries liba.so.3.3.7 and libb.so.1.4.5. However
the binary links to liba.3 and libb.1 (and that is already optimistic. More
chance that it links to liba.so and libb.so)

Now a liba.3.3.8 comes out as "minor" patch, and it gets into circulation.

After a while, there turns to be a compability bug in a relatively little
used function in liba.3.3.8. A * was added, but that changed the interface significantly

However a popular, yet a not extremely, but still often used library z uses
_that_ function in liba.so.3.3.8 in its initialisation. Result: all programs
depending on library z fail.

What to do now?

Current situation: upgrade all libraries and binaries related to the
incident to newer versions. (who knows if your lib z depends on 3.3.8 or
3.3.7?, maybe you have a version depending on that odd behaviour in 3.3.8)

Future situations 
- 3.3.8 is taken out of the liba.so.3.3 compability list.
- A new compability list is made for 3.3.8 with only 3.3.8 in it.

Since all binaries and libs know their version and the version they were
build for, stuff depending on 3.3.8 will remain working. Other programs and
libs will remain usign 3.3.7 OR when newer libs are added to the compability
list, they will use the next in the series (like 3.3.9)


>> Solution: 
>> - binaries must _always_ have full version nr of required library.

>That would make dependency hell worse, as you'd need /every/ minor library
>revision present on your system.

No, since there is that db with compability lists.

However this is not possible with the current dynlinker. (which simply
searches the libraries named in the binary through the path, until it finds
one). Resolving a library because a lot more complex piece of code.

Libraries will probably need to be registered.

>> - I must be able to assign any library version to any binary. 

>You can do this by mucking with symlinks, or hexediting the binary.

Or building a new make world and upgrade all ports, yes:-)

But that is more hacking, not system management doable by a user.
Integration with the package system with the library dependancy system could
pretty much hide the work for the user.

Even intermediate users like me (with over 10 years of BSD/Linux experience)
might do such things for a special system, but don't want to be bothered
with it for each box that comes through this door.

> But in
> my experience, it doesn't work properly anyway, so you're better off just
> meeting the requirements of the application you're trying to run, one way or
> the other.

I usually don't even _have_ the requirements if I get only a binary!

> - there must be a system global db which lists version compabilities. 
>      so if binary lists 2.1.76, the db must list that 2.1.40-2.1.80
>      are compatible, and automatically assign the newest version of that
> 	library available in that range to the binary.
>     I'm still in dubio if this should be done only for libraries as a whole,
> 	or on a per symbol basis ( library version X is compatible with
> 	library version Y as long as you don't use for symbol Z)
> - There must be a database to register exceptions (specially for proprietary
> 	software).  E.g. if Heroes3 links to old versions libc and gtk, I
> 	must be able to list that heroes3 (as old binary only game) should
> 	get assigned some old libc/gtk versions only. (if file=heroes3 then
> 	assignlibc(x<=libc<=y) and assigngtk(z<=libgtk<=a)

>The way it /should/ work is that if the API of a library changes, the major
>version number should be incremented. If the API is left the same, then the
>major version number can be left unincremented, and old applications can
>inherit the improvements.


>The problem comes with library authors who don't follow these rules
>strictly 

A system that doens't account for possible human error is broken IMHO.

>when changing their library's API. You can have all the technical
>solutions in the world, but they're no use if people don't stick to the
>rules. ;-]

But _I_ (and the rest of the users) will have to deal with it.

So this is all _exactly_ what I'm trying to achieve, a way to influence
dependancies in a _systematic_ way _after_ the binaries in question have
been released.

And while there might be oversights in my solution for the problem, (the
dynlinker change would be fairly huge) the problem is real. The solution is
the best I could come up with in 1/2 hour, I'm sure the avg distro-builder
can put in some more time (and save a lot on support calls, "Sir, is your
library dependancy problem solving window empty?")

Also I'm talking about a system that tries to avoid and resolve issues. 

One could also use the version info in binary files (programs and libs), and
the dependancylist db to simply search for possible problems. (this would
remove need for a severely altered dynamic lib loader)