Re: perl breakage in Cooker mostly resolved

On Jan 20, 2012, at 3:57 PM, Denis Silakov wrote:

> On 01/20/12 21:56, Jeffrey Johnson wrote:
>> What would be needed to use in perl to detect a deletion
>> in the perl abi of the symbol "Perl_Gthr_key_ptr" is this:
>> 
>> 1) Extract all exposed external symbols that de facto constitute perl's ABI.
>> 
>> 2) Sort the list of symbols, and compute the digest of the list of symbols.
>> Use any digest: there are few security related issues for using a digest
>> for "identification" rather than for an integrity check. but use SHA-4096
>> if you are concerned, all that is needed is uniqueness.
>> 
>> 3) Generate the dependency (like kernel/ocaml packages do) as
>> 	Requires: perl(abi) =<put the digest in hex here>
>> 
>> 4) Test the results, examine the consequences, filter out symbols
>> and/or split the single dependency into multiple dependencies, as needed.
>> 
>> I expect that the above technique -- because it can be automated reliably -- to
>> be increasingly important for *.rpm dependencies over the next few years.
>> 
> 
> In general it is a good idea, but probably it's really an overkill for Perl where it's not so hard to just rebuild all necessary packages (though I can't estimate their number atm, maybe I'm wrong...). At least perl updates that break ABI happen not as often as kernel updates that can break the modules.
> 

Good that we agree "good idea" as well as "overkill" for perl (because it what take
someone with extensive perl internal experience to filter the symbols intelligently
enough for perl to switch to a scheme like in OCAML and the kernel).

> One more thing is that symbol digests don't give you 100% guarantee that you binary will work with new library; two approaches can be mentioned here:
> 1) when calculating hash, you take into account only binary symbol names - in this case you can generate dependencies relatively fast, but you can't handle e.g. signature changes. So there is still a significant chance that you application will fail with updated ABI
> 2) you can calculate hash on the basis of binary symbol bodies - this gives more guarantees, but requires more tricky implementation, and dependency generation will take more time.
> 

Yes but …

… to call the technique by its real name this is "currying" in widespread
use in "functional" programming languages like Haskell and also
being attempted for "package management" at
	http://nixos.org

There is nothing that depends on "symbol list digests" involved with
"currying". Symbol digests are merely a convenient plaintext that
defines an ABI in a de facto fashion precisely.

You are quite correct that ABI tracking isn't gud enuf (but is better than nothing).

(for extra credit)
There is an additional technique that permits curried plain texts to be chained
(and not only with symbols) that is useful for pinning down exactly what broke
that is also going to be needed for the technique I mentioned to be deployed.

Lets say you split the huge list of perl symbols into a "core" and a "extension"
set.

Each of "core" and "extension" will then have its own digest.

The chaining comes from having a meta-plaintext composed of 2 lines
(for my artificial example) that would look like
	core		<core-digest-here>
	extension	<extension-digest-here>

And that 2-line file is also digested (and included somehow with packaging).

Then when the
	Requires: perl(abi) = <meta-plaintext-digest-here>
"breaks" (as it inevitably will), then the actual plaintext is retrieved to
identify whether "core" or "extension" symbols were changed.

The above is an entirely artificial example used to illustrate the chaining.

A general scheme to handle "configuration changes" (i.e. not just ELF symbols)
has been implemented in WindRiver 3.0 since March 2008.

Just Works afiak (and there is a proposal in <[email protected]> archives
that I can dig out and turn into a blueprint whenever there is interest in
better automation for generating *.rpm package dependencies).

hth

73 de Jeff
> But nothing prevents us from performing such experiments (locally first, not in cooker:)) to get some real estimations.
> 
> --
> Regards,
> Denis.