Re: [Security-Discuss] 10.1 x86_64 update mirrors are out of sync

On 12-Jul-05, at 10:38 AM, Michael Riss wrote:

>>>>> Could someone check the master mirror please?
>>>>> I even tried to sync from the Mandriva corporate rsync server,
>>>>> but the updates packages are also missing there.
>>>>>
>>>
>>> It works now. Thanks.
>>>
>>> As mentioned on the main list, glitches in the Mandriva mirror  
>>> system
>>> happen quite often. Is it possible to improve something here?
>>>
>>
>> Sorry, my mail client was being stupid and cut off my single line
>> response when I changed my from address.
>>
>> What exactly was the problem?  Which packages were missing?
>>
>
> After reading the security-announce mails I tried to do an update this
> morning, but "urpmi --auto-select" came up with no packages.
> So I checked the date of the hdlist.cz file, it was dated as  
> 07.07.2005
> and the cpio-RPM had not the recent version - a typical sync problem.
> I resynced from your corporate rsync server, but nothing changed.
> I looked at ftp.proxad.net - still the old packages.'

Silly question, but you did a "urpmi.update -a" first, before the -- 
auto-select, right?  Otherwise urpmi is working with the hdlist.cz or  
synthesis list on the local machine which wouldn't of course contain  
any reference to the new packages.

Also..  "this morning" doesn't mean much to me.  It's still morning  
for me (11:30am) so could morning have been 8hrs ago?  Longer?  Shorter?

I ask because it of course takes time for the mirrors to catch up.  I  
waited about 3hrs (I think?) before announcing the packages last  
night which usually is enough time to get it to the "primaries"  
anyways.  Not sure if proxad is a "primary" or not.

Also, what do you mean by the corporate rsync server?  I didn't know  
we had one.  Are you updating corporate server or desktop?  Because  
that one, from what I understand, should be damn-near instantaneous  
(AFAIK, I'm uploading directly to the corp repository).

>> Honestly, as much as I'd love to improve the mirror system (or get
>> away with it altogether) there isn't much we can do as we're relying
>> on the goodwill of volunteers and companies to mirror things.  Some
>> mirror hourly, some mirror daily.  Then you get mirrors mirroring
>> from other mirrors and you end up with a nice cascade effect.
>>
>> My preferred solution would be to scrap the mirrors and setup our own
>> self-controlled mirrors at strategic geographic locations, but I
>> don't think that will happen anytime soon.  To some people, the
>> current mirror thing is good enough.  I'm not one who shares that
>> sentiment for the most part.
>>
>> But I'd like to still know what was missing etc.
>>
>
> You are right about the external mirrors, you can't do anything  
> about them.
> But we also have access to the Mandriva corporate rsync server.
> We should not be affected from these problems - but we are.
> In most cases the corporate server is also out of sync, so the  
> glitch has
> to happen somewhere inside Mandriva (according to the main list,
> it was a hard disk running full this time).

What is the main list?  And was this hard disk being full something  
that happened today?  And I still don't know about this corporate  
rsync server.  So many questions... =)

It could very well be something inside of Mandriva, but I (as you  
probably have guessed by now) know almost nothing about the  
infrastructure.  I can guess how many hops the packages take (or at  
least how many they used to... it could very well have changed in the  
last year or two), but I don't know what happens to the intermediate  
steps.

I do know that for the corp products I upload to what I believe is  
the accessible repository and if it isn't, it's one hop away from  
it.  For "regular" updates, I upload to our development system, which  
our rsync system mirrors from, and the primaries get from the rsync  
system; the other mirrors from the primaries, etc.  I assume,  
possibly incorrectly, that the "corporate rsync server" is the rsync  
system the mirrors use.

If it is, then it's 2 hops from the originating system; if it's not  
the same server, then I don't know how many hops it is.

> Don't get me wrong, accidents can happen from time to time.
> But if you look through the archives of this list, you will find  
> several
> postings from me about mirror problems, outdated hdlist.cz files or
> wrong public keys on the packages.
> I guess for every 10 security updates there is one of these problems.

Yes, I know.  You're absolutely right, and each time I forward the  
problems on to the people who can fix them.  How they fix them I do  
not know.  But I don't have the access to fix/change things on my own  
so there are communication delays, etc.  Anyways, I won't make  
excuses.  You and I both agree there are issues with the current  
system.  I don't know how else I can improve the situation beyond  
making my own repository (the master) publically accessible.  But  
that would cost me too much bandwidth.  =)

> Maybe this update mechanism has some hidden depths I do not see,
> but for me it looks like it's a matter of digging through all the  
> steps
> necessary to ship an update package and then to write scripts
> which do the signing of the package, updating of the hdlists and
> sending of the packages and hdlists to the master mirror.

I can dig through my steps.  But I know that's not where the problem  
is.  Packages are signed, hdlists updated, all the "maintenance" is  
done before the packages even get uploaded anywhere.  All I ever do  
is upload the file tree (well, rsync it).  All the "maintenance" is  
done here.  So errors with package signing (which happens on occasion  
although I'm not sure why since nothing ever changes), or hdlist  
generation, etc. would be on my end.  Unless something isn't being  
synced or gets corrupt in transit, etc.

The problem is somewhere once everything gets released from my  
system, but once it's out of my system, it's largely beyond my control.

> Am I wrong?

A little bit, but mostly not.

-- 
"lynx -source http://linsec.ca/vdanen.asc | gpg --import"
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}