Re: [Security-Discuss] 10.1 x86_64 update mirrors are out of sync

On 13-Jul-05, at 3:29 AM, Michael Riss wrote:

>>>> What exactly was the problem?  Which packages were missing?
>>>>
>>>
>>> After reading the security-announce mails I tried to do an update  
>>> this
>>> morning, but "urpmi --auto-select" came up with no packages.
>>> So I checked the date of the hdlist.cz file, it was dated as
>>> 07.07.2005
>>> and the cpio-RPM had not the recent version - a typical sync  
>>> problem.
>>> I resynced from your corporate rsync server, but nothing changed.
>>> I looked at ftp.proxad.net - still the old packages.'
>>>
>>
>> Silly question, but you did a "urpmi.update -a" first, before the --
>> auto-select, right?  Otherwise urpmi is working with the hdlist.cz or
>> synthesis list on the local machine which wouldn't of course contain
>> any reference to the new packages.
>>
>
> Yes, I did a "urpmi.update update_source" first. ;)

Ok, had to check.  Some people don't.  =)

>> Also..  "this morning" doesn't mean much to me.  It's still morning
>> for me (11:30am) so could morning have been 8hrs ago?  Longer?   
>> Shorter?
>>
>
> Sorry, I meant 10:00am CEST.
> Btw. where are you situated?

MST, so quite a few hours behind you (GMT-7).  I'm in Edmonton which  
is in Alberta (western Canada).

>> I ask because it of course takes time for the mirrors to catch up.  I
>> waited about 3hrs (I think?) before announcing the packages last
>> night which usually is enough time to get it to the "primaries"
>> anyways.  Not sure if proxad is a "primary" or not.
>>
>> Also, what do you mean by the corporate rsync server?  I didn't know
>> we had one.  Are you updating corporate server or desktop?  Because
>> that one, from what I understand, should be damn-near instantaneous
>> (AFAIK, I'm uploading directly to the corp repository).
>>
>
> No, this is independent from the Corporate Desktop or Server.
> We are a member of the Corporate Mandriva Club. One benefit of this is
> access to a rsync server at Mandriva with the latest Commercial  
> Club packages
> and the whole update tree.

Ahhh... ok.

> [..]
>
>>> You are right about the external mirrors, you can't do anything
>>> about them.
>>> But we also have access to the Mandriva corporate rsync server.
>>> We should not be affected from these problems - but we are.
>>> In most cases the corporate server is also out of sync, so the
>>> glitch has
>>> to happen somewhere inside Mandriva (according to the main list,
>>> it was a hard disk running full this time).
>>>
>>
>> What is the main list?  And was this hard disk being full something
>> that happened today?  And I still don't know about this corporate
>> rsync server.  So many questions... =)
>
> The main list? ;) [email protected]
> And I'm refering to this posting:
> http://archives.mandrivalinux.com/cooker/2005-07/msg01068.php

Hmmm... shouldn't impact the mirroring.  There have been some network  
issues that caused some problems last week when I was trying to get  
zlib out, but I haven't seen any of that since.

>> It could very well be something inside of Mandriva, but I (as you
>> probably have guessed by now) know almost nothing about the
>> infrastructure.  I can guess how many hops the packages take (or at
>> least how many they used to... it could very well have changed in the
>> last year or two), but I don't know what happens to the intermediate
>> steps.
>>
>> I do know that for the corp products I upload to what I believe is
>> the accessible repository and if it isn't, it's one hop away from
>> it.  For "regular" updates, I upload to our development system, which
>> our rsync system mirrors from, and the primaries get from the rsync
>> system; the other mirrors from the primaries, etc.  I assume,
>> possibly incorrectly, that the "corporate rsync server" is the rsync
>> system the mirrors use.
>>
>> If it is, then it's 2 hops from the originating system; if it's not
>> the same server, then I don't know how many hops it is.
>
> Although there are several hops involved, that shouldn't pose a
> problem as rsync is quite robust ... unless something happens
> like a hard disk running full of course, many systems, many sources
> of errors. :-\

No, you're right, but timing is a thing.  For one, if I upload as the  
rsync machine is finishing it's sync, there's automatically 1hr  
added.  And if a mirror gets from our master in the middle of a  
transfer, it might update some files, but the hdlists may be left out  
because they haven't been synced yet, and if that's hourly, you've  
just added another hour to the wait.  That's why I typically wait  
3-4hrs from when I upload to when I announce due to situations like  
that.

>>> Don't get me wrong, accidents can happen from time to time.
>>> But if you look through the archives of this list, you will find
>>> several
>>> postings from me about mirror problems, outdated hdlist.cz files or
>>> wrong public keys on the packages.
>>> I guess for every 10 security updates there is one of these  
>>> problems.
>>>
>>
>> Yes, I know.  You're absolutely right, and each time I forward the
>> problems on to the people who can fix them.  How they fix them I do
>> not know.  But I don't have the access to fix/change things on my own
>> so there are communication delays, etc.  Anyways, I won't make
>> excuses.  You and I both agree there are issues with the current
>> system.  I don't know how else I can improve the situation beyond
>> making my own repository (the master) publically accessible.  But
>> that would cost me too much bandwidth.  =)
>>
>
> Maybe you can talk with the maintainer of the mirror system and ask
> him whether he can think of some improvements.
> If this doesn't work you could talk with the management and apply for
> a fatter pipe and really open you own rsync server. ;)

LOL.  I'm paying for my own internet connection, so I have my doubts  
they'll pony up for a fatter pipe to my house (not to mention laying  
the fibre will be expensive in the first place!).  Not that I'd  
complain if they volunteered, but somehow I doubt it.  And I have  
forwarded this issue to the mirror maintainer and he is looking into  
it because he says this situation is not normal.

>>> Maybe this update mechanism has some hidden depths I do not see,
>>> but for me it looks like it's a matter of digging through all the
>>> steps
>>> necessary to ship an update package and then to write scripts
>>> which do the signing of the package, updating of the hdlists and
>>> sending of the packages and hdlists to the master mirror.
>>>
>>
>> I can dig through my steps.  But I know that's not where the problem
>> is.  Packages are signed, hdlists updated, all the "maintenance" is
>> done before the packages even get uploaded anywhere.  All I ever do
>> is upload the file tree (well, rsync it).  All the "maintenance" is
>> done here.  So errors with package signing (which happens on occasion
>> although I'm not sure why since nothing ever changes), or hdlist
>> generation, etc. would be on my end.  Unless something isn't being
>> synced or gets corrupt in transit, etc.
>>
>
> Do you have read access to the "master mirrors" at Mandriva?
> If yes, you could write a script which compares the update tree there
> with yours before you send out the advisories (just an idea).

I have write access (gotta upload them somehow).  And I know they're  
all there when rsync tells me there's nothing left to transfer.  I  
have the full updates/ mirror structure here, so I'm not uploading  
files piece-meal, I'm rsyncing the entire structure which is how I  
can quickly test for errors and report definitively if there is  
something wrong with a package.  If it's not broke here, then it's  
not the package at fault, but some corruption in-transit, etc.  But  
that only tells me the master is updated.. that doesn't tell me how  
many mirrors are updated, etc.

>> The problem is somewhere once everything gets released from my
>> system, but once it's out of my system, it's largely beyond my  
>> control.
>>
>
> Yes, seems like a tricky situation. But thanks for the explanation  
> from
> your side. From the outside it looks very odd that there are nearly no
> problems with the RPMS themself, they install smooth, no side effects,
> real good QA. But then on the way to the customer things suddenly
> get bumpy, missing hdlists, wrong keys, some RPMS missing.
> It looks like someone is sitting there and copying the files by hand
> and then - of course - forgets some of them from time to time.
> But it's more like a many cooks, many systems problem.

I think you've nailed it.  Too many cooks in the kitchen, then you  
have to deal with some cooks have days off, working 8hrs apart so  
everything isn't in sync between the cooks, etc.  The mirroring issue  
has been an issue on and off for a long long time.  I'd like to say  
it can be dealt with better, but due to the logistics involved I  
don't think it will get much better.  There will always be delays.   
Hopefully not 8hr delays anymore, but always delays.  Then you get  
some mirrors that sync daily rather than hourly and you really have a  
delay.

-- 
"lynx -source http://linsec.ca/vdanen.asc | gpg --import"
{FEE30AD4 : 7F6C A60C 06C2 4811 FA1C  A2BC 2EBC 5E32 FEE3 0AD4}