Re: oldest requests fetched last

Dan Jacobson <[email protected]> writes:

> A> The current method will tend to get the pages requested offline first,
> A> but won't do anything to guarantee this.
> 
> OK, let's discuss the order requests made offline are later fetched
> when online.
> 
> I notice it seems it is a first in - last out order -- the requests
> you made longest ago will take longest to be fetched.

I don't know why you would see this.  The requests that you make
offline should be fetched in the order that you made them.  There may
be other pages or images fetched in between, but the order of the
requests should be the order they are fetched.

> Sure, if one can finish fetching all in one connection, who will
> notice the difference. All pages got fetched.
> 
> But with short online sessions, we may never get to fetch those older
> requests, as new requests pile on top of their heads, offline or on.
> 
> Hmmm, and I seem to recall /var/cache/wwwoffle/outgoing is fetched in
> ls -U order.  Perhaps efficient, but bad: throw ordering to the winds
> of chance.

Yes it is fetched in this order which is why it is a FIFO, entries are
fetched in the order they are added to the directory.

> Proposal: new variable: wwwoffle-fetch-offline-order: Default "U"
> (random). ls(1) switches used in ordering fetching of request made
> offline. Use "t" to fetch newest requests first, "tr" to fetch oldest
> requests first. Hmmm.

If there is any sorting of the files in the outgoing directory it will
slow down WWWOFFLE.  Currently there is no searching required because
the first file found is used.  If you want any of these sort options
then the time taken will increase with N^2 (When there are N files it
takes N directory lookups for each of them).  This is why this method
was removed in version 1.2e of WWWOFFLE over 7 years ago.

> Or perhaps I can do a hack to reorder the outgoing directory just
> before going online:
> mkdir outgoing.FIFO
> cd outgoing
> ln `ls -U|tac` ../outgoing.FIFO
> cd ..
> mv outgoing outgoing.LIFO #backup
> mn outgoing.FIFO outgoing
> OK, I'll see how that works. "However note, personal hacks do not benefit
> the masses."

This will only change the order of the pages that are requested when
offline.  It won't do anything about any new pages that need to be
fetched as a result of the content of those pages.

-- 
Andrew.
----------------------------------------------------------------------
Andrew M. Bishop                             amb-Uxr6IM1mbv2TY6FTCsQk+9Bc4/[email protected]
                                      http://www.gedanken.demon.co.uk/

WWWOFFLE users page:
        http://www.gedanken.demon.co.uk/wwwoffle/version-2.8/user.html