Re: Multiple Polipos sharing disk cache

On 27.11.2012 21:36, Juliusz Chroboczek wrote:
>> # What
>> I would use two Polipo instances, both using the same disk cache.
>
> Just go ahead, do it. It's safe; the only issue is that if both Polipi
> access the same URL at the same time, one of the two Polipi may
> occasionally drop a connection or do some useless work (write out data
> to an unlinked file). Fixing this would require removing the
> file-descriptor cache, which would be a shame.
>
> You'll see occasional warnings and broken connections, but no data
> should be corrupted. (This is completely untested, of course.)
I'm already doing it. Just seldomly one of them stops serving webpages. But could be something else.


>> Make Polipo use locking when creating new cache files.
>> Make Polipo use locking if changing cache files in-place.
>
> Polipo's update strategy is safe, with the above caveats. No locks are
> needed. (And I don't like using locks, recall that Polipo is a labour
> of love, it's not meant to be actually useful.)
>
> Note that using a lock wouldn't be enough to fix the underlying issue,
> which is that the two Polipi won't asynchronously notify each other when
> the shared on-disk data changes; this is why you'll see some useless
> work being done.
This one I didn't notice *dumb me*, thanks. So basically sharing the disk cache doesn't work well
at all. If polipo needs to have it's internal data structures in sync to serve the data, rather than
just referring the request to a file on disk.. Not to mention if, during runtime, Polipo doesn't
read unknown cache entries from disk, or how often it refreshes known ones from disk.

Well bah. There goes that idea :(


> -- Juliusz

------------------------------------------------------------------------------
Keep yourself connected to Go Parallel: 
DESIGN Expert tips on starting your parallel project right.
http://goparallel.sourceforge.net