Re: CACHE_MUST_REVALIDATE

Hi Juliusz,

I can't wait for the polipo 2.0 so I will keep on playing with 1.x-:)

There are three things that I changed during the holiday season to make
polipo happy with the Microsoft Cougar server. (stream source)

1. A client request queued on a connection my not necessary good to be
served from cached object. e.g. a client request hit a in progress new
object. It is not known if the object is good for caching as polipo haven't
see the reply headers yet. So when the new object is filled from server.
Whether it is good for the queued requests on a connection should be checked
again.

--- smg/contrib/polipo-cache/client.c 2008/06/12 17:58:16 1.4

+++ smg/contrib/polipo-cache/client.c	2009/01/05 16:33:54	1.5
@@ -239,7 +239,7 @@ httpClientFinish(HTTPConnectionPtr conne
            into the queue */
         if(connection->request) {
             if(connection->request->object != NULL)
-                httpClientNoticeRequest(connection->request, 1);
+                httpClientNoticeRequest(connection->request, 0);
             else
                 assert(connection->flags & CONN_READER);
         }

2. Polipo uses HEAD to do re-validating on must-validate and no_cache
object. The microsoft cougar will keep silent on the HEAD request or reply
to say it only support GET and POST. Change the validating method from HEAD
to GET makes the cougar server happy. Does it make sense to use GET if
server warn the proxy that object is like to change?

--- smg/contrib/polipo-cache/client.c	2008/06/12 17:58:16	1.4
+++ smg/contrib/polipo-cache/client.c	2009/01/05 16:33:54	1.5
@@ -239,7 +239,7 @@ httpClientFinish(HTTPConnectionPtr conne
            into the queue */
         if(connection->request) {
             if(connection->request->object != NULL)
-                httpClientNoticeRequest(connection->request, 1);
+                httpClientNoticeRequest(connection->request, 0);
             else
                 assert(connection->flags & CONN_READER);
         }

3. problem when calculating haveData in httpClientNoticeRequest(). When the
server reply have no Content Length header, the haveData will be false.

 *if*(request->method == METHOD_HEAD)
        haveData = !(request->object->flags & OBJECT_INITIAL);
    *else*
        haveData =
            (request->object->length >= 0 &&
             request->object->length <= request->from) ||
            (objectHoleSize(request->object, request->from) == 0);


This causes the httpServeObject() to be skipped later in the function
    *if*(!(request->object->flags & OBJECT_VALIDATING) &&
       ((!validate && haveData) ||
        (request->object->flags & OBJECT_FAILED))) {
        *if*(serveNow) {
            connection->flags |= CONN_WRITER;
            lockChunk(request->object, request->from / CHUNK_SIZE);
            *return* httpServeObject(connection);
        } *else* {
            *return* 1;
        }
    }
....

    request->object->flags |= OBJECT_VALIDATING;
    rc = request->object->request(request->object,


Once it skipped the returns here, OBJECT_VALIDATING will be added to
the object,
making the obj not servisible to the client. If the object has a
finite size, the serverfinish will eventually remove
the OBJECT_VALIDATING flag. Depending on the size of the object, the
end user may or maynot notice
a delay on the page he/she is loading. But for stream, this is deadly.
I don't have a good way to fix the haveData value, so I start to
remove the OBJECT-VALIDATING flags
once the server side started to read reply data.

--- smg/contrib/polipo-cache/server.c	2008/08/27 20:50:43	1.5
+++ smg/contrib/polipo-cache/server.c	2009/01/05 16:33:54	1.6
@@ -2747,6 +2747,14 @@ httpServerReadData(HTTPConnectionPtr con
     if(connection->len == 0)
         httpConnectionDestroyBuf(connection);

+    /* This is the wrong place to fix the haveData calculation in client.c
+     * The OBJECT_VALIDATING is wrongly added if we were asked to
+     * read the body of a reply.
+     * It should be corrected once a fix for haveData is found.
+     */
+    if (! immediate) {
+	object->flags &= ~OBJECT_VALIDATING;
+    }
     httpSetTimeout(connection, serverTimeout);
     do_stream_buf(IO_READ | IO_NOTNOW |
                   ((immediate && connection->len) ? IO_IMMEDIATE : 0),


The three changes make a lot of audio stream sites working. Based on
the protocol trace I see, there is
not much I can say that the traffic is non-HTTP, except the HEAD
method is not welcome.

Happy New Year,
Ming

On Tue, Jan 6, 2009 at 4:49 PM, Juliusz Chroboczek <
[email protected]> wrote:

> > Here is the traffic that lead me to look into how to make polipo NOT
> reply
> > from cache. You can see from the protocol trace that the client is
> > requesting to the same url, but each time it gets a different reply. This
> is
> > captured without proxy. The tcpdump is attached in gzip format.
>
> Hmm, the server is saying ``Cache-control: no-cache'', so Polipo will
> revalidate by sending an ``If-Modified-Since'' request.  Doesn't that work?
>
> By the way, it appears that you are trying to pipe streaming media through
> Polipo.  Polipo was not designed for that kind of usage, so depending on
> what you are trying to achieve, you might be better off with a different
> proxy (such as Polipo 2.0, which should be out by the end of the decade).
>
>                                        Juliusz
>
------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB