Re: Control-Cache max-age when polipo is restarted
Ruben Alleres <[email protected]>
| Newsgroups | gmane.comp.web.polipo.user |
|---|---|
| Message-ID | <[email protected]> |
I've added some tests to check polipo's behaviour with Cache-Control values. I've attached a patch which includes the bug fixing and these tests. This patch have been tested on automated-testing2 branch of https://github.com/wmanley/polipo. Thanks This transmission contains information that may be confidential and contain personal views which are not necessarily those of YouView TV Ltd. YouView TV Ltd (Co No:7308805) is a limited liability company registered in England and Wales with its registered address at YouView TV Ltd, 3rd Floor, 10 Lower Thames Street, London, EC3R 6YT. For details see our web site at http://www.youview.com ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb _______________________________________________ Polipo-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/polipo-users
0001-Fix-Polipo-ignores-cache-control-values-when-it-has-.patch
(text/plain, 9 KB)
From 6ea766a3046c0fcff64d5ae0cf69a54da531aa46 Mon Sep 17 00:00:00 2001 From: Ruben Alleres <[email protected]> Date: Fri, 1 Mar 2013 12:14:01 +0000 Subject: [PATCH] Fix Polipo ignores cache control values when it has restarted. The max-age value of Cache-control could be received in a http header. Polipo web cache proxy should be set this value as the maximum time that this page is stored in cache. When max-age has elapsed the cache is stale and it should be refreshed making a new request to the server. Polipo's is saving the cache control values in memory but they are not saved on disk doing that polipo doesn't know the cache control values when is restarted and setting them with a default value. httpPrintCacheControl is the polipo's function which writes the values from a CacheControlRec object in a buffer. It has two callers: - httpWriteRequest which does pass in a CacheControlRec object. - httpWriteObjectHeaders which did pass a NULL object rather a CacheControlRec object. httpWriteObjectHeaders is called when polipo writes the header values on disk cache so some cache control values was never written on disk. It has been updated to write cache-control values on disk. - Updated the call to httpPrintCacheControl from httpWriteObjectHeaders passing an updated CacheControlRec object as parameter. - Updated validateEntry function to fill the memory cache-control values when they has been read from disk cache. - Added test cases in polipo to check its behaviour when it receives a header which contains Cache-Control values: - Simple test cases using max-age, min-fresh and s-maxage values. - Test case for multiple values in the same header: i.e: max-age=3, min-fresh=20 (it expires when the minimum time has elapsed) --- diskcache.c | 2 + http.c | 9 +++++- test/expect_expire/cache_control_maxage.header | 4 +++ test/expect_expire/cache_control_maxage.payload | 1 + test/expect_expire/cache_control_minfresh.header | 4 +++ test/expect_expire/cache_control_minfresh.payload | 1 + test/expect_expire/cache_control_multiple.header | 4 +++ test/expect_expire/cache_control_multiple.payload | 1 + test/expect_expire/cache_control_smaxage.header | 4 +++ test/expect_expire/cache_control_smaxage.payload | 1 + test/run-test.sh | 30 ++++++++++++++++++++- 11 files changed, 59 insertions(+), 2 deletions(-) create mode 100644 test/expect_expire/cache_control_maxage.header create mode 100644 test/expect_expire/cache_control_maxage.payload create mode 100644 test/expect_expire/cache_control_minfresh.header create mode 100644 test/expect_expire/cache_control_minfresh.payload create mode 100644 test/expect_expire/cache_control_multiple.header create mode 100644 test/expect_expire/cache_control_multiple.payload create mode 100644 test/expect_expire/cache_control_smaxage.header create mode 100644 test/expect_expire/cache_control_smaxage.payload diff --git a/diskcache.c b/diskcache.c index 4dbaa99..d11f15b 100644 --- a/diskcache.c +++ b/diskcache.c @@ -1035,6 +1035,8 @@ validateEntry(ObjectPtr object, int fd, dirty = 1; object->cache_control |= cache_control.flags; + object->max_age = cache_control.max_age; + object->s_maxage = cache_control.s_maxage; if(object->age < 0) object->age = object->date; if(object->age < 0) object->age = 0; /* a long time ago */ diff --git a/http.c b/http.c index db35bbf..4d96a1b 100644 --- a/http.c +++ b/http.c @@ -286,6 +286,13 @@ httpWriteObjectHeaders(char *buf, int offset, int len, ObjectPtr object, int from, int to) { int n = offset; + CacheControlRec cache_control; + + cache_control.flags = object->flags; + cache_control.max_age = object->max_age; + cache_control.s_maxage = object->s_maxage; + cache_control.max_stale = -1; + cache_control.min_fresh = -1; if(from <= 0 && to < 0) { if(object->length >= 0) { @@ -351,7 +358,7 @@ httpWriteObjectHeaders(char *buf, int offset, int len, } n = httpPrintCacheControl(buf, n, len, - object->cache_control, NULL); + object->cache_control, &cache_control); if(n < 0) goto fail; diff --git a/test/expect_expire/cache_control_maxage.header b/test/expect_expire/cache_control_maxage.header new file mode 100644 index 0000000..5098549 --- /dev/null +++ b/test/expect_expire/cache_control_maxage.header @@ -0,0 +1,4 @@ +HTTP/1.1 200 OK +Content-type: text/html +Cache-Control: max-age=3 + diff --git a/test/expect_expire/cache_control_maxage.payload b/test/expect_expire/cache_control_maxage.payload new file mode 100644 index 0000000..ca1ab04 --- /dev/null +++ b/test/expect_expire/cache_control_maxage.payload @@ -0,0 +1 @@ +<html><head><body>Testing Cache-Control values</body></head></html> diff --git a/test/expect_expire/cache_control_minfresh.header b/test/expect_expire/cache_control_minfresh.header new file mode 100644 index 0000000..6dbbbcf --- /dev/null +++ b/test/expect_expire/cache_control_minfresh.header @@ -0,0 +1,4 @@ +HTTP/1.1 200 OK +Content-type: text/html +Cache-Control: min-fresh=3 + diff --git a/test/expect_expire/cache_control_minfresh.payload b/test/expect_expire/cache_control_minfresh.payload new file mode 100644 index 0000000..ca1ab04 --- /dev/null +++ b/test/expect_expire/cache_control_minfresh.payload @@ -0,0 +1 @@ +<html><head><body>Testing Cache-Control values</body></head></html> diff --git a/test/expect_expire/cache_control_multiple.header b/test/expect_expire/cache_control_multiple.header new file mode 100644 index 0000000..99715da --- /dev/null +++ b/test/expect_expire/cache_control_multiple.header @@ -0,0 +1,4 @@ +HTTP/1.1 200 OK +Content-type: text/html +Cache-Control: max-age=3 min-fresh=20 + diff --git a/test/expect_expire/cache_control_multiple.payload b/test/expect_expire/cache_control_multiple.payload new file mode 100644 index 0000000..ca1ab04 --- /dev/null +++ b/test/expect_expire/cache_control_multiple.payload @@ -0,0 +1 @@ +<html><head><body>Testing Cache-Control values</body></head></html> diff --git a/test/expect_expire/cache_control_smaxage.header b/test/expect_expire/cache_control_smaxage.header new file mode 100644 index 0000000..d0ce359 --- /dev/null +++ b/test/expect_expire/cache_control_smaxage.header @@ -0,0 +1,4 @@ +HTTP/1.1 200 OK +Content-type: text/html +Cache-Control: s-maxage=3 + diff --git a/test/expect_expire/cache_control_smaxage.payload b/test/expect_expire/cache_control_smaxage.payload new file mode 100644 index 0000000..ca1ab04 --- /dev/null +++ b/test/expect_expire/cache_control_smaxage.payload @@ -0,0 +1 @@ +<html><head><body>Testing Cache-Control values</body></head></html> diff --git a/test/run-test.sh b/test/run-test.sh index 5042291..2211673 100755 --- a/test/run-test.sh +++ b/test/run-test.sh @@ -36,6 +36,7 @@ stop_polipo () { run_test_case () { FILENAME_ROOT=$1 EXPECTED_REQUESTS_SERVED=$2 + [ "$#" -gt 2 ] && TIME_TO_EXPIRE=`expr $3 + 1` PAYLOAD=${FILENAME_ROOT}.payload HEADER=${FILENAME_ROOT}.header @@ -57,16 +58,18 @@ run_test_case () { for n in $(seq 1 10) do curl -o "${SAVE_LOCATION}" "http://localhost:${HTTP_SERVER_PORT}/" 2>/dev/null + [[ ! -z "${TIME_TO_EXPIRE:-}" && "$n" -eq 5 ]] && sleep $TIME_TO_EXPIRE cmp "${PAYLOAD}" "${SAVE_LOCATION}" || fail "Files don't match" rm "${SAVE_LOCATION}" done stop_polipo - + [[ ! -z "${TIME_TO_EXPIRE:-}" ]] && sleep $TIME_TO_EXPIRE for n in $(seq 1 10) do ( start_polipo curl -o "${SAVE_LOCATION}" "http://localhost:${HTTP_SERVER_PORT}/" 2>/dev/null + [[ ! -z "${TIME_TO_EXPIRE:-}" && "$n" -eq 5 ]] && sleep $TIME_TO_EXPIRE stop_polipo ) cmp "${PAYLOAD}" "${SAVE_LOCATION}" || fail "Files don't match" @@ -91,6 +94,31 @@ do run_test_case ${HEADER%%.header} 20 done +# Check polipo behaviour caching data with Cache-Control values +# in header related with expiration time. +# An expect_expire directory was provide with +# different header test cases. +# run_test_case make 20 request: +# - The 10 first request are consecutives +# (Polipo won't be closed between requests). +# - In the next 10 request polipo will be started before +# every request and closed after. This allow check the +# polipo behaviour when is restarted and there are cached. +# data with expire time specified. +# The third argument of run_test_case is the duration in seconds +# when the cached data time will expire and it must be its value must +# be the same than the Cache-control value field (max-age, min-fresh or +# s-maxage). The test will sleep after 5th and 15th request and between +# the 10 first test cases and the 10 next ones during the time to cause +# cached data time expiration. +# 4 request will be made to the server: +# - Initial request. (response is cached) +# - In the 6th,11st and 16th requests (after cached data expires) +for HEADER in ${POLIPO_TEST_DIR}/expect_expire/*.header +do + run_test_case ${HEADER%%.header} 4 3 +done + rm -R "${TMP_DIR}" echo SUCCESS -- 1.7.1