Use of 304 - not modified

Hi developers

a system manager of a rather large Silva installation, I often think
about performance. A lot of requests gets answered by the HTTP caches,
but this leads to other problems, mainly pages in different caches being
'not fresh enough'; or people, at the same moment, seeing different content.

While looking at the HTTP protocol specs I found a way to optimize CPU
usage without sacrificing document freshness. The protocol offers the
'304 - not modified' answer, which was mainly designed to reduce network
load: a cache will not reload the content of a page, even if that copy
has expired, if it can be revalidated. Possible revalidation keys are
Last-modified and ETag.

For sites where even the most dynamic content changes only every few
minutes, but most of the pages contain some of this dynamic content, a
page freshness revalidation feature could save a lot of CPU time by
revalidating pages instead of rendering them completely. And would make
my users much happier by reducing the base caching time.

Since this issue affects core components and includes the extension of
the object methods, I would like to open the discussion.

The HTTP header Last-modified must be set correctly, so we can use it as
a freshness indicator. It depends on the save date of a document. It may
also depend on some external source objects used in the document. So a
Silva object should respond on this new method, 'last modified
question', by internally asking all external object too, since we don't
(and should not) know about their context.

Every relevant dynamic component must provide a method which compares a
given date to its own freshness, and gives back what it thinks would be
the date for a document generated right now. This leads to an overall
Last-modified date which can be used for validation.

If a cache (public or browser internal) requests a new copy of a page,
we add a relative freshness timer, the HTTP max-age header. After
max-age, a number of seconds, has expired, the cache will ask the
upstream server as usual, but add a 'If-modified-since' header.

If we detect such a header, we could ask a document to compare its date
with the content of the header. Maybe the Zope server can end the
request by returning 304 without any content. Depending on what kind of
site you have, the CPU savings could be huge: compared to the complete
XML-HTML transformation,

- date lookups are cheap
- many external sources don't care about time (formatting stuff)
  and would simply return the same value: very cheap
. others will change regularly on a adjustable base: a comparison
  with the value of If-modified-since is very cheap, too

The 304 answer will make caches and clients happy, and due to the
relative max-age header, the timer will be reset and start again. A
proxy server can use the revalidated copy for answering further
requests. So we could keep caching times low, while the proxy caches
serve the same content even for weeks, as long as it validates.

Regards

Bengt