Date: Sunday, August 3, 2008 @ 16:02:50
Author: eagle
Revision: 4226
Add a section describing the history mechanism and algorithm. Update
references in Duties of an Injecting Agent and Duties of a Serving
Agent accordingly, moving the specifics of date-based rejection from
the Duties sections to the history section and making clearer the
distinction between the protocol requirement and the implications of
specific implementations satisfying that requirement.
Modified:
docs/usefor/usepro.xml
Modified: docs/usefor/usepro.xml
===================================================================
--- docs/usefor/usepro.xml 2008-08-03 21:07:34 UTC (rev 4225)
+++ docs/usefor/usepro.xml 2008-08-03 23:02:50 UTC (rev 4226)
@@ -452,6 +452,66 @@
</section>
</section>
+ <section anchor="history"
+ title="Article History and Duplicate Suppression">
+ <t>Netnews normally uses a flood-fill algorithm for propagation of
+ articles in which each news server offers the articles it accepts
+ to multiple peers and each news server may be offered the same
+ article from multiple other news servers. Accordingly, duplicate
+ suppression is key; if a news server accepted every article it was
+ offered, it may needlessly accept (and then potentially
+ retransmit) dozens of copies of every article.</t>
+
+ <t>Relaying and serving agents therefore MUST keep a record of
+ articles they have already seen and use that record to reject
+ additional offers of the same article. This record is called the
+ "history" file or database.</t>
+
+ <t>Each article is uniquely identified by its message identifier,
+ so a relaying or serving agent could satisfy this requirement by
+ storing a record of every message identifier that agent has ever
+ seen. Such a history database would grow without bound, however,
+ so it is common and permitted to optimize based on the
+ Injection-Date or Date header field of an article as follows. (In
+ the following discussion, the "date" of an article is defined to
+ be the date represented by its Injection-Date header field if
+ present, otherwise its Date header field.)
+ <list style="symbols">
+ <t>Agents MAY select a cutoff interval and reject any article
+ with a date farther in the past than that cutoff interval. If
+ this interval is shorter than the time it takes for an article
+ to propagate through the network, the agent might reject an
+ article it had not yet seen, so it ought not be aggressively
+ short. For Usenet, for example, a cutoff interval of no less
+ than seven days is conventional.</t>
+
+ <t>Agents that enforce such a cutoff MAY then drop records of
+ articles that had dates older than the cutoff from their
+ history databases. If such an article were offered to the
+ agent again, it would be rejected due to the cutoff date, so
+ the history record is no longer required to suppress the
+ duplicate.</t>
+
+ <t>Alternatively, agents MAY drop history records according to
+ the date when the article was first seen by that agent rather
+ than the date of the article. In this case, the history
+ retention interval MUST be at least 24 hours longer than the
+ cutoff interval to allow for articles dated in the future.
+ This interval matches the allowable error in the date of the
+ article (see <xref target="injecting" />).</t>
+ </list>
+ </t>
+
+ <t>These are just two implementation strategies for article
+ history, albeit the most common ones. Relaying and serving agents
+ are not required to use these strategies, only to meet the
+ requirement of not accepting an article more than once. However,
+ these strategies are safe and widely deployed and implementors are
+ encouraged to use one of them, especially if they do not have
+ extensive experience with Netnews and the subtle effects of its
+ flood-fill algorithm.</t>
+ </section>
+
<section anchor="posting" title="Duties of a Posting Agent">
<t>A posting agent is the component of a user agent that assists a
poster in creating a valid proto-article and forwarding it to an
@@ -806,19 +866,19 @@
field or Message-ID header field, or without either an
Injection-Date or Date header field.</t>
- <t>It MUST reject any article that has already been
- successfully sent to it, based on the Message-ID header field
- of the article. To satisfy this requirement, a relaying agent
- normally keeps a database of message identifiers it has
- already accepted.</t>
-
<t>It MUST examine the Injection-Date header field or, if
absent, the Date header field, and reject the article if that
- date predates the earliest articles of which it keeps record
- or if that date is more than 24 hours into the future. It MAY
- reject articles with dates in the future with a smaller margin
- than 24 hours.</t>
+ date is more than 24 hours into the future. It MAY reject
+ articles with dates in the future with a smaller margin than
+ 24 hours.</t>
+ <t>It MUST reject any article that has already been accepted.
+ If it implements one of the mechanisms described in <xref
+ target="history" />, this means that it MUST reject any
+ article whose date falls outside the cutoff interval since it
+ won't know whether such articles had been accepted previously
+ or not.</t>
+
<t>It SHOULD reject any article that does not include all the
mandatory header fields. It MAY reject any article that
contains header fields that do not have valid contents.</t>
@@ -891,16 +951,16 @@
<t>It MUST examine the Injection-Date header field or, if
absent, the Date header field, and reject the article if that
- date predates the earliest articles of which it keeps record
- or if that date is more than 24 hours into the future. It MAY
- reject articles with dates in the future with a smaller margin
- than 24 hours.</t>
+ date is more than 24 hours into the future. It MAY reject
+ articles with dates in the future with a smaller margin than
+ 24 hours.</t>
- <t>It MUST reject any article that has already been
- successfully sent to it, based on the Message-ID header field
- of the article. To satisfy this requirement, a relaying agent
- normally keeps a database of message identifiers it has
- already accepted.</t>
+ <t>It MUST reject any article that has already been accepted.
+ If it implements one of the mechanisms described in <xref
+ target="history" />, this means that it MUST reject any
+ article whose date falls outside the cutoff interval since it
+ won't know whether such articles had been accepted previously
+ or not.</t>
<t>It SHOULD reject any article that matches an
already-received and honored cancel message or Supersedes
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.