Bypass XHDR

This is a multi-part message in MIME format.
--------------59020DDA6BC05388FF973068
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

First, it needs to be stated emphatically that I am not a C programmer! 
All I can do is copy code and then hack it until it does what I want, or
I admit defeat and give up.

The reason for this patch is that I wanted to suck a newsgroup with over
5 million articles; it was taking about 7 hours to run XHDR.  In my
opinion, XHDR is a complete waste of time for a news server because it
downloads the entire article and never just the headers.  In addition to
the wasted 7 hours, my upstream NNTP server - newsguy - places a
limitation on either the number of articles or the number of bytes
downloaded in a single setting, and sends "503 transfer limit, closing
connection." when that limit is reached.  I didn't time it, but my guess
is that the actual article download ran for 10 or 11 hours before the
503 error.  Thus, each attempt consumed 17 or 18 hours, of which
something like 40% was totally wasted.  Worse, if I allowed snfetch to
automatically restart, it would begin with the original .serial number
value and download the same articles again, providing a screenfull of
"oops"s and not progressing at all.

My connection is very fast, via a cable modem.

So I hacked snfetch to skip sending XHDR and go straight to downloading
articles, beginning with the article number in .serial.  In order to
assure that .serial is reasonably correct, the following shell script
runs on a different terminal:

#! /bin/bash
# This is Serial.sh.  It updates .serial and terminates sn when a 503
error occurs.
# The last article requested is received even though there is no 220
line for it.
# Depending on speed, a 15 second sleep may cause a small inaccuracy in
.serial, so it's
# best to look at /tmp/follow and record the correct article number in
.serial manually.
while :; do
  SN=`tail -n20 /tmp/follow | grep " 220 " | tail -n1 | awk '{print
$3}'`
  echo "$SN" >/PATH/TO/FAKE.NEWS.GROUP/.serial
  STOP=`grep " 503 " /tmp/follow | tail -n1 | awk '{print $2}'`
  if [ ! "$STOP" = "" ]; then   # 503 transfer limit, closing
connection.
    pkill snget
    pkill snfetch
    pkill Serial.sh
  else
    sleep 15
  fi
done
# Done.

In order to make the above script work, snget is invoked with

snget -dd -p 6 -c 4 FAKE.NEWS.GROUP 2>>/tmp/follow 1>>/tmp/follow

NOTE: Although newsguy allows 6 simultaneous connections, sn never uses
more than 1, even with the unpatched code.  Therefore, the "-p 6" has
never worked for me, and no attempt to use more than one connection has
been made by my patch.  That parameter is there just to remind me of
what is possible.

WARNING #1:  My patch destroys pipelining, so "-c 4" does nothing.  I
don't care because now I'm no longer pissing away cca 7 hours per
session on a useless XHDR.

WARNING #1:  My patch trashes .serial, leaving it blank when a session
finishes.  Record the correct article number in .serial manually when
the session terminates.  Relying on Serial.sh will cause at least some
duplicated articles, even if only one.

Perhaps someone who is C capable will stumble across this and correct
all my mistakes, then make the fixed code available to us all...  I'll
be dead by then.
--
gypsy
--------------59020DDA6BC05388FF973068
Content-Type: text/plain; charset=us-ascii;
 name="PatchToSnfetch.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="PatchToSnfetch.diff"

--- snfetch.c.unpatched	2004-04-28 11:00:25.000000000 -0700
+++ snfetch.c.patched	2012-02-09 13:01:56.000000000 -0800
@@ -314,6 +314,64 @@
    return last;
 }
 
+/* This replaces fetch */
+int suck (int from, int to)
+{
+   int n, p, collected, last;
+
+   last = collected = 0;
+/* comment begin: pipelining needs to be fixed
+   p = pipelining;
+   if (p > nr_arts)
+      p = nr_arts;
+   for (n = from; n < nr_arts + p; n++)
+comment end */
+   for (n = from; n < to; n++)
+   {
+/* comment begin
+      if (n < nr_arts)
+comment end */
+      if (n < to)
+      {
+/* comment begin: bypassing XHDR makes this is empty
+         struct data d;
+         d.messageid = arts[n].id;
+         if (0 == dh_find(&d, FALSE))
+         {
+            nrhave++;
+            n = -1;
+         }
+         else
+comment end */
+            args_write(7, "ARTICLE %d\r\n", n);
+      }
+      if (n >= p && n - p > -1)
+      {
+         char *end;
+
+         if (doread() < 3)
+            badresponse("ARTICLE");
+         switch (strtoul(args[0], &end, 10))
+         {
+            default:
+               badresponse("ARTICLE");
+            case 220:	/* TODO: Must update .serial for every 220 else restart will DL _everything_ again! */
+               last = strtoul(args[1], &end, 10);
+               readprint();
+               collected++;
+               break;
+            case 430:
+            case 423:
+               if (*end)
+                  fail(3, "fetch:Bad response to ARTICLE, got \"%s\"", args_inbuf);
+         }
+      }
+   }
+   LOG("Fix .serial manually!");	/* need to fix .serial */
+   return last;
+}
+/* end of suck() */
+
 static int cat (char *fn, int *val)
 {
    char buf[64];
@@ -452,6 +510,7 @@
    to = strtoul(args[3], &cp, 10);
    if (to < 0 || *cp)
       badresponse("GROUP");
+   nr_arts = strtoul(args[1], &cp, 10);	/* is this necessary? */
 
    if (to <= from)
       fail(0, "Empty newsgroup:%s", args_inbuf);
@@ -465,19 +524,26 @@
       LOG("out of sync, %d not between %d-%d", serial, last = from, to);
    else
       from = serial; /* normally taken */
-
+/* comment begin: Don't alter from and don't send XHDR
    if (max > 0)
       if (to - from > max)
-         from = to - max - 1; /* -1 added to make the number of arts downloaded == max. */
+         from = to - max - 1;
+ -1 added to make the number of arts downloaded == max.
 
    nrhave = nrdup = 0;
+
    sendxhdr(from, to);
 
    if (!nr_arts || (last = fetch(&nr)) <= 0)
       fail(0, "Nothing to fetch");
+comment end */
+/* Skip XHDR and just suck the articles beginning with .serial's value */
+   nrhave = nrdup = 0;
+   last = suck(from, to);
 
    dh_close();
-   LOG("%d articles (%d bytes) in %d seconds", nr, bytesin, time(NULL) - t);
+   LOG("%d articles %d B in %d S %d Bps", nr, bytesin, time(NULL) - t, bytesin/(time(NULL) -t) ); /* Record Bps */
+/* end of changes */
    LOG1("new/dup/present = %d/%d/%d", nr, nrdup, nrhave);
 
    if (-1 == (fd = open(".serial.tmp", O_WRONLY | O_CREAT, 0644)))

--------------59020DDA6BC05388FF973068--