RE: dnscache/dnsq timeouts on RCODE=5(refused)

(Reprising a 7-year old thread!)

Jeff King helped me out with a problem (thanks Jeff!) and he suggest
I write it up and drop it into the djbdns mailing list, so here it is.
Hopefully, I haven't duplicated somebody else's effort.

Although the patch seems to empirically solve my problems, I don't know
whether there are unwanted side-effects.  Comments are welcome.

--------------------------------------------------------------------------------

Problem:

 	My users have complained about slow connection and timeouts to
 	some of our network services.  The server seems to be hung up
 	waiting for DNS IP->hostname lookups; the user's ISP had
 	delegated their .in-addr.arpa zones to content servers that
 	refuses that query (rcode=5).

 	dnscache's behaviour in this regard is to query each authoritative
 	name server using its full retry scheme.  It discards rcode=5
 	replies and retries in hope it will receive an answer or NXDOMAIN.

 	The resolver API (libresolv or djbtool) may exacerbate the problem
 	by doing its own retry when it gets the SERVFAIL response.  The total
 	wait time is also proportional the number of resolving cache servers
 	specified in /etc/resolv.conf or environment variable DNSCACHEIP.

 	This wait can get quite long, and the server is keeping the user
 	in limbo, sometimes resulting service failure from timeouts.

 	A technical writeup of this situation is summarized by Jeff King:

 		http://marc.info/?l=djbdns&m=103943508202597&w=2

 	Complaints to the ISP got me nowhere.

Possible solution:

 	It seems unreasonable to requery a NS server that has actively
 	refused a query.  A seemingly better strategy for a RCODE=5
 	response is to try the next NS server rather than beat our head
 	against the same NS server that just told us to bugger off.

 	Jeff King kindly suggested a method by which I could take a NS
 	server out of the query list so that it wouldn't be retried.
 	It resulted in the patch to dns_transmit.c below.

Testing and results

 	(Tested on Solaris10)

 	Without the patch, hostname lookups for an IP PTR delegated to
 	a content server (mis)configured to deny queries will result
 	in wait times of

 		gethostbyaddr(): ~40s per resolving cache
 		dnsname: ~58s per resolving cache

 	The same query using the patched dns_transmit.c:

 		gethostbyaddr(): <1s per resolving cache
 		unpatched dnsname, patched dnscache: ~11s per resolving cache
 		patched dnsname+dnscache: <1s per resolving cache

 	The output remained consistent (i.e. temporary failure from
 	SERVFAIL).  dnsq, dnsfilter also seems to have similar reductions
 	in response time.

 	I also tested the scenario where the query was delegated to 2 NS
 	servers: one faulty and one responsive.  It worked as expected:
 	the faulty NS was skipped and the correct answer fetched from
 	the working NS.

 	I have been running the patched dnscache in a production
 	environment (~130K queries/day) for a week without problems.
 	Users with addresses allocated in their ISP's troublesome
 	netblocks report they no longer have service connection hangs.

 	Getting RCODE=5 responses is rare (checked by resolving our web
 	logs and back-checking the timed out IPs).  It's much more common
 	to have slow lookups caused by a RCODE=2 SERVFAIL response.
 	The same actions for RCODE=5 responses could also be employed
 	for RCODE=2.  Doing this seems to makes dnscache empirically
 	behave like BIND named.

Patch for djbdns-1.05:

--- dns_transmit.c~	Thu Dec  3 02:56:51 2009
+++ dns_transmit.c	Mon Dec  7 19:07:43 2009
@@ -19,3 +19,3 @@

-static int serverfailed(const char *buf,unsigned int len)
+static int serverfailed(const char *buf,unsigned int len,char *server)
  {
@@ -25,6 +25,7 @@
    if (!dns_packet_copy(buf,len,0,out,12)) return 1;
-  rcode = out[3];
-  rcode &= 15;
-  if (rcode && (rcode != 3)) { errno = error_again; return 1; }
-  return 0;
+  rcode = out[3] & 15;
+  if ((rcode == 0) || (rcode == 3)) return 0;
+  if (rcode == 5) byte_zero(server,4);
+  errno = error_again;
+  return 1;
  }
@@ -271,3 +272,3 @@
      if (serverwantstcp(udpbuf,r)) return firsttcp(d);
-    if (serverfailed(udpbuf,r)) {
+    if (serverfailed(udpbuf,r,(char *)(d->servers + 4*d->curserver))) {
        if (d->udploop == 2) return 0;
@@ -358,3 +359,3 @@
      if (serverwantstcp(d->packet,d->packetlen)) return nexttcp(d);
-    if (serverfailed(d->packet,d->packetlen)) return nexttcp(d);
+    if (serverfailed(d->packet,d->packetlen,(char *)(d->servers + 4*d->curserver))) return nexttcp(d);


Joseph Tam <[email protected]>