Re: MicroVAX 3100/20 is apocalyptically slow even with 32MB of RAM

A discussion on discord resulted in a lot of testing and experimentation =
over the past several days, and this is the present status of the =
investigation into losing wall clock time / general performance =
degradation of NetBSD/vax.

Previously, it was observed that NetBSD 10.1 was extremely slow on a =
3100/20 with 32MB of RAM and a fast (physical) SCSI disk. Further =
testing showed this behavior not apparent in NetBSD 4.0.1 but present in =
NetBSD 9.1, with the releases in between not being tested because the =
GENERIC kernels for them would not boot on my 3100.

Since the issue involves losing wall clock time, and thus presumably =
clock interrupts, it is assumed that in-kernel profiling will not =
produce valid results since the timing information will be skewed by the =
issue.

Since then, it has been determined that the issue isn=E2=80=99t strictly =
SCSI related as originally suspected. Under NetBSD 9.1, equal-size =
transfers made using dd from a file located on an NFS mount to /dev/null =
and from /dev/rsd0c to /dev/null lost approximately the same amount of =
wall clock time. A transfer from /dev/zero to /dev/null of the same size =
as the previous two tests lost approximately half as much time.

A second 3100 (having only 16MB of RAM) was similarly tested and showed =
the same results, so this is unlikely to be a hardware issue.

The first 3100 (the one with 32MB of RAM) was reinstalled with NetBSD =
4.0.1 and the tests were repeated there. No appreciable loss of wall =
clock time was observed. Notably, the percentage of time spent in system =
during the dd from /dev/zero to /dev/null was much higher than that =
observed under NetBSD 9.1 - staying between 85% and 92% on 4.0.1 - =
leading us to believe that simply being in system state did not cause =
the issue.

Thus it was concluded that 4.0.1 did not exhibit the behavior, but then =
when I made a CVS checkout of the NetBSD 4 source tree on the NFS host =
(to save myself having to ssh from the VAX) and mv=E2=80=99d it to the =
usr filesystem, 70 seconds of time was lost moving the src directory and =
30 seconds of time was lost moving the xsrc directory. Clearly some part =
of the issue is still there. A dd from /dev/zero to the SCSI disk was =
performed to see if SCSI *writes* were the problem, but it did not lose =
any time.

I started a build of tools after that, and the system has not yet lost a =
significant amount of wall clock time. It lost a little under 2 seconds =
in the first 24 hours.

I plan to finish an installation of 4.0.1 on the smaller VAX and try =
this same transfer of src and xsrc again to verify it exhibits the same =
lossage. If it does, then I want to try packing src / xsrc into a tar =
file and extracting it on the VAX to see if the filesystem operations =
are the issue. If nothing makes sense, then I will start an identical =
build on the smaller 3100 to see if the increased paging activity causes =
any divergence vs the bigger one.