Re: dar became very slow (and process in D state?)

Il giorno gio, 09/05/2019 alle 11.23 +0200, Denis Corbin ha scritto:
> 
> [...]
> 
> Hello Andrea,

Hi Neil, Denis and all,
  first of all, thank you for your detailed explanations help.

> 
> > 
> > Hi Neil, thank you very much for this further explanation. I use
> > Fedora, so yes, indeed, it's reasonable that I had installed an
> > updated kernel across the time the problem rose. I will try to use
> > a previous kernel and let you know here.
> > 
> > By the way, just to add a bit more speculation: if it were a
> > kernel related problem, then I'd also have problems with a mere
> > file copy, wouldn't I? On the other hand, if it were a usb related
> > problem, then I wouldn't have problems with SATA to SATA backup.
> > (Where) am I wrong if I assume that those two things leads to
> > suspect a problem (at least) in dar itself?
> 
> All is possible, but to solve a problem we must first follow the
> path
> leading to most probable cause, eliminating them one by one from the
> most to the least probable one, up to the time we find an
> explanation
> and a coherent behavior with that explanation so we can fix and be
> sure the problem is fixed.
> 
> Here dar cannot by itslef decide to stay in uninterrupted sleep:
> this
> is the kernel decision to put processes in that state when it is
> treating the request he received from that process - network, disk,
> whatever.  Second you did not mentioned having changed of dar
> version
> since the last backup while all was working good, so why should this
> behavior be caused by dar while it was working fine in the past? ...
> this seems less probable than an I/O problem as suggested by Neil.
> 
> Now there is something weird I agree, dar is put in the D state when
> it is releasing memory (at that time all disk I/O have completed and
> even if data is still in the cache, pending from writing to disk,
> dar
> is not concerned by that). This points rather toward a memory (RAM)
> problem or a memory management problem, or a swap problem like
> corruption of the swap file/partition or a disk problem where the
> swap
> is stored...
> 
> Usually RAM problem does corrupt data, the kernel returns
> immediately
> but the wrong data. You can still check your RAM using memtest86+
> (the
> best way I know for that is relying on system rescueCD which has a
> boot mode to do that easily - http://www.system-rescue-cd.org/ )

I haven't check the RAM yet, but going to do it in the next night. (I
think it's the only test I still have to do, see below).

> 
> Remains the swap which is to me the most probable cause. I would do
> the following:
> 
> First disabling the swap:
> # swapoff -a
> 
> grab the UUID of the swap partition(s) using /sbin/blkid as root
> program which is available in the util-linux package (if using
> Debian
> or child distro):
> 
> # blkid
> 
> then recreate the swap partition preserving its original UUID (to
> avoid having to modify /etc/fstab with a new UUID)
> 
> # mkswap -c -U <UUID> /dev/sdXY
> 
> where <UUID> is the UUID reported by blkid for your /dev/sdXY swap
> partition
> 
> Then reactive the swap partition(s)
> 
> # swapon -a

I re-created the swap, and the problem still happens.
I also tried to connect the pendrive directly to a USB port connected
directly to the mainboard, and the problem still happens.

Then I installed and tested many kernels and found that problem
doesn't happen with kernel 4.20, and happens with kernel 5.01 (the
next one, as I can understand) and newer (tested 5.0.7, 5.0.9, 5.0.10,
5.0.13, 5.0.14, 5.0.16).

With the "faulty" kernels, dar takes roughly 10-20 times to complete
(I usually made the tests with a 1.1 GB only file to backup, but also
with other backup size), say 10-20 minutes instead of 1-2 minutes.

When process is in D state, I see (iotop) some increased I/O
operation, which stops when the process terminates.

So, I think we can say it's a kernel problem, and I would report a bug
or something to the correct bug tracker/mailing list (which one is the
correct one?), but I'd like to have an advice from you about it (and,
of course, your corrections if I am wrong in some conclusions).

As a side note, I also tried to copy a file to the pendrive and the
behaviour is similar: we have a factor of 5 between copying a 1.1 GB
file from the internal SATA HD to the pendrive, using the working
kernel vs. the faulty one. An odd thing (maybe perfectly normal, but
I've not enough skills to understand it) is that the time needed to
copy a file is sometimes substantially different (15 seconds instead
of 1 minute...)(I usually did some 5-10 identical tries for each
test). I thought it could be related to some low level caching, but
curiously it happens on the first copy and not on the next ones. Btw,
I have also discovered that this seems to happen using the faulty
kernel only.

> 
> 
> Anyway, whatever is asked to the kernel (by mean of system call) if
> it
> is "bad", it should fail immediately, dar would report an error, or
> would be crashed by the system (core dump), or would do ugly things
> looping and consuming a lot of CPU in vain, ... but it would not be
> waiting for the kernel to answer the request it made (which is such
> D
> state)
so, if I understand correctly, the D state is not compatible with
kernel problems, but I'm not sure to have understood :-)

Thanks, and bye,
Andrea