Re: Question regarding multi-threading

On Tue, 18 Apr 2023 at 20:00, J Snukis <[email protected]> wrote:
>
> Upgrading the software will have to be a long term plan, unfortunately these servers are in use now.

Which OS/platform is this?

> I noticed that while we have 10 threads specified in the prf it seems to be running single threaded. Is the value a single number?

Yes, it is a single number. The name of this preference is misleading.
There will always be a single OS thread. This number is actually the
maximum parallel transfers that will be started. I'm not sure if this
will work with 2.48.4 but if you increase this number to be larger
than the number of huge files then even if all huge files are in the
transfer queue, smaller files could still continue to be synced at the
same time.

Is there a 'stream' preference in the profile?

> Also what would be the way to specify large files only? I see the option for maxsize but not for minimum.

There is no way to do this by file size, I think. For your situation,
as a workaround at least, you can do it by path or by ignoring/not
ignoring files/sub-paths.
For example, multiple profiles can specify the same root values. But
each profile can specify different 'path' preferences. So different
profiles can separately sync different non-overlapping sub-paths
within the roots.
Alternatively, it is possible to 'ignore' something in one profile and
'ignorenot' it in the other profile. For example, 'ignore = Name
*.img' in one profile would ignore all *.img files and sync all the
rest. In another profile, 'ignore = Name *' and 'ignorenot = Name
*.img' would ignore everything except *.img files. Instead of 'Name',
you can also use 'Path' and 'Regex' (see the manual).

This may work for you but it will not make the sync go faster, of
course (I'm assuming it is network-bound or disk-bound). In any case,
this will be a workaround. If increasing maxthreads works, that's
better. I'm not sure what may have caused the maxthreads value to be
so low to begin with (default is 1000, unless 'stream' preference is
turned off).

If you can experiment, you may also try the 'sortbysize' preference,
which just syncs smaller files before larger files. See also
'sortfirst' and 'sortlast' preferences in the manual.

If possible, you may describe the dir tree under the sync root. You
don't have to use real names if they're sensitive in any way. It will
suffice with something along the lines of:
sync root has two subdirs
subdir1 has few huge files
subdir2 has thousands of small files.
OR
sync root has hundreds of subdirs
each subdir has hundreds of files of varying sizes.