Re: limits in directory size

Olaf Hering <[email protected]> writes:

> after renaming a huge, synced directory with 'mv dir/a dir/b', and let
> unsion 2.53.3 mirror the result, the sending side crashed in the
> middle of the operation (in caml_hash).

You left out your OS, your ocaml version, and what "huge" means.  You
left out what the process resource limits are.  And the backtrace, if
any.  You didn't explain what "the operation is", if this is before the
list of changes are printed, or during propagation.  And if this is, as
I would expect following the debugging instructions, using the TUI
version without any kind of watch/notify/repeat.

I realize you are asking a question here, which is the right place, and
thus missing information is ok, but providing it might lead to a more
useful answer.

> Are there any limits how many dentries unison can safely handle? Can
> one increase such limits by increasing the stack size, for example?

I am not aware of intentional limits, but I am also unaware of code that
tries to keep most state in disk and limit RAM use :-) I am also
unlikely to want to merge code that does that, as it seems to add vast
complexity for rare gain, and I am pretty sure nobody is going to write
it anyway :-)

Surely things could be improved, but you're going to need to dig into
ocaml to understand what's going on.

> I have not tried to do the same move on the destination side, nor have
> I tried to tweak the stack, to get things going.

It really depends on what resource is being exhausted, heap vs stack, vs
something els?  And whether this is hitting an OS limit or if there is
some limit in the ocaml runtime.

There is a meta ticket open, basically that stack traces should not be
shown.  But that would change your error, not make it work.

  https://urldefense.com/v3/__https://github.com/bcpierce00/unison/issues/378__;!!IBzWLUs!UU7Qo-wtpHRIFqHz6hivlODoM1LFD0Xio1mkRgFWuPdNDqkNYOyEAfvwvjvsE1AUg-WVqmaHRtzhsqZHrg$ 

There is also the possibility of examining all the data that is stored
and trying to squeeze the space requirements down, but this is likely to
make things e.g. 1.3x better rather than being a fundamental change.

My guess (NB I said guess!) is that the biggest leverage, assuming you
are using a CPU architecture with a 64-bit address space and have many
GB of RAM, is to figure out how to increase resource limits.

A possible workaround is to create the new dir and sync it, and then to
move files in chunks, syncing each time.

You may wish to set up a test directory to experiment with rather than
using it on your real data, which of course already should have multiple
backups, at least one of which is offline, totally disconnected from
unison.

Greg


-- 
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].