Re: File not recognized but another file in same folder is recognized

omitheaussie <[email protected]> writes:

> I am running Ubuntu 20.04 on a machine. There is a rc.local service that 

It looks like you are using the most recent unison release.  That's
good; I am only willing to debug that or the tip of git.  It was built
with ancient ocaml, but that is unlikely to be the problem.

> runs as follows:
>
> #!/bin/bash
> sudo -u ubuntu s3fs nameofs3bucket /home/ubuntu/folder1 -0 umask=000 -o 
> allow_others url=<url.com>
> sudo chown -R ubuntu:ubuntu /home/ubuntu/.unison
> /usr/bin/unison -root /home/ubuntu/folder1 -root /home/ubuntu/folder2 
> -batch -auto -prefer newer -repeat=watch
>
> my permissions for the /usr/bin/ applications involved are rwxrwxrwx (the 
> applications are unison, unison-fsmonitor, s3fs).

As long as they can be run (meaning group x bit if you are in the group,
or the user x bit), it should work.  However it is irregular to have
files in /usr/bin be writable.  The normal permissions are 555.

The first thing you should do is to simplify the setup for testing.
There are multiple aspects to this.

Your script is mangled by bad line wrapping, but it looks ilke you are
running unison as a different uid.

You are using repeat.  This adds the watcher, and the advice in the
debugging wiki page is to simplify because the point in debugging is to
find a minimal example, or a working and a broken setup that are close
to each other.  So run unison from the command line, and
  stop using batch
  stop using prefer newer
  stop using repeat (most importantly)

  and hence leaving only auto and the roots

and see what happens.  If that doesn't do what you expect, let's start
from there.

(It's up to you, but I would use more meaningful names than
folder1/folder2.  That makes it harder to follow.  My read of the
situation is that folder1 is s3 and folder2 is local.)

I am having a hard time following, but it looks like you are running
s3fs as user "ubuntu".  And unison as root.

> When I upload a file to s3 bucket the s3fs running in the background is 
> able to mount the file to the folder1 without any problem with public 
> permissions (rwxrwxrwx).

I am unclear on how s3fs works, and if s3 has the concept of file owners
and modes; it seems not given that you are reporting a file with mode
777.

It is possible that some system calls, when invoked on files in s3fs, do
not do what one normally expects.  It would not shock me if one returned
something unusual and this caused some processing to decide to skip a
file.  If so this might or might not be a bug in unison; it might be a
bug in s3fs.  Or it might be much more complicated.


You should be aware that unison expects proper filesystem semantics and
by default tries to sync permissions.  This often does not work well
with filesystems that lack normal filesystem features, like FAT and
probably s3.

> I uploaded 3 files to s3 bucket (temp1.txt, temp2.txt, temp3.txt) and I can 
> see all their permissions when s3fs mounts them to folder1 are rwxrwxrwx 
> (due to the umask=000 option).

You did not post the output of "ls -l".  It is a really interesting
question if one of them is different.

> But in the log for unison I only see two of 
> the 3 files and in the folder2 only 2 files get copied.
>
> [BGN] Copying temp1.txt from /home/ubuntu/folder1 to /home/ubuntu/folder2
> [END] Copying temp1.txt
> [BGN] Copying temp3.txt from /home/ubuntu/folder1 to /home/ubuntu/folder2
> [END] Copying temp3.txt
> Unison 2.53.3 (ocaml 4.08.1) finished propagating changes at 16:17:21.22 on 
> 25 Aug 2023, 85.803 s
>
> I am not sure why unison is missing syncing the third file?

I'm not either, but the debugging instructions tell you to turn on debug
flags.  Also not unison specific but you should use ktrace/ktruss or
whatever it is called on ubuntu to record system calls.

Because you have not simplified you don't know if the problem is that
when unison runs normally it cannot ee temp2, or that it misproceses it,
or that the watcher does not see it as changed, or that the kernel
mechanism that the watcher uses does not report it.

> Also, if I dont set the option for the permissions for the synced files in 
> folder1 look like -rw-------.

That is not necessarily wrong.  Arguably it's right as it limits access
to the s3 files to the user that authenticated to s3.

> In this case unison doesnt sync any files to folder2. I also tried
> running unison and s3fs both as root in rc.local, but that didnt work
> either.

You have not explained the point of what you are doing, but it seems
quiet likely that doing everything as root is inadvisable.


You also might try a different number of files and see if it's every
other file, or not the second.  But first you should get to be able to
run unison without batch/repeat.

-- 
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].