Re: WebDAV folders internally have hundreds of copies of the same few files

 > What are you using as the mailbox name?

#drive/shared/test2


Ken, do you know why cyr_expire -u and -p don't work on shared DAV 
mailboxes? E.g. cyr_expire -X 3 -u "#drive/shared/test2" doesn't clean 
the folder and with -v shows 0 for everything, then unexpunge -l 
"#drive/shared/test2" shows a lot of items with Expg date older that 3 
days, but a general cyr_expire -X 3 (i.e. without the -u or -p) does 
clean the entire store, including the shared DAV folders.

*From:* Ken Murchison
*Sent:* Tuesday, November 20, 2018 10:22
*To:* Info-cyrus
*Subject:* Re: WebDAV folders internally have hundreds of copies of the 
same few files

What are you using as the mailbox name?


On 11/20/18 1:01 AM, Anatoli wrote:
> > Enable the imapmagicplus option in imapd.conf
>
> With this option I could login with imtest specifying "+DAV" for the 
> user, both as the admin and as a regular user, but then when trying to 
> select a folder, I got "NO Permission denied" for the admin and "NO 
> Mailbox does not exist" for a common user. This looks like 
> undocumented magic and mbexamine actually provides all needed 
> information, so I will get file details with it.
>
> Thanks for you help, Ken.
>
> *From:* Ken Murchison
> *Sent:* Friday, November 16, 2018 13:17
> *To:* Info-cyrus
> *Subject:* Re: WebDAV folders internally have hundreds of copies of 
> the same few files
>
>
> On 11/16/18 1:19 AM, Anatoli wrote:
>> Ken,
>>
>> > If you login as an admin, you should be able to SELECT the mailbox 
>> and use normal IMAP commands.  Alternatively, if you add "+DAV" to a 
>> regular userid, this will also allow SELECTing of DAV collection 
>> mailboxes.
>>
>> When I do imtest -a mailadmin and then issue a002 select 
>> #drive/folder, I get:
>>
>> a002 NO Invalid mailbox type
>
>
> It appears that just logging in as an admin won't do the trick.
>
>
>
>>
>> When I do imtest -a mailadmin+DAV, I get (XX replaces the real auth 
>> value):
>>
>> C: A01 AUTHENTICATE PLAIN XXXXXXXXXXXXXXXXXXX
>> S: A01 NO authentication failure
>> Authentication failed. generic failure
>>
>> I've also tried these combinations with the same result: 
>> "[email protected]+DAV" and "[email protected]". Am I doing it wrong?
>
>
> Enable the imapmagicplus option in imapd.conf
>
>
>
>>
>> Independently of this, is there a way to obtain the details about the 
>> flags for each message in a folder directly from the db files? My 
>> idea is to rsync the Cyrus store where the WebDAV is located to some 
>> other folder (probably on another server), purge there the files 
>> marked as deleted and backup the remaining files, all this without 
>> altering the Cyrus store. Is it possible?
>>
>>
>> > If the append of the resource into the mailbox, or updating the DAV 
>> db entry fails, the operation should be reverted, with partial saving 
>> done.  Which version of Cyrus are you using?
>>
>> Currently it is 3.0.5, but I can upgrade if there were any related 
>> changes in the later versions. What I was observing is that if a PC 
>> (LibreOffice Calc on Ubuntu 16.04, the folder mounted with davfs2) 
>> crashes in the middle of saving a file on the server, the result was 
>> a damaged file: some 400 bytes instead of 3-4Mb of a real file.
>>
>> I was thinking that maybe when the TCP stream hangs in the middle, 
>> Cyrus would interpret it as an end of data and write a file, but I've 
>> just tried to make some tests killing mount.davfs2 process and/or 
>> blocking the stream with iptables in the middle of file transmission 
>> and Cyrus responded correctly. For killing mount.davfs2 it showed in 
>> the corresponding log file the bytestream followed by:
>>
>> HTTP/1.1 400 Bad Request
>>
>> Unable to read body data
>>
>> And for blocking the stream with iptables it was showing the 
>> bytestream ending where the transmission stopped, without the 400 Bad 
>> Request, but in both cases the original file was not modified by the 
>> partial upload. So it appears the problem is elsewhere.
>>
>>
>> Regards,
>> Anatoli
>>
>> *From:* Ken Murchison
>> *Sent:* Thursday, November 15, 2018 14:09
>> *To:* Info-cyrus
>> *Subject:* Re: WebDAV folders internally have hundreds of copies of 
>> the same few files
>>
>>
>> On 11/15/18 1:48 AM, Anatoli wrote:
>>> Hi Ken,
>>>
>>> Thanks a lot for the clarification, everything makes sense now. How 
>>> can I list the files marked for deletion and those that are not 
>>> marked? unexpunge can provide the list of files marked for deletion, 
>>> is there any better way to list them, directly reading the DB? How 
>>> to list those that are not marked?
>>
>>
>> If you login as an admin, you should be able to SELECT the mailbox 
>> and use normal IMAP commands.  Alternatively, if you add "+DAV" to a 
>> regular userid, this will also allow SELECTing of DAV collection 
>> mailboxes.
>>
>>
>>> One more question, related: we got a client who's PC was crashing 
>>> exactly during the modify operation (some issue with the PC hardware 
>>> triggered by Excel save operation, probably a RAM spike touching 
>>> some bad blocks). As a result, the file in Cyrus was becoming 
>>> damaged, i.e. partially saved. Is it expected?
>>>
>>> Shouldn't Cyrus update the db with the pointer to the new file (a 
>>> new message in the store) only if the operation completes 
>>> successfully (e.g. the WebDAV messages exchange completes and the 
>>> connection is closed at the right time or something similar)?
>>
>>
>> If the append of the resource into the mailbox, or updating the DAV 
>> db entry fails, the operation should be reverted, with partial saving 
>> done.  Which version of Cyrus are you using?
>>
>>
>>
>>> *From:* Ken Murchison
>>> *Sent:* Wednesday, November 14, 2018 10:54
>>> *To:* Info-cyrus
>>> *Subject:* Re: WebDAV folders internally have hundreds of copies of 
>>> the same few files
>>>
>>>
>>> On 11/13/18 10:15 PM, Anatoli wrote:
>>>> Hi,
>>>>
>>>> I'm not sure this is due to some configuration option, bug or 
>>>> feature, but I'm observing some folders on Cyrus HTTP WebDAV server 
>>>> having hundreds (995 at this moment to be precise) internal files 
>>>> in the format "NNN." that correspond to the same file but different 
>>>> versions in time.
>>>>
>>>> There are 2-3 files (xls) in the folder that are edited constantly 
>>>> during the day and it looks like each save operation creates a new 
>>>> file. The files are of some 3-5Mb each. In the explorer/web view 
>>>> there are only a couple of files with a total size of 17.5Mb, but 
>>>> the reported disk usage for the folder is of 1.6Gb.
>>>>
>>>> Could someone please shed some light on what's going on and how to 
>>>> make each file visible to the users to be stored in only one 
>>>> internal file?
>>>>
>>>> Thanks,
>>>> Anatoli
>>>
>>>
>>> Because *DAV is layered on top of an IMAP store, we have to abide by 
>>> IMAP semantics in which messages (in this case DAV resources) are 
>>> immutable.  Therefore, we can NOT overwrite an existing message in 
>>> the mailbox.  Each change MUST result in a new message.  However, 
>>> the server does mark the previous version(s) as deleted and 
>>> expunged, which means that they will eventually be removed by 
>>> cyr_expire.  If you aren't running cyr_expire, you should consider 
>>> adding an event to cyrus.conf to remove expunged messages (see -X 
>>> option).
>>>
>>> -- 
>>> Ken Murchison
>>> Cyrus Development Team
>>> FastMail US LLC
>>>
>>> ----
>>> Cyrus Home Page:http://www.cyrusimap.org/
>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>
>>>
>>> ----
>>> Cyrus Home Page:http://www.cyrusimap.org/
>>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> -- 
>> Ken Murchison
>> Cyrus Development Team
>> FastMail US LLC
>>
>> ----
>> Cyrus Home Page:http://www.cyrusimap.org/
>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>>
>> ----
>> Cyrus Home Page:http://www.cyrusimap.org/
>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> -- 
> Ken Murchison
> Cyrus Development Team
> FastMail US LLC
>
> ----
> Cyrus Home Page:http://www.cyrusimap.org/
> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>
> ----
> Cyrus Home Page:http://www.cyrusimap.org/
> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC


----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus