Re: WebDAV folders internally have hundreds of copies of the same few files

 > Enable the imapmagicplus option in imapd.conf

With this option I could login with imtest specifying "+DAV" for the 
user, both as the admin and as a regular user, but then when trying to 
select a folder, I got "NO Permission denied" for the admin and "NO 
Mailbox does not exist" for a common user. This looks like undocumented 
magic and mbexamine actually provides all needed information, so I will 
get file details with it.

Thanks for you help, Ken.

*From:* Ken Murchison
*Sent:* Friday, November 16, 2018 13:17
*To:* Info-cyrus
*Subject:* Re: WebDAV folders internally have hundreds of copies of the 
same few files


On 11/16/18 1:19 AM, Anatoli wrote:
> Ken,
>
> > If you login as an admin, you should be able to SELECT the mailbox 
> and use normal IMAP commands.  Alternatively, if you add "+DAV" to a 
> regular userid, this will also allow SELECTing of DAV collection 
> mailboxes.
>
> When I do imtest -a mailadmin and then issue a002 select 
> #drive/folder, I get:
>
> a002 NO Invalid mailbox type


It appears that just logging in as an admin won't do the trick.



>
> When I do imtest -a mailadmin+DAV, I get (XX replaces the real auth 
> value):
>
> C: A01 AUTHENTICATE PLAIN XXXXXXXXXXXXXXXXXXX
> S: A01 NO authentication failure
> Authentication failed. generic failure
>
> I've also tried these combinations with the same result: 
> "[email protected]+DAV" and "[email protected]". Am I doing it wrong?


Enable the imapmagicplus option in imapd.conf



>
> Independently of this, is there a way to obtain the details about the 
> flags for each message in a folder directly from the db files? My idea 
> is to rsync the Cyrus store where the WebDAV is located to some other 
> folder (probably on another server), purge there the files marked as 
> deleted and backup the remaining files, all this without altering the 
> Cyrus store. Is it possible?
>
>
> > If the append of the resource into the mailbox, or updating the DAV 
> db entry fails, the operation should be reverted, with partial saving 
> done.  Which version of Cyrus are you using?
>
> Currently it is 3.0.5, but I can upgrade if there were any related 
> changes in the later versions. What I was observing is that if a PC 
> (LibreOffice Calc on Ubuntu 16.04, the folder mounted with davfs2) 
> crashes in the middle of saving a file on the server, the result was a 
> damaged file: some 400 bytes instead of 3-4Mb of a real file.
>
> I was thinking that maybe when the TCP stream hangs in the middle, 
> Cyrus would interpret it as an end of data and write a file, but I've 
> just tried to make some tests killing mount.davfs2 process and/or 
> blocking the stream with iptables in the middle of file transmission 
> and Cyrus responded correctly. For killing mount.davfs2 it showed in 
> the corresponding log file the bytestream followed by:
>
> HTTP/1.1 400 Bad Request
>
> Unable to read body data
>
> And for blocking the stream with iptables it was showing the 
> bytestream ending where the transmission stopped, without the 400 Bad 
> Request, but in both cases the original file was not modified by the 
> partial upload. So it appears the problem is elsewhere.
>
>
> Regards,
> Anatoli
>
> *From:* Ken Murchison
> *Sent:* Thursday, November 15, 2018 14:09
> *To:* Info-cyrus
> *Subject:* Re: WebDAV folders internally have hundreds of copies of 
> the same few files
>
>
> On 11/15/18 1:48 AM, Anatoli wrote:
>> Hi Ken,
>>
>> Thanks a lot for the clarification, everything makes sense now. How 
>> can I list the files marked for deletion and those that are not 
>> marked? unexpunge can provide the list of files marked for deletion, 
>> is there any better way to list them, directly reading the DB? How to 
>> list those that are not marked?
>
>
> If you login as an admin, you should be able to SELECT the mailbox and 
> use normal IMAP commands.  Alternatively, if you add "+DAV" to a 
> regular userid, this will also allow SELECTing of DAV collection 
> mailboxes.
>
>
>> One more question, related: we got a client who's PC was crashing 
>> exactly during the modify operation (some issue with the PC hardware 
>> triggered by Excel save operation, probably a RAM spike touching some 
>> bad blocks). As a result, the file in Cyrus was becoming damaged, 
>> i.e. partially saved. Is it expected?
>>
>> Shouldn't Cyrus update the db with the pointer to the new file (a new 
>> message in the store) only if the operation completes successfully 
>> (e.g. the WebDAV messages exchange completes and the connection is 
>> closed at the right time or something similar)?
>
>
> If the append of the resource into the mailbox, or updating the DAV db 
> entry fails, the operation should be reverted, with partial saving 
> done.  Which version of Cyrus are you using?
>
>
>
>> *From:* Ken Murchison
>> *Sent:* Wednesday, November 14, 2018 10:54
>> *To:* Info-cyrus
>> *Subject:* Re: WebDAV folders internally have hundreds of copies of 
>> the same few files
>>
>>
>> On 11/13/18 10:15 PM, Anatoli wrote:
>>> Hi,
>>>
>>> I'm not sure this is due to some configuration option, bug or 
>>> feature, but I'm observing some folders on Cyrus HTTP WebDAV server 
>>> having hundreds (995 at this moment to be precise) internal files in 
>>> the format "NNN." that correspond to the same file but different 
>>> versions in time.
>>>
>>> There are 2-3 files (xls) in the folder that are edited constantly 
>>> during the day and it looks like each save operation creates a new 
>>> file. The files are of some 3-5Mb each. In the explorer/web view 
>>> there are only a couple of files with a total size of 17.5Mb, but 
>>> the reported disk usage for the folder is of 1.6Gb.
>>>
>>> Could someone please shed some light on what's going on and how to 
>>> make each file visible to the users to be stored in only one 
>>> internal file?
>>>
>>> Thanks,
>>> Anatoli
>>
>>
>> Because *DAV is layered on top of an IMAP store, we have to abide by 
>> IMAP semantics in which messages (in this case DAV resources) are 
>> immutable.  Therefore, we can NOT overwrite an existing message in 
>> the mailbox.  Each change MUST result in a new message.  However, the 
>> server does mark the previous version(s) as deleted and expunged, 
>> which means that they will eventually be removed by cyr_expire.  If 
>> you aren't running cyr_expire, you should consider adding an event to 
>> cyrus.conf to remove expunged messages (see -X option).
>>
>> -- 
>> Ken Murchison
>> Cyrus Development Team
>> FastMail US LLC
>>
>> ----
>> Cyrus Home Page:http://www.cyrusimap.org/
>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>>
>> ----
>> Cyrus Home Page:http://www.cyrusimap.org/
>> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> -- 
> Ken Murchison
> Cyrus Development Team
> FastMail US LLC
>
> ----
> Cyrus Home Page:http://www.cyrusimap.org/
> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>
> ----
> Cyrus Home Page:http://www.cyrusimap.org/
> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC


----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus