RE: event notifier problem

Hi,

I would have the same conclusion than Ellie regarding the proposal for a custom patch here, we indeed use Cyrus 3.0.x in production and the code mentioned in this issue is present in this version without any issue for long events (length > 1400 bytes for us)

From which version of Cyrus Imap do you come from for encountering this regression ? Or is it a change in your system ?

Also, the socket(7) man page describes how SO_RCVBUF and SO_SNDBUF are defined by default :

SO_SNDBUF
              Sets or gets the maximum socket send buffer in bytes.  The
              kernel doubles this value (to allow space for bookkeeping
              overhead) when it is set using setsockopt(2)<https://man7.org/linux/man-pages/man2/setsockopt.2.html>, and this
              doubled value is returned by getsockopt(2)<https://man7.org/linux/man-pages/man2/getsockopt.2.html>.  The default
              value is set by the /proc/sys/net/core/wmem_default file

SO_RCVBUF

              Sets or gets the maximum socket receive buffer in bytes.

              The kernel doubles this value (to allow space for

              bookkeeping overhead) when it is set using setsockopt(2)<https://man7.org/linux/man-pages/man2/setsockopt.2.html>,

              and this doubled value is returned by getsockopt(2)<https://man7.org/linux/man-pages/man2/getsockopt.2.html>.  The

              default value is set by the

              /proc/sys/net/core/rmem_default file

In our Linux system hosting Cyrus we configure /proc/sys/net/core/rmem_default and /proc/sys/net/core/wmem_default with the same value of 212992

To have an understanding of the reason for dividing by 10 minus 32 there is a good technical response here : https://stackoverflow.com/a/22007520
Indeed, SO_SNDBUF is the size of the buffer for the socket, not the size of a single message send with sendto.

Cdt,
Sébastien

On Thu, 23 Feb 2023, at 8:04 AM, Matteo A. wrote:
In imap/notify.c, data is written to the socket using a buffer sized at 556; in nodityd/notifyd.c the buffer at reading is limited to 480 by the following: bufsiz = MIN(bufsiz / 10 - 32, NOTIFY_MAXSIZE);
(https://github.com/cyrusimap/cyrus-imapd/issues/1674<https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcyrusimap%2Fcyrus-imapd%2Fissues%2F1674&data=05%7C01%7Csebastien.michel%40worldline.com%7C0b913c4bf1a74f3e46de08db15f2daa3%7Cfda9decfe89243ac9d9f1a493f9f98d0%7C0%7C0%7C638127905027293261%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NX7WibeAvZAmGAcJrCxcl8F1IcpC3Okvdk0lI%2BE4fdw%3D&reserved=0>)

Applying a little math, for us seems that the value of SO_RCVBUF read by getsockopt is 5120.
So, after patching the two sources, removing the division by 10, MessageMove events come back to work again.

Ahh! I did see that code and wondered later if maybe it was the culprit...

It seems like there's an alternate code path in there that uses dlist instead (of ??), and it looks like it relies on "notifysocket" in imapd.conf having a value prefixed with "dlist:" -- have a look at notify_dlist() and its caller in imap/notify.c.  It doesn't seem to be limited in length, so it might avoid this truncation.

I don't see it documented anywhere though, just noticed it in the code while I was rummaging around, so I don't know what it's intended for or how to set it up properly.  I imagine your external_event.pl script would need to be updated to parse dlist instead of whatever the usual format is.  There's a Cyrus::DList perl module in perl/imap/Cyrus/DList.pm that will help with that, and it's used by the cassandane tests, so you can look there for working examples of its usage.  That might be preferable to custom patches tinkering with datagram lengths?

If you end up chasing that path, I'd appreciate a writeup that can be incorporated into the docs!  (Or even better, a PR that updates the docs directly.)

Cheers,

ellie

Worldline, Cardlink, GoPay and Santeos are registered trademarks and trade names owned by the Worldline Group. This e-mail and any documents attached are confidential and intended solely for the addressee. It may also be privileged. If you are not the intended recipient of this e-mail, you are not authorized to copy, disclose, use or retain it. Please notify the sender immediately and delete this e-mail (including any attachments) from your systems. As e-mails may be intercepted, amended or lost, they are not secure. Therefore, Worldline's and its subsidiaries' liability cannot be triggered for the message content. Although the Worldline Group endeavors to maintain a virus-free network, we do not warrant that this e-mail is virus-free and do not accept liability for any damages, losses or consequences resulting from any transmitted virus if any. The risks are deemed to be accepted by anyone who communicates with Worldline or its subsidiaries by e-mail.


------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T84dc31747d015b6c-M9daa39dc1d5b922b6064d38a
Delivery options: https://cyrus.topicbox.com/groups/info/subscription