Re: [Imap-protocol] Best way to support HTTP PUT based 'push' notifications from IMAP?

On 03/09/2013 11:53, Andrew Sutherland wrote:
> On devices that either have difficulty maintaining persistent TCP 
> connections or have power concerns about doing so, maintaining an IMAP 
> connection for IDLE/NOTIFY purposes is undesirable and it would be 
> great if the IMAP server could generate a notification via other means.
>
> For the Firefox OS e-mail client, one of the Web APIs the Firefox OS 
> platform makes available and that Mozilla and other contributors are 
> trying to standardize is the "Push API" 
> (http://www.w3.org/TR/push-api/) under the auspices of the webapps WG 
> (http://www.w3.org/2008/webapps/). The API is somewhat decoupled from 
> the actual notification mechanism.  Mozilla's reference server 
> protocol is documented at 
> https://wiki.mozilla.org/WebAPI/SimplePush/Protocol.
>
> From the IMAP server's perspective, the process is that the client 
> uses the Push API to get a URL and then the client tells the IMAP 
> server the URL.  To notify the client, the IMAP server issues a PUT 
> request to the URL.  The only data payload is the 'version' which is a 
> monotonically increasing 64-bit integer and allows for a degree of 
> idempotency.  The only key thing for the version is that if the server 
> doesn't increase the value since the last time the client heard it, 
> the client won't receive a new notification.
>
> *Our application-level goal is that rather than checking every N 
> minutes, after each synchronization attempt we refresh our 
> notification request before disconnecting from the IMAP server. The 
> IMAP server would then send a ping notification when new messages are 
> received for the account/inbox/whatever. *
>
> Details/thoughts from my investigation are found below.  In general, 
> I'm very interested in what server implementers think and would be 
> willing to adopt, especially for things that might be 
> enabled-by-default in deployments.  It's been indicated to me that we 
> (Mozilla) have engineering resources that could be used to help 
> provide patches to open source implementation to help make this happen.
>
> Note that any solution that satisfies our needs could probably also 
> satisfy similar needs.  For example, the Firefox OS Calendar App makes 
> use of the Google Calendar API's push notifications mechanism as 
> documented at https://developers.google.com/google-apps/calendar/v3/push
>
>
> # Existing Work
>
> ## XAPPLEPUSHSERVICE
>
> The only thing I've seen in the wild seems to be XAPPLEPUSHSERVICE 
> advertised by Yahoo's IMAP server and Apple's dovecot fork.  The most 
> useful information I've found on this is 
> http://www.dovecot.org/list/dovecot/2012-August/067682.html which 
> leads us to the implementation of XAPPLEPUSHSERVICE at 
> http://www.opensource.apple.com/source/dovecot/dovecot-239.8/dovecot/src/imap/cmd-x-apple-push-service.c 
> which tells us the syntax of the command.  While the idea is similar, 
> XAPPLEPUSHSERVICE does not seem to support alternate endpoints to 
> Apple's Push Notification Service and so is not useful for our purposes.
>
> ## sieve/enotify
>
> The sieve enotify extension (http://tools.ietf.org/html/rfc5435) seems 
> like the only standadized mechanism capable of implementing such 
> logic.  Unfortunately, there is no HTTP/URL notification listed in the 
> registry at 
> https://www.iana.org/assignments/sieve-notification/sieve-notification.xhtml. 
> 'mailto' is a defined mechanism that has some support (ex: Pigeonhole 
> on Dovecot), and it's possible for Mozilla to stand-up a proxy to help 
> bridge this gap.

mailto: can be sufficient for some gatewaying cases, but I suggest you 
define a new "http" notification mechanism. I would be happy to help you 
out with that.

> The main problem with sieve is that adoption does not seem 
> particularly wide-spread and discoverability when supported is 
> limited, although I'm hoping that http://tools.ietf.org/html/rfc6785 
> could help improve that.
If you only care about notifications generated due to new mail 
deliveries, than you can use ManageSieve: 
<http://www.rfc-editor.org/rfc/rfc5804.txt>. This is quite common when 
Sieve is supported on delivery.

> # Potential Workarounds
>
> I think the following options exist to make this work today:
>
> ## sieve mailto bridge
>
> Stand up an SMTP-receiving proxy so that sieve/enotify/mailto can be 
> used to trigger instead of HTTP PUT.  Use sieve to set up a rule to 
> generate mail notifications every time a message is received.  No 
> message contents would be sent.  This might result in a large number 
> of notifications since it's not obvious to me how to write a stateful 
> sieve script.

Metadata can be used, as you suggested below.

> ## Hosted IMAP Proxy (Undesirable due to inherent credential/data 
> exposure)
>
> A proposal from those working on the back-end of Mozilla's push 
> notification service is an HTTPS-accessible proxy.  Instead of having 
> the device maintain the long-lived IMAP connection, the device has a 
> server do that on its behalf.  The very undesirable aspect of such a 
> solution is that while the proxy need not persist the credentials to 
> disk, it does need to know them to establish the connection using 
> traditional LOGIN.  And even if a challenge-response authentication 
> mechanism was used in such a way that the proxy server could not 
> establish additional connections without device support, the proxy (or 
> whatever pretends to be the proxy) would still have a fully authorized 
> IMAP connection to use as it chooses until the connection is dropped.
>
>
> # Potential Standards Approaches
>
> ## Create a new capability, ex: PUSH-NOTIFY-HTTP=PUT
>
> Create a "PUSH-NOTIFY-HTTP PUT <URL> <BODY PAYLOAD>" capability with 
> very simple semantics:
Did you mean an IMAP capability?
> - Each request is one-shot.  The request lives until it is fired.
> - At once most one request per URL; new requests replace the old request.
> - An HTTP PUT is triggered to the given URL with content-type 
> application/x-www-form-urlencoded with the provided body payload.
> - Body payload is limited to something really small; we only need to 
> be able to encode "version=<64bitnumber>"
> - Delivery to the Inbox or maybe just delivery to the account in 
> general in case a filter files the message somewhere.  Messages filed 
> to the junk folder would not trigger the notification.
>
> ## sieve enotify HTTP PUT extension

This would be my preferred method.

> Since it's not likely for adoption to happen immediately anyways and 
> sieve is extremely powerful it seems like a reasonable choice to avoid 
> introducing a new feature which is doomed to feature creep.
>
> Because sieve is not particularly stateful the main issue would be how 
> to achieve single-shot semantics.  This might be best addressed by 
> doing something like the servermetadata example at 
> http://tools.ietf.org/html/rfc5490#section-4.1 where the client would 
> use metadata to control the process and the URL notification function 
> would have built-in idempotent semantics.
A particular notification mechanism can support duplicate suppression, 
although I am not sure that that is what you are asking for.

Metadata works otherwise though.
> Each time the client connected it would refresh the 'verson' it uses 
> and which is used in the notification invocation.  Then when the sieve 
> rule next fires, the URL/payload changes so that the notify action 
> actually goes through.
>
> Multiple consumers of such notifications would be handled during 
> rewrites of the sieve rules.  A new device setting up for push 
> notifications would identify its rule based on the URL the push API 
> tells it.  It would write its rule to the sieve store along with the 
> current timestamp for the rule.  It would also cull any other rules of 
> the same variety whose timestamp more than twice as old as the 
> rule-rewrite-interval based on the timestamp.
>
> ## custom sieve extension
>
> Realize the above state management logic is too crazy. Internalize the 
> one-shot state management, still using metadata somehow.

It is better to extend enotify, unless you can see some severe 
limitations in the enotify framework.

> ## Extend NOTIFY somehow
>
> This seems like it would be a mismatch since the NOTIFY subscription 
> is inherently associated with the connection lifetime and any other 
> lifetime would be weird.

Agreed.

> ## Define CONTEXT=pushnotify
>
> Similar to NOTIFY, this seems like a mismatch.
>
>
> Thanks for any/all responses,
> Andrew Sutherland

Best Regards,
Alexey

_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol