[jira] [Closed] (JAMES-3728) ElaticSearch race condition between email moves and flag changes
"Benoit Tellier (Jira)" <[email protected]>
| Newsgroups | gmane.comp.jakarta.james.devel |
|---|---|
| Message-ID | <[email protected]> |
[ https://issues.apache.org/jira/browse/JAMES-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benoit Tellier closed JAMES-3728.
---------------------------------
Resolution: Fixed
I need to work on this problem for a customer for which this data race is critical.
It turns out this roblem can now "easily besolved".
In december 2024 we added the ability to publish a List<Event> in a attempt to group relevant notifications together.
This semantic can be easily reused n order to:
- implement a emailUpdate semantic in MessageIdManager that bundles flag update and move
make it publish all relevant events in a list.
- Wire it down in the JMAP implementation so hat it is used when relevant
- Ensure the coupled JMAP state recording keeps on handling this well
- Finally verify the listeners all respect to the letter the orderin.
> ElaticSearch race condition between email moves and flag changes
> ----------------------------------------------------------------
>
> Key: JAMES-3728
> URL: https://issues.apache.org/jira/browse/JAMES-3728
> Project: James Server
> Issue Type: Improvement
> Components: elasticsearch, mailbox
> Affects Versions: 3.7.0
> Reporter: Benoit Tellier
> Priority: Major
>
> h3. Description
> We are writting a JMAP client app called TMail written in FLutter.
> In the sent mailbox we noticed the following behaviour:
> {code:java}
> Request
> {
> "using": [
> "urn:ietf:params:jmap:core",
> "urn:ietf:params:jmap:mail"
> ],
> "methodCalls": [
> [
> "Email/query",
> {
> "accountId": "{{accountId}}",
> "limit": 20,
> "filter": {
> "inMailbox": "{{Sent}}",
> "notKeyword": "$seen"
> },
> "sort": [{
> "property":"sentAt",
> "isAscending": false
> }],
> "position": 0
> },
> "c2"
> ],
> [
> "Email/get",
> {
> "accountId": "{{accountId}}",
> "properties": [
> "subject",
> "from",
> "receivedAt",
> "sentAt",
> "preview",
> "hasAttachment",
> "mailboxIds",
> "keywords"
> ],
> "#ids": {
> "resultOf": "c2",
> "name": "Email/query",
> "path": "ids/*"
> }
> },
> "c3"
> ]
> ]
> }
> Response
> {
> "sessionState": "2c9f1b12-b35a-43e6-9af2-0106fb53a943",
> "methodResponses": [
> [
> "Email/query",
> {
> "accountId": "3ce33c876a726662c627746eb9537a1d13c2338193ef27bd051a3ce5c0fe5b12",
> "queryState": "11dc2cfd",
> "canCalculateChanges": false,
> "ids": [
> "c331d3c0-90ac-11ec-984e-e3f8b83572b4",
> "36128e70-32e8-11ec-a759-2fef1ee78d9e",
> "4b0db860-32e6-11ec-a759-2fef1ee78d9e",
> "4f769140-2cc6-11ec-a759-2fef1ee78d9e",
> "3b21cb60-2cc6-11ec-a759-2fef1ee78d9e",
> "254999d0-2cc6-11ec-a759-2fef1ee78d9e",
> "19707570-2cc6-11ec-a759-2fef1ee78d9e",
> "0e877d70-2cc6-11ec-a759-2fef1ee78d9e",
> "8b5a5280-2cc3-11ec-a759-2fef1ee78d9e",
> "826ba3e0-2cc3-11ec-a759-2fef1ee78d9e",
> "5f771f90-2cc3-11ec-a759-2fef1ee78d9e"
> ],
> "position": 0
> },
> "c2"
> ],
> [
> "Email/get",
> {
> "accountId": "3ce33c876a726662c627746eb9537a1d13c2338193ef27bd051a3ce5c0fe5b12",
> "notFound": [],
> "state": "c55b33d0-a52e-11ec-90ba-e169393c493e",
> "list": [
> {
> "preview": "------- Forwarded message ------- Subject: [linshare-mobile-flutter-app-android-release] - Build [success] Date: October 8, 2021 12:50 PM From: [email protected] To: [email protected] <[email protected]>, [email protected] <[email protected]",
> "hasAttachment": false,
> "keywords": {
> "$seen": true
> },
> "subject": "Fwd: [linshare-mobile-flutter-app-android-release] - Build [success]",
> "mailboxIds": {
> "abb99c10-18d9-11eb-a677-2990b970028d": true
> },
> "from": [
> {
> "name": "Lê Nguyễn User B",
> "email": "[email protected]"
> }
> ],
> "sentAt": "2021-10-14T07:51:15Z",
> "id": "826ba3e0-2cc3-11ec-a759-2fef1ee78d9e",
> "receivedAt": "2021-10-14T07:51:15Z"
> },
> ....
> {code}
> As we can see, `Email/query` and `Email/get` disagrees on wether the message is seen or not!
> This happens primarily in the Sent mailbox.
> h3. Explanation ?
> Email/query relies on elasticSearch while Email/get relies on cassandra. Getting those two in sync can be an issue and you can have inconsistencies across the 2 data-stores. Now the indexing is done asynchronously which means that if two actions happens veeeery quickly (say read then move emails or vice versa) we can get into a data race in elasticsearch where we do the operation in the wrong order which could result into what you actually see.
> Moving emails to sent box does just that: mark as seen and move in one request
> To be very techish our event system don't guaranty ordering, and the underlying operation don't commute.
> Ordering don't seem achievable on RabbitMQ (contrary to Pulsar/Kafka)
> Working on commutativity can likely help!
> h3. Fix?
> I got an idea of a fix. Basically the key problem is that we use a mutable id as an indexing key which results in poor, non transactional behaviour if indexing requests get reordered. Of course it demands significant redesign of ES layers... And might bring additional concerns!
> h3. Original report from Dat Pham Hoang
> https://github.com/linagora/tmail-flutter/issues/340#issuecomment-1069188877
--
This message was sent by Atlassian Jira
(v8.20.10#820010)