OutOfMemoryError: Java heap space when using imapsync to transfer mail to James

Wojtek <[email protected]>
Newsgroups gmane.comp.jakarta.james.user
Message-ID <[email protected]> 
Hi,
I was attempting to transfer data from my old account to James and ran into an issue: 
"OutOfMemoryError: Java heap space". Upon closer inspection, James was using 4,7g virt / 2,3g res 
memory (system usage of the generated docker image was 7,6g with 0,1g free reported). This is a 
fairly simple setup of james-jpa backed by MySQL.

Has anyone ran into similar problem? imap sync seemed to process messages one-by-one, so the load 
wasn't great. I'm currently collecting heapdump.

Error stacktraces entry seems to point to MySQL/openJPA - one assumption would be that James itself 
can process incomming messages faster than the repository can process/store them? Isn't there a 
feature that would slow down processing incomming messages in thise case?


21:44:10.161 [WARN ] r.c.Exceptions - throwIfFatal detected a jvm fatal exception, which is thrown 
and logged below:
java.lang.OutOfMemoryError: Java heap space
         at java.base/java.math.BigInteger.shiftRightImpl(Unknown Source)
         at java.base/java.math.BigInteger.shiftRight(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(Unknown 
Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValue(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.setValue(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.<init>(Unknown Source)
         at 
java.base/sun.security.util.math.intpoly.IntegerPolynomial$ImmutableElement.<init>(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown Source)
         at jdk.crypto.ec/sun.security.ec.XECOperations.encodedPointMultiply(Unknown Source)
         at jdk.crypto.ec/sun.security.ec.XDHKeyAgreement.engineDoPhase(Unknown Source)
         at java.base/javax.crypto.KeyAgreement.doPhase(Unknown Source)
         at java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(Unknown Source)
         at java.base/sun.security.ssl.KAKeyDerivation.deriveKey(Unknown Source)
         at java.base/sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(Unknown Source)
         at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(Unknown Source)
         at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(Unknown Source)
         at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
         at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
         at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
         at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
         at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
         at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:347)
         at 
com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:194)
         at 
com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:101)
         at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:308)
         at 
com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:204)
         at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1369)

21:44:14.037 [ERROR] r.c.s.Schedulers - Scheduler worker in group main failed with an uncaught exception
java.lang.OutOfMemoryError: Java heap space
         at java.base/java.math.BigInteger.shiftRightImpl(Unknown Source)
         at java.base/java.math.BigInteger.shiftRight(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(Unknown 
Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValue(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.setValue(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.<init>(Unknown Source)
         at 
java.base/sun.security.util.math.intpoly.IntegerPolynomial$ImmutableElement.<init>(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown Source)
         at java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown Source)
         at jdk.crypto.ec/sun.security.ec.XECOperations.encodedPointMultiply(Unknown Source)
         at jdk.crypto.ec/sun.security.ec.XDHKeyAgreement.engineDoPhase(Unknown Source)
         at java.base/javax.crypto.KeyAgreement.doPhase(Unknown Source)
         at java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(Unknown Source)
         at java.base/sun.security.ssl.KAKeyDerivation.deriveKey(Unknown Source)
         at java.base/sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(Unknown Source)
         at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(Unknown Source)
         at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(Unknown Source)
         at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
         at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
         at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
         at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
         at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
         at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
         at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:347)
         at 
com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:194)
         at 
com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:101)
         at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:308)
         at 
com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:204)
         at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1369)





21:47:57.114 [ERROR] o.a.j.i.p.AbstractMailboxProcessor - Unexpected error during IMAP processing
org.apache.openjpa.lib.jdbc.ReportingSQLException: Java heap space {prepstmnt 2016315081 SELECT 
t0.MAILBOX_ID, t0.MAIL_UID, t0.MAIL_IS_ANSWERED, t0.MAIL_BODY_START_OCTET, 
t0.MAIL_CONTENT_OCTETS_COUNT, t0.MAIL_IS_DELETED, t0.MAIL_IS_DRAFT, t0.MAIL_IS_FLAGGED, 
t0.MAIL_DATE, t1.MAILBOX_ID, t1.MAILBOX_HIGHEST_MODSEQ, t1.MAILBOX_LAST_UID, t1.MAILBOX_NAME, 
t1.MAILBOX_NAMESPACE, t1.MAILBOX_UID_VALIDITY, t1.USER_NAME, t0.MAIL_MIME_TYPE, t0.MAIL_MODSEQ, 
t0.MAIL_IS_RECENT, t0.MAIL_IS_SEEN, t0.MAIL_MIME_SUBTYPE, t0.MAIL_TEXTUAL_LINE_COUNT FROM JAMES_MAIL 
t0 LEFT OUTER JOIN JAMES_MAILBOX t1 ON t0.MAILBOX_ID = t1.MAILBOX_ID WHERE (t0.MAILBOX_ID = ?) ORDER 
BY t0.MAIL_UID ASC} [code=0, state=S1000]
         at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:219)
         at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:203)
         at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:58)
         at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeQuery(LoggingConnectionDecorator.java:1171)
         at 
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:300)
         at 
org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1870)
         at 
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:290)
         at org.apache.openjpa.jdbc.sql.SelectImpl.executeQuery(SelectImpl.java:531)
         at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:456)
         at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:423)
         at org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:477)
         at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:259)
         at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:248)
         at 
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:95)
         at org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:36)
         ... 78 common frames omitted
Wrapped by: org.apache.openjpa.persistence.PersistenceException: Java heap space {prepstmnt 
2016315081 SELECT t0.MAILBOX_ID, t0.MAIL_UID, t0.MAIL_IS_ANSWERED, t0.MAIL_BODY_START_OCTET, 
t0.MAIL_CONTENT_OCTETS_COUNT,
  t0.MAIL_IS_DELETED, t0.MAIL_IS_DRAFT, t0.MAIL_IS_FLAGGED, t0.MAIL_DATE, t1.MAILBOX_ID, 
t1.MAILBOX_HIGHEST_MODSEQ, t1.MAILBOX_LAST_UID, t1.MAILBOX_NAME, t1.MAILBOX_NAMESPACE, 
t1.MAILBOX_UID_VALIDITY, t1.USER_NAME,
t0.MAIL_MIME_TYPE, t0.MAIL_MODSEQ, t0.MAIL_IS_RECENT, t0.MAIL_IS_SEEN, t0.MAIL_MIME_SUBTYPE, 
t0.MAIL_TEXTUAL_LINE_COUNT FROM JAMES_MAIL t0 LEFT OUTER JOIN JAMES_MAILBOX t1 ON t0.MAILBOX_ID = 
t1.MAILBOX_ID WHERE (t0
.MAILBOX_ID = ?) ORDER BY t0.MAIL_UID ASC} [code=0, state=S1000]
         at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:5320)
         at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:5280)
         at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:134)
         at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:115)
         at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:67)
         at 
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.handleCheckedException(SelectResultObjectProvider.java:162)
         at org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:42)
         at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1314)
         at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1061)
         at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:911)
         at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:842)
         at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:601)
         at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:297)
         at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:314)
         at 
org.apache.james.mailbox.jpa.mail.JPAMessageMapper.findMessagesInMailbox(JPAMessageMapper.java:414)
         at org.apache.james.mailbox.jpa.mail.JPAMessageMapper.findAsList(JPAMessageMapper.java:113)
         at org.apache.james.mailbox.jpa.mail.JPAMessageMapper.findInMailbox(JPAMessageMapper.java:100)
         at 
org.apache.james.mailbox.store.mail.MessageMapper.findInMailboxReactive(MessageMapper.java:88)
         at 
org.apache.james.mailbox.jpa.mail.JPAMessageMapper.listAllMessageUids(JPAMessageMapper.java:92)
         at 
org.apache.james.mailbox.store.StoreMessageManager.lambda$listAllMessageUids$23(StoreMessageManager.java:986)
         at 
org.apache.james.mailbox.store.transaction.TransactionalMapper.execute(TransactionalMapper.java:38)
         ... 64 common frames omitted


Wrapped by: org.apache.openjpa.persistence.RollbackException: Java heap space {prepstmnt 2016315081 
SELECT t0.MAILBOX_ID, t0.MAIL_UID, t0.MAIL_IS_ANSWERED, t0.MAIL_BODY_START_OCTET, 
t0.MAIL_CONTENT_OCTETS_COUNT, t0
.MAIL_IS_DELETED, t0.MAIL_IS_DRAFT, t0.MAIL_IS_FLAGGED, t0.MAIL_DATE, t1.MAILBOX_ID, 
t1.MAILBOX_HIGHEST_MODSEQ, t1.MAILBOX_LAST_UID, t1.MAILBOX_NAME, t1.MAILBOX_NAMESPACE, 
t1.MAILBOX_UID_VALIDITY, t1.USER_NAME, t0.
MAIL_MIME_TYPE, t0.MAIL_MODSEQ, t0.MAIL_IS_RECENT, t0.MAIL_IS_SEEN, t0.MAIL_MIME_SUBTYPE, 
t0.MAIL_TEXTUAL_LINE_COUNT FROM JAMES_MAIL t0 LEFT OUTER JOIN JAMES_MAILBOX t1 ON t0.MAILBOX_ID = 
t1.MAILBOX_ID WHERE (t0.MA
ILBOX_ID = ?) ORDER BY t0.MAIL_UID ASC} [code=0, state=S1000]
         at org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:713)
         at org.apache.james.mailbox.jpa.JPATransactionalMapper.commit(JPATransactionalMapper.java:71)
         ... 65 common frames omitted
Wrapped by: org.apache.james.mailbox.exception.MailboxException: Commit of transaction failed
         at org.apache.james.mailbox.jpa.JPATransactionalMapper.commit(JPATransactionalMapper.java:73)
         at 
org.apache.james.mailbox.store.transaction.TransactionalMapper.execute(TransactionalMapper.java:39)
         at 
org.apache.james.mailbox.store.StoreMessageManager.listAllMessageUids(StoreMessageManager.java:985)
         at org.apache.james.mailbox.store.StoreMessageManager.search(StoreMessageManager.java:835)
         at org.apache.james.mailbox.store.StoreMessageManager.search(StoreMessageManager.java:132)
         at 
org.apache.james.imap.processor.base.SelectedMailboxImpl.finishInit(SelectedMailboxImpl.java:167)
         at 
org.apache.james.imap.processor.AbstractSelectionProcessor.selectMailbox(AbstractSelectionProcessor.java:426)
         at 
org.apache.james.imap.processor.AbstractSelectionProcessor.lambda$selectMailbox$25(AbstractSelectionProcessor.java:399)
         at com.github.fge.lambdas.functions.FunctionChainer.doApply(FunctionChainer.java:20)
         at com.github.fge.lambdas.functions.ThrowingFunction.apply(ThrowingFunction.java:17)
         ... 56 common frames omitted
Wrapped by: com.github.fge.lambdas.ThrownByLambdaException: 
org.apache.james.mailbox.exception.MailboxException: Commit of transaction failed
         at com.github.fge.lambdas.functions.ThrowingFunction.apply(ThrowingFunction.java:21)
         at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
         at 
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
         at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
         at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
         at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
         at reactor.core.publisher.MonoCallable$MonoCallableSubscription.request(MonoCallable.java:156)
         at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341)
         at 
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
         at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:48)
         at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
         at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:109)
         at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:102)
         at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
         at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
         at reactor.core.publisher.Mono.subscribe(Mono.java:4444)
         at reactor.core.publisher.Mono.subscribeWith(Mono.java:4510)
         at reactor.core.publisher.Mono.subscribe(Mono.java:4272)
         at 
org.apache.james.imapserver.netty.ImapChannelUpstreamHandler.channelRead(ImapChannelUpstreamHandler.java:367)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
         at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
         at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
         at 
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
         at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
         at java.base/java.lang.Thread.run(Unknown Source)


-- 
Wojtek
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.