QtWebEngine uses up all memory and will not give it back!

J Barchan <[email protected]> Fri, 4 Oct 2019 14:08:11 +0100
Newsgroups	gmane.comp.python.pyqt-pykde
Message-ID	<CABz3M_8kc2Cdb5vcv==-9dQ_GojjmwqX2a+zD-r_vNHyY82Jxg@mail.gmail.com>
Hello Experts,

I do not know whether this issue is PyQt-related, but it might be, as I
think it's to do with deleteLater() and *possibly* Python memory
management.  I should be so grateful if someone could take the time to look
through and advise me what to try next?

Qt 5.12.2. Python/PyQt (though that does not help, it should not be the
issue). Tested under Linux, known from user to happen under Windows too. I
am in trouble, and I need help from someone who knows their QtWebEngine!

Briefly: I have to create and delete QtWebEngines (for non-interactive use,
read below) a *large* number of times from a loop which cannot call the
*main* event loop. Every instance holds onto its memory allocation ---
which is "large" --- until code finally returns to *main* event loop. I
*cannot* find any way of getting Qt to release the memory being use by
QtWebEngine as it proceeds, *only* when return to main event loop. Result
is whole machine runs out of memory + swap space, until it dies/freezes
machine, requiring reboot!

   -

   In large body of code, QWebEngineView is employed in a QDialog.
   -

   Sometimes that dialog is used interactively by user.
   -

   But it is also used *non*-interactively in order to use its ability to
   print from HTML to PDF file.
   -

   Code will do a non-interactive "batch run" of hundreds/thousands of
   pieces of HTML, exporting to PDF file(s).
   -

   During this *large* amounts of memory will be gobbled by QtWebEngine. Of
   the order of hundreds of create/delete taking Gigabytes of machine memory.
   So much so that machine can even run out of all memory and die!
   -

   *Only* a return to top-level, main Qt event loop allows that memory to
   be recouped. I *need* something better than that!

I paste below about as minimal an example of code I am using in a test to
prove behaviour.

import sys

from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWebEngineWidgets import QWebEngineView

class MyDialog(QtWidgets.QDialog):
    def __init__(self, parent=None):
        super(MyDialog, self).__init__(parent)

        self.wev = QWebEngineView(self)

        self.renderLoop = QtCore.QEventLoop(self)
        self.rendered = False
        self.wev.loadFinished.connect(self.synchronousWebViewLoaded)

        # set some HTML
        html = "<html><body>Hello World</body></html>"
        self.wev.setHtml(html)

        # wait for the HTML to finish rendering asynchronously
        # see synchronousWebViewLoaded() below
        if not self.rendered:
            self.renderLoop.exec()

        # here I would do the printing in real code
        # but it's not necessary to include this to show the memory problem

    def synchronousWebViewLoaded(self, ok: bool):
        self.rendered = True
        # cause the self.renderLoop.exec() above to exit now
        self.renderLoop.quit()

class MyMainWindow(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        super(MyMainWindow, self).__init__(parent)

        self.btn = QtWidgets.QPushButton("Do Test", self)
        self.btn.clicked.connect(self.doTest)

    def doTest(self):
        print("Started\n")
        # create & delete 500 non-interactive dialog instances
        for i in range(500):
            # create the dialog, it loads the HTML and waits till it
has finished rendering
            dlg = MyDialog(self)
            # try desperately to get to delete the dialog/webengine to
reclaim memory...
            dlg.deleteLater()
            # next lines do not help from Python
            # del dlg
            # dlg = None
        QtWidgets.QMessageBox.information(self, "Dismiss to return to
main event loop", "At this point memory is still in use :(")

if __name__ == '__main__':
    # -*- coding: utf-8 -*-

    app = QtWidgets.QApplication(sys.argv)

    mainWin = MyMainWindow()
    mainWin.show()

    sys.exit(app.exec())

I have tried various flavours of processEvents() in the loop after
deleteLater() but none releases the memory in use. *Only, only* when the
code returns to the top-level, main Qt event loop does it get released.
Which is too late.

To monitor what is going on, under Linux I used

watch -n 1 free mem
watch -n 1 ps -C QtWebEngineProc
top -o %MEM

There are two areas of memory hogging:

   - The process itself uses up considerable memory per QtWebEngine
   - It will run 26 (yes, precisely 26) QtWebEngineProc processes to
   service the requests, each also taking memory.

Both of these disappear as & when return to Qt top-level event loop, so we
know the memory can & should be released. I do not know if this behaviour
is QtWebEngine specific.

Anyone kind enough to answer will need to be specific about what to put
where to resolve or try out, as I say I have tried a lot of fiddling!
Unfortunately, advising to do the whole thing "a different way" (e.g. "do
not use QtWebEngineView", "rewrite code so it does not have to do hundreds
at a time", etc.) is really not what I am looking for, I need to understand
why I can't get it to release its memory as it is now? Can anyone make my
deleteLater() release its memory without going back to the top-level Qt
event loop??

-- 
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [email protected]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt