Re: QtWebEngine uses up all memory and will not give it back!

Hi Kevin,

In your example script, one potential fix is to not give MyDialog a parent,
> so change "dlg = MyDialog(self)" to "dlg = MyDialog()".  That should let
> PyQt destroy the dialog as soon as the Python reference count drops to
> zero, which would happen with "del dlg", "dlg=None",
>

Tried this carefully.  Did not make any difference, nothing released as it
goes along.  I note that where I have

self.wev = QWebEngineView(self)

if I remove *that* self parameter I do get memory back.  So something all
to do with the QWebEngineView having the QDialog as parent is
relevant/problematic?  But that's not good for my code, as elsewhere I do
display the dialog with the webview on it.

If you need to parent the dialogs, another option is to use
> "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger
> immediate deletion of the dialog instead of scheduling it for later.
>

Much better!  sip.delete(dlg) is the one thing which *does* result in the
memory being freed as it goes along, so even after doing hundreds I see
machine memory usage staying constant & acceptable!  Thank you :)

So is it perfectly safe to use that?  It seems to delete/reclaim the
webview which the dialog owns, so is all well or any there any lurking
"gotchas"?

Thanks,
Jonathan

On Sat, 5 Oct 2019 at 10:18, J Barchan <[email protected]> wrote:

> Wow, that's very useful, thank you so much for replying.  I shall try
> these out next week.  I am aware plain processEvents() won't do the
> "deferred deletes", indeed my problem is to try to find an alternative
> which does.  I tried the processEvents(event-type, timeout) overload
> which seems to say it handles these differently, but no improvement.  I
> tried destroy(), which from a previous post of mine I found from a year
> ago I seemed to have claimed resolved, but that was just worse!  If you
> would be kind, enough please keep an eye on this thread as I will report
> back, and if it's failure I should welcome any further thoughts!
>
> I did read up about processing events and event loops.  One thing I do not
> understand is *how* the deleteLater()s only take effect in the "main"
> loop, not elsewhere e.g. if I call my own explicit processEvents().  Is
> there a simple conceptual explanation of how/why?  For example, the
> existing app I am working is very heavily designed around having some kind
> of modal dialog (which may call other modals) up most of the time.  I
> believe I found from my testing that deleteLater()s do *not* get released
> during modal dialog event loop, only when return to "top" loop?  This is
> not so good if the program does not spend a lot of its time at that level!
>
> Meanwhile, an observation about this email-forum.  You have put your reply
> at the top of mine, e.g. just where Gmail puts you for *Reply*, so I have
> done the same time this time.  The very first time I replied in this
> message group I was *shouted at* immediately by someone saying that order
> of replying was unacceptable here, for people with other readers, and I
> must adhere to "reply at end".  I felt I nearly got banned!  However I have
> seen increasingly people reply as you have.  Does it matter any longer?  I
> don't want to offend anyone!
>
>
> On Fri, 4 Oct 2019 at 21:58, Kevin Keating <[email protected]>
> wrote:
>
>> processEvents() will process everything other than DeferredDelete events,
>> which is the type of event that gets created by deleteLater() calls.
>> That's intended (but confusing) behavior.  See
>> https://doc.qt.io/qt-5/qcoreapplication.html#processEvents.  In your
>> example script, one potential fix is to not give MyDialog a parent, so
>> change "dlg = MyDialog(self)" to "dlg = MyDialog()".  That should let PyQt
>> destroy the dialog as soon as the Python reference count drops to zero,
>> which would happen with "del dlg", "dlg=None", or on the next loop
>> iteration.
>>
>> If you need to parent the dialogs, another option is to use
>> "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger
>> immediate deletion of the dialog instead of scheduling it for later.
>>
>> - Kevin
>>
>> On 10/4/2019 9:09:48 AM, J Barchan <[email protected]> wrote:
>> Hello Experts,
>>
>> I do not know whether this issue is PyQt-related, but it might be, as I
>> think it's to do with deleteLater() and *possibly* Python memory
>> management.  I should be so grateful if someone could take the time to look
>> through and advise me what to try next?
>>
>> Qt 5.12.2. Python/PyQt (though that does not help, it should not be the
>> issue). Tested under Linux, known from user to happen under Windows too. I
>> am in trouble, and I need help from someone who knows their QtWebEngine!
>>
>> Briefly: I have to create and delete QtWebEngines (for non-interactive
>> use, read below) a *large* number of times from a loop which cannot call
>> the *main* event loop. Every instance holds onto its memory allocation
>> --- which is "large" --- until code finally returns to *main* event
>> loop. I *cannot* find any way of getting Qt to release the memory being
>> use by QtWebEngine as it proceeds, *only* when return to main event
>> loop. Result is whole machine runs out of memory + swap space, until it
>> dies/freezes machine, requiring reboot!
>>
>>    -
>>
>>    In large body of code, QWebEngineView is employed in a QDialog.
>>    -
>>
>>    Sometimes that dialog is used interactively by user.
>>    -
>>
>>    But it is also used *non*-interactively in order to use its ability
>>    to print from HTML to PDF file.
>>    -
>>
>>    Code will do a non-interactive "batch run" of hundreds/thousands of
>>    pieces of HTML, exporting to PDF file(s).
>>    -
>>
>>    During this *large* amounts of memory will be gobbled by QtWebEngine.
>>    Of the order of hundreds of create/delete taking Gigabytes of machine
>>    memory. So much so that machine can even run out of all memory and die!
>>    -
>>
>>    *Only* a return to top-level, main Qt event loop allows that memory
>>    to be recouped. I *need* something better than that!
>>
>> I paste below about as minimal an example of code I am using in a test to
>> prove behaviour.
>>
>> import sys
>>
>> from PyQt5 import QtCore, QtWidgets
>> from PyQt5.QtWebEngineWidgets import QWebEngineView
>>
>> class MyDialog(QtWidgets.QDialog):
>>     def __init__(self, parent=None):
>>         super(MyDialog, self).__init__(parent)
>>
>>         self.wev = QWebEngineView(self)
>>
>>         self.renderLoop = QtCore.QEventLoop(self)
>>         self.rendered = False
>>         self.wev.loadFinished.connect(self.synchronousWebViewLoaded)
>>
>>         # set some HTML
>>         html = "<html><body>Hello World</body></html>"
>>         self.wev.setHtml(html)
>>
>>         # wait for the HTML to finish rendering asynchronously
>>         # see synchronousWebViewLoaded() below
>>         if not self.rendered:
>>             self.renderLoop.exec()
>>
>>         # here I would do the printing in real code
>>         # but it's not necessary to include this to show the memory problem
>>
>>     def synchronousWebViewLoaded(self, ok: bool):
>>         self.rendered = True
>>         # cause the self.renderLoop.exec() above to exit now
>>         self.renderLoop.quit()
>>
>> class MyMainWindow(QtWidgets.QMainWindow):
>>     def __init__(self, parent=None):
>>         super(MyMainWindow, self).__init__(parent)
>>
>>         self.btn = QtWidgets.QPushButton("Do Test", self)
>>         self.btn.clicked.connect(self.doTest)
>>
>>     def doTest(self):
>>         print("Started\n")
>>         # create & delete 500 non-interactive dialog instances
>>         for i in range(500):
>>             # create the dialog, it loads the HTML and waits till it has finished rendering
>>             dlg = MyDialog(self)
>>             # try desperately to get to delete the dialog/webengine to reclaim memory...
>>             dlg.deleteLater()
>>             # next lines do not help from Python
>>             # del dlg
>>             # dlg = None
>>         QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(")
>>
>> if __name__ == '__main__':
>>     # -*- coding: utf-8 -*-
>>
>>     app = QtWidgets.QApplication(sys.argv)
>>
>>     mainWin = MyMainWindow()
>>     mainWin.show()
>>
>>     sys.exit(app.exec())
>>
>> I have tried various flavours of processEvents() in the loop after
>> deleteLater() but none releases the memory in use. *Only, only* when the
>> code returns to the top-level, main Qt event loop does it get released.
>> Which is too late.
>>
>> To monitor what is going on, under Linux I used
>>
>> watch -n 1 free mem
>> watch -n 1 ps -C QtWebEngineProc
>> top -o %MEM
>>
>> There are two areas of memory hogging:
>>
>>    - The process itself uses up considerable memory per QtWebEngine
>>    - It will run 26 (yes, precisely 26) QtWebEngineProc processes to
>>    service the requests, each also taking memory.
>>
>> Both of these disappear as & when return to Qt top-level event loop, so
>> we know the memory can & should be released. I do not know if this
>> behaviour is QtWebEngine specific.
>>
>> Anyone kind enough to answer will need to be specific about what to put
>> where to resolve or try out, as I say I have tried a lot of fiddling!
>> Unfortunately, advising to do the whole thing "a different way" (e.g. "do
>> not use QtWebEngineView", "rewrite code so it does not have to do hundreds
>> at a time", etc.) is really not what I am looking for, I need to understand
>> why I can't get it to release its memory as it is now? Can anyone make my
>> deleteLater() release its memory without going back to the top-level Qt
>> event loop??
>>
>> --
>> Kindest,
>> Jonathan
>> _______________________________________________ PyQt mailing list
>> [email protected]
>> https://www.riverbankcomputing.com/mailman/listinfo/pyqt
>>
>>
>
> --
> Kindest,
> Jonathan
>


-- 
Kindest,
Jonathan