Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6

I have a first result on this issue:

The problem does not occur if TomEE is running on Linux. I've tested my 
own built 8.0.8, a freshly extracted 8.0.8 and 8.0.15 from the releases, 
and none of them showed the dev mode performance problem.

Using TomEE 8.0.8 on Windows does show the dev mode performance problem. 
All of the tested ones, means the released 8.0.8 and my own built 8.0.8 
(the one I've built on Linux).

My development environment is Windows 10 (Netbeans and therefore also 
the TomEE for easy deployment and debugging). The problem does occur on 
stand alone started TomEE and also if started out of Netbeans, so 
Netbeans can be sorted out as the root of the problem. Java Versions 11, 
13, 15 and 17 tested, all the same, so Java should also not be the 
problem. BTW: It is very unlikely a problem of my individual Windows 
installation, because this problem also shows up on the Windows system 
of my colleague, which btw. was the root cause why I'm started to dig 
deeper into this problem now and why I subscribed to this mailing list ;-)

I'm afraid, this will complicate my git bisect session a little bit :-(

However, I'll keep you posted.

cu
Jens


Am 27.10.2023 um 13:12 schrieb Jens Zurawski:
> Thanks Richard (and Vicente),
>
> I've changed to JDK 8 and now the builds are successful (both 8.0.6 
> and 8.0.8).
>
> I'll script me some test scripts to ease the build, run and test for 
> every bisect step.
> A build needs around 3 to 4 minutes. Running a test with my 
> application will need some manual interaction and checks, I guess also 
> something around 4 minutes. This sums up to roughly 8 minutes per step.
>
> I've started a bisect and it told me:
> git bisect good 20441eb
> git bisect bad 4c8a616
> Bisecting: 136 revisions left to test after this (roughly 7 steps)
>
> That means, if everything works fine, I should be able to identify the 
> commit after about an hour? (That's cool.... why nobody has told me 
> about git bisect before? ;-) )
>
> Anyway, I will have to find some time for this, maybe I need a couple 
> of days to find some.
> I will let you know about the outcome.
>
> cu
> Jens
>
>
> Am 27.10.2023 um 11:50 schrieb Richard Zowalla:
>> Hi,
>>
>> you need Java/JDK 8 to build TomEE 8.
>>
>> The Maven command is
>>
>> mvn --fail-at-end clean install -pl tomee/apache-tomee -am -
>> Dfile.encoding=UTF-8 -DskipTests=true
>>
>> You will find the tar.gz/zip in ./tomee/apache-tomee/target
>>
>> Gruß
>> Richard
>>
>> P.S. 8.0.7 is here: https://archive.apache.org/dist/tomee/tomee-8.0.7/
>>
>> Am Freitag, dem 27.10.2023 um 10:50 +0200 schrieb Jens Zurawski:
>>> Hi Richard,
>>>
>>> to identify the "bad" version was easy. I didn't find a 8.0.7 release
>>> on
>>> the download page, so I started with 8.0.8. And this version already
>>> has
>>> the slow dev mode. So for my problem I can say:
>>> good: 8.0.6
>>> bad: 8.0.8
>>>
>>> Now, I think, I need a little help from you. I tried to build a TomEE
>>> out of git (I've cloned the github repository
>>> https://github.com/apache/tomee.git). I've tried these two commits:
>>> 20441eb (tag: tomee-8.0.6)
>>> 4c8a616 (tag: tomee-project-8.0.8)
>>>
>>> But the build fails every time on "TomEE :: Container :: Core" with:
>>>
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ---------------------------------------------------------------------
>>> ---
>>> [INFO] Total time:  01:19 min
>>> [INFO] Finished at: 2023-10-27T10:29:40+02:00
>>> [INFO]
>>> ---------------------------------------------------------------------
>>> ---
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile
>>> (default-compile) on project openejb-core: Compilation failure:
>>> Compilation failure:
>>> [ERROR]
>>> /sandbox/tomee/container/openejb-
>>> core/src/main/java/org/apache/openejb/core/OrbFactory.java:[21,21]
>>> package org.omg.CORBA does not exist
>>> [ERROR]
>>> /sandbox/tomee/container/openejb-
>>> core/src/main/java/org/apache/openejb/core/OrbFactory.java:[24,12]
>>> cannot find symbol
>>> [ERROR]   symbol:   class ORB
>>> [ERROR]   location: class org.apache.openejb.core.OrbFactory
>>>
>>> I've tried to build with:
>>> mvn clean install
>>>
>>> My build environment is a linux ubuntu 20.04.6 LTS VM with:
>>> git 2.25.1
>>> Maven 3.6.3
>>> java 14.0.2
>>>
>>> Is it some easy thing to fix, or do I have to switch some versions of
>>> my
>>> environment?
>>>
>>> cu
>>> Jens
>>>
>>> Am 26.10.2023 um 19:48 schrieb Richard Zowalla:
>>>> Hi,
>>>>
>>>> without knowing code or having a reproducer, it might be not that
>>>> obvious to find a reason / answer for the "academic problem" ;-)
>>>>
>>>> If you really want to boil it down, I recommend the following
>>>> approach:
>>>>
>>>> (1) From your description it seems, that 8.0.6 is the last known
>>>> "good"
>>>> version.
>>>>
>>>> (2) You've investigated versions starting from 8.0.12 onwards,
>>>> which
>>>> seems to be "bad".
>>>>
>>>> Following the approach recommended by Jonathan:
>>>>
>>>> - Can you check if it happens in 8.0.7 (8.0.8, ...) until we find
>>>> the
>>>> first "bad" released version?
>>>>
>>>> - If we have that, you could start a "git bisect" session between
>>>> 8.0.6
>>>> and the first version, which is known to be bad (reducing the
>>>> amount of
>>>> needed bisect cycles dramatically), so we find a commit / reason
>>>> (if it
>>>> is in TomEE).
>>>>
>>>> - It would involve some quick builds without executing tests (~10-
>>>> 15min
>>>> each) for each bisect step and testing on your side.
>>>>
>>>> Don't know if your inner monk is willing to go that route, but I am
>>>> happy to help with the related Maven commands for quick building :)
>>>>
>>>> Gruß
>>>> Richard
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Am Donnerstag, dem 26.10.2023 um 18:00 +0200 schrieb Jens Zurawski:
>>>>> Hi Vicente,
>>>>>
>>>>> thank you for the tip. This is a good workaround. When I switch
>>>>> to
>>>>> Production mode and insert this configuration, it reloads the
>>>>> view
>>>>> from
>>>>> the file if it has changed. This means, I can use a decent
>>>>> version of
>>>>> TomEE in my dev environment.
>>>>>
>>>>> However, it doesn't completely solve my problem, as the error
>>>>> reporting
>>>>> isn't as verbose as in Dev mode as far as I can see. But I have
>>>>> to
>>>>> check, maybe it's enough for my development.
>>>>>
>>>>> And, well.... the academic problem: "Why is it still broken in
>>>>> Dev
>>>>> mode?" still runs circles through my brain ;-) Because if it's a
>>>>> problem
>>>>> of my coding, it might be a good idea to change this, before it
>>>>> might
>>>>> get a problem on Production mode, too, at some time in the
>>>>> future.
>>>>>
>>>>> But anyway, you've helped me a good step forward, thanks.
>>>>> cu
>>>>> Jens
>>>>>
>>>>>
>>>>> Am 26.10.2023 um 17:04 schrieb Vicente Rossello:
>>>>>> We are using the production stage with
>>>>>> javax.faces.FACELETS_REFRESH_PERIOD
>>>>>> = 1. It's faster and more similar to production, just try it if
>>>>>> it
>>>>>> solves
>>>>>> your problem
>>>>>>
>>>>>> On Thu, Oct 26, 2023 at 4:18 PM Jens Zurawski <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> The one bundled with TomEE.
>>>>>>> myfaces 2.3.7 on TomEE 8.0.6
>>>>>>> and  e.g. myfaces 2.3.9 on TomEE 8.0.12
>>>>>>>
>>>>>>> Am 26.10.2023 um 16:08 schrieb Richard Zowalla:
>>>>>>>> Which JSF lib are you using?
>>>>>>>>
>>>>>>>> Am 26. Oktober 2023 16:00:57 MESZ schrieb Jens Zurawski
>>>>>>>> <[email protected]
>>>>>>>> :
>>>>>>>>> Thank you for your quick reply, Jonathan.
>>>>>>>>>
>>>>>>>>> It is good to know, that normally the Dev mode should
>>>>>>>>> work as
>>>>>>>>> fast as
>>>>>>> before even with 8.0.15. So it obviously has something to do
>>>>>>> with
>>>>>>> my
>>>>>>> environment, configuration and/or code.
>>>>>>>>> And also thank you for the reference to tomitribe.
>>>>>>>>> Unfortunately I'm
>>>>>>> the only developer on this project, and it will be difficult
>>>>>>> to
>>>>>>> convince my
>>>>>>> customer to an additional support subscription, because he on
>>>>>>> production
>>>>>>> doesn't have issues ;-) But, ok, that's my problem, and maybe
>>>>>>> I'll consult
>>>>>>> tomitribe in the future for other projects.
>>>>>>>>> So, I'm afraid, for the near future I'm on my own to
>>>>>>>>> solve
>>>>>>>>> this
>>>>>>> problem. Therefore any hints to restrict the places where I
>>>>>>> should take a
>>>>>>> look will be more than welcome.
>>>>>>>>> I'm a senior programmer with 20 years of experience in
>>>>>>>>> Java
>>>>>>>>> programming
>>>>>>> (SE and JSP). But I'm relative new to JSF/JavaEE/JakartaEE
>>>>>>> and
>>>>>>> sometimes I
>>>>>>> have a hard time to understand which component of the whole
>>>>>>> stack
>>>>>>> is
>>>>>>> responsible for what.
>>>>>>>>> The project meanwhile has over a 1/4 million lines of
>>>>>>>>> code
>>>>>>>>> (most of
>>>>>>> them in the back-end, but also the front-end is big enough to
>>>>>>> not
>>>>>>> trace
>>>>>>> through every lane if I want to get to the point in this life
>>>>>>> :-D
>>>>>>> ).
>>>>>>>>> The application now runs for nearly 2 years on the
>>>>>>>>> customers
>>>>>>>>> site and
>>>>>>> is very reliable and fast in its daily usage. Apart from
>>>>>>> occasional updates
>>>>>>> every few month, it will never be restarted or need any
>>>>>>> manual
>>>>>>> maintenance
>>>>>>> at all. So I assume the code is not the badest on earth. At
>>>>>>> least
>>>>>>> because
>>>>>>> of that I'm able to out-rule things like memory leaks, gc
>>>>>>> issues
>>>>>>> or
>>>>>>> bottlenecks (max Heap is never reached, neither half of it,
>>>>>>> even
>>>>>>> after
>>>>>>> running for month).
>>>>>>>>> Of course there could be aspects of JSF which I may not
>>>>>>>>> have
>>>>>>>>> understood
>>>>>>> correctly until now. But because there are so many new things
>>>>>>> to
>>>>>>> a JSF
>>>>>>> beginner like me, I'm running out of ideas what this could
>>>>>>> be. It
>>>>>>> is
>>>>>>> something that worked well 'til TomEE 8.0.6 (Prod and Dev
>>>>>>> mode)
>>>>>>> and
>>>>>>> suddenly don't work anymore on 8.0.12 in Dev mode (but still
>>>>>>> works great in
>>>>>>> Prod mode). There are only minor versions in-between. So I
>>>>>>> think
>>>>>>> there were
>>>>>>> no revolutionary changes anywhere. My hope was (and still
>>>>>>> is),
>>>>>>> that anybody
>>>>>>> here on the list might have had a similar problem and has
>>>>>>> solved
>>>>>>> it, or
>>>>>>> maybe someone knows about some changes in Dev mode which
>>>>>>> could
>>>>>>> explain such
>>>>>>> behaviour (don't know, maybe xml parsing or component tree
>>>>>>> generation has
>>>>>>> changed, or scope handling of managed beans, or whatever
>>>>>>> could
>>>>>>> cause some
>>>>>>> code to be entered far more often than before, or some
>>>>>>> cache/pool
>>>>>>> was
>>>>>>> removed or has different behaviour... such things). Then it
>>>>>>> would
>>>>>>> be far
>>>>>>> more easy for me to track down the real problem.
>>>>>>>>> Anyway, when I get some spare time (currently have some
>>>>>>>>> deadlines in
>>>>>>> sight), I'll try to build a small test case. Either I can get
>>>>>>> a
>>>>>>> grip on the
>>>>>>> issue myself on doing this, or I can supply it to anyone who
>>>>>>> might want to
>>>>>>> have a look at it.
>>>>>>>>> cu
>>>>>>>>> Jens
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 26.10.2023 um 14:01 schrieb Jonathan S. Fisher:
>>>>>>>>>>> newer versions are so incredibly slow in dev mode
>>>>>>>>>> I can assure you that Dev/Prod mode works very very
>>>>>>>>>> swiftly
>>>>>>>>>> on 8.0.15
>>>>>>>>>> for sizable applications. For example, I have a giant
>>>>>>>>>> application
>>>>>>>>>> (thousands of LOC, 50+ jsf views) that has sub-8ms
>>>>>>>>>> response
>>>>>>>>>> times
>>>>>>>>>> (minus database time). More than likely they are doing
>>>>>>>>>> something they
>>>>>>>>>> should not be doing with JSF apis and are causing
>>>>>>>>>> problems.
>>>>>>>>>>
>>>>>>>>>> I would check the obvious things first with VisualVM:
>>>>>>>>>> heap
>>>>>>>>>> size, gc
>>>>>>>>>> frequency, etc. You can also run their profiler and see
>>>>>>>>>> if
>>>>>>>>>> you can
>>>>>>>>>> spot anything immediately obvious.
>>>>>>>>>>
>>>>>>>>>> After that, the easiest way to locate the code slowing
>>>>>>>>>> you
>>>>>>>>>> down is
>>>>>>>>>> bisection. Cut half the code out, run, and continue
>>>>>>>>>> cutting
>>>>>>>>>> until you
>>>>>>>>>> can locate the problematic code. Then after that,
>>>>>>>>>> isolate a
>>>>>>>>>> test case
>>>>>>>>>> where flipping a boolean flag triggers the issue and
>>>>>>>>>> post
>>>>>>>>>> the results
>>>>>>>>>> here.
>>>>>>>>>>
>>>>>>>>>> I would get ahold of: https://www.tomitribe.com who are
>>>>>>>>>> literal
>>>>>>>>>> experts in this stuff. They offer professional services
>>>>>>>>>> to
>>>>>>>>>> handle
>>>>>>>>>> exactly these situations and can work directly in your
>>>>>>>>>> codebase to
>>>>>>>>>> help you find and fix the problem.
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 26, 2023 at 5:48 AM Jens Zurawski
>>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>>> Hi altogether,
>>>>>>>>>>>
>>>>>>>>>>> I'm new to this list and hope it's the right place to
>>>>>>>>>>> ask
>>>>>>>>>>> this
>>>>>>> question.
>>>>>>>>>>> If not, and someone know the right place to ask this,
>>>>>>>>>>> please give
>>>>>>> advise.
>>>>>>>>>>> I'm developing a big JSF Application for a customer.
>>>>>>>>>>> It's
>>>>>>>>>>> a long
>>>>>>> running
>>>>>>>>>>> project and development and when I started with it,
>>>>>>>>>>> TomEE
>>>>>>>>>>> 8.0.6 was
>>>>>>> the
>>>>>>>>>>> most recent Version of TomEE. The switch to Jakarta
>>>>>>>>>>> is
>>>>>>>>>>> planned for
>>>>>>> next
>>>>>>>>>>> year, so I'm still on the 8.x path for the time being
>>>>>>>>>>> and
>>>>>>>>>>> wasn't able
>>>>>>> to
>>>>>>>>>>> test if this problem still exists in the 9.x branch.
>>>>>>>>>>>
>>>>>>>>>>> The problem:
>>>>>>>>>>> On the customers site I'm with the latest TomEE
>>>>>>>>>>> version
>>>>>>>>>>> 8.0.15
>>>>>>> (running
>>>>>>>>>>> on Java 11) and everything works fine, because it is
>>>>>>>>>>> running in
>>>>>>>>>>> production mode. But in my development environment
>>>>>>>>>>> I'm
>>>>>>>>>>> still stuck
>>>>>>> with
>>>>>>>>>>> 8.0.6, because there I need the development mode
>>>>>>>>>>> (javax.faces.PROJECT_STAGE: Development). I need to
>>>>>>>>>>> be
>>>>>>>>>>> able to see
>>>>>>>>>>> changes in facelets without restarting everything
>>>>>>>>>>> everytime, and to
>>>>>>> get
>>>>>>>>>>> more detailed error messages.
>>>>>>>>>>> My attempts to update my dev TomEE to something newer
>>>>>>>>>>> than 8.0.6 all
>>>>>>>>>>> failed, because all (at least all I've tested so far)
>>>>>>>>>>> newer versions
>>>>>>> are
>>>>>>>>>>> so incredibly slow in dev mode, that it's unbearable
>>>>>>>>>>> to
>>>>>>>>>>> use the
>>>>>>>>>>> application. Several very looong seconds on every
>>>>>>>>>>> request
>>>>>>>>>>> (even the
>>>>>>>>>>> little AJAX requests in a view) is simply not a
>>>>>>>>>>> practical
>>>>>>>>>>> environment.
>>>>>>>>>>> After switching to Production mode everything works
>>>>>>>>>>> very
>>>>>>>>>>> fast, and all
>>>>>>>>>>> my views have response times of very few ms (even the
>>>>>>>>>>> big
>>>>>>>>>>> ones with
>>>>>>> max.
>>>>>>>>>>> around 400ms). When switching back to Development
>>>>>>>>>>> mode I
>>>>>>>>>>> have response
>>>>>>>>>>> times of up to 30s on big views. Not with 8.0.6,
>>>>>>>>>>> there
>>>>>>>>>>> even in
>>>>>>>>>>> development mode it's reasonable fast in not getting
>>>>>>>>>>> higher than 1s.
>>>>>>>>>>>
>>>>>>>>>>> My question:
>>>>>>>>>>> What causes this enormous performance degradation?
>>>>>>>>>>> I'm
>>>>>>>>>>> hoping, it is
>>>>>>>>>>> just a configuration which now has another default
>>>>>>>>>>> value
>>>>>>>>>>> or the like.
>>>>>>> If
>>>>>>>>>>> yes, maybe someone can point me in the right
>>>>>>>>>>> direction of
>>>>>>>>>>> where to
>>>>>>> find
>>>>>>>>>>> this configuration? If it's not a configuration
>>>>>>>>>>> thing:
>>>>>>>>>>> what can I do
>>>>>>> to
>>>>>>>>>>> get around this?
>>>>>>>>>>>
>>>>>>>>>>> I haven't tested all Versions of TomEE from 8.0.6 to
>>>>>>>>>>> 8.0.15, so I
>>>>>>> can't
>>>>>>>>>>> say at what version exactly this behaviour changes.
>>>>>>>>>>> If it
>>>>>>>>>>> helps or is
>>>>>>>>>>> needed, I can make some tests to find the exact
>>>>>>>>>>> version
>>>>>>>>>>> where this
>>>>>>>>>>> happens. Versions I've already tested are: 8.0.12,
>>>>>>>>>>> 8.0.13
>>>>>>>>>>> and 8.0.15.
>>>>>>>>>>> All of them are very slow in dev mode. If you need
>>>>>>>>>>> more
>>>>>>>>>>> information,
>>>>>>>>>>> I'll try to provide it.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance for any help
>>>>>>>>>>> cu
>>>>>>>>>>> Jens
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>