Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6

Thanks Richard (and Vicente),

I've changed to JDK 8 and now the builds are successful (both 8.0.6 and 
8.0.8).

I'll script me some test scripts to ease the build, run and test for 
every bisect step.
A build needs around 3 to 4 minutes. Running a test with my application 
will need some manual interaction and checks, I guess also something 
around 4 minutes. This sums up to roughly 8 minutes per step.

I've started a bisect and it told me:
git bisect good 20441eb
git bisect bad 4c8a616
Bisecting: 136 revisions left to test after this (roughly 7 steps)

That means, if everything works fine, I should be able to identify the 
commit after about an hour? (That's cool.... why nobody has told me 
about git bisect before? ;-) )

Anyway, I will have to find some time for this, maybe I need a couple of 
days to find some.
I will let you know about the outcome.

cu
Jens


Am 27.10.2023 um 11:50 schrieb Richard Zowalla:
> Hi,
>
> you need Java/JDK 8 to build TomEE 8.
>
> The Maven command is
>
> mvn --fail-at-end clean install -pl tomee/apache-tomee -am -
> Dfile.encoding=UTF-8 -DskipTests=true
>
> You will find the tar.gz/zip in ./tomee/apache-tomee/target
>
> Gruß
> Richard
>
> P.S. 8.0.7 is here: https://archive.apache.org/dist/tomee/tomee-8.0.7/
>
> Am Freitag, dem 27.10.2023 um 10:50 +0200 schrieb Jens Zurawski:
>> Hi Richard,
>>
>> to identify the "bad" version was easy. I didn't find a 8.0.7 release
>> on
>> the download page, so I started with 8.0.8. And this version already
>> has
>> the slow dev mode. So for my problem I can say:
>> good: 8.0.6
>> bad: 8.0.8
>>
>> Now, I think, I need a little help from you. I tried to build a TomEE
>> out of git (I've cloned the github repository
>> https://github.com/apache/tomee.git). I've tried these two commits:
>> 20441eb (tag: tomee-8.0.6)
>> 4c8a616 (tag: tomee-project-8.0.8)
>>
>> But the build fails every time on "TomEE :: Container :: Core" with:
>>
>> [INFO] BUILD FAILURE
>> [INFO]
>> ---------------------------------------------------------------------
>> ---
>> [INFO] Total time:  01:19 min
>> [INFO] Finished at: 2023-10-27T10:29:40+02:00
>> [INFO]
>> ---------------------------------------------------------------------
>> ---
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile
>> (default-compile) on project openejb-core: Compilation failure:
>> Compilation failure:
>> [ERROR]
>> /sandbox/tomee/container/openejb-
>> core/src/main/java/org/apache/openejb/core/OrbFactory.java:[21,21]
>> package org.omg.CORBA does not exist
>> [ERROR]
>> /sandbox/tomee/container/openejb-
>> core/src/main/java/org/apache/openejb/core/OrbFactory.java:[24,12]
>> cannot find symbol
>> [ERROR]   symbol:   class ORB
>> [ERROR]   location: class org.apache.openejb.core.OrbFactory
>>
>> I've tried to build with:
>> mvn clean install
>>
>> My build environment is a linux ubuntu 20.04.6 LTS VM with:
>> git 2.25.1
>> Maven 3.6.3
>> java 14.0.2
>>
>> Is it some easy thing to fix, or do I have to switch some versions of
>> my
>> environment?
>>
>> cu
>> Jens
>>
>> Am 26.10.2023 um 19:48 schrieb Richard Zowalla:
>>> Hi,
>>>
>>> without knowing code or having a reproducer, it might be not that
>>> obvious to find a reason / answer for the "academic problem" ;-)
>>>
>>> If you really want to boil it down, I recommend the following
>>> approach:
>>>
>>> (1) From your description it seems, that 8.0.6 is the last known
>>> "good"
>>> version.
>>>
>>> (2) You've investigated versions starting from 8.0.12 onwards,
>>> which
>>> seems to be "bad".
>>>
>>> Following the approach recommended by Jonathan:
>>>
>>> - Can you check if it happens in 8.0.7 (8.0.8, ...) until we find
>>> the
>>> first "bad" released version?
>>>
>>> - If we have that, you could start a "git bisect" session between
>>> 8.0.6
>>> and the first version, which is known to be bad (reducing the
>>> amount of
>>> needed bisect cycles dramatically), so we find a commit / reason
>>> (if it
>>> is in TomEE).
>>>
>>> - It would involve some quick builds without executing tests (~10-
>>> 15min
>>> each) for each bisect step and testing on your side.
>>>
>>> Don't know if your inner monk is willing to go that route, but I am
>>> happy to help with the related Maven commands for quick building :)
>>>
>>> Gruß
>>> Richard
>>>
>>>
>>>
>>>
>>>
>>> Am Donnerstag, dem 26.10.2023 um 18:00 +0200 schrieb Jens Zurawski:
>>>> Hi Vicente,
>>>>
>>>> thank you for the tip. This is a good workaround. When I switch
>>>> to
>>>> Production mode and insert this configuration, it reloads the
>>>> view
>>>> from
>>>> the file if it has changed. This means, I can use a decent
>>>> version of
>>>> TomEE in my dev environment.
>>>>
>>>> However, it doesn't completely solve my problem, as the error
>>>> reporting
>>>> isn't as verbose as in Dev mode as far as I can see. But I have
>>>> to
>>>> check, maybe it's enough for my development.
>>>>
>>>> And, well.... the academic problem: "Why is it still broken in
>>>> Dev
>>>> mode?" still runs circles through my brain ;-) Because if it's a
>>>> problem
>>>> of my coding, it might be a good idea to change this, before it
>>>> might
>>>> get a problem on Production mode, too, at some time in the
>>>> future.
>>>>
>>>> But anyway, you've helped me a good step forward, thanks.
>>>> cu
>>>> Jens
>>>>
>>>>
>>>> Am 26.10.2023 um 17:04 schrieb Vicente Rossello:
>>>>> We are using the production stage with
>>>>> javax.faces.FACELETS_REFRESH_PERIOD
>>>>> = 1. It's faster and more similar to production, just try it if
>>>>> it
>>>>> solves
>>>>> your problem
>>>>>
>>>>> On Thu, Oct 26, 2023 at 4:18 PM Jens Zurawski <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> The one bundled with TomEE.
>>>>>> myfaces 2.3.7 on TomEE 8.0.6
>>>>>> and  e.g. myfaces 2.3.9 on TomEE 8.0.12
>>>>>>
>>>>>> Am 26.10.2023 um 16:08 schrieb Richard Zowalla:
>>>>>>> Which JSF lib are you using?
>>>>>>>
>>>>>>> Am 26. Oktober 2023 16:00:57 MESZ schrieb Jens Zurawski
>>>>>>> <[email protected]
>>>>>>> :
>>>>>>>> Thank you for your quick reply, Jonathan.
>>>>>>>>
>>>>>>>> It is good to know, that normally the Dev mode should
>>>>>>>> work as
>>>>>>>> fast as
>>>>>> before even with 8.0.15. So it obviously has something to do
>>>>>> with
>>>>>> my
>>>>>> environment, configuration and/or code.
>>>>>>>> And also thank you for the reference to tomitribe.
>>>>>>>> Unfortunately I'm
>>>>>> the only developer on this project, and it will be difficult
>>>>>> to
>>>>>> convince my
>>>>>> customer to an additional support subscription, because he on
>>>>>> production
>>>>>> doesn't have issues ;-) But, ok, that's my problem, and maybe
>>>>>> I'll consult
>>>>>> tomitribe in the future for other projects.
>>>>>>>> So, I'm afraid, for the near future I'm on my own to
>>>>>>>> solve
>>>>>>>> this
>>>>>> problem. Therefore any hints to restrict the places where I
>>>>>> should take a
>>>>>> look will be more than welcome.
>>>>>>>> I'm a senior programmer with 20 years of experience in
>>>>>>>> Java
>>>>>>>> programming
>>>>>> (SE and JSP). But I'm relative new to JSF/JavaEE/JakartaEE
>>>>>> and
>>>>>> sometimes I
>>>>>> have a hard time to understand which component of the whole
>>>>>> stack
>>>>>> is
>>>>>> responsible for what.
>>>>>>>> The project meanwhile has over a 1/4 million lines of
>>>>>>>> code
>>>>>>>> (most of
>>>>>> them in the back-end, but also the front-end is big enough to
>>>>>> not
>>>>>> trace
>>>>>> through every lane if I want to get to the point in this life
>>>>>> :-D
>>>>>> ).
>>>>>>>> The application now runs for nearly 2 years on the
>>>>>>>> customers
>>>>>>>> site and
>>>>>> is very reliable and fast in its daily usage. Apart from
>>>>>> occasional updates
>>>>>> every few month, it will never be restarted or need any
>>>>>> manual
>>>>>> maintenance
>>>>>> at all. So I assume the code is not the badest on earth. At
>>>>>> least
>>>>>> because
>>>>>> of that I'm able to out-rule things like memory leaks, gc
>>>>>> issues
>>>>>> or
>>>>>> bottlenecks (max Heap is never reached, neither half of it,
>>>>>> even
>>>>>> after
>>>>>> running for month).
>>>>>>>> Of course there could be aspects of JSF which I may not
>>>>>>>> have
>>>>>>>> understood
>>>>>> correctly until now. But because there are so many new things
>>>>>> to
>>>>>> a JSF
>>>>>> beginner like me, I'm running out of ideas what this could
>>>>>> be. It
>>>>>> is
>>>>>> something that worked well 'til TomEE 8.0.6 (Prod and Dev
>>>>>> mode)
>>>>>> and
>>>>>> suddenly don't work anymore on 8.0.12 in Dev mode (but still
>>>>>> works great in
>>>>>> Prod mode). There are only minor versions in-between. So I
>>>>>> think
>>>>>> there were
>>>>>> no revolutionary changes anywhere. My hope was (and still
>>>>>> is),
>>>>>> that anybody
>>>>>> here on the list might have had a similar problem and has
>>>>>> solved
>>>>>> it, or
>>>>>> maybe someone knows about some changes in Dev mode which
>>>>>> could
>>>>>> explain such
>>>>>> behaviour (don't know, maybe xml parsing or component tree
>>>>>> generation has
>>>>>> changed, or scope handling of managed beans, or whatever
>>>>>> could
>>>>>> cause some
>>>>>> code to be entered far more often than before, or some
>>>>>> cache/pool
>>>>>> was
>>>>>> removed or has different behaviour... such things). Then it
>>>>>> would
>>>>>> be far
>>>>>> more easy for me to track down the real problem.
>>>>>>>> Anyway, when I get some spare time (currently have some
>>>>>>>> deadlines in
>>>>>> sight), I'll try to build a small test case. Either I can get
>>>>>> a
>>>>>> grip on the
>>>>>> issue myself on doing this, or I can supply it to anyone who
>>>>>> might want to
>>>>>> have a look at it.
>>>>>>>> cu
>>>>>>>> Jens
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 26.10.2023 um 14:01 schrieb Jonathan S. Fisher:
>>>>>>>>>> newer versions are so incredibly slow in dev mode
>>>>>>>>> I can assure you that Dev/Prod mode works very very
>>>>>>>>> swiftly
>>>>>>>>> on 8.0.15
>>>>>>>>> for sizable applications. For example, I have a giant
>>>>>>>>> application
>>>>>>>>> (thousands of LOC, 50+ jsf views) that has sub-8ms
>>>>>>>>> response
>>>>>>>>> times
>>>>>>>>> (minus database time). More than likely they are doing
>>>>>>>>> something they
>>>>>>>>> should not be doing with JSF apis and are causing
>>>>>>>>> problems.
>>>>>>>>>
>>>>>>>>> I would check the obvious things first with VisualVM:
>>>>>>>>> heap
>>>>>>>>> size, gc
>>>>>>>>> frequency, etc. You can also run their profiler and see
>>>>>>>>> if
>>>>>>>>> you can
>>>>>>>>> spot anything immediately obvious.
>>>>>>>>>
>>>>>>>>> After that, the easiest way to locate the code slowing
>>>>>>>>> you
>>>>>>>>> down is
>>>>>>>>> bisection. Cut half the code out, run, and continue
>>>>>>>>> cutting
>>>>>>>>> until you
>>>>>>>>> can locate the problematic code. Then after that,
>>>>>>>>> isolate a
>>>>>>>>> test case
>>>>>>>>> where flipping a boolean flag triggers the issue and
>>>>>>>>> post
>>>>>>>>> the results
>>>>>>>>> here.
>>>>>>>>>
>>>>>>>>> I would get ahold of: https://www.tomitribe.com who are
>>>>>>>>> literal
>>>>>>>>> experts in this stuff. They offer professional services
>>>>>>>>> to
>>>>>>>>> handle
>>>>>>>>> exactly these situations and can work directly in your
>>>>>>>>> codebase to
>>>>>>>>> help you find and fix the problem.
>>>>>>>>>
>>>>>>>>> On Thu, Oct 26, 2023 at 5:48 AM Jens Zurawski
>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>> Hi altogether,
>>>>>>>>>>
>>>>>>>>>> I'm new to this list and hope it's the right place to
>>>>>>>>>> ask
>>>>>>>>>> this
>>>>>> question.
>>>>>>>>>> If not, and someone know the right place to ask this,
>>>>>>>>>> please give
>>>>>> advise.
>>>>>>>>>> I'm developing a big JSF Application for a customer.
>>>>>>>>>> It's
>>>>>>>>>> a long
>>>>>> running
>>>>>>>>>> project and development and when I started with it,
>>>>>>>>>> TomEE
>>>>>>>>>> 8.0.6 was
>>>>>> the
>>>>>>>>>> most recent Version of TomEE. The switch to Jakarta
>>>>>>>>>> is
>>>>>>>>>> planned for
>>>>>> next
>>>>>>>>>> year, so I'm still on the 8.x path for the time being
>>>>>>>>>> and
>>>>>>>>>> wasn't able
>>>>>> to
>>>>>>>>>> test if this problem still exists in the 9.x branch.
>>>>>>>>>>
>>>>>>>>>> The problem:
>>>>>>>>>> On the customers site I'm with the latest TomEE
>>>>>>>>>> version
>>>>>>>>>> 8.0.15
>>>>>> (running
>>>>>>>>>> on Java 11) and everything works fine, because it is
>>>>>>>>>> running in
>>>>>>>>>> production mode. But in my development environment
>>>>>>>>>> I'm
>>>>>>>>>> still stuck
>>>>>> with
>>>>>>>>>> 8.0.6, because there I need the development mode
>>>>>>>>>> (javax.faces.PROJECT_STAGE: Development). I need to
>>>>>>>>>> be
>>>>>>>>>> able to see
>>>>>>>>>> changes in facelets without restarting everything
>>>>>>>>>> everytime, and to
>>>>>> get
>>>>>>>>>> more detailed error messages.
>>>>>>>>>> My attempts to update my dev TomEE to something newer
>>>>>>>>>> than 8.0.6 all
>>>>>>>>>> failed, because all (at least all I've tested so far)
>>>>>>>>>> newer versions
>>>>>> are
>>>>>>>>>> so incredibly slow in dev mode, that it's unbearable
>>>>>>>>>> to
>>>>>>>>>> use the
>>>>>>>>>> application. Several very looong seconds on every
>>>>>>>>>> request
>>>>>>>>>> (even the
>>>>>>>>>> little AJAX requests in a view) is simply not a
>>>>>>>>>> practical
>>>>>>>>>> environment.
>>>>>>>>>> After switching to Production mode everything works
>>>>>>>>>> very
>>>>>>>>>> fast, and all
>>>>>>>>>> my views have response times of very few ms (even the
>>>>>>>>>> big
>>>>>>>>>> ones with
>>>>>> max.
>>>>>>>>>> around 400ms). When switching back to Development
>>>>>>>>>> mode I
>>>>>>>>>> have response
>>>>>>>>>> times of up to 30s on big views. Not with 8.0.6,
>>>>>>>>>> there
>>>>>>>>>> even in
>>>>>>>>>> development mode it's reasonable fast in not getting
>>>>>>>>>> higher than 1s.
>>>>>>>>>>
>>>>>>>>>> My question:
>>>>>>>>>> What causes this enormous performance degradation?
>>>>>>>>>> I'm
>>>>>>>>>> hoping, it is
>>>>>>>>>> just a configuration which now has another default
>>>>>>>>>> value
>>>>>>>>>> or the like.
>>>>>> If
>>>>>>>>>> yes, maybe someone can point me in the right
>>>>>>>>>> direction of
>>>>>>>>>> where to
>>>>>> find
>>>>>>>>>> this configuration? If it's not a configuration
>>>>>>>>>> thing:
>>>>>>>>>> what can I do
>>>>>> to
>>>>>>>>>> get around this?
>>>>>>>>>>
>>>>>>>>>> I haven't tested all Versions of TomEE from 8.0.6 to
>>>>>>>>>> 8.0.15, so I
>>>>>> can't
>>>>>>>>>> say at what version exactly this behaviour changes.
>>>>>>>>>> If it
>>>>>>>>>> helps or is
>>>>>>>>>> needed, I can make some tests to find the exact
>>>>>>>>>> version
>>>>>>>>>> where this
>>>>>>>>>> happens. Versions I've already tested are: 8.0.12,
>>>>>>>>>> 8.0.13
>>>>>>>>>> and 8.0.15.
>>>>>>>>>> All of them are very slow in dev mode. If you need
>>>>>>>>>> more
>>>>>>>>>> information,
>>>>>>>>>> I'll try to provide it.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance for any help
>>>>>>>>>> cu
>>>>>>>>>> Jens
>>>>>>>>>>
>>>>>>>>>>
>>>>>>