Re: Scons swallowing exception stack trace for jinja2.exceptions.TemplateNotFound

Thanks Mats and Bill for your valuable feedback here.

Yes, I’m catching the exception with a try-except and re-raising as a different exception to make forward progress.

Would you like me to file an scons bug?

Thanks
Samir

From: Scons-users <[email protected]> on behalf of [email protected] <[email protected]>
Date: Tuesday, May 14, 2024 at 10:36 PM
To: [email protected] <[email protected]>
Subject: Scons-users Digest, Vol 153, Issue 2
WARNING: This email originated from outside of Qualcomm. Please be wary of any links or attachments, and do not enable macros.

Send Scons-users mailing list submissions to
        [email protected]

To subscribe or unsubscribe via the World Wide Web, visit
        https://pairlist4.pair.net/mailman/listinfo/scons-users
or, via email, send a message with subject or body 'help' to
        [email protected]

You can reach the person managing the list at
        [email protected]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Scons-users digest..."


Today's Topics:

   1. Re: Scons swallowing exception stack trace for
      jinja2.exceptions.TemplateNotFound (Mats Wichmann)
   2. Re: Scons swallowing exception stack trace for
      jinja2.exceptions.TemplateNotFound (Bill Deegan)


----------------------------------------------------------------------

Message: 1
Date: Tue, 14 May 2024 08:04:23 -0600
From: Mats Wichmann <[email protected]>
To: [email protected]
Subject: Re: [Scons-users] Scons swallowing exception stack trace for
        jinja2.exceptions.TemplateNotFound
Message-ID: <[email protected]>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 5/13/24 22:16, Samir Mahaboob Khan Kagadkar via Scons-users wrote:
> Hi SCons Team,
>
> I use the Jinja2 templating language
> (https://jinja.palletsprojects.com/en/3.1.x/
> <https://jinja.palletsprojects.com/en/3.1.x/>) to generate scripts from
> within SCons.
>
> Jinja2 raises the TemplateNotFound (jinja2.exceptions.TemplateNotFound)
> exception when, say, a file included in the template cannot be found.
>
> Unfortunately scons does not show the exception stack trace when this is
> raised as part of a builder.
>
> Here is a simple SConstruct that demonstrates this:
>
> import jinja2
>
>  ?def small_builder(target, source, env):
>
>  ???? raise jinja2.exceptions.TemplateNotFound("You should see this
> message")
>
>  ?env=Environment()
>
> env.Command(['dummy/TEST.COMPLETE'], [], small_builder, chdir=0)
>
> Here is the output not showing any stack trace:
>
> $ scons -f <path_to_sconstruct_dir>/SConstruct dummy/TEST.COMPLETE
>
> scons: Reading SConscript files ...
>
> scons: done reading SConscript files.
>
> scons: Building targets ...
>
> small_builder(["dummy/TEST.COMPLETE"], [])
>
> scons: done building targets.
>
> The TemplateNotFound can be raised in a simple standalone python3 script
> invocation:
>
> import jinja2
>
> raise jinja2.exceptions.TemplateNotFound("You should see this message")
>
> with the below output
>
> $ python3 test1.py
>
> Traceback (most recent call last):
>
>  ? File "test1.py", line 2, in <module>
>
>  ??? raise jinja2.exceptions.TemplateNotFound("You should see this message")
>
> jinja2.exceptions.TemplateNotFound: You should see this message
>
> Other exceptions work in SCons builders, of course. For example:
>
> import jinja2
>
>  ?def small_builder(target, source, env):
>
>  ???? #raise jinja2.exceptions.TemplateNotFound("You should see this
> message")
>
>  ???? raise ValueError("You should see this message")
>
>  ?env=Environment()
>
> env.Command(['dummy/TEST.COMPLETE'], [], small_builder, chdir=0)
>
> with the following output:
>
> scons: Reading SConscript files ...
>
> scons: done reading SConscript files.
>
> scons: Building targets ...
>
> small_builder(["dummy/TEST.COMPLETE"], [])
>
> scons: *** [dummy/TEST.COMPLETE] ValueError : You should see this message
>
> Traceback (most recent call last):
>
>  ? File ?<?.>/SCons/Action.py", line 1441, in execute
>
>  ??? result = self.execfunction(target=target, source=rsources, env=env)
>
>  ? File "<?.> /SConstruct", line 5, in small_builder
>
>  ??? raise ValueError("You should see this message")
>
> ValueError: You should see this message
>
> scons: building terminated because of errors.

>
> Could you please take a look?

SCons has a routine which tries to synthesize a BuildError exception
upon detecting other exceptions that may have been raised during the
call to the action function.  If doesn't do that right for the jinja
template exception - which I see is itself synthesized. I don't really
see how to cause it to do better at the moment.

convert_to_BuildError recognizes the template error as an instance of
IOError. In that case it tries to do this:

         filename = getattr(status, 'filename', None)

         strerror = getattr(status, 'strerror', str(status))

         errno = getattr(status, 'errno', 2)



         buildError = BuildError(

             errstr=strerror,

             status=errno,

             exitstatus=2,

             filename=filename,

             exc_info=exc_info)

the TemplateNotFound has all three attributes that are checked for - set
to None, so the getattr defaults aren't used. The BuildError thus ends
up with an error message and status of None, which not surprisingly,
doesn't yield any interesting message.

I suppose the attribute check could be a little more pedantic, like this:

         strerror = getattr(status, 'strerror', None)

         if strerror is None:

             strerror = str(status)

@bdbaddog - any thoughts?




------------------------------

Message: 2
Date: Tue, 14 May 2024 09:08:56 -0700
From: Bill Deegan <[email protected]>
To: SCons users mailing list <[email protected]>
Subject: Re: [Scons-users] Scons swallowing exception stack trace for
        jinja2.exceptions.TemplateNotFound
Message-ID:
        <CAEyG4CEEgxgHC9fz1H-Fw1e562JfHwS-caZqr9eZgOacndh4+A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I'm guessing you've created a python action to process the jinja2 template?
If so you should catch whatever exceptions jinja2 may through and re-raise
them as the relevant SCons exception.

That's likely the best fit.

Doing a lot of processing, especially file operations in python actions,
can hold the GIL and limit parallelism, and generally not encouraged.
So another way to resolve that is to move your jinja2 processing to a
standalone script, then when that script fails, scons would see the error.

-Bill

On Tue, May 14, 2024 at 7:04?AM Mats Wichmann <[email protected]> wrote:

> On 5/13/24 22:16, Samir Mahaboob Khan Kagadkar via Scons-users wrote:
> > Hi SCons Team,
> >
> > I use the Jinja2 templating language
> > (https://jinja.palletsprojects.com/en/3.1.x/
> > <https://jinja.palletsprojects.com/en/3.1.x/>) to generate scripts from
> > within SCons.
> >
> > Jinja2 raises the TemplateNotFound (jinja2.exceptions.TemplateNotFound)
> > exception when, say, a file included in the template cannot be found.
> >
> > Unfortunately scons does not show the exception stack trace when this is
> > raised as part of a builder.
> >
> > Here is a simple SConstruct that demonstrates this:
> >
> > import jinja2
> >
> >   def small_builder(target, source, env):
> >
> >       raise jinja2.exceptions.TemplateNotFound("You should see this
> > message")
> >
> >   env=Environment()
> >
> > env.Command(['dummy/TEST.COMPLETE'], [], small_builder, chdir=0)
> >
> > Here is the output not showing any stack trace:
> >
> > $ scons -f <path_to_sconstruct_dir>/SConstruct dummy/TEST.COMPLETE
> >
> > scons: Reading SConscript files ...
> >
> > scons: done reading SConscript files.
> >
> > scons: Building targets ...
> >
> > small_builder(["dummy/TEST.COMPLETE"], [])
> >
> > scons: done building targets.
> >
> > The TemplateNotFound can be raised in a simple standalone python3 script
> > invocation:
> >
> > import jinja2
> >
> > raise jinja2.exceptions.TemplateNotFound("You should see this message")
> >
> > with the below output
> >
> > $ python3 test1.py
> >
> > Traceback (most recent call last):
> >
> >    File "test1.py", line 2, in <module>
> >
> >      raise jinja2.exceptions.TemplateNotFound("You should see this
> message")
> >
> > jinja2.exceptions.TemplateNotFound: You should see this message
> >
> > Other exceptions work in SCons builders, of course. For example:
> >
> > import jinja2
> >
> >   def small_builder(target, source, env):
> >
> >       #raise jinja2.exceptions.TemplateNotFound("You should see this
> > message")
> >
> >       raise ValueError("You should see this message")
> >
> >   env=Environment()
> >
> > env.Command(['dummy/TEST.COMPLETE'], [], small_builder, chdir=0)
> >
> > with the following output:
> >
> > scons: Reading SConscript files ...
> >
> > scons: done reading SConscript files.
> >
> > scons: Building targets ...
> >
> > small_builder(["dummy/TEST.COMPLETE"], [])
> >
> > scons: *** [dummy/TEST.COMPLETE] ValueError : You should see this message
> >
> > Traceback (most recent call last):
> >
> >    File ?<?.>/SCons/Action.py", line 1441, in execute
> >
> >      result = self.execfunction(target=target, source=rsources, env=env)
> >
> >    File "<?.> /SConstruct", line 5, in small_builder
> >
> >      raise ValueError("You should see this message")
> >
> > ValueError: You should see this message
> >
> > scons: building terminated because of errors.
>
> >
> > Could you please take a look?
>
> SCons has a routine which tries to synthesize a BuildError exception
> upon detecting other exceptions that may have been raised during the
> call to the action function.  If doesn't do that right for the jinja
> template exception - which I see is itself synthesized. I don't really
> see how to cause it to do better at the moment.
>
> convert_to_BuildError recognizes the template error as an instance of
> IOError. In that case it tries to do this:
>
>          filename = getattr(status, 'filename', None)
>
>          strerror = getattr(status, 'strerror', str(status))
>
>          errno = getattr(status, 'errno', 2)
>
>
>
>          buildError = BuildError(
>
>              errstr=strerror,
>
>              status=errno,
>
>              exitstatus=2,
>
>              filename=filename,
>
>              exc_info=exc_info)
>
> the TemplateNotFound has all three attributes that are checked for - set
> to None, so the getattr defaults aren't used. The BuildError thus ends
> up with an error message and status of None, which not surprisingly,
> doesn't yield any interesting message.
>
> I suppose the attribute check could be a little more pedantic, like this:
>
>          strerror = getattr(status, 'strerror', None)
>
>          if strerror is None:
>
>              strerror = str(status)
>
> @bdbaddog - any thoughts?
>
>
> _______________________________________________
> Scons-users mailing list
> [email protected]
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20240514/32ae8863/attachment-0001.htm>

------------------------------

Subject: Digest Footer

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users


------------------------------

End of Scons-users Digest, Vol 153, Issue 2
*******************************************

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users