Re: Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

Just as a tip, in Ansible when running tests we normally try to figure out
the absolute executable path and then pass that to all process spawning
functions to be explicit about what we run:
https://github.com/ansible/ansible/blob/6352e67/test/runner/lib/util.py#L129

2018-05-06 2:37 GMT+02:00 Jibun no Kage <[email protected]>:

> Red Hat!  Wow.  I have quite a few friends at Red Hat, US based.  Cool.  I
> worked for a Fortune 50 Bank, for many years, designed several
> virtualization standards for same.  Red Hat has recruited from the same
> Fortunate 50 Bank for years, even a former boss of mine now is at RH.  I
> know a number of the guys on the CloudForms sales and consulting team, if
> you are familiar with CloudForms.
>
> I read through the VirtualEnv documentation, yup, it is just isolation in
> simple form, in my world (of virtualization) we would call this a 'type' of
> application isolation, but... it not as sophisticated as Docker even,
> true.  Agree, I think your comment "My point is that when running app and
> subprocesses you may unintentionally switch versions" is exactly what
> might, likely is happening at some point, this is what I was thinking when
> I said the CherryPy software stack is just exposing something odd... it
> just happens to happen, when CherryPy is used.  It could happen via other
> scenarios as well of course.
>
> To tackle the scenario, I am setting up a 'qualified' reference platform,
> that I am going to make everyone validate against.  The reference platform
> will be explicitly documented, configuration via Ansible, etc., to make
> sure it is always consistent.  Since I will need explicit control of what
> modules, versions etc. used, will be able to have greater control.  I do
> plan to use Pipenv as well, but will need to spend some time with it.
>
> Schorschi
>
>
> On 05/05/2018 17:19, Sviatoslav Sydorenko wrote:
>
>
>
> 2018-05-06 0:48 GMT+02:00 Jibun no Kage <[email protected]>:
>
>> Not to debate, from my perspective, you are talking about virtualization,
>> about application virtualization.  Per the link (
>> https://virtualenv.pypa.io/en/stable/) you sent me, "In all these cases,
>> virtualenv can help you. It creates an environment that has its own
>> installation directories, that doesn’t share libraries with other
>> virtualenv environments (and optionally doesn’t access the globally
>> installed libraries either)."  That is 'application' virtualization from my
>> perspective.  :)
>>
>
> Well, my assumption might be wrong, but I have a feeling that you might
> not realize how virtualenv works. While it contains word "virtual" it's
> still not a typical virtualization, nor isolation. Yes, it creates a
> separate "env" folder with binaries linked against some other (system)
> installation of python and site-packages, which won't collide with that
> system install of python. So you add that thing to $PATH or invoke directly.
> My point is that when running app and subprocesses you may unintentionally
> switch versions when being not careful (for example, when you reset/erase
> env vars available for subshell). And since I don't know anything about how
> you run web app and scripts it is hard for me to point where to look closer.
> So logging executable path and env vars would help you figure out more
> details about this.
>
>
>>
>> As for code examples, I can't publish the explicit code, if I could, I
>> would have.  I realize this limits your assistance, just the same, thanks
>> for the insight, and suggestions, you have helped, I appreciate it.
>>
>
> You don't have to expose any of your super secret business logic or so.
> The rule of thumb is to post minimal amount of similar code, which might
> contain dummy "hello world", but still show how exactly to spawn
> subprocesses, which from your words looks like:
>
> *cp_app.py:*
> import os
> import cherrypy
>
> class RootWebApp:
>     @cherrypy.expose
>     def index(self):
>         exec_result = os.subprocess.run(['/path/to/specific/py/env/bin/python',
> 'some_script.py'])  # might be different and in fact has lots of arguments
> to play with.
>         return exec_result.stdout
>
>     @classmethod
>     def run(cls):
>         cherrypy.quickstart(cls(), '', {})
>
>
> def main():
> *    RootWebApp.run()*
>
>
> *__name__ == '__main__' and main()*
>
>
> *some_script.py: def main():*
>     print('Hello world!')
>
>
> *__name__ == '__main__' and main()*
>
> As you can see, your example doesn't have to include any logic specific to
> your project, but to your env and the way your components interact with
> each other. This would help with troubleshooting.
>
>
>> Since your domain is UA... you are in Ukraine?  I spent most of my
>> summers as a kid in Germany, even got to visit Hungry, Poland and Yugoslavia
>> (when it still existed), even East Germany once, but I never got to the
>> Ukraine.  Maybe when I retire and have time, I can get back to Eastern
>> Europe again.
>>
>
> I used to live in Ukraine for my whole life, but 3 months ago I moved to
> Brno, Czech Republic for Red Hat :)
>
>
>>
>>
>> Schorschi
>>
>>
>> On 05/05/2018 02:28, Sviatoslav Sydorenko wrote:
>>
>> I'm not talking about virtualization at all. I only mentioned
>> https://virtualenv.pypa.io/en/stable/
>> There's also Pipenv which exposes a bit more full feature set
>> https://packaging.python.org/tutorials/managing-dependencies/
>>
>> Besides this, I cannot help you with your problem, because this would
>> require you posting a real code example with a reproducible issue.
>>
>> 2018-05-05 2:57 GMT+02:00 Jibun no Kage <[email protected]>:
>>
>>> I am under the impression you did not see my other note, showing an an
>>> example of the odd scenario I found, where the non-CherryPy environment
>>> software 'stack' accepts an enum based on a simple class definition, but
>>> the CherryPy inclusive environment software 'stack' does not.  Believe
>>> CherryPy is exposing something odd, rather than creating something odd, as
>>> I noted before.
>>>
>>> Yes, virtualization is a best practice, and where and when we can use
>>> it, it is, but not all micro-controller environments can be emulated, for
>>> example BCM processors are very hard to emulate under Qemu.  I have design
>>> and supported virtual labs and large scale production environments for over
>>> 16 years, just about every variant of VMware, KVM, and even Hyper-V (and
>>> Microsoft Virtual Server before Hyper-V).  Moreover, running applications
>>> in slices, in such as things like docker (even LXC) is a best practice when
>>> possible, application virtualization versus OS virtualization.  But
>>> micro-controllers are often so specialized and narrow in resource scope,
>>> that is not an option.  The interesting scenario is when ARM (as now ATOM)
>>> based systems continue to scale up, from 32bit to 64bit, greater than 1GB
>>> RAM, etc., more CPU cores, etc.  Then, realistic partitioning of
>>> micro-controllers will be possible.  Looking forward to it.
>>>
>>> Side note, much of VMware's ESXi server solution, under the hood/behind
>>> the scenes, leverages quite a bit of python in its control (console)
>>> environment.  My first experience with python came from its internal use in
>>> ESXi.
>>>
>>> Schorschi
>>>
>>>
>>> On 05/04/2018 15:16, Sviatoslav Sydorenko wrote:
>>>
>>> First of all, it's not CherryPy picking interpreter, but vice versa -
>>> you run CherryPy app under a certain interpreter (better explicitly chosen).
>>> Also, I advise you to check sys.executable to learn which binary exactly
>>> you use.
>>> Finally, it is best practice to use a virtualenv to isolate environments
>>> for different apps you run.
>>>
>>> --Sviatoslav.
>>>
>>> Sent from my phone, please pardon any typos.
>>>
>>> On Fri, May 4, 2018, 21:45 Jibun no Kage <[email protected]> wrote:
>>>
>>>> Ah... Yes, I know how shebang works.  But, I had not worried about it,
>>>> since the expectation was that everything would be completely python 2.  I
>>>> went snooping... all the 'existing' child scripts, are 'banged' explicitly
>>>> to Python2.  However, one of my test scripts, I was using/learning CherryPy
>>>> was not explicitly qualified.  So I added a bit of code to the CherryPy
>>>> script and left it NOT banged on purpose to see what CherryPy was
>>>> defaulting to:
>>>>
>>>> >>> Sys Get Version 2
>>>> >>> Platform Get Version 2.7.13
>>>>
>>>> So CherryPy defaults to system default, and it is version 2, the child
>>>> scripts are directed to version 2 via shebang.  All the modules used by the
>>>> child scripts were developed in a version 2 context.  Moreover, I modified
>>>> one of the child scripts to report what version bash was defaulting to when
>>>> invoked by bash, a sanity check to validate, and it also reported version 2.
>>>>
>>>> Child script direct invocation...
>>>> [2018-05-04 19:27:13 UTC]  >>> Platform Get Version 2.7.13
>>>> [2018-05-04 19:27:13 UTC]  >>> Sys Get Version 2
>>>>
>>>> Child script indirect invocation via the os shell/bash trick...
>>>> [2018-05-04 19:29:43 UTC]  >>> Platform Get Version 2.7.13
>>>> [2018-05-04 19:29:43 UTC]  >>> Sys Get Version 2
>>>>
>>>> So the OS default is 2, the scripts with os module direction are
>>>> version 2,  the CherryPy defaults to 2, the subprocess defaults to 2.
>>>>
>>>> So there is still something different between module load into CherryPy
>>>> and invoking it, and module load into child script and then invoking it,
>>>> the child script invoking always works, the module load to CherryPy then
>>>> invoking, never works, the modules that exhibit the issue.  I will find it,
>>>> in due course, because like we agreed, really want to import the modules
>>>> and just use them directly, and avoid the os shell/bash trick.  At this
>>>> point it has to be something really odd, tripping it.  Likely, it is some
>>>> funky code in the module that is not very pythontic as it should be, since
>>>> we do a lot of micro-controller/hardware driving python coding.
>>>>
>>>> Not to confuse things, but I already found one issue and resolved it,
>>>> in our development lab.  It had to do with different enum modules clashing,
>>>> I discovered that 'enum34' was conflicting with 'enum' module.  On one
>>>> specific system, the modules installed was incorrect, got to love when
>>>> (many) developers touch stuff... oh... wait, I am one of the developers.
>>>> Ouch.
>>>>
>>>> And, thanks for the comments and insight, appreciate it.
>>>>
>>>> Schorschi
>>>>
>>>>
>>>>
>>>> On 05/03/2018 22:45, Sviatoslav Sydorenko wrote:
>>>>
>>>> 1) You run CherryPy correctly, but it doesn't matter, because you
>>>> obviously run the scripts themselves in a way that they might choose
>>>> different Python env or even version. Do you know about shebang, for
>>>> example? Anyway, you didn't provide sufficient details to know what happens
>>>> exactly.
>>>> 2) Running scripts is not CherryPy specific, it's just Python. There's
>>>> no connection of your problem with CherryPy, this is not something which
>>>> has to be documented.
>>>> 3) My guess is that you just don't know how os.subprocess or bash
>>>> executable lookup works. However I cannot be sure, because you did not show
>>>> any reproducible examples of code you think you have problems with.
>>>>
>>>> --Sviatoslav.
>>>>
>>>> Sent from my phone, please pardon any typos.
>>>>
>>>> On Fri, May 4, 2018, 03:28 Jibun no Kage <[email protected]> wrote:
>>>>
>>>>> That could be, but how would anyone know?  When I run python <script
>>>>> name>.py (which is a CherryPy) script, I am executing the default python
>>>>> for the environment, which happens to be 2.7.x.  The CherryPy examples to
>>>>> not qualify doing anything different.  For example...
>>>>>
>>>>> import cherrypy
>>>>>
>>>>> class HelloWorld(object):
>>>>>
>>>>>     @cherrypy.expose
>>>>>
>>>>>     def index(self):
>>>>>         return "Hello world!"
>>>>>
>>>>> if __name__ == '__main__':
>>>>>     cherrypy.quickstart(HelloWorld())
>>>>>
>>>>> Run as "python <script name>.py"  This is taken straight from the
>>>>> online CherryPy official documentation.  So, I should be 100% python
>>>>> version consistent, regardless of the invocation.  Of it was otherwise,
>>>>> then CherryPy documentation should be explicit on this point, no?  None of
>>>>> the modules I am calling other than CherryPy as imported python 3
>>>>> specific.  Well, less the pending backports, when I get to that point.
>>>>>
>>>>> Schorschi
>>>>>
>>>>> On 05/02/2018 14:53, Sviatoslav Sydorenko wrote:
>>>>>
>>>>> It *is supported*, but you are doing it wrong. So it looks to me that
>>>>> you are spawning python3 interpreter from within python2 process.
>>>>>
>>>>> 2018-05-02 23:36 GMT+02:00 Jibun no Kage <[email protected]>:
>>>>>
>>>>>> Thanks for the suggestions.  Unfortunately, at this time I can't
>>>>>> address Python 3.  Python 2 only environment at the moment.  Odd, CherryPy
>>>>>> documentation states that Python 2.7 is supported, but not surprised
>>>>>> something tripped up.  Long term, as we revise the classes, migrate off of
>>>>>> Python 2, expect to import and use them directly, as you suggest.
>>>>>>
>>>>>> Schorschi
>>>>>>
>>>>>>
>>>>>> On 04/28/2018 23:21, Sviatoslav Sydorenko wrote:
>>>>>>
>>>>>> 1) don't use os module to spawn subprocesses, just have a function in
>>>>>> those modules, import it and call it then.
>>>>>> 2) you probably wrote scripts in a manner/syntax, which is
>>>>>> incompatible with Python 3 and run CherryPy under Python 3. Or maybe just a
>>>>>> way you run scripts is different when you spawn subprocesses. Anyway, I'd
>>>>>> recommend you to not use subprocesses and do normal direct imports.
>>>>>> To mitigate cross python compatibility problems I strongly recommend
>>>>>> you writing scripts using syntax, which works in both or completely switch
>>>>>> to Python 3.
>>>>>> Look for a package distribution called "six", it may provide you some
>>>>>> helpers for cross compat.
>>>>>>
>>>>>> --Sviatoslav.
>>>>>>
>>>>>> Sent from my phone, please pardon any typos.
>>>>>>
>>>>>>
>>>>>> On Sun, Apr 29, 2018, 06:12 <[email protected]> wrote:
>>>>>>
>>>>>>> Yeah, I got a variant of it working.  Not sure it is pretty, but
>>>>>>> works.  We have a lot of existing python scripts we want to continue to use
>>>>>>> from console (Linux).  We just wanted to wrapper these with some simple web
>>>>>>> pages, to make them a bit easier to use.  So, I just imported the os
>>>>>>> module, can execute the script.  This is all internal stuff, nothing that
>>>>>>> has to be production hardened or such.  This lets the guys writing the
>>>>>>> python scripts for console do their thing, and lets me provide another
>>>>>>> avenue for execution.
>>>>>>>
>>>>>>> I plan to swing back around on this one, and do more work, but using
>>>>>>> os module for now works.  I did notice one odd thing, a couple of our
>>>>>>> python classes/functions that work great from console, break when I import
>>>>>>> them to a CherryPy script, and then execute as you note.  One error is that
>>>>>>> bit shifting with the '<<' operator gripes about how it is used.  Only get
>>>>>>> that error, when the given function/routine is imported to CherryPy
>>>>>>> environment.  From console (python 2), no error at all.
>>>>>>>
>>>>>>>
>>>>>>> On Saturday, April 28, 2018 at 1:04:56 PM UTC-7, [email protected]
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Have existing python script file that I want to execute from within
>>>>>>>> a cherrypy web site/page.  How can this be done 100% native to cherrypy?
>>>>>>>> Many examples reference java or CGI mods, that is not acceptable nor should
>>>>>>>> it be IMHO.  There has to be a 100% cherrypi method to do this, no?
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "cherrypy-users" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to cherrypy-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
>>>>>>> To post to this group, send email to cherrypy-users-/[email protected]
>>>>>>> .
>>>>>>> Visit this group at https://groups.google.com/group/cherrypy-users.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "cherrypy-users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to cherrypy-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
>>>>>> To post to this group, send email to cherrypy-users-/[email protected].
>>>>>> Visit this group at https://groups.google.com/group/cherrypy-users.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kind regards,
>>>>>
>>>>> Sviatoslav Sydorenko
>>>>> email: [email protected]
>>>>> cell #: +380978963757
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Kind regards,
>>
>> Sviatoslav Sydorenko
>> email: [email protected]
>> cell #: +380978963757
>> --
>> You received this message because you are subscribed to the Google Groups
>> "cherrypy-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to cherrypy-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
>> To post to this group, send email to cherrypy-users-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
>> Visit this group at https://groups.google.com/group/cherrypy-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>
>
> --
> Kind regards,
>
> Sviatoslav Sydorenko
> email: [email protected]
> cell #: +380978963757
> --
> You received this message because you are subscribed to the Google Groups
> "cherrypy-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cherrypy-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
> To post to this group, send email to cherrypy-users-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
> Visit this group at https://groups.google.com/group/cherrypy-users.
> For more options, visit https://groups.google.com/d/optout.
>
>
>


-- 
Kind regards,

Sviatoslav Sydorenko
email: [email protected]
cell #: +380978963757

-- 
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cherrypy-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
To post to this group, send email to cherrypy-users-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.