Re: rough edges

This is a multi-part message in MIME format.
--===============3041188371699906357==
Content-Type: multipart/alternative;
 boundary="------------020206050303010209010806"

This is a multi-part message in MIME format.
--------------020206050303010209010806
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Hi,

>> general silva
>> =======
>>
>> * I've found minimal documentation on creating a silva service and having it be zmi addable.
>> To figure this out, I needed to debug in the startup process and the grokker for zmiobjects in order
>> to learn my interface for the service needs to extend ISilvaLocalService.  There's no mention of this in the Silva 3.0 developer documentation for services either.
>>
>     Yes the documentation is not complete, but this technical documentation is rather a recent thing, that still need work.
>
>     I worked on it recently, but there is still much to be done, concerning the services in 3.0 for example (few things got improved).
Please keep improving it!  It is so incredibly helpful.  I'm fairly good 
at reading the source, but comments are often lacking or aren't 
descriptive enough.  To me these docs serve as a good companion to the 
source, providing more of the "how do I use this" rather than "what this 
does".
>> zeam
>> ====
>>
>> * zeam forms (the<form>  tag) has no 'name' or 'id' attribute.  As such it impossible to test
>> zeam forms using infrae.testbrowser's "get_form" method.  I have a local clone and branch of zeam, and have adjusted some of the form templates to simply put the form prefix as the name attribute.  Not the best, but quick enough to enable writing of functional tests using infrae.testbrowser.
>>
>     They should have a name, in 2.3. That's how I test it using infrae.testbrowser, and there is rather correct functional testing suite in 2.3 (compared to before), so I am sure of it. If name is missing in some the templates, you should have posted it on the mailing list, and I would have gladly fixed it (or sent me a patch).

Sure.  zmisubformtemplate is missing a 'name' attribute.  You can see my 
patch here:
https://hg.infrae.com/zeam.form.silva/rev/47b1b129f46e
>> * If a zeam form fails to initialize you get a 404 Not Found, and no error is
>> recorded in the error log.  Example: a custom composed form with an __init__
>> that creates subforms based on criteria other than adapters, i.e. a subform
>> for each value of a property on the content object.  If the __init__ fails, a 404 is raised.
>>
>    Yes, that's a limitation of Zope. In Zope, views are adapters. They you have an exception raised in an __init__ of an adapter, the ZCA raise a ComponentLookupError (same exception that is raised when no adapter is found). This is translated to a 404 not found in case of a view.
> I as well hate this feature, if you can call it feature, and lost many hours because of it. This is not specific to zeam form or Silva.
>
>    If you wish to change the subforms, and that your are not sure that your code won't trigger errors, I would recommend doing it in update, before calling the super() to the call the update of the Composed form (that will call update on the sub-forms). This will be done before any form business is done (which is handled by updateForm and called after update).
Thanks for the explanation.  It's sad but encouraging to hear you have 
struggled with it too. The approach I took is a pattern I developed 
about a year ago.  I don't recall the full specifics, but I tried doing 
this in the update form (replacing the subforms with a custom list) and 
it didn't work.
>> * what is a field / widget to use to manage an arbitrary set (of strings)?
>> I want to be able to add strings to the set and remove strings from the
>> set.  I thought a textarea would work, but I ended up creating a custom
>> DataManager to convert the text area value into a set and back into
>> a list.  I see that the tuple schema field has a difficult to use UI, is there something better?
>>
>    Kit asked me to create a 'lines' widget. It is located in zeam.form.silva (use. mode = 'lines' on your zeam field). The correct fashion for this would be to create a widget, that extract to the correct type of data (a set of string). DataManager have the only responsibility of setting (and getting, deleting now) the value on the content, when the content is an object, or a dictionary, or a reference.
I tried setting mode="lines" in with a schema.Tuple having a value_type 
of schema.URI, and I received a componentlookuperror.  I agree the 
"correct" way would be to create a new widget and widget extractor for 
this one thing.
>> infrae.wsgi
>> =======
>>
>> * I would like ability to write to the error log from code sources while swallowing the error (if a CS has an error, the page should still display)
>>
>    This is fixed in 3.0, and was regarded as a bug. And there is a function you can call if you wish to add errors to the error log, it is possible.
Was this a fix in infrae.wsgi, or where else?  Can it be used in Silva 
2.3, or back-ported?
>> * I would like ability to add additional exceptions to ignore in a
>> persistent manner, e.g. in zope.conf or something.  The old zope error log
>> also recorded a full suite of data about an error; all we get is a short list
>> of state variables and a traceback. I usually find the recorded state to be
>> insufficient.  It would also be nice to be able to have more than 25 errors in
>> the log, and an improved errorlog view (collapsible errors?) would also be helpful.
>>
>> Basically, I miss the old zope error log :-)
>>
>    I didn't use the Zope error log because:
>
>    1. There is far less information written in the filesystem log, and that is the information that our customers provides us with. Now you have the URL, referer, user, the various published Zope objects in this log. Before you only had a traceback without context.
>
>    2. Using the log through the web interface is not really useful, as far as I think, because even the Zope error log only keep the latest errors, and usually not the most interesting ones.
>
>    2. When you have an error logged in the Zope error log, it often create a commit (even if the error is not stored in the database). Having lot of errors creates conflict errors, that bring down the site, even if parts of it works.
>
>    3. Lot of information is lost in the Zope error log. It actually going through 5 or 6 different API, loosing and replacing correct information with wrong one, and at the end, when it renders the error message to the user, well you don't always have the correct context, error, message. Having correct error pages is more important to me than a buggy, anyway unusable error log that is the Zope one.
>
>    4. Going through all those API make rendering an error absolutely slow.
>
>    But for sure, we can improve the current web interface, and make the list of exception to ignore configurable. I already thought of that, some customers have extra exceptions they want to be ignored as well, but never got the time to do it.
I agree with all of what you say about the old error log.  From a 
consultant stand point it really makes sense.  But from a silva manager 
standpoint, the old error log was nice.  A richer set of context data 
for each error, improvements to ignored exceptions, and a nicer UI would 
be much appreciated.
>> Writing tests
>> ========
>>
>> Silva seems to be in between two unit test frameworks.  I say "seems" because I'm not sure, and I find getting the initial testing framework set up for a new package to be time consuming and frought with dead ends.  I typically spend a day getting the layer and inital test case setup to install the packages and products I need, install the extensions, and such.  It's a good thing I'm persistent or I would have given up writing tests long ago.
>>
>    The new way to write test introduced in 2.3 is I think way simpler, and more reliable than the old one. First of all, we write test by using directly the TestCase from Python, and not some mega class with thousands of methods. Second of all, you write test in the same environment than your code will be used in production, not some fake environment that behave differently, and does half of the things differently.
I'm very happy about that.
>> Is there any thorough documentation on setting up a complete testing suite for a
>> new silva extension?  I usually end up digging through poorly documented tests
>> for Silva trying to find what I need.  There's little documentation on setting
>> up a configure.zcml for testing and what's needed in the layer and such.
>     That is something I can improve in the developer documentation.
Thank you!
>> * How do I test a local service in a publication or folder (NOT the silva root)?
>> E.g. how do I make a folder or publication a site in order to add a local service?
>> One would think there would be tests for local
>> services already, and that there would be a standard body of helper functions
>> to do these sorts of things.  There seems to be a transition in process for
>> testing suites, and it's hard to know what's where.
>>
>    You use the corresponding adapter, ISiteManager, that is used in real life. You can have a look at the tests for the customization service in silva.core.layout if you want, or the test for the SiteManager adapter in Products.Silva.
>
>    I am against helper functions in tests. They means that: your API sucks and is not usable. Over time, they will do things differently than what your application is doing in real life. And doing so means you are not testing your application, but building something else.
>
>    Creating a local site with the real adapter is one line of code, and one import (from interfaces, that would have already imported I think anyway). This is as much of code as calling an helper function to do it.
>
>    (http://docs.infrae.com/silva/trunk/api/folders.html#managing-local-sites, yes that was not in the 2.3 documentation).
Partly my point was to say that the Silva framework is huge and while 
single apis are typically designed ok, digging through them all to find 
the one pattern I'm looking for, which I need in order to test what I 
actually want to test, is a pain.  I disagree with the "your api sucks" 
comment.  Even your new functional tests have helper functions to do 
things that are common in the ui (like add an object) -- does that mean 
your UI sucks?  Or the api layer for the browser?  No, perhaps it's just 
a common pattern that has an irreducible number of steps.  I think the 
same happens in unittests.  One would hope that, as an api drifts, the 
tests would break and need to be adjusted anyways...

Perhaps a better example is the old addObject helper method. I really 
like that as it makes the testing code uniform and simple.  I have put 
copies of it in most of my test suites.
>> * The TestRequest used by Silva only partially implements a real request object. As such
>> it is impossible to write any tests to cover virtual hosting situations.  Is there any way around this?
>>
>    The TestRequest used is the Zope official TestRequest. There is no Zope2 TestRequest, but following its interface it implements the same API (difficult to believe). And yes, it doesn't behave like a real life request, and that is an another example of tests that doesn't do the same thing than in real life.
>
>    Having a fake request in Zope 2 is quite difficult, because most of the code of Zope 2 is in its request (response, publication process, virtual hosting, database connection ...), so you cannot create one easily. Your application object on the test layer should have one I think however (self.root.REQUEST).
>
>    To test things like virtual hosting, I would use infrae.testbrowser, and verify the whole publication process creates correct URLs.
>
>    In infrae.wsgi 2.0, I integrated much more of the Zope 2 publication process in the extension. This was to cut it in pieces instead of having it in one large method like it is done in Zope 2. The advantages are that we can change the virtual hosting process (with silva.app.forest, that provides an official support for multiple virtual hosting inside the same Silva site, and the capacity to ask an URL of an object that is not in the same virtual host, so having for instance having different publication tree for backend and frontend in Silva), or to change the authentication process (using repoze.who instead of Zope 2 acl_users, in case where Silva is used in conjunction with pyramid applications, in a SSO solution), or to change the publication process (we got a request, no longer vali
 d, to disable the acquisition during the URL resolution).

Sounds nice.  Wish it was in Silva 2.3 :-)

peace,
Andy

>
>     I hope this answer some of your questions.
>
>     Regards,
>
>     Sylvain,
>


--------------020206050303010209010806
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <br>
    Hi,<br>
    <br>
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">general silva
=======

* I've found minimal documentation on creating a silva service and having it be zmi addable.
To figure this out, I needed to debug in the startup process and the grokker for zmiobjects in order
to learn my interface for the service needs to extend ISilvaLocalService.  There's no mention of this in the Silva 3.0 developer documentation for services either.

</pre>
      </blockquote>
      <pre wrap="">
   Yes the documentation is not complete, but this technical documentation is rather a recent thing, that still need work.

   I worked on it recently, but there is still much to be done, concerning the services in 3.0 for example (few things got improved).</pre>
    </blockquote>
    Please keep improving it!&nbsp; It is so incredibly helpful.&nbsp; I'm fairly
    good at reading the source, but comments are often lacking or aren't
    descriptive enough.&nbsp; To me these docs serve as a good companion to
    the source, providing more of the "how do I use this" rather than
    "what this does".
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">zeam
====

* zeam forms (the &lt;form&gt; tag) has no 'name' or 'id' attribute.  As such it impossible to test
zeam forms using infrae.testbrowser's "get_form" method.  I have a local clone and branch of zeam, and have adjusted some of the form templates to simply put the form prefix as the name attribute.  Not the best, but quick enough to enable writing of functional tests using infrae.testbrowser.

</pre>
      </blockquote>
      <pre wrap="">
   They should have a name, in 2.3. That's how I test it using infrae.testbrowser, and there is rather correct functional testing suite in 2.3 (compared to before), so I am sure of it. If name is missing in some the templates, you should have posted it on the mailing list, and I would have gladly fixed it (or sent me a patch).</pre>
    </blockquote>
    <br>
    Sure.&nbsp; zmisubformtemplate is missing a 'name' attribute.&nbsp; You can
    see my patch here:<br>
    <meta http-equiv="content-type" content="text/html;
      charset=ISO-8859-1">
    <a href="https://hg.infrae.com/zeam.form.silva/rev/47b1b129f46e">https://hg.infrae.com/zeam.form.silva/rev/47b1b129f46e</a>
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">* If a zeam form fails to initialize you get a 404 Not Found, and no error is
recorded in the error log.  Example: a custom composed form with an __init__
that creates subforms based on criteria other than adapters, i.e. a subform
for each value of a property on the content object.  If the __init__ fails, a 404 is raised.

</pre>
      </blockquote>
      <pre wrap="">
  Yes, that's a limitation of Zope. In Zope, views are adapters. They you have an exception raised in an __init__ of an adapter, the ZCA raise a ComponentLookupError (same exception that is raised when no adapter is found). This is translated to a 404 not found in case of a view.
I as well hate this feature, if you can call it feature, and lost many hours because of it. This is not specific to zeam form or Silva.

  If you wish to change the subforms, and that your are not sure that your code won't trigger errors, I would recommend doing it in update, before calling the super() to the call the update of the Composed form (that will call update on the sub-forms). This will be done before any form business is done (which is handled by updateForm and called after update).</pre>
    </blockquote>
    Thanks for the explanation.&nbsp; It's sad but encouraging to hear you
    have struggled with it too. The approach I took is a pattern I
    developed about a year ago.&nbsp; I don't recall the full specifics, but
    I tried doing this in the update form (replacing the subforms with a
    custom list) and it didn't work.
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">* what is a field / widget to use to manage an arbitrary set (of strings)?
I want to be able to add strings to the set and remove strings from the
set.  I thought a textarea would work, but I ended up creating a custom
DataManager to convert the text area value into a set and back into
a list.  I see that the tuple schema field has a difficult to use UI, is there something better?

</pre>
      </blockquote>
      <pre wrap="">
  Kit asked me to create a 'lines' widget. It is located in zeam.form.silva (use. mode = 'lines' on your zeam field). The correct fashion for this would be to create a widget, that extract to the correct type of data (a set of string). DataManager have the only responsibility of setting (and getting, deleting now) the value on the content, when the content is an object, or a dictionary, or a reference.</pre>
    </blockquote>
    I tried setting mode="lines" in with a schema.Tuple having a
    value_type of schema.URI, and I received a componentlookuperror.&nbsp; I
    agree the "correct" way would be to create a new widget and widget
    extractor for this one thing.
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">infrae.wsgi
=======

* I would like ability to write to the error log from code sources while swallowing the error (if a CS has an error, the page should still display)

</pre>
      </blockquote>
      <pre wrap="">
  This is fixed in 3.0, and was regarded as a bug. And there is a function you can call if you wish to add errors to the error log, it is possible.</pre>
    </blockquote>
    Was this a fix in infrae.wsgi, or where else?&nbsp; Can it be used in
    Silva 2.3, or back-ported?
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">* I would like ability to add additional exceptions to ignore in a
persistent manner, e.g. in zope.conf or something.  The old zope error log
also recorded a full suite of data about an error; all we get is a short list
of state variables and a traceback. I usually find the recorded state to be
insufficient.  It would also be nice to be able to have more than 25 errors in
the log, and an improved errorlog view (collapsible errors?) would also be helpful.

Basically, I miss the old zope error log :-)

</pre>
      </blockquote>
      <pre wrap="">
  I didn't use the Zope error log because:

  1. There is far less information written in the filesystem log, and that is the information that our customers provides us with. Now you have the URL, referer, user, the various published Zope objects in this log. Before you only had a traceback without context.

  2. Using the log through the web interface is not really useful, as far as I think, because even the Zope error log only keep the latest errors, and usually not the most interesting ones.

  2. When you have an error logged in the Zope error log, it often create a commit (even if the error is not stored in the database). Having lot of errors creates conflict errors, that bring down the site, even if parts of it works.

  3. Lot of information is lost in the Zope error log. It actually going through 5 or 6 different API, loosing and replacing correct information with wrong one, and at the end, when it renders the error message to the user, well you don't always have the correct context, error, message. Having correct error pages is more important to me than a buggy, anyway unusable error log that is the Zope one.

  4. Going through all those API make rendering an error absolutely slow.

  But for sure, we can improve the current web interface, and make the list of exception to ignore configurable. I already thought of that, some customers have extra exceptions they want to be ignored as well, but never got the time to do it.</pre>
    </blockquote>
    I agree with all of what you say about the old error log.&nbsp; From a
    consultant stand point it really makes sense.&nbsp; But from a silva
    manager standpoint, the old error log was nice.&nbsp; A richer set of
    context data for each error, improvements to ignored exceptions, and
    a nicer UI would be much appreciated.
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">Writing tests
========

Silva seems to be in between two unit test frameworks.  I say "seems" because I'm not sure, and I find getting the initial testing framework set up for a new package to be time consuming and frought with dead ends.  I typically spend a day getting the layer and inital test case setup to install the packages and products I need, install the extensions, and such.  It's a good thing I'm persistent or I would have given up writing tests long ago.

</pre>
      </blockquote>
      <pre wrap="">
  The new way to write test introduced in 2.3 is I think way simpler, and more reliable than the old one. First of all, we write test by using directly the TestCase from Python, and not some mega class with thousands of methods. Second of all, you write test in the same environment than your code will be used in production, not some fake environment that behave differently, and does half of the things differently.</pre>
    </blockquote>
    I'm very happy about that.
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">Is there any thorough documentation on setting up a complete testing suite for a
new silva extension?  I usually end up digging through poorly documented tests
for Silva trying to find what I need.  There's little documentation on setting
up a configure.zcml for testing and what's needed in the layer and such.
</pre>
      </blockquote>
      <pre wrap="">
   That is something I can improve in the developer documentation.</pre>
    </blockquote>
    Thank you!
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">* How do I test a local service in a publication or folder (NOT the silva root)?
E.g. how do I make a folder or publication a site in order to add a local service?
One would think there would be tests for local
services already, and that there would be a standard body of helper functions
to do these sorts of things.  There seems to be a transition in process for
testing suites, and it's hard to know what's where.

</pre>
      </blockquote>
      <pre wrap="">
  You use the corresponding adapter, ISiteManager, that is used in real life. You can have a look at the tests for the customization service in silva.core.layout if you want, or the test for the SiteManager adapter in Products.Silva.

  I am against helper functions in tests. They means that: your API sucks and is not usable. Over time, they will do things differently than what your application is doing in real life. And doing so means you are not testing your application, but building something else.

  Creating a local site with the real adapter is one line of code, and one import (from interfaces, that would have already imported I think anyway). This is as much of code as calling an helper function to do it.

  (<a class="moz-txt-link-freetext" href="http://docs.infrae.com/silva/trunk/api/folders.html#managing-local-sites">http://docs.infrae.com/silva/trunk/api/folders.html#managing-local-sites</a>, yes that was not in the 2.3 documentation).</pre>
    </blockquote>
    Partly my point was to say that the Silva framework is huge and
    while single apis are typically designed ok, digging through them
    all to find the one pattern I'm looking for, which I need in order
    to test what I actually want to test, is a pain.&nbsp; I disagree with
    the "your api sucks" comment.&nbsp; Even your new functional tests have
    helper functions to do things that are common in the ui (like add an
    object) -- does that mean your UI sucks?&nbsp; Or the api layer for the
    browser?&nbsp; No, perhaps it's just a common pattern that has an
    irreducible number of steps.&nbsp; I think the same happens in
    unittests.&nbsp; One would hope that, as an api drifts, the tests would
    break and need to be adjusted anyways...<br>
    <br>
    Perhaps a better example is the old addObject helper method. I
    really like that as it makes the testing code uniform and simple.&nbsp; I
    have put copies of it in most of my test suites.
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <blockquote type="cite">
        <pre wrap="">* The TestRequest used by Silva only partially implements a real request object. As such
it is impossible to write any tests to cover virtual hosting situations.  Is there any way around this?

</pre>
      </blockquote>
      <pre wrap="">
  The TestRequest used is the Zope official TestRequest. There is no Zope2 TestRequest, but following its interface it implements the same API (difficult to believe). And yes, it doesn't behave like a real life request, and that is an another example of tests that doesn't do the same thing than in real life.

  Having a fake request in Zope 2 is quite difficult, because most of the code of Zope 2 is in its request (response, publication process, virtual hosting, database connection ...), so you cannot create one easily. Your application object on the test layer should have one I think however (self.root.REQUEST).

  To test things like virtual hosting, I would use infrae.testbrowser, and verify the whole publication process creates correct URLs.

  In infrae.wsgi 2.0, I integrated much more of the Zope 2 publication process in the extension. This was to cut it in pieces instead of having it in one large method like it is done in Zope 2. The advantages are that we can change the virtual hosting process (with silva.app.forest, that provides an official support for multiple virtual hosting inside the same Silva site, and the capacity to ask an URL of an object that is not in the same virtual host, so having for instance having different publication tree for backend and frontend in Silva), or to change the authentication process (using repoze.who instead of Zope 2 acl_users, in case where Silva is used in conjunction with pyramid applications, in a SSO solution), or to change the publication process (we got a request, no longer valid, 
 to disable the acquisition during the URL resolution).</pre>
    </blockquote>
    <br>
    Sounds nice.&nbsp; Wish it was in Silva 2.3 :-)<br>
    <br>
    peace,<br>
    Andy<br>
    <br>
    <blockquote
      cite="mid:E218DA9A-8E72-429B-8DEB-3C6051A193CE-9ywwomju6vjQT0dZR+AlfA@public.gmane.org"
      type="cite">
      <pre wrap="">

   I hope this answer some of your questions.

   Regards,

   Sylvain,

</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------020206050303010209010806--

--===============3041188371699906357==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
silva-dev mailing list
silva-dev-IAPFreCvJWM6s/[email protected]
https://lists.infrae.com/mailman/listinfo/silva-dev

--===============3041188371699906357==--