[JIRA] Updated: (CC-991) Piping STDOUT->STDIN of scripts (piped exec builder)

     [ http://jira.public.thoughtworks.org/browse/CC-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Tihelka updated CC-991:
------------------------------

    Priority: Major  (was: Minor)

Hallo.
Just recently we have found two issues with the PipedExecBuilder:

The first was the memory problem - one of our scripts generated so much data, which were cached by PipedExecBuilder as they are distributed to several other scripts, that CC used all memory which it was allowed to use, and java runtime terminated itself.

The second appeared when we enabled debug output of piped exec builder. It caused that also binary data passed through scripts were printed to the log, messing it to virtually unreadable state.

Fortunately, both issues are very easy to fix, introducing trivial changes to the PipedExecBuilder implementation:

The first can be bypassed by increasing the -Xmx value. It was, however, not very suitable for our server (other programs must run on it as well). Therefore, we defined new optional configuration attribute 'gzipstdout' for the builder, which causes that the data will be kept gzipped in the cache (transparently gzipped by PipedExecBuilder when read from the STDOUT and unzipped before written to the piped STDIN). Although it requires some additional computer power, it uses about five to ten times less memory now. Without the attribute defined, the data are kept  as-are in the cache.

To solve the second issue, we defined one more configuration attribute 'binarystdout', which simply disables the printing of piped data in debug mode (note that piped data can only be printed in debug mode).

I can easily prepare the patch solving those issues (including unit tests and documentation), but I would prefer to wait until the earlier issues (those targetted by PipedExecBuilder_10.patch) are fixed.

Thank you very much,
DT

> Piping STDOUT->STDIN of scripts (piped exec builder)
> ----------------------------------------------------
>
>                 Key: CC-991
>                 URL: http://jira.public.thoughtworks.org/browse/CC-991
>             Project: CruiseControl
>          Issue Type: New Feature
>          Components: Core Application
>         Environment: Any
>            Reporter: Daniel Tihelka
>         Assigned To: Dan Rollo
>            Priority: Major
>         Attachments: Documentation.patch01, PipedExecBuilder-07_1.patch, PipedExecBuilder.java, PipedExecBuilder.patch01, PipedExecBuilder.patch02, PipedExecBuilder.patch03, PipedExecBuilder.patch04, PipedExecBuilder.patch05, PipedExecBuilder.patch06, PipedExecBuilder.patch07, PipedExecBuilder_08.patch, PipedExecBuilder_09.patch, PipedExecBuilder_10.patch, PipedExecBuilderTest.java, StdoutBuffer.java, StdoutBufferTest.java
>
>   Original Estimate: 2 weeks
>  Remaining Estimate: 2 weeks
>
> The build process may sometime consist of the set of scripts piped one into another (the unix-way of filtering). This issue targets CruiseControl extension which would allow it.
> Almost a year ago, I have created PipedExecBuilder class as 3rd party CC plugin. It contains several scripts to execute, while the output of any script can be piped into the input of an arbitrary number of other scripts. Moreover, the execution of a script may be hold until another script is running (to prevent several scripts with large memory requirements running at the same time).
> The configuration may look as follows:
> <schedule>
>    <pipedexec>
>      <exec id="" command="" args="" />
>      <exec id="" command="" args="" pipefrom="" />
>      <exec id="" command="" args="" waitfor="" />
>      <exec id="" command="" args="" pipefrom="" waitfor="" />
>    </pipedexec>
> </schedule>
> where the <exec /> tag is the same as the original <exec /> tag, with additional attributes:
> - id - unique identifier of the script for the piping
> - pipefrom - the id of script from which to pipe stdout into the stdin of the current script (optional)
> - waitfor - the id of script to wait for - it holds the execution of the current script until the script which we wait for terminates.
> The pipedexec builder tries to run as many scripts in parallel as possible, while it finishes when all the scripts were finished, or when a script failed.
> For further details see cruisecontrol-devel mailing list thread:
> http://sourceforge.net/mailarchive/forum.php?thread_name=201005271341.32553.dtihelka%40kky.zcu.cz&forum_name=cruisecontrol-devel
> Kind regards,
> DanT

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.public.thoughtworks.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

------------------------------------------------------------------------------
Lotusphere 2011
Register now for Lotusphere 2011 and learn how
to connect the dots, take your collaborative environment
to the next level, and enter the era of Social Business.
http://p.sf.net/sfu/lotusphere-d2d