Re: running out of file descriptors

I think it is caused by this problem:

http://jira.public.thoughtworks.org/browse/CC-831

Jtf


On Thu, Jan 22, 2009 at 11:14 AM, Joshua Ross <[email protected]> wrote:
> This morning after heavy use this file descriptors problem reocurred after a
> few short hours:
>
>
> [cc]Jan-22 11:10:04 Project       - exception notifying listener
> net.sourceforge.cruisecontrol.listeners.CurrentBuildStatusListener for
> project myproject net.sourceforge.cruisecontrol.CruiseControlException:
> Error writing file: /myproject/logs/status.txt
>        at net.sourceforge.cruisecontrol.util.IO.write(IO.java:173)
>        at net.sourceforge.cruisecontrol.util.IO.write(IO.java:158)
>        at
> net.sourceforge.cruisecontrol.util.CurrentBuildFileWriter.writefile(CurrentBuildFileWriter.java:56)
>
>        at
> net.sourceforge.cruisecontrol.listeners.CurrentBuildStatusListener.handleEvent(CurrentBuildStatusListener.java:71)
>        at
> net.sourceforge.cruisecontrol.Project.notifyListeners(Project.java:896)
>        at net.sourceforge.cruisecontrol.Project.setState(Project.java:633)
>        at net.sourceforge.cruisecontrol.Project.bootstrap(Project.java:758)
>        at net.sourceforge.cruisecontrol.Project.build(Project.java:192)
>        at net.sourceforge.cruisecontrol.Project.execute(Project.java:147)
>        at
> net.sourceforge.cruisecontrol.ProjectConfig.execute(ProjectConfig.java:402)
>        at
> net.sourceforge.cruisecontrol.ProjectWrapper.run(ProjectWrapper.java:69)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: projects/ops_opsdb/logs/status.txt
> (Too many open files)
>        at java.io.FileOutputStream.open(Native Method)
>        at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
>        at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
>        at java.io.FileWriter.<init>(FileWriter.java:73)
>        at net.sourceforge.cruisecontrol.util.IO.write(IO.java:170)
>        ... 11 more
>
>
> "Joshua Ross" <[email protected]> wrote in message
> news:[email protected]...
>>I just install 2.8.2 on Tuesday and just ran into the same issue this
>>morning.  I opened the dashboard and the error was
>>
>> FileNotFoundException: root/WEB-INF/web.xml (Too many open files)
>>
>> --Josh
>>
>> "Matthijs van der Klip" <[email protected]> wrote in
>> message
>> news:E9F74EB5065FFA4CA4BBC6855E52334C4B735DD5A1@mail.spill.spillgroup.com...
>> Hello all,
>>
>> I am new to Cruisecontrol and this list, so forgive me if I'm asking
>> questions which have been answered before.
>>
>> The version of Cruisecontrol I'm using is the binary release of 2.8.2 on
>> RedHat Linux, using JDK 1.6.0_11 downloaded from the Sun site. This is
>> running a relatively simple setup based on the instructions as found on
>> http://www.phpundercontrol.org/. I'm attaching my config.xml and
>> build.xml.
>>
>> My problem is that CC is running out of filedescriptors within several
>> hours. Sample error from cruisecontrol.log:
>>
>> 2009-01-22 09:32:04,242 [Project myproject thread] ERROR
>>         - exception notifying listener
>> net.sourceforge.cruisecontrol.listeners.CurrentBuildStatusListener for
>> project hyscor.dev.spilgames.com
>> net.sourceforge.cruisecontrol.CruiseControlException: Error writing file:
>> /opt/soft/LIBS/cruisecontrol/logs/hyscor.dev.spilgames.com/status.txt
>>    at net.sourceforge.cruisecontrol.util.IO.write(IO.java:173)
>>    at net.sourceforge.cruisecontrol.util.IO.write(IO.java:158)
>>    at
>> net.sourceforge.cruisecontrol.util.CurrentBuildFileWriter.writefile(CurrentBuildFileWriter.java:56)
>>    at
>> net.sourceforge.cruisecontrol.listeners.CurrentBuildStatusListener.handleEvent(CurrentBuildStatusListener.java:71)
>>    at
>> net.sourceforge.cruisecontrol.Project.notifyListeners(Project.java:896)
>>    at net.sourceforge.cruisecontrol.Project.setState(Project.java:633)
>>    at net.sourceforge.cruisecontrol.Project.run(Project.java:301)
>>    at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.FileNotFoundException: logs/myproject/status.txt (Too
>> many open files)
>>    at java.io.FileOutputStream.open(Native Method)
>>    at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
>>    at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
>>    at java.io.FileWriter.<init>(FileWriter.java:73)
>>    at net.sourceforge.cruisecontrol.util.IO.write(IO.java:170)
>>    ... 7 more
>>
>> This also happens to the internal Jetty server, all operations within CC
>> basically freeze and only a restart can solve this.
>>
>>
>> Using lsof I can see the amount of allocated file descriptors increase
>> quite rapidly:
>>
>> $ sudo lsof -p `pidof java` | wc -l; sleep 60; sudo lsof -p `pidof java` |
>> wc -l
>> 271
>> 299
>>
>> There are no other Java processes running on the machine besides CC.
>>
>>
>> Looking at the output of both netstat and lsof I can see that CC is
>> opening four listening TCP ports: 1099 (RMI), 8082 (JMX), 8083 (HTTP) and
>> an unassigned high port. The function of that fourth port is a mystery to
>> me.
>>
>> Looking at the full output of lsof the huge amount of file descriptors
>> seem to be caused by lots of TCP connections to and from that high port
>> (34520 in this case):
>>
>> java    2807 cruisec  140u  IPv6          226277658                TCP
>> la111.priv.spillgroup.org:38696->la111.priv.spillgroup.org:34520
>> (ESTABLISHED)
>> java    2807 cruisec  141u  IPv6          226277659                TCP
>> la111.priv.spillgroup.org:34520->la111.priv.spillgroup.org:38697
>> (ESTABLISHED)
>> java    2807 cruisec  142u  IPv6          226277661                TCP
>> la111.priv.spillgroup.org:38697->la111.priv.spillgroup.org:34520
>> (ESTABLISHED)
>> java    2807 cruisec  143u  IPv6          226277662                TCP
>> la111.priv.spillgroup.org:34520->la111.priv.spillgroup.org:38708
>> (ESTABLISHED)
>> java    2807 cruisec  144u  IPv6          226277708                TCP
>> la111.priv.spillgroup.org:34520->la111.priv.spillgroup.org:38715
>> (ESTABLISHED)
>> java    2807 cruisec  145u  IPv6          226277747                TCP
>> la111.priv.spillgroup.org:38715->la111.priv.spillgroup.org:34520
>> (ESTABLISHED)
>> java    2807 cruisec  146u  IPv6          226277748                TCP
>> la111.priv.spillgroup.org:34520->la111.priv.spillgroup.org:38718
>> (ESTABLISHED)
>> java    2807 cruisec  147u  IPv6          226277779                TCP
>> la111.priv.spillgroup.org:38725->la111.priv.spillgroup.org:34520
>> (ESTABLISHED)
>> java    2807 cruisec  148u  IPv6          226277753                TCP
>> la111.priv.spillgroup.org:38718->la111.priv.spillgroup.org:34520
>> (ESTABLISHED)
>> ...
>>
>>
>> Can somebody explain to me what that high port is for and what might be
>> the cause of this huge amount of TCP connections?
>>
>> Oh, and I know I can allocate more file descriptors by using ulimit, but
>> that's only a workaround. I'd like to find the reason why CC is 'leaking'
>> TCP connections and solve it from the root.
>>
>>
>> Thanks,
>>
>> --
>> Matthijs van der Klip
>> Director of IT Architecture
>>
>> Phone:    +31 (0) 35 646 6300
>> Fax:      +31 (0) 35 646 6301
>> Email:    [email protected]
>> Linkedin: http://www.linkedin.com/in/mvdklip
>>
>> SPIL GAMES BV
>> Arendstraat 23
>> 1223 RE  Hilversum
>> The Netherlands
>>
>>
>>
>>
>>
>> --------------------------------------------------------------------------------
>>
>>
>>> ------------------------------------------------------------------------------
>>> This SF.net email is sponsored by:
>>> SourcForge Community
>>> SourceForge wants to tell your story.
>>> http://p.sf.net/sfu/sf-spreadtheword
>>
>>
>> --------------------------------------------------------------------------------
>>
>>
>>> _______________________________________________
>>> Cruisecontrol-user mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/cruisecontrol-user
>>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by:
>> SourcForge Community
>> SourceForge wants to tell your story.
>> http://p.sf.net/sfu/sf-spreadtheword
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by:
> SourcForge Community
> SourceForge wants to tell your story.
> http://p.sf.net/sfu/sf-spreadtheword
> _______________________________________________
> Cruisecontrol-user mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/cruisecontrol-user
>



-- 
 Free hour of CI consulting: http://the.ci-guys.com

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H