Re: Segmentation fault issue with new install

Jan-Pieter <[email protected]> Tue, 28 Apr 2015 16:27:49 +0200
Newsgroups gmane.comp.audio.icecast.devel
Message-ID <CAB=23ASQjoQ5pAfjO5rnedx_KjdkFnoKvt1uVrxSS+BvSLdWRg@mail.gmail.com>
Hi all,
see my reply inline.
Cheers, Jan-Pieter


2015-04-26 7:23 GMT+02:00 "Thomas B. Rücker" <[email protected]>:

> On 04/25/2015 07:32 PM, Jan-Pieter wrote:
> > Hi all,
> >
> > I am a long-time user of Icecast, always without problems. However i
> > made a new setup using icecast 2.4.1 on a fresh and quick CentOS VPS.
>
> EPEL package or from somewhere else?
>
> it's just a standard CentOS 6.6 release, no EPEL. Distri from hosting
provider. [root@h2429770 tmp]# uname -a
Linux h2429770.stratoserver.net 2.6.32-042stab103.6 #1 SMP Wed Jan 21
13:07:39 MSK 2015 x86_64 x86_64 x86_64 GNU/Linux


>
> > The issue is that starting icecast gives a segmentation fault at
> > around 80% of all starts. No core file nor a entry in error.log is
> posted.
>
> Enable core dumps using "ulimit -c unlimited". The PWD from where you
> start Icecast needs to be writeable.
>

Even with ulimit -c unlimited and running from a writable PWD no core is
dumped.


>
>
> > In the remaining 20% of the starts icecast start with no problem.
> >
> > Contents of the error.log with a succesfull start (20% of icecast
> starts):
> > [2015-04-25  21:08:40] INFO main/main Icecast 2.4.1 server started
> > [2015-04-25  21:08:40] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-25  21:08:40] INFO stats/_stats_thread stats thread started
> > [2015-04-25  21:08:40] INFO yp/yp_update_thread YP update thread started
> > [2015-04-25  21:08:40] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-25  21:08:40] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic minimal
> > handler for best effort.
> > [2015-04-25  21:08:40] WARN source/source_fallback_file mountpoint
> > "/geen-uitzending.mp3" already reserved
> > [2015-04-25  21:08:40] INFO source/source_main listener count on
> > /geen-uitzending.mp3 now 0
> >
> >
> > Contents of error.log with a segmentation fault direct after start
> > (80% of icecast starts):
> > 2015-04-25  21:09:05] INFO main/main Icecast 2.4.1 server started
> > [2015-04-25  21:09:05] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-25  21:09:05] INFO yp/yp_update_thread YP update thread started
> > [2015-04-25  21:09:05] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-25  21:09:05] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic minimal
> > handler for best effort.
> > [2015-04-25  21:09:05] INFO stats/_stats_thread stats thread started
>
> Can you attach your icecast.xml please (with passwords removed)?
> This might help reproduce the problem.
>
>
I attached icecast.xml. I also turned debug errorlog level on.
Results with debug on:

good start:
[2015-04-28  15:56:39] INFO main/main Icecast 2.4.1 server started
[2015-04-28  15:56:39] DBUG yp/yp_recheck_config Updating YP configuration
[2015-04-28  15:56:39] INFO connection/get_ssl_certificate No SSL capability
[2015-04-28  15:56:39] INFO stats/_stats_thread stats thread started
[2015-04-28  15:56:39] INFO yp/yp_update_thread YP update thread started
[2015-04-28  15:56:39] INFO source/source_fallback_file mountpoint
/geen-uitzending.mp3 is reserved
[2015-04-28  15:56:39] DBUG connection/connection_complete_source sources
count is 0
[2015-04-28  15:56:39] WARN format/format_get_type Unsupported or legacy
stream type: "audio/mpeg". Falling back to generic minimal handler for best
effort.
[2015-04-28  15:56:39] DBUG connection/connection_complete_source source is
ready to start
[2015-04-28  15:56:39] DBUG source/source_init Source creation complete
[2015-04-28  15:56:39] WARN source/source_fallback_file mountpoint
"/geen-uitzending.mp3" already reserved
[2015-04-28  15:56:39] INFO source/source_main listener count on
/geen-uitzending.mp3 now 0
[2015-04-28  15:56:39] DBUG stats/modify_node_event update global sources
(1)
[2015-04-28  15:56:39] DBUG stats/process_source_event new source stat
/geen-uitzending.mp3
[2015-04-28  15:56:39] DBUG stats/modify_node_event update global
source_client_connections (1)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node listeners
(0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node listenurl (
http://stream3.server7.nl:80/geen-uitzending.mp3)
[2015-04-28  15:56:39] DBUG stats/modify_node_event update global
source_total_connections (1)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
slow_listeners (0)
[2015-04-28  15:56:39] DBUG stats/modify_node_event update
"/geen-uitzending.mp3" listeners (0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
listener_peak (0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
stream_start (Tue, 28 Apr 2015 15:56:39 +0200)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
stream_start_iso8601 (2015-04-28T15:56:39+0200)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
total_bytes_read (0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
total_bytes_sent (0)
[2015-04-28  15:56:39] DBUG stats/modify_node_event update
"/geen-uitzending.mp3" listeners (0)
[2015-04-28  15:56:40] DBUG slave/_slave_thread checking master stream list

bad start (segfault):
[2015-04-28  15:56:53] INFO main/main Icecast 2.4.1 server started
[2015-04-28  15:56:53] DBUG yp/yp_recheck_config Updating YP configuration
[2015-04-28  15:56:53] INFO connection/get_ssl_certificate No SSL capability
[2015-04-28  15:56:53] INFO yp/yp_update_thread YP update thread started
[2015-04-28  15:56:53] INFO source/source_fallback_file mountpoint
/geen-uitzending.mp3 is reserved
[2015-04-28  15:56:53] DBUG connection/connection_complete_source sources
count is 0
[2015-04-28  15:56:53] WARN format/format_get_type Unsupported or legacy
stream type: "audio/mpeg". Falling back to generic minimal handler for best
effort.
[2015-04-28  15:56:53] DBUG connection/connection_complete_source source is
ready to start
[2015-04-28  15:56:53] DBUG source/source_init Source creation complete
[2015-04-28  15:56:53] INFO stats/_stats_thread stats thread started
[2015-04-28  15:56:53] DBUG stats/process_source_event new source stat
/geen-uitzending.mp3





>
> > Both the stream and the fallback file are .mp3 type. I tried removing
> > the fallback entry. Then icecast starts always normally. So it seems
> > related to the .mp3 fallback.
> >
> > Also I tried to launch icecast with strace, and this NEVER fails. No
> > segmentation faults!
> >
> >
> > Any idea what's going wrong? Something going wrong with timing on a
> > fast server?
>
> It sounds like you've found a race condition. I'm not exactly sure
> what's happening though.
> By the log output something seems to be instantly accessing the stream.
> Notice the order of the stats_thread is different in both.
>
>
Yes this is the only interesting thing. The segfault only happens when
"INFO stats/_stats_thread stats thread started"
is missing in the error.log. So with every succesful start this line gets
logged, and if the line is missing the segfault occurs.


I also tried running using valgrind on advice of Philipp, but when running
with valgrind the segfault does not occur and icecast starts normally...
(same as with gdb)


>
> Cheers
>
> Thomas
>
>
>
> _______________________________________________
> Icecast-dev mailing list
> [email protected]
> http://lists.xiph.org/mailman/listinfo/icecast-dev
>

_______________________________________________
Icecast-dev mailing list
[email protected]
http://lists.xiph.org/mailman/listinfo/icecast-dev
icecast.xml (text/xml, 11.6 KB)
<icecast>
    <!-- location and admin are two arbitrary strings that are e.g. visible
         on the server info page of the icecast web interface
         (server_version.xsl). -->
    <location>Lelystad</location>
    <admin>[email protected]</admin>

    <!-- IMPORTANT!
         Especially for inexperienced users:
         Start out by ONLY changing all passwords and restarting Icecast.
         For detailed setup instructions please refer to the documentation.
         It's also available here: http://icecast.org/docs/
    -->

    <limits>
        <clients>100</clients>
        <sources>2</sources>
        <queue-size>524288</queue-size>
        <client-timeout>30</client-timeout>
        <header-timeout>15</header-timeout>
        <source-timeout>10</source-timeout>
        <!-- If enabled, this will provide a burst of data when a client
             first connects, thereby significantly reducing the startup
             time for listeners that do substantial buffering. However,
             it also significantly increases latency between the source
             client and listening client.  For low-latency setups, you
             might want to disable this. -->
        <burst-on-connect>1</burst-on-connect>
        <!-- same as burst-on-connect, but this allows for being more
             specific on how much to burst. Most people won't need to
             change from the default 64k. Applies to all mountpoints  -->
        <burst-size>65535</burst-size>
    </limits>

    <authentication>
        <!-- Sources log in with username 'source' -->
        <source-password>pass</source-password>
        <!-- Relays log in with username 'relay' -->
        <relay-password>pass</relay-password>

        <!-- Admin logs in with the username given below -->
        <admin-user>admin</admin-user>
        <admin-password>pass</admin-password>
    </authentication>

    <!-- set the mountpoint for a shoutcast source to use, the default if not
         specified is /stream but you can change it here if an alternative is
         wanted or an extension is required
    <shoutcast-mount>/live.nsv</shoutcast-mount>
    -->

    <!-- Uncomment this if you want directory listings -->
    <!--
    <directory>
        <yp-url-timeout>15</yp-url-timeout>
        <yp-url>http://dir.xiph.org/cgi-bin/yp-cgi</yp-url>
    </directory>
    -->

    <!-- This is the hostname other people will use to connect to your server.
         It affects mainly the urls generated by Icecast for playlists and yp
         listings. You MUST configure it properly for YP listings to work!
    -->
    <hostname>stream3.server7.nl</hostname>

    <!-- You may have multiple <listener> elements -->
    <listen-socket>
        <port>80</port>
        <bind-address>85.214.159.197</bind-address>
        <!-- <shoutcast-mount>/stream</shoutcast-mount> -->
    </listen-socket>
    <listen-socket>
        <port>8000</port>
    </listen-socket>
    <listen-socket>
        <port>7999</port>
    </listen-socket>
    <!--
    <listen-socket>
        <port>8443</port>
        <ssl>1</ssl>
    </listen-socket>
    -->


    <!-- Global header settings
         Headers defined here will be returned for every HTTP request to Icecast.

         The ACAO header makes Icecast public content/API by default
         This will make streams easier embeddable (some HTML5 functionality needs it).
         Also it allows direct access to e.g. /status-json.xsl from other sites.
         If you don't want this, comment out the following line or read up on CORS.
    -->
    <http-headers>
        <header name="Access-Control-Allow-Origin" value="*" />
    </http-headers>


    <!-- Relaying
         You don't need this if you only have one server.
         Please refer to the config for a detailed explanation.
    -->
    <!--<master-server>127.0.0.1</master-server>-->
    <!--<master-server-port>8001</master-server-port>-->
    <!--<master-update-interval>120</master-update-interval>-->
    <!--<master-password>hackme</master-password>-->

    <!-- setting this makes all relays on-demand unless overridden, this is
         useful for master relays which do not have <relay> definitions here.
         The default is 0 -->
    <!--<relays-on-demand>1</relays-on-demand>-->

    <!--
    <relay>
        <server>127.0.0.1</server>
        <port>8080</port>
        <mount>/example.ogg</mount>
        <local-mount>/different.ogg</local-mount>
        <on-demand>0</on-demand>

        <relay-shoutcast-metadata>0</relay-shoutcast-metadata>
    </relay>
    -->


    <!-- Mountpoints
         Only define <mount> sections if you want to use advanced options,
         like alternative usernames or passwords
    -->

    <!-- Default settings for all mounts that don't have a specific <mount type="normal">.
    -->
    <!--
    <mount type="default">
        <public>0</public>
        <intro>/server-wide-intro.ogg</intro>
        <max-listener-duration>3600</max-listener-duration>
        <authentication type="url">
                <option name="mount_add" value="http://auth.example.org/stream_start.php"/>
        </authentication>
        <http-headers>
                <header name="foo" value="bar" />
        </http-headers>
    </mount>
    -->

    <!-- Normal mounts -->
    <mount type="normal">
        <mount-name>/lichtbron</mount-name>
        <username></username>
        <password>pass</password>
        <max-listeners>100</max-listeners>
        <max-listener-duration>14400</max-listener-duration>
        <dump-file>/tmp/dumplichtbron.mp3</dump-file>
        <intro>1000_welkom_lichtbron.ogg</intro>
        <fallback-mount>/geen-uitzending.mp3</fallback-mount>
        <fallback-override>1</fallback-override>
        <fallback-when-full>0</fallback-when-full>
        <charset>ISO8859-1</charset>
        <public>1</public>
        <stream-name>GKV Lichtbron</stream-name>
        <stream-description>1Live vanuit de Lichtbron in Lelystad</stream-description>
        <stream-url>http://www.pauwenburg.nl/beamer/beamerhq.html</stream-url>
        <genre>kerkdienst</genre>
        <bitrate>128</bitrate>
        <type>application/ogg</type>
        <subtype>vorbis</subtype>
        <hidden>0</hidden>
        <burst-size>65536</burst-size>
        <mp3-metadata-interval>4096</mp3-metadata-interval>
    </mount>
    <mount type="normal">
        <mount-name>/pijlerjp</mount-name>
        <username></username>
        <password>pass</password>
        <max-listeners>25</max-listeners>
        <max-listener-duration>14400</max-listener-duration>
        <dump-file>/var/www/vhosts/pijlervideo.server7.nl/httpdocs/Audio/dienst.mp3</dump-file>
        <intro></intro>
        <fallback-mount>/geen-uitzending.mp3</fallback-mount>
        <fallback-override>1</fallback-override>
        <fallback-when-full>0</fallback-when-full>
        <charset>ISO8859-1</charset>
        <public>1</public>
        <stream-name>TestStream Pijler</stream-name>
        <stream-description>Live teststream Pijler - info jpveen</stream-description>
        <stream-url>http://www.pijler.nl/</stream-url>
        <genre>test</genre>
        <bitrate>128</bitrate>
        <type>application/mp3</type>
        <subtype>mp3</subtype>
        <hidden>0</hidden>
        <burst-size>65536</burst-size>
        <mp3-metadata-interval>4096</mp3-metadata-interval>
    </mount>
   <!--
    <mount type="normal">
        <mount-name>/example-complex.ogg</mount-name>

        <username>othersource</username>
        <password>hackmemore</password>

        <max-listeners>1</max-listeners>
        <dump-file>/tmp/dump-example1.ogg</dump-file>
        <burst-size>65536</burst-size>
        <fallback-mount>/example2.ogg</fallback-mount>
        <fallback-override>1</fallback-override>
        <fallback-when-full>1</fallback-when-full>
        <intro>/example_intro.ogg</intro>
        <hidden>1</hidden>
        <public>1</public>
        <authentication type="htpasswd">
                <option name="filename" value="myauth"/>
                <option name="allow_duplicate_users" value="0"/>
        </authentication>
        <http-headers>
                <header name="Access-Control-Allow-Origin" value="http://webplayer.example.org" />
                <header name="baz" value="quux" />
        </http-headers>
        <on-connect>/home/icecast/bin/stream-start</on-connect>
        <on-disconnect>/home/icecast/bin/stream-stop</on-disconnect>
    </mount>
    -->

    <!--
    <mount type="normal">
        <mount-name>/auth_example.ogg</mount-name>
        <authentication type="url">
            <option name="mount_add"       value="http://myauthserver.net/notify_mount.php"/>
            <option name="mount_remove"    value="http://myauthserver.net/notify_mount.php"/>
            <option name="listener_add"    value="http://myauthserver.net/notify_listener.php"/>
            <option name="listener_remove" value="http://myauthserver.net/notify_listener.php"/>
            <option name="headers"         value="x-pragma,x-token"/>
            <option name="header_prefix"   value="ClientHeader."/>
        </authentication>
    </mount>
    -->

    <fileserve>1</fileserve>

    <paths>
        <!-- basedir is only used if chroot is enabled -->
        <basedir>/opt/icecast/latest/share/icecast</basedir>

        <!-- Note that if <chroot> is turned on below, these paths must both
             be relative to the new root, not the original root -->
        <!-- <logdir>/opt/icecast/latest/var/log/icecast</logdir> -->
        <webroot>/opt/icecast/latest/share/icecast/web</webroot>
        <adminroot>/opt/icecast/latest/share/icecast/admin</adminroot>
        <!-- <pidfile>/opt/icecast/latest/share/icecast/icecast.pid</pidfile> -->
       <logdir>/var/log/icecast</logdir>
       <pidfile>/var/run/icecast/icecast.pid</pidfile>

        <!-- Aliases: treat requests for 'source' path as being for 'dest' path
             May be made specific to a port or bound address using the "port"
             and "bind-address" attributes.
          -->
        <!--
        <alias source="/foo" destination="/bar"/>
        -->
        <!-- Aliases: can also be used for simple redirections as well,
             this example will redirect all requests for http://server:port/ to
             the status page
        <alias source="/" destination="/status.xsl"/>
        -->
        <alias source="/" dest="/status.xsl"/>
        <!-- The certificate file needs to contain both public and private part.
             Both should be PEM encoded.
        <ssl-certificate>/opt/icecast/latest/share/icecast/icecast.pem</ssl-certificate>
        -->
    </paths>

    <logging>
        <accesslog>access.log</accesslog>
        <errorlog>error.log</errorlog>
        <!-- <playlistlog>playlist.log</playlistlog> -->
        <loglevel>4</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
        <logsize>10000</logsize> <!-- Max size of a logfile -->
        <!-- If logarchive is enabled (1), then when logsize is reached
             the logfile will be moved to [error|access|playlist].log.DATESTAMP,
             otherwise it will be moved to [error|access|playlist].log.old.
             Default is non-archive mode (i.e. overwrite)
        -->
        <!-- <logarchive>1</logarchive> -->
    </logging>

    <security>
        <chroot>0</chroot>
        <changeowner>
            <user>icecast</user>
            <group>icecast</group>
        </changeowner>
    </security>
</icecast>