Re: Agent not responding after initial startup

Here is an update on this;

I have five agents actively connected

[grinder@ip-10-82-151-77 ~]$ curl --noproxy localhost -s -X GET
http://localhost:6373/agents/status

[{"id":"ip-10-82-151-68.localdomain:396883763|1429731199058|-581571585:0","name":"i-9544d563","number":-1,"state":"running","workers":[]},{"id":"ip-10-82-151-67.localdomain:396883763|1429731216470|948975004:0","name":"i-9844d56e","number":-1,"state":"running","workers":[]},{"id":"ip-10-82-151-66.localdomain:396883763|1429731223428|118753153:0","name":"i-9944d56f","number":-1,"state":"running","workers":[]},{"id":"ip-10-82-151-65.localdomain:396883763|1429731205574|-2087968126:0","name":"i-9a44d56c","number":-1,"state":"running","workers":[]},{"id":"ip-10-82-151-64.localdomain:396883763|1429731213895|-520891325:0","name":"i-9b44d56d","number":-1,"state":"running","workers":[]}]

>From the console:
[grinder@ip-10-82-151-77 ~]$ netstat -an | grep 637
tcp        0      0 0.0.0.0:6372                0.0.0.0:*
     LISTEN
tcp        0      0 0.0.0.0:6373                0.0.0.0:*
     LISTEN
tcp        0      0 10.82.151.77:6372           10.82.151.68:49863
     ESTABLISHED
tcp        0      0 10.82.151.77:6372           10.82.151.66:35165
     ESTABLISHED
tcp        0      0 10.82.151.77:6372           10.82.151.67:59181
     ESTABLISHED
tcp        0      0 10.82.151.77:6372           10.82.151.65:33245
     ESTABLISHED
tcp        0      0 10.82.151.77:6372           10.82.151.64:34232
     ESTABLISHED

[grinder@ip-10-82-151-77 ~]$ curl --noproxy localhost -s -X POST
http://localhost:6373/files/distribute
{"id":3,"state":"started","files":[]}

Files status - per-cent-complete 100, but stale is true?
[grinder@ip-10-82-151-77 ~]$ curl --noproxy localhost -s -X GET
http://localhost:6373/files/status
{"stale":true,"last-distribution":{"per-cent-complete":100,"id":3,"state":"finished","files":["README.md","createapp.py","grinder.properties"]}}

On one of the agents - the log is just sitting waiting for console
signal - no files have been distributed:

[grinder@ip-10-82-151-66]$ cat grinderagent.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/grinder/uber/logback-classic-1.0.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/grinder/uber/slf4j-nop-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/grinder/lib/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
{"timestamp":"2015-04-22T19:33:43.430+00:00","logger":"agent","thread":"main","level":"INFO","HOSTNAME":"ip-10-82-151-66.localdomain","message":"The
Grinder 3.12-SNAPSHOT"}
{"timestamp":"2015-04-22T19:33:43.505+00:00","logger":"agent","thread":"main","level":"INFO","HOSTNAME":"ip-10-82-151-66.localdomain","message":"connected
to console at grinderconsole1/10.82.151.77:6372"}
{"timestamp":"2015-04-22T19:33:43.505+00:00","logger":"agent","thread":"main","level":"INFO","HOSTNAME":"ip-10-82-151-66.localdomain","message":"waiting
for console signal"}

[grinder@ip-10-82-151-66 ~]$ netstat -an | grep 637
tcp        0      0 10.82.151.66:35165          10.82.151.77:6372
     ESTABLISHED


Any ideas?


-Darren


On Wed, Apr 22, 2015 at 12:40 PM, Darren Ball <[email protected]> wrote:
> Hi All,
>
> I currently have jenkins jobs that are used to fire up the console on a
> remote machine via SSH.
> After the console is up and ports are listening, I wait and then fire up
> agents remotely via SSH as well.
>
> Console comes up and I can talk to it via service interface.  It looks like
> the agents are all up.  Java processes are there, port is connected and
> logging indicates that it is waiting for a console signal.
>
> My startup for the agents is scripted as follows:
> java -classpath
> "/home/grinder/uber/*":/home/grinder/myapp-uber.jar:/opt/grinder/lib/grinder.jar
> \
>  -Dlogback.configurationFile=/home/grinder/logback.xml \
>  -Dgrinder.consoleHost=$1 \
>  -Dgrinder.hostID=$2 \
>  -Dgrinder.logDirectory="/var/log/appbase/$(date +%y%m%d)" \
>  net.grinder.Grinder -daemon 30 > /var/log/qbase/grinderagent.log 2>&1 &
>
>
> I have the agent logging out to a file of which contains:
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/grinder/uber/logback-classic-1.0.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/grinder/uber/slf4j-nop-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/grinder/lib/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> {"timestamp":"2015-04-22T16:18:59.373+00:00","logger":"agent","thread":"main","level":"INFO","HOSTNAME":"ip-10-82-151-41.localdomain","message":"The
> Grinder 3.12-SNAPSHOT"}
> {"timestamp":"2015-04-22T16:18:59.439+00:00","logger":"agent","thread":"main","level":"INFO","HOSTNAME":"ip-10-82-151-41.localdomain","message":"connected
> to console at grinderconsole1/10.82.151.109:6372"}
> {"timestamp":"2015-04-22T16:18:59.439+00:00","logger":"agent","thread":"main","level":"INFO","HOSTNAME":"ip-10-82-151-41.localdomain","message":"waiting
> for console signal"}
>
>
> When I attempt to distribute files, the agents are not responding (or
> accepting the request to distribute).
>
> If I kill the process and re-run the startup script a second time directly
> on the machine, everything seems to work fine.
> Maybe this is due to launching from within a Jenkins job via SSH, but if
> that was the case, I would assume the console would have similar issues as
> well?
>
> Weird antics here.
>
> Looks like the process is coming up orphaned as well (most likely due to the
> non terminal launch) when launched from the CI.
>
> [grinder@ip-10-82-151-41]$ netstat -an | grep 637
>
> tcp        0      0 10.82.151.41:54335          10.82.151.109:6372
> ESTABLISHED
>
> [grinder@ip-10-82-151-41]$ ps -ef | grep grinder
>
> grinder    3722      1  1 16:11 ?        00:00:02 java -classpath
> /home/grinder/uber/*:/home/grinder/myapp-uber.jar:/opt/grinder/lib/grinder.jar
> -Dlogback.configurationFile=/home/grinder/logback.xml
> -Dgrinder.consoleHost=grinderconsole1 -Dgrinder.hostID=i-a9b7265f
> -Dgrinder.logDirectory=/var/log/qbase/150422 net.grinder.Grinder -daemon 30
>
>
> Any insight on how to get the agents to continue to be listening or how to
> debug this would be great.
>
> Thanks,
> Darren

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF