Re: head node has an extremely high load average.

The new top showed the same problem after the reboot.  The queued jobs
continued during the reboot of the head node.  These jobs started up the
problem with the io as soon as the head node was back up.  Killing one of
the queued jobs stopped the problem.

I need a way to determine which queued job is causing the io problem, so
that I can kill it.  (And hopefully train the user to write their program
to write to the compute node and not the head node during execution.)

Thanks to everyone for their help.



On Thu, Jun 27, 2013 at 9:15 AM, Yixin Luo <[email protected]> wrote:

> It is true - %wa is too high if non-parallel job run. Ann, what is the new
> top after reboot?
>
> Yixin
>
>
> On Thu, Jun 27, 2013 at 7:31 AM, Doll, Margaret Ann <
> [email protected]
> > wrote:
>
> > I installed the iozone program and ran ./iozone -a.
> > How does this information help me find the offending program?
> >
> > Sorry for my ignorance.
> >
> > I do have 10 nfsd programs running.   I only have four jobs on the queues
> > none of which are running parallel code.
> >
> >
> > On Thu, Jun 27, 2013 at 7:42 AM, Miner, Jonathan W (US SSA) <
> > [email protected]> wrote:
> >
> > >
> > > > From: [email protected] [[email protected]
> ]
> > > on behalf of Yixin Luo [[email protected]]
> > > > Sent: Wednesday, June 26, 2013 17:56
> > > > To: General Red Hat Linux discussion list
> > > > Subject: Re: head node has an extremely high load average.
> > > >
> > > > NFS may hang up. Have you tried running autofs?
> > >
> > > Can you explain why "autofs" would be better than NFS?   I have not
> > > managed any NFS-based systems for nearly a decade, but from what I
> > > remember, autofs simplifies the management aspect of network
> filesystems;
> > > but NFS is still the underlaying protocol.  Without autofs, things were
> > > mounted all the time, and you'd have to push changes out to all the
> > > clients' /etc/fstab files.
> > >
> > > As for Margaret's original problem, her system looks very I/O bound.
> >  Like
> > > someone else suggested, I'd start looking at the local disk performance
> > and
> > > see if one disk, or one bus was in contention for most of the traffic.
> > >  Then look at the number of nfsd processes and make sure they're
> > > appropriate for the expected load. The iozone program should help you
> > with
> > > this task.
> > >
> > > http://www.thegeekstuff.com/2011/05/iozone-examples/
> > >
> > > - Jon
> > >
> > >
> > >
> > > --
> > > redhat-list mailing list
> > > unsubscribe mailto:[email protected]?subject=unsubscribe
> > > https://www.redhat.com/mailman/listinfo/redhat-list
> > >
> > --
> > redhat-list mailing list
> > unsubscribe mailto:[email protected]?subject=unsubscribe
> > https://www.redhat.com/mailman/listinfo/redhat-list
> >
> --
> redhat-list mailing list
> unsubscribe mailto:[email protected]?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>
-- 
redhat-list mailing list
unsubscribe mailto:[email protected]?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list