Re: Client deadlock?
Garrett Wollman <[email protected]> Thu, 31 Mar 2011 12:58:34 -0400
| Newsgroups | gmane.comp.file-systems.openafs.freebsd |
|---|---|
| Message-ID | <[email protected]> |
<<On Wed, 30 Mar 2011 21:18:05 -0400 (EDT), Benjamin Kaduk <[email protected]> said: > cmdebug -cache would be more authoritative (the last time I tripped the > "small cache" behavior was when I was passing enough arguments to afsd > that the memcache size was determined by them and the cachinfo > specification was ignored). It reports: Chunk files: 1464 Stat caches: 2196 Data caches: 1464 Volume caches: 200 Chunk size: 1048576 Cache size: 1499136 kB Set time: no Cache type: memory It turns out that even dd(1) hangs it nearly immediately: $ dd if=/dev/zero of=test bs=64k count=1048576 load: 0.04 cmd: dd 7375 [afsslp] 17.63r 0.01u 0.45s 0% 1088k load: 0.03 cmd: dd 7375 [afsslp] 21.97r 0.01u 0.45s 0% 1088k Similar symptoms: two afsd's sleeping on afsslp, one in sbwait, and rxdebug shows no connection with the server; other afsds doing nothing, but the client does respond to rxdebug requests on the callback port: wollman@modesty:~$ rxdebug incoming-new 7001 Trying 128.30.2.181 (port 7001): Free packets: 269, packet reclaims: 6, calls: 113, used FDs: 64 not waiting for packets. 0 calls waiting for a thread 1 threads are idle Connection from host 128.30.2.188, port 7000, Cuid b40e39c4/43383920 serial 57003675, natMTU 1444, flags DESTROYED pktCksum, security index 2, client conn rxkad: level crypt, flags pktCksum Received -1197234936 bytes in 7808384 packets Sent 1414335526 bytes in 49636663 packets call 0: # 4370276, state dally, mode: receiving, flags: receive_done call 1: # 2057007, state not initialized call 2: # 0, state not initialized call 3: # 0, state not initialized Done. (incoming-new is the client, modesty is the server this particular volume is on). cmdebug -long reports: $ cmdebug -long localhost Lock afs_xvcache status: (none_waiting) Lock afs_xdcache status: (none_waiting) Lock afs_xserver status: (none_waiting) Lock afs_xvcb status: (none_waiting) Lock afs_xbrs status: (none_waiting) Lock afs_xcell status: (none_waiting) Lock afs_xconn status: (none_waiting) Lock afs_xuser status: (none_waiting) Lock afs_xvolume status: (none_waiting) Lock puttofile status: (none_waiting) Lock afs_ftf status: (none_waiting) Lock afs_xcbhash status: (none_waiting) Lock afs_xaxs status: (none_waiting) Lock afs_xinterface status: (none_waiting) Lock afs_xosi status: (none_waiting) Lock afs_xsrvAddr status: (none_waiting) Lock afs_xvreclaim status: (none_waiting) Lock afsdb_client_loc status: (none_waiting) Lock afsdb_req_lock status: (none_waiting) Lock afs_discon_lock status: (none_waiting) Lock afs_disconDirtyL status: (none_waiting) Lock afs_discon_vc_di status: (none_waiting) Lock dynroot status: (none_waiting) Lock rl.ac.uk status: (none_waiting) Lock hep.man.ac.uk status: (none_waiting) Lock ic.ac.uk status: (none_waiting) Lock inf.ed.ac.uk status: (none_waiting) Lock phy.bris.ac.uk status: (none_waiting) Lock hep-ex.physics.m status: (none_waiting) Lock ihep.su status: (none_waiting) Lock p-ng.si status: (none_waiting) Lock f9.ijs.si status: (none_waiting) Lock su.se status: (none_waiting) Lock sanchin.se status: (none_waiting) Lock physto.se status: (none_waiting) Lock syd.kth.se status: (none_waiting) Lock stacken.kth.se status: (none_waiting) Lock pdc.kth.se status: (none_waiting) Lock nada.kth.se status: (none_waiting) Lock mech.kth.se status: (none_waiting) Lock md.kth.se status: (none_waiting) Lock it.kth.se status: (none_waiting) Lock isk.kth.se status: (none_waiting) Lock kth.se status: (none_waiting) Lock riscpkg.org status: (none_waiting) Lock pfriedma.org status: (none_waiting) Lock oc7.org status: (none_waiting) Lock nomh.org status: (none_waiting) Lock mstacm.org status: (none_waiting) Lock mrph.org status: (none_waiting) Lock mrow.org status: (none_waiting) Lock jeaton.org status: (none_waiting) Lock idahofuturetruck status: (none_waiting) Lock dementia.org status: (none_waiting) Lock coed.org status: (none_waiting) Lock bazquux.org status: (none_waiting) Lock adrake.org status: (none_waiting) Lock acm-csuf.org status: (none_waiting) Lock 1ts.org status: (none_waiting) Lock nikhef.nl status: (none_waiting) Lock interdose.net status: (none_waiting) Lock tproa.net status: (none_waiting) Lock slackers.net status: (none_waiting) Lock sinenomine.net status: (none_waiting) Lock laroia.net status: (none_waiting) Lock gorlaeus.net status: (none_waiting) Lock es.net status: (none_waiting) Lock epitech.net status: (none_waiting) Lock crossproduct.net status: (none_waiting) Lock lcp.nrl.navy.mil status: (none_waiting) Lock cmf.nrl.navy.mil status: (none_waiting) Lock italia status: (none_waiting) Lock vn.uniroma3.it status: (none_waiting) Lock dia.uniroma3.it status: (none_waiting) Lock ing.uniroma1.it status: (none_waiting) Lock math.unifi.it status: (none_waiting) Lock tgrid.it status: (none_waiting) Lock psm.it status: (none_waiting) Lock roma3.infn.it status: (none_waiting) Lock pi.infn.it status: (none_waiting) Lock lngs.infn.it status: (none_waiting) Lock lnf.infn.it status: (none_waiting) Lock le.infn.it status: (none_waiting) Lock kloe.infn.it status: (none_waiting) Lock ba.infn.it status: (none_waiting) Lock infn.it status: (none_waiting) Lock ictp.it status: (none_waiting) Lock icemb.it status: (none_waiting) Lock fusione.it status: (none_waiting) Lock enea.it status: (none_waiting) Lock caspur.it status: (none_waiting) Lock kfki.hu status: (none_waiting) Lock bme.hu status: (none_waiting) Lock doe.atomki.hu status: (none_waiting) Lock nersc.gov status: (none_waiting) Lock jpl.nasa.gov status: (none_waiting) Lock ic-afs.arc.nasa. status: (none_waiting) Lock fnal.gov status: (none_waiting) Lock usatlas.bnl.gov status: (none_waiting) Lock rhic.bnl.gov status: (none_waiting) Lock anl.gov status: (none_waiting) Lock mcc.ac.gb status: (none_waiting) Lock in2p3.fr status: (none_waiting) Lock dapnia.saclay.ce status: (none_waiting) Lock biocenter.helsin status: (none_waiting) Lock ific.uv.es status: (none_waiting) Lock ifca.unican.es status: (none_waiting) Lock ciemat.es status: (none_waiting) Lock physics.wisc.edu status: (none_waiting) Lock hep.wisc.edu status: (none_waiting) Lock engr.wisc.edu status: (none_waiting) Lock cs.wisc.edu status: (none_waiting) Lock cs.uwm.edu status: (none_waiting) Lock eng.utah.edu status: (none_waiting) Lock uncc.edu status: (none_waiting) Lock physics.unc.edu status: (none_waiting) Lock isis.unc.edu status: (none_waiting) Lock sph.umich.edu status: (none_waiting) Lock citi.umich.edu status: (none_waiting) Lock atlas.umich.edu status: (none_waiting) Lock umich.edu status: (none_waiting) Lock wam.umd.edu status: (none_waiting) Lock glue.umd.edu status: (none_waiting) Lock umbc.edu status: (none_waiting) Lock ncsa.uiuc.edu status: (none_waiting) Lock illigal.uiuc.edu status: (none_waiting) Lock acm.uiuc.edu status: (none_waiting) Lock cats.ucsc.edu status: (none_waiting) Lock physics.ucsb.edu status: (none_waiting) Lock slac.stanford.ed status: (none_waiting) Lock ir.stanford.edu status: (none_waiting) Lock cs.stanford.edu status: (none_waiting) Lock hep.sc.edu status: (none_waiting) Lock rpi.edu status: (none_waiting) Lock cs.rose-hulman.e status: (none_waiting) Lock rose-hulman.edu status: (none_waiting) Lock cede.psu.edu status: (none_waiting) Lock scoobydoo.psc.ed status: (none_waiting) Lock psc.edu status: (none_waiting) Lock cs.pitt.edu status: (none_waiting) Lock pitt.edu status: (none_waiting) Lock crc.nd.edu status: (none_waiting) Lock nd.edu status: (none_waiting) Lock msu.edu status: (none_waiting) Lock soap.mit.edu status: (none_waiting) Lock sipb.mit.edu status: (none_waiting) Lock numenor.mit.edu status: (none_waiting) Lock net.mit.edu status: (none_waiting) Lock dev.mit.edu status: (none_waiting) Lock athena.mit.edu status: (none_waiting) Lock iastate.edu status: (none_waiting) Lock eecs.harvard.edu status: (none_waiting) Lock cs.hm.edu status: (none_waiting) Lock northstar.dartmo status: (none_waiting) Lock dbic.dartmouth.e status: (none_waiting) Lock msc.cornell.edu status: (none_waiting) Lock math.cornell.edu status: (none_waiting) Lock cnf.cornell.edu status: (none_waiting) Lock sbp.ri.cmu.edu status: (none_waiting) Lock qatar.cmu.edu status: (none_waiting) Lock scotch.ece.cmu.e status: (none_waiting) Lock ece.cmu.edu status: (none_waiting) Lock cs.cmu.edu status: (none_waiting) Lock chem.cmu.edu status: (none_waiting) Lock club.cc.cmu.edu status: (none_waiting) Lock mw.andrew.cmu.ed status: (none_waiting) Lock andrew.cmu.edu status: (none_waiting) Lock clarkson.edu status: (none_waiting) Lock ugcs.caltech.edu status: (none_waiting) Lock hep.caltech.edu status: (none_waiting) Lock eecs.berkeley.ed status: (none_waiting) Lock mars.asu.edu status: (none_waiting) Lock asu.edu status: (none_waiting) Lock ies.auc.dk status: (none_waiting) Lock s-et.aau.dk status: (none_waiting) Lock physik.uni-wuppe status: (none_waiting) Lock uni-paderborn.de status: (none_waiting) Lock uni-mannheim.de status: (none_waiting) Lock physik.uni-mainz status: (none_waiting) Lock rrz.uni-koeln.de status: (none_waiting) Lock meteo.uni-koeln. status: (none_waiting) Lock impetus.uni-koel status: (none_waiting) Lock rz.uni-jena.de status: (none_waiting) Lock uni-hohenheim.de status: (none_waiting) Lock ziti.uni-heidelb status: (none_waiting) Lock urz.uni-heidelbe status: (none_waiting) Lock mathi.uni-heidel status: (none_waiting) Lock physnet.uni-hamb status: (none_waiting) Lock physik.uni-freib status: (none_waiting) Lock uni-freiburg.de status: (none_waiting) Lock atlass01.physik. status: (none_waiting) Lock e18.ph.tum.de status: (none_waiting) Lock tu-chemnitz.de status: (none_waiting) Lock tu-bs.de status: (none_waiting) Lock combi.tfh-wildau status: (none_waiting) Lock i1.informatik.rw status: (none_waiting) Lock nicsys.de status: (none_waiting) Lock mpe.mpg.de status: (none_waiting) Lock ipp-garching.mpg status: (none_waiting) Lock lrz-muenchen.de status: (none_waiting) Lock integra-ev.de status: (none_waiting) Lock ifh.de status: (none_waiting) Lock cms.hu-berlin.de status: (none_waiting) Lock gppc.de status: (none_waiting) Lock desy.de status: (none_waiting) Lock zcu.cz status: (none_waiting) Lock ics.muni.cz status: (none_waiting) Lock ruk.cuni.cz status: (none_waiting) Lock sodre.cx status: (none_waiting) Lock setfilepointer.c status: (none_waiting) Lock membrain.com status: (none_waiting) Lock freedaemon.com status: (none_waiting) Lock extundo.com status: (none_waiting) Lock psi.ch status: (none_waiting) Lock ethz.ch status: (none_waiting) Lock epfl.ch status: (none_waiting) Lock ams.cern.ch status: (none_waiting) Lock cern.ch status: (none_waiting) Lock ualberta.ca status: (none_waiting) Lock sums.math.mcgill status: (none_waiting) Lock itp.tugraz.at status: (none_waiting) Lock cgv.tugraz.at status: (none_waiting) Lock hephy.at status: (none_waiting) Lock wu-wien.ac.at status: (none_waiting) Lock grand.central.or status: (none_waiting) Lock csail.mit.edu status: (none_waiting) ** Cache entry @ 0x01916000 for 0.1.1.1 [dynroot] 2048 bytes DV 208 refcnt 1 callback 00000000 expires 0 0 opens 0 writers volume root states (0x5), stat'd, read-only ** Cache entry @ 0x0195ac40 for 203.536870916.1.1 [csail.mit.edu] 2048 bytes DV 256 refcnt 0 callback 0a8bb280 expires 1301597388 0 opens 0 writers volume root states (0x5), stat'd, read-only ** Cache entry @ 0x019a47d0 for 203.536870916.4.213 [csail.mit.edu] 3 bytes DV 1 refcnt 0 callback 0a8bb280 expires 1301597388 0 opens 0 writers mount point states (0xd), stat'd, read-only, mt pt valid ** Cache entry @ 0x0191ef70 for 203.536870921.1.1 [csail.mit.edu] 18432 bytes DV 4048 refcnt 0 callback 0a158280 expires 1301604556 0 opens 0 writers volume root states (0x1), stat'd ** Cache entry @ 0x019e5090 for 203.536870928.1.1 [csail.mit.edu] 2048 bytes DV 261 refcnt 0 callback 0a8bb280 expires 1301597388 0 opens 0 writers volume root states (0x5), stat'd, read-only ** Cache entry @ 0x019b8000 for 203.536870928.24.650 [csail.mit.edu] 11 bytes DV 0 refcnt 0 callback 0a8bb280 expires 1301597388 0 opens 0 writers mount point states (0xd), stat'd, read-only, mt pt valid ** Cache entry @ 0x01918d90 for 203.536870928.47.636 [csail.mit.edu] 6144 bytes DV 181 refcnt 0 callback 0a8bb280 expires 1301597388 0 opens 0 writers normal file states (0x5), stat'd, read-only ** Cache entry @ 0x0192a400 for 203.536870921.110.888004 [csail.mit.edu] 19 bytes DV 0 refcnt 2 callback 0a158280 expires 1301604556 0 opens 0 writers mount point states (0x9), stat'd, mt pt valid ** Cache entry @ 0x0198c560 for 203.537011527.1.1 [csail.mit.edu] 370688 bytes DV 38156 refcnt 0 callback 0a8ba500 expires 1301604556 0 opens 0 writers volume root states (0x1), stat'd !!!!!!!!!!!!!!!!!!!!!!!!!! this vvvvvvvvvvvvvvvvvvvvvvv is the output file ** Cache entry @ 0x0194ea30 for 203.537011527.2.2096586 [csail.mit.edu] 1504706560 bytes DV 0 refcnt 1 callback 0a8ba500 expires 1301604555 1 opens 1 writers normal file states (0x21), stat'd !!!!!!!!!!!!!!!!!!!!!!!!!! ** Cache entry @ 0x019dd710 for 203.537011527.3.1184272 [csail.mit.edu] 2048 bytes DV 12 refcnt 0 callback 0a8ba500 expires 1301604556 0 opens 0 writers normal file states (0x1), stat'd ** Cache entry @ 0x019793f0 for 203.537011527.5.1184273 [csail.mit.edu] 2048 bytes DV 12 refcnt 0 callback 0a8ba500 expires 1301604556 0 opens 0 writers normal file states (0x1), stat'd ** Cache entry @ 0x01960400 for 0.1.16778028.1 [dynroot] 24 bytes DV 1 refcnt 0 callback 00000000 expires 0 0 opens 0 writers mount point states (0xd), stat'd, read-only, mt pt valid -GAWollman