[ ssic-linux-Bugs-1283892 ] Cluster Hangs on write()
"SourceForge.net" <[email protected]>
| Newsgroups | gmane.linux.cluster.ssic.devel |
|---|---|
| Message-ID | <[email protected]> |
Bugs item #1283892, was opened at 2005-09-07 08:30 Message generated for change (Settings changed) made by rogertsang You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=405834&aid=1283892&group_id=32541 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Filesystem Group: v1.9.1 >Status: Closed Resolution: Fixed Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: Cluster Hangs on write() Initial Comment: On my 11 node cluster running FC3 and OpenSSi 1.9.1 the cluster hangs on write() if the file is bigger than approx. 300MB. mounting the cfs disk with sync option solves the problem, but provides bad performance. I found another entry in the list telling that with a manual sync everthing springs to live again. ---------------------------------------------------------------------- Comment By: Roger Tsang (rogertsang) Date: 2007-03-25 03:57 Message: Logged In: YES user_id=1246761 Originator: NO Seems to be resolved by the patch at http://article.gmane.org/gmane.linux.cluster.ssic.devel/5253 ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-10-30 10:28 Message: Logged In: NO One way to reproduce this is mount a physical partition on a non-initnode non-chard. Copy to it a large file twice the size of that node's vm.dirty_threshold. Wait until dirty pages in /proc/meminfo rises above vm.dirty_threshold limit and for the copy process to hang at blk_congestion_wait. Do a sync at that node. Look up /proc/meminfo on that node, dirty pages should still be above vm.dirty_threshold. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-10-30 10:04 Message: Logged In: NO It seems the problem stems from pdflush kthread not being woken on non-initnodes; when the CFS server (CFS mount) is not on the initnode. In fact it even looks like the pdflush process is woken for background_writeout on the initnode, but should be woken on the CFS server (non-initnode). So when writing very large files - bigger than vm.dirty_threshold of the remote node - a number of icssvr_daemon's writing to the same file would hang the cluster waiting for IO in balance_dirty_pages and the CFS client would be hung waiting for rcfsd_write to complete. I can't reproduce this when writing to the CFS server on the initnode. There either is a failure at writeback_inodes call or pdflush_operation call to pdflush's background_writeout function. ---------------------------------------------------------------------- Comment By: Roger Tsang (rogertsang) Date: 2005-09-29 01:14 Message: Logged In: YES user_id=1246761 A follow up of this bug is here http://marc.theaimsgroup.com/?l=ssic-linux-devel&m=112744789721893&w=2 ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-09-09 01:55 Message: Logged In: NO >What is the underlying filesystem and device that you were >writing to? Is this a remote CFS write or write to local >device? Usually its a Remote device, a node writes back to the masternode. It allways happens wenn two nodes write at the same time. > Are you writing to device-mapper device? If you ask wether the device is a distributed block device, the answer is no. >Can you print out the backtrace of the process from kdb on the node >that you are writing from? because it usually happens with parallel processes, I will see to do that. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-09-09 01:55 Message: Logged In: NO >What is the underlying filesystem and device that you were >writing to? Is this a remote CFS write or write to local >device? Usually its a Remote device, a node writes back to the masternode. It allways happens wenn two nodes write at the same time. > Are you writing to device-mapper device? If you ask wether the device is a distributed block device, the answer is no. >Can you print out the backtrace of the process from kdb on the node >that you are writing from? because it usually happens with parallel processes, I will see to do that. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-09-08 11:04 Message: Logged In: NO >What is the underlying filesystem and device that you were >writing to? Is this a remote CFS write or write to local >device? Usually its a Remote device, a node writes back to the masternode. It allways happens wenn two nodes write at the same time. > Are you writing to device-mapper device? If you ask wether the device is a distributed block device, the answer is no. >Can you print out the backtrace of the process from kdb on the node >that you are writing from? because it usually happens with parallel processes, I will see to do that. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-09-08 06:51 Message: Logged In: NO >What is the underlying filesystem and device that you were >writing to? Is this a remote CFS write or write to local >device? Usually its a Remote device, a node writes back to the masternode. It allways happens wenn two nodes write at the same time. > Are you writing to device-mapper device? If you ask wether the device is a distributed block device, the answer is no. >Can you print out the backtrace of the process from kdb on the node >that you are writing from? because it usually happens with parallel processes, I will see to do that. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-09-08 06:47 Message: Logged In: NO An occational sync solves the problem and a possible woarkaround is to do a sync, say every minute or so. But its not a very convinient way. Any ideas ? ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2005-09-07 17:33 Message: Logged In: NO What is the underlying filesystem and device that you were writing to? Is this a remote CFS write or write to local device? Are you writing to device-mapper device? Can you print out the backtrace of the process from kdb on the node that you are writing from? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=405834&aid=1283892&group_id=32541 ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/