New buffer cache code, softdep, and the syncer
Thor Lancelot Simon <[email protected]>
| Newsgroups | gmane.os.netbsd.devel.performance,gmane.os.netbsd.devel.kernel |
|---|---|
| Message-ID | <[email protected]> |
I've been trying to understand the interaction of the new NetBSD buffer cache code and the syncer. In particular, it appears that, when soft dependencies are in use, the following condition can now occur: 1) User runs something like "rm -rf /usr/obj" that causes a large number of directory/metadata changes 2) Buffer cache grows to absorb all relevant changes 3) All changes are entered into the syncer worklist in a second or two 4) When, many seconds later, the syncer gets around to the list that has all the changes in it, the entire system bogs down trying to flush all the I/O in one giant slug. It's my belief that the only reason this couldn't happen before was that the buffer cache couldn't supply enough buffers to alllow it to happen; waiting for buffers to come around via the buffer cache LRU tended to inherently pace the I/O such that it would end up on many worklists for the syncer rather than one. Even if speedup_syncer's called somehow -- which is unlikely, since the buffer cache won't grow this large if there's memory pressure -- it will not really help with the situation, since it can only really cause worklists to be run at twice the normal speed, and here what we would really like would be to spread the I/O load evenly from now into the future, rather than simply changing the second in which it will all land on us at once! It seems like some form of rebalancing I/O towards "now" each time the syncer runs might be the way to go. But I am not sure what the best way to do so might be. Count all items for all seconds, then dole them out in-order evenly to each queue? That seems simple and likely to be highly effective, but perhaps it too severely limits the amount of I/O that may be avoided entirely by caching for a sufficient interval? -- Thor Lancelot Simon [email protected] But as he knew no bad language, he had called him all the names of common objects that he could think of, and had screamed: "You lamp! You towel! You plate!" and so on. --Sigmund Freud