Kernel Traffic #248 by Zack Brown
[email protected] Tue, 20 Jan 2004 08:08:42 -0800
| Newsgroups | gmane.linux.kernel.kernel-traffic |
|---|---|
| Message-ID | <[email protected]> |
Kernel Traffic #248 For 9 Jan
By Zack Brown
Table Of Contents
* Standard Format
* Text Format
* XML Source
* Introduction
* Mailing List Stats For This Week
* Threads Covered
1. 22 Dec 2003 - 2 Jan (54 Researching SCO's Infringement Claims
posts)
2. 22 Dec 2003 - 2 Jan (31 Some Discussion Of Process Load
posts) Balancing And Priority Handling
3. 23 Dec 2003 - 29 Dec 2003 (13 SysFS Class Patches To Ease Driver
posts) Support
4. 24 Dec 2003 - 29 Dec 2003 (4 udev 011 Released
posts)
5. 26 Dec 2003 - 29 Dec 2003 (28 Increasing PAGE_SIZE For 2.7
posts)
6. 26 Dec 2003 - 2 Jan (15 Adaptec/DPT I2O Gone In 2.6;
posts) Replacement Sought
7. 28 Dec 2003 - 30 Dec 2003 (18 Some RAID Recommendations
posts)
8. 28 Dec 2003 - 29 Dec 2003 (11 Status Of ATARAID In 2.6
posts)
9. 28 Dec 2003 - 29 Dec 2003 (8 Problems With Accusys Drives
posts)
10. 29 Dec 2003 - 30 Dec 2003 (27 Status Of 2.6 Maintainership
posts)
11. 29 Dec 2003 - 1 Jan (13 Patch Submission Policies
posts)
12. 29 Dec 2003 - 31 Dec 2003 (16 BitKeeper Usage Policy And Advice
posts)
13. 29 Dec 2003 (1 Status Of BitKeeper Snapshots For 2.6
post)
14. 29 Dec 2003 (1 USB Updates For 2.6
post)
15. 29 Dec 2003 - 30 Dec 2003 (2 Experimental Net Driver Updates For
posts) 2.6
16. 30 Dec 2003 (1 SCSI Updates
post)
17. 30 Dec 2003 (1 Prototype For BitKeeper 'Undo
post) Changeset' Improvement
18. 30 Dec 2003 (2 Status Of kernel.bkbits.net
posts)
19. 31 Dec 2003 - 2 Jan (22 Linux 2.6.1-rc1 Released; Unanswered
posts) Questions About 2.6 Release Policy
20. 31 Dec 2003 (4 Linux 2.4.24-pre3 Released
posts)
21. 31 Dec 2003 (1 udev 012 Released
post)
22. 1 Jan - 3 Jan (8 Trouble With udev Removable Media
posts) Handling
23. 2 Jan (2 Fair Scheduling In 2.4
posts)
Introduction
In response to Issue #247, Section #3 (18 Dec 2003: Real-Time Maintainership;
Nanokernel Maintainership; Patent Policy) , Karim Yaghmour (author of Building
Embedded Linux Systems (http://www.oreilly.com/catalog/belinuxsys/) ) sent me
an email saying:
I was going through KT247 and noticed the summary of the real-time
maintainership discussion. Most of the summary is right on the spot and I
have nothing to add about it. The only thing I'd like to bring to your
attention is Linus' statement on the issue.
As I pointed out in my response to him (http://marc.theaimsgroup.com/?l=
linux-kernel&m=107208752427058&w=2), Linus is making major mistakes on this
issue. For example, the mail you quote has him saying: "It's doubly
discgusting with some of the people who were trying to spread all the FUD
and mis-information were doing so because they were themselves doing a
non-GPL microkernel, and they complained about how the patents were somehow
against the GPL and wanted to get community support by trying to make out
the situation to be somehow different from what it was."
Yet, neither I or anyone involved in this debate has ever asked for a
non-GPL microkernel. In fact, I was quoted by KT in June 2002 (http://
kt.zork.net/kernel-traffic/kt20020609_170.html#1) as saying: "We don't want
a non-GPL real-time executive or a non-GPL OS. All we want is the right to
develop applications using our licenses as others are for Linux."
In addition, Linus' claim that I'm somehow spreading FUD is contradicted by
Victor Yodaiken's own statements on the issues, as I explained in my reply
to Linus (see above URL.)
I do understand that given Linus' community stance, his word is worth
reporting. That being said, Linus' stance doesn't mean he's infallible and
I think KT readers would profit from having the full facts presented to
them, even when such facts show that a prominent figure of the community is
mistaken.
I'm including Karim's comments (with permission) because it seems like a
relevant part of the debate. Further discussion should of course go to
linux-kernel.
Mailing List Stats For This Week
We looked at 1221 posts in 5976K.
There were 389 different contributors. 189 posted more than once. 144 posted
last week too.
The top posters of the week were:
* 50 posts in 245K by Greg KH
* 47 posts in 300K by Geert Uytterhoeven
* 37 posts in 149K by Linus Torvalds
* 33 posts in 195K by Jeff Garzik
* 22 posts in 109K by Andrew Morton
* Full Stats
1. Researching SCO's Infringement Claims
22 Dec 2003 - 2 Jan (54 posts) Archive Link: "SCO's infringing files list"
Topics: Ioctls, POSIX
People: Tom Felker, Erik Andersen, Linus Torvalds, J.W. Schultz, Stan Bubrouski
, Florian Weimer, Giacomo A. Catenazzi, Alexander Viro, Xose Vazquez Perez,
Mitchell Blank Jr., H. J. Lu, Andries Brouwer, Alan Cox, Mitchell Blank Jr
Stan Bubrouski posted a list of files the SCO claimed were infringed by Linux.
Among these were 'include/asm-i386/errno.h'. Tom Felker said, "The original
errno.h, from linux-0.01, says it was taken from minix, and goes up to 40.
Between linux-0.96c and linux-0.97, that file was replaced with the present
version, which includes the error strings and goes up to 121. Where did the
0.97 to present version come from?" Erik Andersen replied, "For errno.h,
according to this: http://www.geocrawler.com/archives/3/360/1997/11/0/1999771/.
I got Linus to add ENOMEDIUM and EMEDIUMTYPE into the kernel for 2.0.32, as
part of my work on (what was to become) the Uniform cdrom driver, based on
original work from David van Leeuwen. http://www.ussg.iu.edu/hypermail/linux/
kernel/9704.0/0105.html. I then helped push these defines into glibc and libc5.
So at least those two defines are clearly not SCO derived..." Close by, Linus
Torvalds also said to Stan:
Good eyes - I only analysed the ctype.h thing, and didn't look up errno.h
in the original sources. errno.h has a _big_ comment saying where the
numbers came from (and some swearwords about POSIX ;)
Looking at signal.h, those numbers also seem to largely match minix. Which
makes sense - I actually had access to them.
In both cases it's only the numbers that got copied, though. And not all of
them either - for some reason I tried to make the signal numbers match
(probably lazyness - not so much that I cared about the numbers themselves,
but about the list of signal names), but for example the SA_xxxx macros -
in the very same file - bear no relation to the minix ones.
J.W. Schultz said, "And for the names, perhaps they would care to sue The Open
Group? http://www.opengroup.org/onlinepubs/007904975/basedefs/signal.h.html.
And that probably applies to the rest of these header files." Stan replied:
Just to shore up what jw was talking about:
http://www.opengroup.org/onlinepubs/007904975/basedefs/signal.h.html
http://www.opengroup.org/onlinepubs/007904975/basedefs/errno.h.html
http://www.opengroup.org/onlinepubs/007904975/basedefs/ctype.h.html
and for a listing of all the defines the open group has (dir listing):
http://www.opengroup.org/onlinepubs/007904975/basedefs/
Florian Weimer also replied to J. W., saying that the comments were more of a
problem than the names. For 'errno.h', he said:
The comments were added in Linux 0.99.1, and I'm not sure what was the
source. For example, Linux has:
#define ENOTTY 25 /* Not a typewriter */
Solaris:
#define ENOTTY 25 /* Inappropriate ioctl for device */
Current POSIX:
[ENOTTY]
Inappropriate I/O control operation.
I couldn't find any historic Minix header files. Minix 2 has:
#define ENOTTY (_SIGN 25) /* inappropriate I/O control operation */
Giacomo A. Catenazzi replied:
In http://www.opensource.apple.com/darwinsource/DevToolsMay2002/gcc-937.2/
libiberty/strerror.c
/* Extended support for using errno values.
Written by Fred Fish. [email protected]
This file is in the public domain. --Per Bothner. */
(...)
#if defined (ENOTTY)
ENTRY(ENOTTY, "ENOTTY", "Not a typewriter"),
#endif
FYI there was a proposed patch to change "Not a typewriter" to
"Inappropriate ioctl for device". Check the interesting thread of lkml:
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/0471.html
Linus replied:
Something like that may well be the source of the string. Fred Fish was
active long before this timeframe (if it's the same Fred Fish - he used to
do freeware collections for the Amiga in the '80's).
But there were multiple libc's around (estdio, libc5, glibc..), and it
could be any of them.
Trying to find the kernel list archives from that timeframe would likely
clarify the issue. There were several lists back then: "linux-activists"
mailing list, and of course the "comp.os.linux" newsgroup (this was before
it split into multiple newsgroups).
I've found some archives for linux-activists, but no newsgroup archives
going that far back.. Anybody?
Alexander Viro replied:
http://groups.google.com/advanced_group_search
They'd done what DN had promised and never did - merged old usenet archives
into their database, all way back to '81.
The earliest they have on comp.os.linux is March 1992 and results of voting
on newgroup had been posted by tytso on Mar 25 1992. There's a bogus
crosspost into (still not existing) c.o.l on Mar 21 and regular postings
starting on Mar 31. IOW, archive goes all way back to the group creation.
alt.os.linux archives there start on Jan 19 1992 (newgrouped somewhere
around Jan 17, took several days to propagate). Before that it's c.o.minix,
but by the time you are looking for migration from c.o.m should've been
over.
Xose Vazquez Perez also said that Alan Cox "used to keep some archives at:
http://ftp.linux.org.uk/pub/linux/alan/Kernel/Documents/Old-Funet-Lists/" .
Mitchell Blank Jr. also pointed out:
the linux-activists messages for that period would be in:
http://www.ibiblio.org/pub/historic-linux/ftp-archives/tsx-11.mit.edu/
Oct-07-1996/mail-archive/linux-activists/Volume2/Linux-V2-5XX.tar.z (http:/
/www.ibiblio.org/pub/historic-linux/ftp-archives/tsx-11.mit.edu/Oct-07-1996
/mail-archive/linux-activists/Volume2/Linux-V2-5XX.tar.z)
Specifically, the files "digest51[789] digest52[012]" are the neighborhood
around july 25th. I did a quick grep for "errno" and a few strings from the
errno.h file and didn't see anything relevant pop up, though.
Andries Brouwer and others found a relevant post (http://groups.google.com/
groups?&selm=1992Jul19.154248.9076%40serval.net.wsu.edu) from that era (July
1992), in which H. J. Lu apparently authored or collected some errno.h changes
that might have been relevant. As Mitchell pointed out close by, "this is back
when gcc and libc were distributed together, both by H J Lu." Mitchell also
noticed that in H. J.'s announcement long ago, the file list of his patches
included 0.96bp2inc.tar.Z, "the kernel header files for 0.96b patch level 2" .
Mitchell said that the file "seems to be a modified version of the 0.96bp2
header files needed in order to work with the new gcc release (searching for
that filename turns up a message discussing it a little) So I'm guessing that
the July 25, 1992 errno.h in the linux tree is a merge from this code. Now,
does anyone have a copy of "0.96bp2inc.tar.Z" lying around?" Linus replied:
Ok, this is the source.
In particular, I can re-create _exactly_ the linux-0.97 "errno.h" file by
using the "sys_errlist[]" contents from "libc-2.2.2". In particular, this
trivial loop will generate the exact (byte-for-byte) list that is in the
kernel:
int i;
for (i = 1; i < 122; i++) {
const char *name = names[i];
int n = strlen(name);
char *tabs = "\t\t"+(n > 7);
const char *expl = libc222_errlist[i];
printf("#define\t%s%s%2d\t/* %s */\n",
name, tabs, i, expl);
}
here, the "names[]" array was filled in with the error names, ie
const char *names[] = { "none",
"EPERM", "ENOENT", "ESRCH", "EINTR", "EIO", "ENXIO", "E2BIG",
...
and the "libc222_errlist[]" array was filled in with the strings found by
just downloading the old "libc-2.2.2" binary that can still be found at
http://www.ibiblio.org/pub/Linux/libs/oldlibs/libc-2.2.2/
and then just doing a "strings - libc-2.2.2" and "sys_errlist[]" will be
obvious:
static char *libc222_errlist[] = {
"Unknown error",
"Operation not permitted",
...
This was literally a five-minute hack (I wrote the silly loop yesterday to
see what it does with the current "strerror()" - there is very good
correlation even today, but using the libc-2.2.2 sys_nerrlist[] you get
_exactly_ the same result).
So this is definitely the source of the kernel error header. It's either a
file from the libc sources, or it is literally auto-generated like the
above (I actually suspect the latter - now that I did the auto-generation
it all felt very familiar, but that may just be my brain rationalizing
things. Humans are good at rationalizing reality.).
Can anybody find the actual libc _sources_? Not the kernel headers that hjl
mentions (those are the old ones from _before_ the change), but the file
"libc-2.2.2.tar.Z"?
Anyway, we know where the kernel header comes from. Let's figure out where
the libc data comes from.
Elsewhere, and still regarding 0.96bp2inc.tar.Z, Mitchell said:
BTW, a few more details on this file - the linux GCC 2.2.2 release was
originally announced 28-Jun-1992. The 0.96bp2inc.tar.Z file originally
lived on the then-primary linux ftp site banjo.concert.net in directory pub
/Linux/GCC.
banjo stopped being an FTP server a couple months later - however, Jonathan
Magid announced on 13-Aug-1992 that the entire banjo site was being
reincarnated at host reggae.oit.unc.edu in directory ftp/pub/pc-stuff/
Linux. Here's a copy of the announcement: http://www.kclug.org/old_archives
/linux-activists/1992/aug/1/0708.shtml
My understanding is that reggae.oit morphed at some point into
sunsite.unc.edu (which is now, of course, ibiblio.org) Jonathan still
appears to be there, so I'm cc:ing him on this (apologies in advance if its
an intrusion, Jonathan) on the off-chance that there might still be a
1992-era archive of the linux files once hosted by banjo.
The only other person likely to have access to a copy is H J Lu himself
(also cc:'ed although I'm 99% sure he's still on lkml :-)
Linus replied:
Note that we really don't care about that "0.96bp2inc.tar.Z" file: that's
just the kernel headers, and 0.96b-pl2 did _not_ contain the comments yet.
But libc used to use the kernel headers for other things (for things like
system call numbers etc).
It's almost certainly the "libc-2.2.2.tar.Z" file that we want - that's the
one that is going to contain the sys_errlist[] lists etc. Note how this
libc-2.2.2 announcement predates the merging of the kernel header by almost
a month - the kernel header information came from libc, not the other way
around.
He went on:
Does anybody have old CD-ROM's lying around?
In particular, the Yggdrasil Linux/GNU/X alpha CD-ROM was apparently
released just a few months later. It would quite possibly contain the
libc-2.2.2 sources... Adam Richter is still active, and I added him to the
cc..
Who else was doing CD's back then? SLS? If nobody has the thing on a
web-site any more, maybe they exist in physical format on somebodys
bookshelf? The only reason that the really historic kernel archives still
exist is that people saved them, and even so we're missing versions 0.02
and 0.03, but by the latter half of -92 there were already CD-ROMs being
manufactured...
Of course, maybe the CD's are unreadable by now.
Andries said, "I just uploaded a copy" [of libc-2.2.2.tar.gz] "to ftp://
ftp.win.tue.nl/pub/linux-local/libc.archive/libc/libc-222.taz" Linus replied:
Yup, and I can confirm two things:
+ the strings match 100% (well, duh, we already saw that from the binary)
+ it doesn't even have an "errno.h"
that, together with the timing, pretty much proves that the kernel header
was indeed just auto-generated from sys_errlist[] of that timeframe, with a
program very much like the one I already posted.
Now, the libc file just says
/* This is a list of all known signal numbers. */
(which is obviously just a cut-and-paste from siglist.c n the same
directory). But it shouldn't much matter, since I don't think SCO really is
going to try to claim copyright ownership of the result of standard C
library interactions like using "sys_errlist[]".
(I take that back - _of_course_ they are going to try to claim ownership.
After all, they already claimed ownership of code I provably wrote).
2. Some Discussion Of Process Load Balancing And Priority Handling
22 Dec 2003 - 2 Jan (31 posts) Archive Link: "[PATCH] 2.6.0 batch scheduling,
HT aware"
Topics: Hyperthreading
People: Con Kolivas, Nick Piggin, Jun Nakajima, Bill Davidsen
Con Kolivas said:
I've done a resync and update of my batch scheduling that is also
hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use
cpu time if there is idle time available, rather than having a proportion
of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
If you have a hyperthread (P4HT) processor and run it as two logical cpus
you can have a very low priority task running that can consume 50% of your
physical cpu's capacity no matter how high priority tasks you are running.
For example if you use the distributed computing client setiathome you will
be effectively be running at half your cpu's speed even if you run
setiathome at nice 20. Batch scheduling for normal cpus allows only idle
time to be used for batch tasks, and for HT cpus only allows idle time when
both logical cpus are idle.
This is not being pushed for mainline kernel inclusion, but the issue of
how to prevent low priority tasks slowing down HT cpus needs to be
considered for the mainline HT scheduler if it ever gets included. This
patch provides a temporising measure for those with HT processors, and a
demonstrative way to handle them in mainline.
Patch available here:
http://ck.kolivas.org/patches/2.6/2.6.0/
Nick Piggin replied:
I wonder how does Intel suggest we handle this problem? Batch scheduling
aside, I wonder how to do any sort of priorities at all? I think POWER5 can
do priorities in hardware, that is the only sane way I can think of doing
it.
I think this patch is much too ugly to get into such an elegant scheduler.
No fault to you Con because its an ugly problem.
How about this: if a task is "delta" priority points below a task running
on another sibling, move it to that sibling (so priorities via timeslice
start working). I call it active unbalancing! I might be able to make it
fit if there is interest. Other suggestions?
Jun Nakajima replied:
Today utilization of execution resources of a logical processor is around
60% as you can find in public papers, and it's dependent on the processor
implementation and the workload. It could be higher in the future, and
their relative priority could be much higher then. So I don't think it's a
good idea to hard code such a implementation-specific factor into the
generic scheduler code.
Regarding H/W-based priority, I'm not sure it's very useful especially
because so many events happen inside the processor and a set of the
execution resources required changes very rapidly at runtime, i.e. the H/W
knows what it should do to run faster at runtime, and imposing priority on
those logical processor could make them run slower.
Nick said his idea would not involve hard-coding implementation-specific stuff
into the generic scheduler code; "The mechanism would be generic, but the
parameters would be arch specific" . He added that he agreed with Jun, that a
hardware-based priority system would not be entirely sufficient, if only
because hardware that didn't provide the means for a hardware-based solution
would still need software to do the same thing; so the software would need to
be written regardless.
Elsewhere, Con replied to Nick's specific 'active unbalancing' idea, saying, "I
discussed this with Ingo and that's the sort of thing we thought of. Perhaps a
relative crossover of 10 dynamic priorities and an absolute crossover of 5
static priorities before things got queued together. This is really only
required for the UP HT case." Nick disagreed, and over the course of a little
back-and-forth, said that the multi-CPU situation was "the same problem. A nice
-20 process can still lose 40-55% of its performance to a nice 19 process, a
figure of 10% is probably too high and we'd really want it <= 5% like what
happens with a single logical processor." And Con agreed.
Close by, Bill Davidsen also said:
There are two goals here. Not having a batch process on one siling makes
sense, and I'm going to try Con's patch after I try Nick's latest.
Actually, if they play nicely I would use both, batch would be very useful
for nightly report generation on servers.
But WRT the whole HT scheduling, it would seem that ideally you want to
schedule the two (or N) processes which have the lowest aggregate cache
thrash, if you had a way to determine that. I suspect that a process which
had a small itterative inner loop with a code+data footprint of 2-3k would
coexist well with almost anything else. Minimizing the FPU contention also
would improve performance, no doubt. I don't know that there are the tools
at the moment to get this information, but it seems as though until it's
available any scheduling will be working in the dark to some extent.
Con said the patches would never play nicely, but they might be able to live
together in some way. Regarding the existence of tools to get the information
Bill wanted, Con added, "Impossible with current tools. Only userspace would
have a chance of predicting this and the simple rule we work off is that
userspace can't be trusted so this does not appear doable in the foreseeable
future." Bill thought it would be hard to make much headway without a solution
to this problem.
3. SysFS Class Patches To Ease Driver Support
23 Dec 2003 - 29 Dec 2003 (13 posts) Archive Link: "[PATCH] sysfs class patches
- take 2 [0/5]"
Topics: FS: sysfs
People: Greg KH, Jeff Garzik
Greg KH said:
Here are the sysfs class patches reworked against a clean 2.6.0 tree. I've
created a class_simple.c file that contains a "simple" class device
interface. I've then converted the tty core to use this interface (the
combo of these two patches makes for no extra code added).
Then there are 3 patches, adding class support for misc, mem, and vc class
devices. As the interface to add simple class support for devices is now so
low, I feel that we do need to have mem class support as to not special
case any char device.
With these patches, it's now much easier for others to implement class
support for remaining char drivers/subsystems that do not have it yet.
Jeff Garzik remarked, "Interesting... I bet that will be useful to the iPAQ
folks (I've been wading through their patches lately), as they have created a
couple ultra-simple classes for SoC devices and such." Greg KH replied, "I bet
it will. I've ported my old frame buffer patch to use it, and it saved a lot of
code."
4. udev 011 Released
24 Dec 2003 - 29 Dec 2003 (4 posts) Archive Link: "[ANNOUNCE] udev 011 release"
Topics: FS: devfs, FS: sysfs, Hot-Plugging, Klibc, Version Control
People: Greg KH
Greg KH announced:
I've released the 011 version of udev. It can be found at: kernel.org/pub/
linux/utils/kernel/hotplug/udev-011.tar.gz (http://www.kernel.org/pub/linux
/utils/kernel/hotplug/udev-011.tar.gz)
rpms built against Red Hat FC1 are available at: kernel.org/pub/linux/utils
/kernel/hotplug/udev-011-1.i386.rpm (http://www.kernel.org/pub/linux/utils/
kernel/hotplug/udev-011-1.i386.rpm) with the source rpm at: kernel.org/pub/
linux/utils/kernel/hotplug/udev-011-1.src.rpm (http://www.kernel.org/pub/
linux/utils/kernel/hotplug/udev-011-1.src.rpm)
udev is a implementation of devfs in userspace using sysfs and /sbin/
hotplug. It requires a 2.6 kernel to run. Please see the udev FAQ for any
questions about it: kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ (
http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ)
The major changes since the 010 release are:
+ The _long_ time delay on startup if you used the udev init script is
now gone. We now fork a new process of udev for every device in the
sysfs tree, causing any slow udev proceses (due to the type of device)
to be not noticeable by the user.
+ The RPM package is built against klibc. If you have any problems with
this image, please let me know.
+ The big "always wait 1 second" problem with the 010 release is now
pretty much fixed. I still need some libsysfs changes to work around
some of the delays I had to hard code in, but for any device that is
not a partition, and has a "device" symlink, udev will not sleep at
all. If a device does not have a "device" symlink, we will wait around
for a few seconds to see if one is going to be created or not, and then
move on. It's the only sane way of handling the issue of not knowing
what kinds of devices have symlinks and which ones do not.
+ fixed a few bugs when devices did not have a "device" symlink. Rules
were getting applied when they should not have been.
+ LABEL and CALLOUT rules now no longer require the BUS key. If the BUS
key is not present, the LABEL or CALLOUT rule will be always be
checked. This allows us to use these types of rules for devices without
a "device" symlink.
+ a few other minor tweaks and bug fixes too.
Thanks again to everyone who has send me patches for this release, a full
list of everyone, and their changes is below.
udev development is done in a BitKeeper repository located at: bk://
linuxusb.bkbits.net/udev
Daily snapshots of this tree used to be found at: http://
www.codemonkey.org.uk/projects/bitkeeper/udev/. But that box seems to be
down now. Hopefully it will be restored someday. If anyone ever wants a
tarball of the current bk tree, just email me.
5. Increasing PAGE_SIZE For 2.7
26 Dec 2003 - 29 Dec 2003 (28 posts) Archive Link: "Page Colouring (was: 2.6.0
Huge pages not working as expected)"
Topics: Executable File Format, Virtual Memory
People: Linus Torvalds, William Lee Irwin III, Mike Fedyk, Daniel Phillips,
Rusty Russell, Hugh Dickins, Eric W. Biederman, Andrew Morton
In the course of discussion, Eric W. Biederman suggested increasing PAGE_SIZE
in the kernel. Linus Torvalds replied:
Yes. This is something I actually want to do anyway for 2.7.x. Dan Phillips
had some patches for this six months ago.
You have to be careful, since you have to be able to mmap "partial pages",
which is what makes it less than trivial, but there are tons of reasons to
want to do this, and cache coloring is actually very much a secondary
concern.
William Lee Irwin III said, "I've not seen Dan Phillips' code for this. I've
been hacking on something doing this since late last December." Mike Fedyk
said, "I remember his work on pagetable sharing, but haven't heard anything
about changing the page size from him. Could this be what Linus is remembering?
" William said, "Doubtful. I suspect he may be referring to pgcl (sometimes
called "subpages"), though Dan Phillips hasn't been involved in it. I guess
we'll have to wait for Linus to respond to know for sure." And Linus said:
I didn't see the patch itself, but I spent some time talking to Daniel
after your talk at the kernel summit. At least I _think_ it was him I was
talking to - my memory for names and faces is basically zero.
Daniel claimed to have it working back then, and that it actually shrank
the kernel source code. The basic approach is to just make PAGE_SIZE
larger, and handle temporary needs for smaller subpages by just dynamically
allocating "struct page" entries for them. The size reduction came from
getting rid of the "struct buffer_head", because it ends up being just
another "small page".
He asked Daniel Phillips for the details, and Daniel said:
Your description is accurate. Another reason for code size shrinkage is
getting rid of the loops across buffers in the block IO library, e.g.,
block_read_full_page.
Subpages only make sense for file-backed memory, which conveniently lets
the page cache keep track of subpages. Each address_space has pages of all
the same size, which may be smaller, larger or the same as PAGE_CACHE_SIZE.
The first case, "subpages", is the interesting one.
An address_space with subpages has base pages of PAGE_CACHE_SIZE for its
"even" entries and up to N-1 dynamically allocated struct pages for the
"odd" entries where N is PAGE_CACHE_SIZE divided by the subpage size. Base
pages are normal members of mem_map. Subpages are not referenced by
mem_map, but only by the page cache. They are created by operations such as
find_or_create_page, which first creates a base page if necessary. A
counter field in the page flags of the base page keeps track of how many
subpages share a base page's physical memory; when this field goes to zero
the base page may be removed from the page cache.
Subpages always have a ->virtual field regardless of whether mem_map pages
do. This is used for virt_to_phys and to locate the base page when a
subpage is freed.
Page fault handling doesn't change much if at all, since the faulting
address is rounded down to a physical page, which will be a base page.
Most of the changes for subpages are in the buffer.c page cache operations
and are largely transparent to the VMM, though PAGE_CACHE_SHIFT becomes
mapping->page_shift, which touches a lot of files. As you noted,
buffer_head functionality can be taken over by struct page and buffers
become expendible. However it is not necessary to cross that bridge
immediately; page buffer lists continue to work though the buffer list is
never longer than one.
With a little more work, subpages can be used to shrink mem_map: implement
a larger PAGE_CACHE_SIZE then use subpages to handle ABI problems. In this
case faults on subpages are possible and the fault path probably needs to
know something about it. With a larger-than-physical PAGE_CACHE_SIZE we can
finally have large buffers, though the kernel would have to be compiled for
it. Some more work to allowing mapping->page_shift to be larger than
PAGE_CACHE_SIZE would complete the process of generalizing the page size.
My impression is, this isn't too messy, most of the impact is on faulting.
Bill and others are already familiar with this I think. The work should
dovetail.
I took a stab at implementing subpages some time ago in 2.4 and got it
mostly working but not quite bootable. I did find out roughly how invasive
the patch is, which is: not very, unless I've overlooked something major.
I'll get busy on a 2.6 prototype, and of course I'll listen attentively for
reasons why this plan won't work.
Close by, William said to Linus, "I did get a positive reaction from you at KS,
and I've also been slaving away at keeping this thing current and improving it
when I can for a year. Would you mind telling me what the Hell is going on
here? I guess I already know I'm screwed beyond all hope of recovery, but I
might as well get official confirmation." Linus replied:
I haven't even _looked_ at any 2.7.x timeframe patches, and I'm not even
going to for the next few months.
I don't care what does it, I want a bigger PAGE_CACHE_SIZE, and working
patches are the only thing that matters. But for now, I have my 2.6.x
blinders on.
William still felt discouraged, though he supposed there was still some hope.
Linus replied:
Well, I don't even know what your approach is - mind giving an overview?
My original plan (and you can see some of it in the fact that
PAGE_CACHE_SIZE is separate from PAGE_SIZE), was to just have the page
cache be able to use bigger pages than the "normal" pages, and the normal
pages would continue to be the hardware page size.
However, especially with mem_map[] becoming something of a problem, and all
the problems we'd have if PAGE_SIZE and PAGE_CACHE_SIZE were different, I
suspect I'd just be happier with increasing PAGE_SIZE altogether (and
PAGE_CACHE_SIZE with it), and then just teaching the VM mapping about
"fractional pages".
What's your approach?
William replied:
I presented on this at KS. Basically, it's identical to Hugh Dickins'
approach from 2000. The only difference is really that it had to be forward
ported (or unfortunately in too many cases reimplemented) to mix with
current code and features.
Basically, elevate PAGE_SIZE, introduce MMUPAGE_SIZE to be a nice macro
representing the hardware pagesize, and the fault handling is done with
some relatively localized complexity. Numerous s/PAGE_SIZE/MMUPAGE_SIZE/
bits are sprinkled around, along with a few more involved changes because a
large number of distributed changes are required to handle oddities that
occur when PAGE_SIZE changes from 4KB. The more involved changes are often
for things such as the only reason it uses PAGE_SIZE is really that it just
expects 4KB and says PAGE_SIZE, or that it wants some fixed (even across
compiles) size and needs updating for more general PAGE_SIZE numbers, or
sometimes that it expects PAGE_SIZE to be what a pte maps when this is now
represented by MMUPAGE_SIZE. I have a bad feeling the diligence of the
original code audit could be bearing against me (and though I'm trying to
be equally diligent, I'm not hugh).
The fact merely elevating PAGE_SIZE breaks numerous things makes me rather
suspicious of claims that minimalistic patches can do likewise.
The only new infrastructures introduced are the MMUPAGE_SIZE and a couple
of related macros (defining numbers, not structures or code) and the fault
handler implementations. The diff size is not small. The memory footprint
is, and demonstrably so (c.f. March 27 2003).
My 2.6 code has been heavily leveraging the pfn abstraction in its favor to
represent physical addresses measured in units of the hardware pagesize.
Generally, my maintenance approach has been incrementally advancing the
state of the thing while keeping it working on as broad a cross section of
i386 systems as I can test or get testers on. It has been verified to run
userspace on Thinkpad T21's and 16x/32GB and 32x/64GB NUMA-Q's at every
point release it's been ported to, which since 2.5.68 or so has been every
point release coming out of kernel.org.
Rusty Russell asked, "Can you give an example? One approach is to simply
present a larger page size to userspace w/ getpagesize(). This does break ELF
programs which have been laid out assuming the old page size (presumably they
try to mprotect the read-only sections). On PPC, the ELF ABI already insists on
a 64k boundary between such sections, and maybe for others you could simply
round appropriately and pray, or do fine-grained protections (ie. on real
pagesize) for that one case." And William replied:
Apps must, of course, be relinked for that, but that's userspace. This ABI
change is largely out of the picture due to legacy binaries, user
virtualspace fragmentation (most likely an issue for 32-bit threading), and
so on. The choice of PAGE_SIZE in such schemes is also restricted to no
larger than whatever choice used for userspace linking, which is a
relatively ugly dependency. There's also a question of "smooth transition":
the only way to "incrementally deploy" it on a mixture "ready" userspace
and "unready" userspace is to turn it off. I suppose it has the minor
advantage of being trivial to program.
I had in mind pure kernel internal issues, not ABI.
The issues from raising PAGE_SIZE alone are things like interpreting
hardware descriptions in arch code, some shifts underflowing for things
like hashtables, certain drivers doing ioremap() and the like either
filling up vmallocspace or getting their math wrong, and some other drivers
doing calculations on physical addresses getting them wrong, or using
PAGE_SIZE to represent some 4KB or other fixed-size memory area interpreted
by hardware, and filesystems that assume blocksize == PAGE_SIZE or assume
PAGE_SIZE is less than some particular value (e.g. short offsets into
pages, worst of all being signed shorts), and tripping BUG()'s in
ll_rw_blk.c when 512*q->max_sectors < PAGE_SIZE.
These issues are the bulk of the work needing to be done for the driver and
fs sweeps. Actual concerns about MMUPAGE_SIZE in drivers/ and fs/ are
rather limited in scope, though drivers/char/drm/ was somewhat painful to
get going (Zwane actually did most of this for me, as I have no DRM/DRI
-capable graphics cards at my disposal).
Mike asked if there were some way for William to split his patch into smaller
chunks, and merge into Andrew Morton's -mm tree; and William said:
I talked about this for a little while. Basically, there is only one
concept in the entire patch, despite its large size. The vast bulk of the
"distributed changes" are s/PAGE_SIZE/MMUPAGE_SIZE/.
At some point I was told to keep the whole shebang rolling out of tree or
otherwise not answered by akpm and/or Linus, after I sent in what a split
up (this is actually very easy to split up file-by-file) version of what
just some of the totally trivial arch/i386/ changes would look like. The
nontrivial changes are stupid in nature, but touch "fragile" or otherwise
"scary to touch" code, and so sort of relegate them to 2.7. This is not
entirely unjustified, as changes of a similar code impact wrt. the GDT
appear to have affected some APM systems' suspend ability (I know for a
fact my changes do not have impacts on APM suspend, but other, analogous
support issues could arise after broader testing.)
Basically, the MMUPAGE_SIZE introductions didn't interest anyone a while
ago, and I suspect people probably just want them all at once, since it's
unlikely people want to repeat the pain analogous to PAGE_CACHE_SIZE (I
should clarify later how this is different) where the incremental
introduction never culminated in the introduction of functionality.
6. Adaptec/DPT I2O Gone In 2.6; Replacement Sought
26 Dec 2003 - 2 Jan (15 posts) Archive Link: "Adaptec/DPT I2O Option Omitted
>From Linux 2.6.0 Kernel Configuration Tool"
Topics: Ioctls
People: Samuel Flory, Go Taniguchi
Leon Toh asked whatever became of the Adapec/DPT I2O option in the 2.6 kernel.
It was gone. Samuel Flory replied, "The DPT I2O driver was never converted to
the new driver model. The driver from what I can see is a mess. It doesn't even
compile in 2.4 for a number of archs like amd64. A while back a bunch of people
(including myself) raised the concern through various channels with adaptec. In
theory someone at adaptec is working on it, but there was not an ETA." Leon
started tinkering, trying to get the thing working again under 2.6; and Samuel
said, "You might want to hold off on doing a lot of work for a bit. I think
there was a beta driver that was being passed around." He found it in his own
archives, and said he'd try to find out the current status. Go Taniguchi
replied:
This is my patch. http://pkgcvs.turbolinux.co.jp/~go/patch-2.6/
dpt_i2o.patch
Worked fine for me on quad xeon with 4G mem and 64bit PCI. It include.
+ Support 2.6 kernel and DMA-mapping
+ ioctl fix for raid tools
+ use schedule_timeout in long long loop
+ not support 64bit CPU yet.
However, It may differ from the Adaptec policy (linux-scsi ML).
7. Some RAID Recommendations
28 Dec 2003 - 30 Dec 2003 (18 posts) Subject: "Best Low-cost IDE RAID Solution
For 2.6.x? (OT?)"
Topics: Disk Arrays: MD, Disk Arrays: RAID, Disks: IDE, Disks: SCSI, Serial ATA
People: Johannes Ruscheinski, Joel Jaeggli, Johannes, Bert Hubert, Arjan van de
Ven, H. Peter Anvin, Wakko Warner, Samuel Flory, Tomas Szepe
Johannes Ruscheinski said, "We're looking for a low-cost high-reliability IDE
RAID solution that works well with the 2.6.x series of kernels. We have about 1
TB (8 disks) that we'd like to access in a non-redundant raid mode. Yes, I
know, that lack of redundancy and high reliability are contradictory. Let's
just say that currently we lack the funding to do anything else but we may be
able to obtain more funding for our disk storage needs in the near future."
Joel Jaeggli replied:
well if you currently have 1tb in 8 non-redundant drives then you using
160GB disks... no?
the biggest p-ata disks right now are ~320GB so you can do a ~1TB software
raid 5 stripe on a single 4 port ata controller such as a promise tx4000
using regular software raid rather than the promise raid. that would end up
being fairly inexpensive and buy you more protection.
linux software raid has been as releiable as anything else we've used over
the years, the lack of reliabilitiy in your situation will come entirely
from failing disks, lose one and your filesystem is toast.
Johannes asked, "why not use the hardware raid capability of the Promise
tx4000? and if we'd use software raid instead, what would be the CPU overhead?"
Bert Hubert said:
For the cost differential between linux native RAID and an external device
of similar capabilities, outfit yourself with an additional CPU. I don't
use RAID5 a lot but to a modern CPU, checksumming dozens of megabytes/
second is child's play:
raid5: measuring checksumming speed
8regs : 1479.600 MB/sec
32regs : 744.400 MB/sec
pIII_sse : 1649.200 MB/sec
pII_mmx : 1806.000 MB/sec
p5_mmx : 1915.200 MB/sec
raid5: using function: pIII_sse (1649.200 MB/sec)
This is on a 800MHz Celeron, so a recent >2Ghz system will do lots better
still.
Arjan van de Ven also offered Johannes a word of advice, saying, "be careful,
almost all ata raid controllers out there are *software raid* hidden in a
binary only driver. Also generally the on-disk format of these is quite
unfortionate resulting in slower access than linux software raid can do..." H.
Peter Anvin added, "Not to mention, well, *proprietary*. Consider this: with
Linux swraid, you don't have to worry about your manufacturer discontinuing
your product or going out of business; as long as you can connect your disks to
a CPU using any kind of controller you can recover your data. If a proprietary
RAID controller croaks, and you can't get another one of the same brand/model,
you might have no more data..." Wakko Warner also said:
Speaking of which, since most DIE^WIDE RAID controllers are really software
driven, it may be possible to get the sw module to read the disks off of
any controller
My machine at work has an onboard promise raid controller but I run linux
sw raid0 on 2 of the disks (4 total, other 2 are not in a raid)
I'm not sure about real hardware raid. Like the Mylex or Adaptec (SCSI)
Elsewhere, Samuel Flory also replied to Johannes' initial inquiry, saying:
It really depends on what you mean by low cost? The ony ide raid controller
that does 8 PATA drives well under linux is the 3ware controller. For SATA
drives you have the 3ware, and adaptec controller.
In theroy the highpoint 8 port sata card would be a good canidate for
software raid, but highpoint has yet to cough up an open source drive yet.
If you want to go the software raid route and have 2 spare pci solts. You
can go with either the high point rocket raid 454 (PATA), or the promise
SATA150 TX4.
I really don't recommend any of promise's cards that use use the i2o
driver, or any sort of binary only driver.
PS- Why not at least run software raid 5? It takes far less cpu than you'd
think, and can save your ass.
Tomas Szepe said, "Absolutely. With eight low-cost IDE disks, you'd be nuts to
go raid0 or linear." Johannes replied that he'd probably go with raid5 and the
Promise tx4000 card Joel had recommended, adding, "It looks like I'll have the
funding to buy another box and another 1 TiB of disk space."
Samuel offered some additional advice, saying, "Be sure to run badblocks on all
the disks before creating your array. Software raid isn't as nice about bad
sectors as most hardware raid controllers. On the other hand the md driver
kicks the ass of nearly every raid controller I've tried." Tomas mentioned that
bad-sector-niceness was only worse with initial bad sectors, not necessarily
sectors that went bad during the course of operation.
A few posts down the road, Wakko remarked, "One thing that keeps me from using
the linux raid sw is the fact it can't be partitioned. I thought about lvm/
evms, but I'm unwilling to make an initrd to set it up (mounting root).
Unfortunately boot loaders don't seem to support anything other than raid1.
(Mostly lilo, but I'm not sure grub would do this either)" Samuel replied,
"You're thinking of it the wrong way. You just create a bunch of partitions and
make them into raid devices. You shouldn't be using the entire disk or you will
break autodetection." And added, "Lilo deals well with raid 1 devices. I
typical create a small raid 1 mirror as /boot. Just be sure to install your
bootloader on to all drives. Newer versions of lilo will do the right thing if
told to use /dev/mdwhatever."
8. Status Of ATARAID In 2.6
28 Dec 2003 - 29 Dec 2003 (11 posts) Archive Link: "ataraid in 2.6.?"
Topics: Disk Arrays: LVM, Disk Arrays: RAID, FS: initramfs, FS: ramfs
People: Nicklas Bondesson, Arjan van de Ven, Christophe Saout
Nicklas Bondesson asked, "Is the ataraid framework planned to be ported to
2.6.x? If so, when could one expect it?" Arjan van de Ven replied, "the plan is
to have a userspace device mapper app take it's place. As for the timeframe;
I'm looking at it but the userspace device mapper code is still a bit of a
mystory to me right now." A few posts down the road, Nicklas replied, "How do
you set this (device mapping) up using the 2.6 kernel. I like the ease of using
ataraid in 2.4.x. Why not have both alternatives as options (both ataraid and
devicemapper)?" Christophe Saout replied:
I think the reason is to avoid unnecessary code duplication. device-mapper
provides a generic method to do such things. Also the developers are
heading towards removing code from the kernel that can be done in
userspace. There are plans to remove partition detection from the kernel in
2.7 and move the detection and setup code to a userspace program (using
device-mapper) which can be placed in the initramfs (so that the user won't
notice any difference). Ataraid detection and setup could also be placed
there later.
If someone writes an ataraid detection and setup program in userspace it
could be placed on an initrd.
You can find the dmsetup tool in Sistina's device-mapper package.
Arjan agreed that code duplication was the big reason not to have both options;
he added, "The outcome is to be a /sbin/ataraid binary or some such that will
do all the magic to detect the raid and tell the kernel device mapper to set it
all up."
Nicklas then said, "I'm planning to go with the device mapping in 2.6.0 to
setup RAID1 using my Promise TX2000 card. How do I know which device name it
will choose? I have looked in the 'devices.txt' but there is neither a ide-raid
nor a specific Promise device name mapping." And Christophe replied, "If you
cannot wait for Arjan or anybody else to write an ataraid setup tool, you can
go with dmsetup and choose any name you want (see the dmsetup man page). The
only restriction is that dmsetup creates device under /dev/mapper, you can
though use symlinks to it like LVM2 does if you need it under /dev/ataraid or
something (in your initscript). If you don't have a better idea you can just
give it the same name as before."
9. Problems With Accusys Drives
28 Dec 2003 - 29 Dec 2003 (8 posts) Subject: "ide: "lost interrupt" with 2.6.0"
Topics: Disks: IDE
People: Andre Hedrick, Mike Fedyk, Andrew Miller
Andrew Miller had some problems with his Accusys ACS7500 C5VL, ATA disk drive,
and Andre Hedrick said:
Accusys is a stolen technology from Dupli-Disk, there are law suits pending
the last time I heard.
If it was based on the original firmware of the Dupli-Disk, it has totally
wrong state machines for executing hdparm style callers.
I know the FSM's are wrong because I fixed them for Dupli-Disk. How they
operate, I can not disclose. But Accusys can not handle correct settings
for FSM to Taskfile.
Mike Fedyk asked, "Does that mean that if you use taskfile on Dupli-Disk
controllers that they will fail, and that disabling taskfile access might help
(is that still an option in 2.6?)" And Andre replied:
Dupli-Disk requires taskfile access and will correctly operate.
Accusys is total CRAP, unless they licensed the technology proper.
Elsewhere, Andrew said he contacted Accusys, and that they had "provided me
with a firmware upgrade tool and the firmware version CDVL(as opposed to C5VL).
It seems they fixed the problem with their firmware in the latest version. I'm
not sure whether or not it is worth fixing the 2.6.x kernels to support the
broken firmware, as they seem to offer the upgrade to all customers anyway."
10. Status Of 2.6 Maintainership
29 Dec 2003 - 30 Dec 2003 (27 posts) Archive Link: "2.6.0-mm2"
Topics: Disks: SCSI
People: Andrew Morton, Stef van der Made, Mike Fedyk, Linus Torvalds
Andrew Morton announced 2.6.0-mm2:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.0/
2.6.0-mm2/
+ Added the adaptec SCSI driver update from Justin. There's some
disagreement over whether the changes in this patch are appropriate,
but we may as well get some external testing underway.
+ Included an update from the ieee1394 team. Anyone who was having
firewire problems, please retest.
+ A couple of patches here should fix the CDROM-related problems which
some people saw in 2.6.0-mm1.
Stef van der Made replied, "Is it possible to use the old schema of pre1, pre2
und so weiter releases so that we can use the incremental patch sets again."
Someone explained that Andrew did not intend to use the -mm tree as the
official sources. Mike Fedyk added:
I think Linus will be releasing the 2.6-pre kernels, and things will
continue like that until 2.7 opens up.
I think Andrew is trying to get all of the after 2.6.0 fixes in -mm and
tested before syncing up with Linus.
And Linus Torvalds said:
Indeed. The fact is, we do need a "testing ground" for some experimental
fixes to stuff that needs to be fixed, and that's one of the things the -mm
kernels used to do for 2.5.x.
Since everybody was pretty comfy with that setup, we'll just continue that
way. By the time 2.7.x opens up, things should have calmed down, and
commercial vendors have their support trees in shape etc, but for now the
-mm tree is a testing ground, and I'll make -pre trees that should be
fairly stable, and then we'll do the 2.6.1 etc "real releases" based off
them.
11. Patch Submission Policies
29 Dec 2003 - 1 Jan (13 posts) Subject: "[CFT/PATCH] give sound/oss/trident a
holiday cleanup for 2.6"
People: Linus Torvalds, Muli, Jeff Garzik, Muli Ben-Yehuda
Muli Ben-Yehuda posted a patch, and Linus Torvalds said:
When doing things like this, can you split up the patches into two separate
things: one that _only_ does whitespace changes, and that is guaranteed not
to change anything else, and another that does the rest.
It's a total b*tch to try to figure out which change resulted in some
difference, if the changes are intermixed with large whitespace cleanups.
Muli replied, "You're 100% right. Internally, the patch I sent is composed of
30 different patches. The reason I didn't seperate it into two patches is that
the changes were interleaved and inter-dependant and seperating them was a
b*tch." Jeff Garzik said, "Thirty separate patches is OK. We have scripts to
handle "patchbombs"." And Linus said:
Yes and no.
Thirty separate patches make sense if they are independent and really do
conceptually different things. Then it makes sense to have them as separate
checkins, and be able to tell people "ok, try undoing that one, maybe
that's the problem".
However, if they are all just "fix silly bugs in xxx", then I'd much rather
see it as one big patch. Having it split up into "fix bug on line 50" and
"fix bug on line 75" just doesn't make any sense - it only makes the patch
history harder to follow.
So "many small patches" aren't automatically any better than one big one.
The thing that matters is to keep things _conceptually_ separated. If one
patch fixes whitespace, and another one fixes bugs, then that's good.
Jeff replied, "There's certainly a middle ground. For drivers I generally
request that bug fixes for separate bugs be split up, since inevitably one bug
fix out of twenty breaks for somebody on that somebody's weird hardware."
12. BitKeeper Usage Policy And Advice
29 Dec 2003 - 31 Dec 2003 (16 posts) Archive Link: "[PATCH] 2.6.0 - Watchdog
patches"
Topics: Disks: SCSI, Version Control
People: Linus Torvalds, Jeff Garzik, Paul Jackson, Matthias Andree, Ed
Tomlinson, Wim Van Sebroeck, Andrew Morton
Wim Van Sebroeck wanted Linus Torvalds and Andrew Morton to pull some changes
from his BitKeeper tree, but Linus said:
This tree has 38 deltas, all just merges.
The end result is a horribly messy revision tree, for a few one-liners.
I'm going to take the patch as a patch instead, and hope that you'll throw
your BK tree away.
Please don't follow the release tree in your development trees, it makes it
impossible to see how the revision history happened.
Jeff Garzik added:
Agreed. Several BK developers do this, forgetting that one of things that
makes BK so useful is its merge technology.
I recommend (assuming no patches outstanding),
+ clone latest tree
+ do development
+ only 'bk pull' from latest tree iff (a) you are about to submit to
Linus/Andrew or (b) you know there is a conflicting change in upstream
Pulling the latest, just to be up-to-date, just obfuscates things and
needlessly increases the size of the master ChangeSet file.
Paul Jackson said:
Another possibility I like is to recreate my changes (what few so far ...)
against a clean bk tree, before sending. Hide all my internal iterations
and changes from others.
I will pull frequently and liberally into the bk clones that I use to track
2.6, 2.6-mm and whatever else I am based on. These in turn I pull into my
main working bk tree, along with pulling in the various changes I have in
progress, each from their own bk clone.
Then when it comes time to send out a patch, I:
1. Generate an old fashioned patch (bk export -tpatch), containing just
the revisions relevant to what I will send.
2. Clone a fresh bk tree that is closest to whatever the recipient of my
patch would like to work with
3. Apply the patch to the fresh clone, generating a clean history of one
change for just that patch.
4. Double check that that builds and boots.
5. Then send that change out, usually by exporting it as a -second- old
fashioned patch, since for reasons not relevant here, I end up sending
patches, not bk pulls, down stream.
The objective being:
My final "published work" is that patch - it should be as clean as
practical.
By going into and back out of old fashioned patches, I isolate the anal
history that bk kept of all my interim changes from the rest of the world.
Elsewhere, Wim explained that his messy BK tree was the result of postponing
those patches until 2.6.0 had come out. Linus replied:
Yeah, I know. It's one of the downsides of having anal revision control,
and BK is more anal than most.
I do end up taking patches that have this syndrome if it looks like the
pain of not taking the messy revision history is larger than the pain of
just fixing it. Sometimes it's hard to avoid.
But most of the time the proper thing to do is to just not merge
unnecessarily - if something is pending for a while, Bk does the merge
correctly anyway, so you can just leave it pending and have me pull from an
old tree (after you have verified in your own tree that the pull will
succeed and do the right thing).
That way it ends up being trivial to see where/when the changes happened.
Matthias Andree asked:
Not being very used to BK, does that mean I have several trees around:
1. the official release tree
2. an "old tree" with my local change that I'm forwarding
3. a temporary test tree to see if the merge would succeed, which I'll get
by cloning (1) and then pulling from (2)?
Well, talk about FAAAAAAST drives (10,025/min SCSI kind) unless you have
time to waste on all those BK consistency checks (which are, of course,
what #3 is all about).
Or am I missing some obvious short cut?
To the idea of keeping several trees around, Linus replied:
The answer to that question is always a resounding "yes".
BK really thrives on having several trees around. You don't _have_ to have
them, but basically the default rule should be: use separate trees for
everything you can think of that isn't directly dependent on stuff in
another tree.
That does _not_ mean that you should necessarily create "temporary trees".
I actually do that a lot of the time, because I tend to create a totally
new clone when I start applying long series of patches or do anything even
half-way strange: it's just a lot easier to throw failures away, than it is
to try to sort it out later.
But most people probably do _not_ want to have that kind of "temporary
tree" mentality in general. People should realize that it's ok, and in
particular that if you're doing something experimental it's fine to just
create a new tree and later on decide that it was crap and just do a full
"rm -rf" on it, and realize that the only thing you lost was some time.
But to Matthias' temporary tree described in item 3 of his list, Linus also
said, "The tree doesn't really need to be temporary per se. It can be your
"work tree" - the tree where you merge all the different sources of BK input. I
realize that a lot of people only really have two sources of input (the
standard tree and their own development tree), but if you get that concept
early, you'll find it trivial to merge in other peoples trees into your "work
tree", and keep track of many different development trees at the same time and
just let BK do the merging for you." And regarding any available short-cuts,
Linus said:
Basically, the obvious shortcut is to keep your work-tree around, so you
don't have to clone and re-pull it all the time.
After a while, your work-tree is really messy (especially if you pull from
multiple different development trees), but the point should be that no
actual development gets done there, so you don't care: you can always just
flush it entirely, and re-create it anew.
But you don't have to flush and re-create it _all_ the time. That would
just be wasteful. Although if you have the hardware, it isn't that
painful..
Elsewhere, Ed Tomlinson asked if it were possible to tell BK to defer its
consistency checks, since they typically took 15 to 20 minutes each time. Andy
Isaacson was surprised by this, saying the checks shouldn't take more than
about 30 seconds, depending on the hardware. Eric D. Mudama reported 2 to 3
minute consistency checks. Andy turned around and agreed that perhaps 30
seconds was too optimistic for all machines, and that RAM size would have a
powerful influence on consistency check times. Ed reported his box as "an old
K6-III 400 with 512MB with UDMA2 harddrives."
13. Status Of BitKeeper Snapshots For 2.6
29 Dec 2003 (1 post) Archive Link: "Daily 2.6.x BK snapshots going again"
Topics: Version Control
People: Jeff Garzik
Jeff Garzik said:
Updated my snapshot script for 2.6.0 release, and snapshots are once again
flowing into
http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/
The snapshots for the 2.6.0-testX series were moved to
http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/old/
14. USB Updates For 2.6
29 Dec 2003 (1 post) Archive Link: "[BK PATCH] USB patches for 2.6.0"
Topics: USB
People: Greg KH
Greg KH said:
Here are some USB patches for 2.6.0. There are a number of different fixes
and a few new drivers added. Some of the highlights are:
+ lots of usb-storage fixes
+ lots of usb-storage unusual-devs updates
+ added new w9968cf driver
+ added new lego tower driver
+ core tweaks for non-compliant USB devices.
+ lots of other little fixes
Please pull from: bk://linuxusb.bkbits.net/usb-devel-2.6
Patches will be posted to linux-usb-devel as a follow-up thread for those
who want to see them.
15. Experimental Net Driver Updates For 2.6
29 Dec 2003 - 30 Dec 2003 (2 posts) Archive Link: "[BK PATCHES] 2.6.x
experimental net driver updates"
Topics: Networking
People: Jeff Garzik
Jeff Garzik said:
Summary of new changes:
+ bonding fixes
+ e100/e1000 fixes
+ other fixes
+ via-rhine netpoll support
Summary of patchkit:
+ new e100 driver (rewritten from scratch)
+ new nVidia nForce NIC driver
+ new pc200syn WAN driver
+ tg3 bug fixes
+ r8169 major bug fixes
+ e1000 minor updates / fixes
+ sk98lin vendor updates / fixes
+ misc bug fixes
+ 8139too NAPI support
+ tulip NAPI support
+ netconsole / netdump support
+ net_device allocation and reference counting work
Patch:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/patchkits/2.6/
2.6.0-bk2-netdrvr-exp1.patch.bz2 (NOTE: _requires_ 2.6.0-bk2 snapshot, or
IOW Linus-latest from BK, in order to apply successfully)
Full changelog:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/patchkits/2.6/
2.6.0-bk2-netdrvr-exp1.log
BK repo:
bk://gkernel.bkbits.net/net-drivers-2.5-exp
16. SCSI Updates
30 Dec 2003 (1 post) Archive Link: "[BK PATCH] SCSI updates"
Topics: Disks: SCSI
People: James Bottomley
James Bottomley said, "This represents the driver updates and other fairly
stable changes that have been floating around in the SCSI trees for a while.
The only controversial element is updating the aic7xxx/79xx drivers. I've been
receiving reports that the 1.3.9 version of the aic79xx was non-functional in
2.6, so I updated it to 1.3.11 based on Justin's tree (after confirming with
the reporters that this fixed their problems)."
17. Prototype For BitKeeper 'Undo Changeset' Improvement
30 Dec 2003 (1 post) Archive Link: "[BK] cset -x improvement (prototype)"
Topics: Version Control
People: Larry McVoy, Larry McVoy
Larry McVoy said:
Some of you use the cset -x feature in BK (which is a way of undoing the
effects of a particular changeset).
The documented behaviour of cset -x is that it only undoes content changes,
not renames, creates, permissions, etc. This interface is unique in that
respect, all the other interfaces in BK operate on all attributes, not just
contents.
I've prototyped a version of the interface which works on contents, names,
and permissions, and in addition also uses the BK merge tools to merge any
conflicting changes as a result of the undo of the changeset. This is JUST
A PROTOTYPE and has a lot of limitations but I'd like some feedback on
whether this is a good direction and we should productize this or if this
is not helpful to you.
If you use cset -x and want a better version of that interface, drop me an
email and I'll send you the shell script (please make sure to send mail to
me directly, not to the lists, because (a) they don't need to see all the
noise and (b) I'm no longer subscribed to the Linux kernel mailing list).
18. Status Of kernel.bkbits.net
30 Dec 2003 (2 posts) Archive Link: "kernel.bkbits.net is up"
People: Larry McVoy, Larry McVoy , Nigel Cunningham
Larry McVoy said that kernel.bkbits.net was up, adding, "If you don't have an
account on this and you want one (it's mostly for BK users but has sort of
turned into a public machine for any of the kernel developers, DaveM describes
it as "a friendly place") send me or davem a ssh2 key and we'll set you up."
Nigel Cunningham asked what the benefit of an account would be, but there was
no reply.
19. Linux 2.6.1-rc1 Released; Unanswered Questions About 2.6 Release Policy
31 Dec 2003 - 2 Jan (22 posts) Archive Link: "2.6.1-rc1"
Topics: Disks: SCSI, USB
People: Linus Torvalds, Mike Fedyk
Linus Torvalds announced 2.6.1-rc1, saying:
Ok, I've merged a lot of pending patches into 2.6.1-rc1, and will now calm
down for a while again, to make sure that the final 2.6.1 is ok.
Most of the updates is for stuff that has been in -mm for a long while and
is stable, along with driver updates (SCSI, network, i2c and USB).
Mike Fedyk pointed out, "Well there goes the -pre series. ;)" And asked, "Are
we going to have 2.6.1-rc1-mm1? :-D" , but there was no reply.
20. Linux 2.4.24-pre3 Released
31 Dec 2003 (4 posts) Archive Link: "Linux 2.4.24-pre3"
Topics: Disk Arrays: LVM
People: Marcelo Tosatti
Marcelo Tosatti released 2.4.24-pre3, saying, "It contains a PPC32/SPARC
update, some i2c cleanups, LVM update, network update, a new WAN driver,
amongst others."
21. udev 012 Released
31 Dec 2003 (1 post) Archive Link: "[ANNOUNCE] udev 012 release"
Topics: FS: devfs, FS: sysfs, Hot-Plugging, Version Control
People: Greg KH
Greg KH announced:
I've released the 012 version of udev. It can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-012.tar.gz (http://
www.kernel.org/pub/linux/utils/kernel/hotplug/udev-012.tar.gz)
rpms built against Red Hat FC1 are available at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-012-1.i386.rpm (http://
www.kernel.org/pub/linux/utils/kernel/hotplug/udev-012-1.i386.rpm)
with the source rpm at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-012-1.src.rpm
udev allows users to have a dynamic /dev and provides the ability to have
persistent device names. It uses sysfs and /sbin/hotplug and runs entirely
in userspace. It requires a 2.6 kernel with CONFIG_HOTPLUG enabled to run.
Please see the udev FAQ for any questions about it:
kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ (http://www.kernel.org/
pub/linux/utils/kernel/hotplug/udev-FAQ)
For any udev vs devfs questions anyone might have, please see:
kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs (http://
www.kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs)
The major changes since the 011 release are:
+ updated ide devfs compatible script
+ command line options are now available, allowing you to query the udev
database. This is just a first cut, the query functionality will be
extended later.
+ lots of minor bug fixes and tweaks.
Thanks again to everyone who has send me patches for this release, a full
list of everyone, and their changes is below.
udev development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/udev
Daily snapshots of this tree used to be found at:
http://www.codemonkey.org.uk/projects/bitkeeper/udev/
But that box seems to be down now. Hopefully it will be restored someday.
If anyone ever wants a tarball of the current bk tree, just email me.
22. Trouble With udev Removable Media Handling
1 Jan - 3 Jan (8 posts) Archive Link: "removable media revalidation - udev
vs. devfs or static /dev"
Topics: FS: devfs, Hot-Plugging
People: Andrey Borzenkov, Greg KH
Andrey Borzenkov reported:
udev names are created when kernel detects corr. device. Unfortunately for
removable media kernel rescans for partitions only when I try to access
device. Meaning - because kernel does not know partition table it did not
send hotplug event so udev did not create device nodes. But without device
nodes I have no way to access device in Unix :(
specifically I have now my Jaz and I have no (reasonable) way to access
partition 4 assuming device nodes are managed by udev.
devfs solved this problem by
+ always exporting at least handle to the whole disk (sda as example)
+ using something simple like dd if=/dev/sda count=1 on lookup for
non-existing partition (/dev/sda4) that would rescan partitions and
create device nodes for them.
static /dev simply has all nodes available and does not suffer from this
problem at all.
unfortunately there are no lookup events in case if udev ... meaning at
this moment user must manually rescan partitions after inserting new media.
I do not see any way to solve this problem at all given current
implementation. The closest is to blindly create nodes for all partitions
as soon as block device is available.
Greg KH said, "Doesn't the kernel always create the main block device for this
device? If so, udev will catch that. If not, there's no way udev will work for
this kind of device, sorry. You could make a script that just creates the
device node in /tmp, runs dd on it, and then cleans it all up to force
partition scanning." Andrey agreed that the kernel did make the main block
device /dev/sda, but he didn't see how this was helpful, since he needed /dev/
sda4 specifically. He could write a script to do what Greg suggested, but he
could see no way to cause that script to execute at the proper time. He said,
"There is no event when you just insert Jaz disk; nor is there any way to
trigger revalidation on access to non-existing device like is the case without
udev." But Greg pointed out that udev "does provide that mechanism. See the
CALLOUT rule. It can run any program or script when a new device is seen by the
kernel."
23. Fair Scheduling In 2.4
2 Jan (2 posts) Archive Link: "Fair scheduling in 2.4 ?"
People: Robert Mena, Rik van Riel
Robert Mena asked:
Back in the days of kernel 2.2 there was a patch (from Rik van Riel if I
recall) to the kernel so no single user could use all available cpu.
I was wondering if this feature (or something like it) has been ported or
integrated in 2.4.x series ?
Rik van Riel replied, "There's a few versions for 2.4 on my patches page ;)
http://surriel.com/patches/"
We Hope You Enjoy Kernel Traffic
Kernel Traffic is hosted by the generous folks at Tux.Org. All pages on this
site are copyright their original authors, and distributed under the terms of
the GNU General Public License, version 2.0.