Kernel Traffic #300 by Zack Brown
[email protected] Wed, 30 Mar 2005 00:52:47 -0800
| Newsgroups | gmane.linux.kernel.kernel-traffic |
|---|---|
| Message-ID | <[email protected]> |
Kernel Traffic #300 For 29 Mar
By Zack Brown
Table Of Contents
* Standard Format
* Text Format
* XML Source
* Threads Covered
1. 10 Feb - 18 Feb (51 Linux 2.6.11-rc3-mm2 Released; Some Debate
posts) Over Security
2. 10 Feb - 16 Feb (76 New Hotplug Project; Reducing Boot-Speed
posts)
3. 13 Feb - 23 Feb (159 BitKeeper Licensing; Kernel Developers
posts) Unhappy
4. 16 Feb - 17 Feb (11 An Attempt At /proc/cpumem
posts)
5. 17 Feb - 21 Feb (4 posts) New yaird Replacement For mkinitrd
6. 22 Feb (3 posts) Extended Attribute Support For JFFS3
7. 23 Feb (1 post) Linux 2.4.30-pre2 Released
8. 23 Feb (1 post) yaird 0.0.4 Released
1. Linux 2.6.11-rc3-mm2 Released; Some Debate Over Security
10 Feb - 18 Feb (51 posts) Archive Link: "2.6.11-rc3-mm2"
Topics: Capabilities, FS: accessfs, Kernel Release Announcement, Real-Time
People: Andrew Morton, Christoph Hellwig, Olaf Dietsche, Jack O'Quin, Chris
Wright, Ingo Molnar, Matt Mackall
Andrew Morton announced Linux 2.6.11-rc3-mm2, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.11-rc3/
2.6.11-rc3-mm2/
+ Added the mlock and !SCHED_OTHER Linux Security Module for the audio
guys. It seems that nothing else is going to come along and this is
completely encapsulated.
+ Various other stuff. If anyone has a patch in here which they think
should be in 2.6.11, please let me know. I'm intending to merge the
following into 2.6.11:
alpha-add-missing-dma_mapping_error.patch
fix-compat-shmget-overflow.patch
fix-shmget-for-ppc64-s390-64-sparc64.patch
binfmt_elf-clearing-bss-may-fail.patch
qlogic-warning-fixes.patch
oprofile-exittext-referenced-in-inittext.patch
force-read-implies-exec-for-all-32bit-processes-in-x86-64.patch
oprofile-arm-xscale1-pmu-support-fix.patch
Regarding the security module patches, Christoph Hellwig said, "Even if we
accept a module that grants capabilities to groups this isn't fine yet because
it only supports two specific capabilities (and even those two in different
ways!) instead of adding generic support to bind capabilities to groups." Olaf
Dietsche remarked:
Unless I misunderstood the code, this one is available for quite some time:
<http://www.olafdietsche.de/linux/accessfs/> or a newer, self-contained
version <http://lkml.org/lkml/2005/1/11/221>
Or you could use a real solution - filesystem capabilities: <http://
www.olafdietsche.de/linux/capability/> and if you don't like this one :-),
there's also an alternative existing here: <http://www.stanford.edu/~luto/
linux-fscap/>
Andrew also said to Christoph, "I'm sure that got discussed somewhere in the
1000 emails which flew past last time. Jack?" And Jack O'Quin replied:
Most people felt that a more general capabilities module would be nice to
have. But, no one offered any code, or volunteered to work on it.
I have no objection to that approach, but am not willing or able to do it
myself. My opinion is that expanding the scope of the LSM would
significantly increase its security risk. That job needs to be done very
carefully, by someone with a deep understanding of the kernel's internal
use of capabilities.
Perhaps, Christoph's suggestion could become part of a more general module,
which might replace the RT-LSM in the 2.8 timeframe. Our LSM is a modest
solution aimed at solving the immediate needs of audio developers and users
with minimal impact on kernel security or correctness.
Matt Mackall asked what happened to the RT rlimit code from Chris Wright, and
Chris replied, "I still have it, but I had the impression Ingo didn't like it
as a long term solution/hack (albeit small) to the scheduler. Whereas the
rt-lsm patch is wholly self-contained."
It turned out there were many opinions about the various implementations. Chris
was right that his rlimit patch was considered too invasive. Actually, the main
reason Andrew chose the RT LSM solution was because none of the alternatives
were self-contained enough. And although, as Matt pointed out, Chris' patches
were more straightforward than the other althernatives considered; there was
still the problem that, as Ingo Molnar pointed out, "it didnt solve the problem
(unprivileged user can lock up the system) in any way. So after it became
visible that all the existing 'dont allow users to lock up' solutions are too
invasive, we went to recommend the solution that introduces the least
architectural problems: RT-LSM." Elsewhere in the thread, Ingo made another
point about Chris' rlimit patch. He said, "if it turns out to be a mistake then
it's already codified into the ABI, while RT-LSM is much less 'persistent' and
could be replaced much easier." Christoph and Matt, however, were very dubious
about the possibility of removing RT-LSM later; they said once RT-LSM was in
the kernel, people would begin to rely on it, and removing it would not be just
a simple technical issue. Ingo replied:
i somewhat share that view. (despite all the promises from the audio folks
- if they are just half as agressive resisting removal as they were pushing
integration then it will never be removed ;-)
but i'm not sure how rlimits will contain the whole problem - can rlimits
be restricted to a single app (jackd)? The most canonical use of rlimits is
per-user (per-group), so the rlimit could end up _widening_ the effects of
the hack ...
Matt confirmed that rlimits could indeed be restricted to a single application.
Ingo asked how this could be accomplished, and Matt replied that the
application had to be invoked by a setuid launcher; it was not possible to use
rlimits to elevate the rlimits of a running process. Ingo argued that if there
was already a requirement for a setuid application launcher, there was no need
for anything else; the launcher itself could give the needed privileges. Matt
argued that the rlimits patch would do a lot more than just this; but the
discussion ended abrubtly, with no resolution.
2. New Hotplug Project; Reducing Boot-Speed
10 Feb - 16 Feb (76 posts) Archive Link: "[ANNOUNCE] hotplug-ng 001 release"
Topics: FS: devfs, FS: initramfs, FS: ramfs, Hot-Plugging, Klibc, Version
Control, Virtual Memory
People: Greg KH, Patrick McFarland, Vojtech Pavlik
Greg KH said:
I'd like to announce, yet-another-hotplug based userspace project:
hotplug-ng. This collection of code replaces the existing linux-hotplug
package with very tiny, compiled executable programs, instead of the
existing bash scripts.
It currently provides the following:
+ a /sbin/hotplug multiplexer. Works identical to the existing bash /sbin
/hotplug.
+ autoload programs for usb, scsi, and pci modules. These programs
determine what module needs to be loaded when the kernel emits a
hotplug event for these types of devices. This works just like the
existing linux-hotplug scripts, with a few exceptions.
But why redo this all in .c code? What's wrong with shell scripts? Nothing
is wrong with shell scripts, unless you don't want to have an interpreter
in your initramfs/initrd and you want to provide /sbin/hotplug and autoload
module functionality. Or if you have a huge box that spawns a zillion
hotplug events all at once, and you need to be able to handle all of that
with the minimum amount of processing time and memory.
So, how small are these programs? Take a look:
text data bss dec hex filename
4669 32 124 4825 12d9 hotplug
5077 8 348 5433 1539 module_pci
4925 8 412 5345 14e1 module_scsi
5349 8 348 5705 1649 module_usb
Those are all static binaries, linked with klibc (which is included in the
hotplug-ng package, just like udev.)
This compares to the following bash scripts:
-rwxr-xr-x 1 root root 4412 Feb 10 15:28 /sbin/hotplug
-rw-r--r-- 1 root root 702 Sep 24 08:04 /etc/hotplug/blacklist
-rw-r--r-- 1 root root 5293 Sep 24 08:04 /etc/hotplug/hotplug.functions
-rwxr-xr-x 1 root root 3739 Sep 24 08:04 /etc/hotplug/pci.agent
-rwxr-xr-x 1 root root 1459 Sep 24 08:04 /etc/hotplug/scsi.agent
-rwxr-xr-x 1 root root 13466 Sep 24 08:04 /etc/hotplug/usb.agent
-rw-r--r-- 1 root root 39306 Sep 24 08:04 /etc/hotplug/usb.distmap
-rw-r--r-- 1 root root 4364 Sep 24 08:04 /etc/hotplug/usb.handmap
-rw-r--r-- 1 root root 189 Sep 24 08:04 /etc/hotplug/usb.usermap
All of which are loaded into memory for each hotplug event (for specific
hotplug events, only that bus type of file is loaded.)
But what about speed? With a completely unscientific measurement on my old,
slow laptop, it takes about 2 seconds from the time I plug a usb device
into the machine, for the proper module to be loaded. With the hotplug-ng
program, it takes less than a second.
And for those of you who might remember the old dietHotplug program that
also did the same thing in a tiny amount of space, this project obsoletes
that one. dietHotplug had to be rebuilt for every kernel that was used on
the system, hotplug-ng uses the ability for modprobe to determine the
module that needs to be loaded based on the module aliases (modprobe as it
currently works stops loading modules when it finds an alias that matches.
This does not work for drivers that claim to support "all devices" and then
later on fail on devices that they really don't support. For that, all
matching drivers need to be loaded for the system to work properly. The
linux-hotplug scripts handle this correctly, so if you rely on this
functionality, please stick with that package for now. I'll be modifying
modprobe to add this feature in the near future.).
The code can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/hotplug-ng-001.tar.gz (http://
www.kernel.org/pub/linux/utils/kernel/hotplug/hotplug-ng-001.tar.gz)
for those who wish to poke around in it.
I still have a few more programs to write to get it up to the same
functionality as the existing hotplug scripts (firmware, ieee1392, etc.)
but those will be done soon. I'd like to get people's comments on the idea,
and welcome suggestions and even patches :)
hotplug-ng development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/hotplug-ng
Patrick McFarland replied:
Wow, thats pretty awesome. Just the other day I said, "Why is hotplug
written in sh? Isn't that horribly inefficient way of handling something
that needs to be done quickly using the least amount of resources possible?
" It seems you were reading my mind.
Please, continue this project and encourage distros to switch to it (when
it exceeds hotplug in functionality and stability). Ubuntu currently is
trying to reduce boot time, and I bet something like this would factor in
(even a few seconds helps).
Greg was grateful for the praise, and he did confirm that he'd been talking to
the Ubuntu folks; but he didn't think his code would really have an impact on
boot-time. Vojtech Pavlik, however, said, "Hotplug scripts were identified as
one of the major culprits of slow boot when we did the analysis for SuSE 9.2.
They took 40+ seconds from the total boot time."
At this point the debate veered off, into which distributions put in how much
effort to reduce boot time.
3. BitKeeper Licensing; Kernel Developers Unhappy
13 Feb - 23 Feb (159 posts) Subject: "[BK] upgrade will be needed"
Topics: Version Control
People: Larry McVoy, Alan Cox
Larry McVoy said:
This is a heads up that the BK tree for the kernel is currently at 59,000
changesets give or take a few. The BK that you are using uses unsigned
shorts for the internal names of each delta which means you folks are about
100 days away from things no longer working.
The good news is that the openlogging tree for the kernel has 135,000
changesets so we've obviously long since fixed this problem.
The bad news is that you will need to upgrade your BK binary in order to
pass over the 64K changeset boundary. The data is stored on disk in ascii
so it doesn't matter if you upgrade until you hit the problem but sooner or
later you will need to upgrade.
We'll get the fix into bk-3.2.4 which should be out by the end of the
month. When we release that we'll send out notice and it would be good if
people gave it a try and let us know if they hit issues because in a couple
of months everyone is going to have to upgrade.
It's possible that we'll be changing the BK license to conform more with
our commercial license but we won't do that without running it by Linus &
Co to make sure that it's acceptable. One change we'd like to make there is
to clarify the non-compete stuff. We've had some people who have indicated
that they believed that if they used BK they were agreeing that they would
never work on another SCM system. We can see how it is possible that people
would interpret the license that way but that wasn't our intent. What we
would like to do is change the language to say that if you use BK you are
agreeing that you won't work on another SCM for 1 year after you stop using
BK. But after that you would be able to hack on anything that you wanted.
That was more of what we had in mind in the first place but we didn't make
it clear. If you all thought that using BK meant you had no right to hack
on SCM ever again, that's just not fair. We need to protect our IP but you
should have the right to choose to go hack SCM if that's what you (our
first choice is that you came and worked here, we really like kernel
hackers, but if you don't want to that's cool too).
Various folks protested that it was absurd to have a license that prohibited
working on a competing project for a year; and some folks said it was
unenforcible, or that it had no standing in many parts of the world. A big
discussion started up, in which some folks accused Larry of misrepresenting his
objectives; while others pleaded with him to use a more sensible license. At
one point Alan Cox remarked, "The real fix is to replace BK with something free
and better, but thats *not* a trivial task."
4. An Attempt At /proc/cpumem
16 Feb - 17 Feb (11 posts) Archive Link: "[PATCH] /proc/cpumem"
People: Itsuro Oda, Eric W. Biederman
Itsuro Oda said:
Attached is an implementation of /proc/cpumem. /proc/cpumem shows the valid
physical memory ranges.
+ i386 and x86_64
+ implement valid_phys_addr_range() and use it. (the first argument of
the i386 version is little uncomfortable.)
+ /dev/mem of the i386 version should be mofified. but not yet.
example: amd64 8GB Mem
# cat /proc/cpumem
0000000000000000 000000000009b800
0000000000100000 00000000fbe70000
0000000100000000 0000000100000000
#
start address and size. hex digit.
Eric W. Biederman replied, "Interesting. My imagination when I proposed this
was something based on struct resource that works like /proc/iomem on x86 but
can be meaningfully be used on systems where ram lives in a separate address
space from io device memory." He still felt that adding a type field to /proc/
cpumem would be a more appropriate solution, though he didn't feel it would be
necessary for x86 machines, as the information was available elsewhere.
5. New yaird Replacement For mkinitrd
17 Feb - 21 Feb (4 posts) Archive Link: "[ANNOUNCE] yaird, a mkinitrd based
on hotplug concepts"
Topics: Backward Compatibility, Device Mapper, Disk Arrays: EVMS, Disk Arrays:
LVM, Disks: IDE, Disks: SCSI, FS: NFS, FS: devfs, FS: initramfs, FS: ramfs, FS:
sysfs, Hot-Plugging, Klibc, Serial ATA, Software Suspend, USB
People: Erik van Konijnenburg, Jeff Garzik
Erik van Konijnenburg said:
OK, time to stop polishing and start publishing.
This is to announce yaird, Yet Another mkInitRD, a rewrite of mkinitrd
based on hotplug algorithms.
MOTIVATION
Why a rewrite? The versions of mkinitrd that I studied, Debian sid and
Fedora FC3, have some problems: they capture a lot of knowledge about the
boot process, but don't always understand when a new kernel uses a
different module name than the old kernel, may rely on modules compiled
into the kernel, and don't always catch errors at the earliest opportunity.
Assumption: there are three issues that cause most of these problems.
+ Backward compatibility: sysfs provides a wealth of information about
hardware, but if you have to support 2.4 kernels, you have to limit
yourself in the use you can make of that information.
+ Originality: hotplug does a pretty good job of finding the appropriate
modules for a device, basically because it closely follows the
algorithms the kernel uses for matching hardware with modules.
Deviating from those algorithms is unlikely to produce a more robust
result.
+ Shell scripting: beyond a certain level of complexity, the shell syntax
makes it difficult to focus on the data structures you're trying to
process and makes it difficult to do rigorous error checking.
Yaird is intended to find out whether that assumption is correct: if so, a
program to build initrd images will be more reliable if it's written in
perl, if it uses sysfs to determine what hardware needs to be supported,
and if it closely follows the methods hotplug uses find the modules needed
to support some hardware.
STATUS
There is working code: yaird booted the machine this note is written on.
Code is available online, there also is a paper that discusses the workings
of the application in detail.
http://www.xs4all.nl/~ekonijn/yaird/
So far, the program works correctly on every machine it's been tested on,
but with only two test boxes that does not mean much.
Basic creature comforts are in place: "configure; make" but no RPM or deb
files, a README and help option but no manual page.
Features:
+ understands SATA and IDE. USB sticks and SCSI should work, but are
untested.
+ understands MDADM and LVM2, activates only necessary devices,
understands stacks like LVM on top of stripe on top of mirror.
+ handles both initrd and initramfs.
+ understands USB keyboards.
+ understands hotplug blacklist.
+ knows that a module compiled into the kernel does not need insmod.
+ understands /etc/fstab, including niceties such as labels, uuids and
octal escapes.
+ image generation understands symbolic links and shared libraries.
(should support 32bit emulation on 64bit kernels; untested).
+ templating system to simplify tuning the initrd image to the
distribution.
+ avoids hard-coded device numbers; does not require devfs.
+ testing mode gives detailed overview of the systems hardware and the
modules needed to support that.
There are rough edges in practically every feature: this is suitable for
testing, but not for production use.
TODO
1. Testing so far is 100% successful, but with just two boxes to play
with, that's not saying much. If you can find space to test the code on
your system, your results are highly appreciated. At this point,
hardware testing is most valuable: I already know that dm-crypt is
unsupported for now, but whether this stuff can boot a powerbook,
sparc, or just about anything else is an open question.
2. Feedback. This may be an interesting idea, but how does it relate to
other new stuff floating about? In particular, what about the work
that's going on putting udev on initramfs: are these approaches
complementary or alternatives? Different perspectives on where this
stuff fits in would help.
3. Support more configurations. dm-crypt is unsupported for now, and so
are multipath, swsusp, EVMS, NFS and loopback mounts. Implementing this
stuff becomes interesting once there are some tests results that the
basic ideas work in practice.
Jeff Garzik replied:
Having a mkinitrd that's not a shell script is a godsend. I would endorse
yaird on that fact alone :)
I've long wanted a "mkinitfoo" that would create .cpio.gz for initramfs by
default. So, good job there, too.
Eventually your script will need pull in, into the initramfs, klibc and
programs linked against klibc.
6. Extended Attribute Support For JFFS3
22 Feb (3 posts) Archive Link: "JFFS2 Extended attributes support & SELinux in
handhelds"
Topics: Extended Attributes, FS: ReiserFS
People: Lorenzo Hern?ndez Garc?a-Hierro, James Morris
Lorenzo Hern?ndez Garc?a-Hierro said:
I've been working in implementing extended attributes support in the JFFS2
filesystem.
During the (short) time I worked on it, I just decided to try to bring back
the thread on SELinux in hand-held and embedded devices and see if there's
someone interested in contributing to it and collaborating to make
something out of it.
The current work is just a draft, I've started with the standard Vanilla
kernel sources plus mtd JFFS2 sources, used to patch the vanilla ones for
latest code (I'm confused on which one has the most updated tree or if
there are special differences between mtd's trees and vanilla's), and
implemented the skeleton using the reiserfs xattr code base.
Then Russell Coker commented to me that I should use the handhelds.org
kernel, so, I have the doubt on which one use, even if porting the changes
made to JFFS2 code base to vanilla sources wouldn't be difficult, at least
I suppose.
I've opened a few wiki pages for discussion and documenting until it gets
further developed:
http://wiki.tuxedo-es.org/tuxedo/JFFS2xattr
http://wiki.tuxedo-es.org/tuxedo/SELinuxEmbedded
In addition, having someone experienced with xattr API could be great, as
development documentation seems inexistent, among James Morris' merged
xattr consolidation code.
Josh Boyer suggested talking to the JFFS2 developers; he said xattr support
would probably be added to JFFS3. After an IRC discussion between the two of
them, Lorenzo migrated his code to JFFS3, and posted a new patch.
7. Linux 2.4.30-pre2 Released
23 Feb (1 post) Archive Link: "Linux 2.4.30-pre2"
Topics: FS: JFS, USB
People: Marcelo Tosatti
Marcelo Tosatti announced Linux 2.4.30-pre2, saying:
Here goes the second pre of v2.4.30.
It contains a bunch of important networking fixes, most noticeably the
brlocks rework.
Plus USB fixes, megaraid2 driver update, JFS update, amongst others.
8. yaird 0.0.4 Released
23 Feb (1 post) Archive Link: "[ANNOUNCE] yaird 0.0.4, a mkinitrd based on
hotplug concepts"
Topics: Disks: SCSI, FS: initramfs, FS: ramfs, FS: sysfs, Hot-Plugging, Klibc
People: Erik van Konijnenburg
Erik van Konijnenburg said:
Version 0.0.4 of yaird is now available at:
http://www.xs4all.nl/~ekonijn/yaird/yaird-0.0.4.tar.gz
Yaird is a perl rewrite of mkinitrd. It aims to reliably identify the
necessary modules by using the same algorithms as hotplug, and comes with a
template system to to tune the tool for different distributions and
experiment with different image layouts. It requires a 2.6 kernel with
hotplug. There is a paper discussing it at:
http://www.xs4all.nl/~ekonijn/yaird/yaird.html
There are rough edges in practically every feature, and numerous features
still need to be added: this is suitable for testing, but not for
production use.
Thanks to all who gave feedback, sent patches and were brave enough to test
this stuff. Below are highlights from the change log and todo list.
Summary of user visible changes for version 0.0.4:
+ Process kernel command line options: init=, ro, rw.
+ Boot into single user mode supported
+ Support modules outside /lib/modules
+ Support modules with extension other than .ko
+ Warn about duplicates in modules.dep
+ Generated image now waits for device to become visible in /sys, and
gives error message if it doesn't
+ Support 2.6.10 sysfs layout: SCSI now has a new subdirectory 'target'.
(Thanks to Harald Dunkel for testing)
+ Warn about unrecognised paths in /sys
+ Empty lines in /etc/fstab are valid. (Patch Goffredo Baroncelli)
On top of the todo list are now:
+ add command line option (--root=/dev/hdb) to simplify testing.
+ add tree copying to the templates, to allow all of firmware to be
copied to the image. Or all of /lib/modules, if you want to have
hotplug on the image.
+ get klibc run_init working. Test by switching the Debian template to
initramfs. This should make Debian and Fedora templates more similar,
it is also groundwork for possible hotplug-ng support.
+ any patches you may wish to send :)
We Hope You Enjoy Kernel Traffic
Kernel Traffic is grateful to be developed on a computer donated by Professor
Greg Benson and Professor Allan Cruse in the Department of Computer Science at
the University of San Francisco. This is the same department that invented
FlashMob Computing. Kernel Traffic is hosted by the generous folks at
kernel.org. All pages on this site are copyright their original authors, and
distributed under the terms of the GNU General Public License, version 2.0.