Kernel Traffic #263 by Zack Brown
[email protected] Mon, 14 Jun 2004 10:04:18 -0700
| Newsgroups | gmane.linux.kernel.kernel-traffic |
|---|---|
| Message-ID | <[email protected]> |
Kernel Traffic #263 For 14 Jun
By Zack Brown
Table Of Contents
* Standard Format
* Text Format
* XML Source
* Mailing List Stats For This Week
* Threads Covered
1. 5 Apr - 15 Apr (6 Linux 2.4.26-rc2 Released
posts)
2. 6 Apr - 16 Apr (22 ReiserFS Version 3 Fixes And Updates
posts)
3. 12 Apr - 21 Apr (20 Page Attribute Table Support In 2.6
posts)
4. 12 Apr - 19 Apr (23 Linux 2.6.5-mm5 Released
posts)
5. 13 Apr - 23 Apr (31 Obfuscating Data For Security At The DM Level
posts)
6. 13 Apr - 15 Apr (15 Toward Variable, Concurrent Page Sizes
posts)
7. 14 Apr - 18 Apr (7 Linux 2.6.5-mm6 Released
posts)
8. 15 Apr - 17 Apr (27 SATA Going Into 2.4.27
posts)
9. 16 Apr - 20 Apr (19 Automatic Bug Checker Reveals Some Kernel
posts) Holes
10. 18 Apr - 21 Apr (30 Linux 2.6.6-rc1-mm1 Released; Some Developer
posts) Disconnect
11. 19 Apr - 20 Apr (7 PS/2 Mouse Fixes; Targus Idiosyncrasy
posts) Workarounds
12. 20 Apr - 21 Apr (4 Documentation On Building External Modules
posts)
13. 20 Apr (1 Linux 2.6.6-rc2 Released
post)
14. 20 Apr (6 Identifying Reverted Patches Based On
posts) Changelog 'Cset exclude' Entry
15. 20 Apr - 23 Apr (2 High Resolution Timers Update For 2.6
posts)
16. 20 Apr - 22 Apr (13 Kernel Mailing List Marked As Spammer By
posts) SpamCop
17. 21 Apr - 24 Apr (10 Linux 2.6.6-rc2-mm1 Released
posts)
18. 22 Apr (1 ELSA (Enhanced Linux System Accounting)
post) Project Begun
19. 24 Apr (2 libata Updates, Including Hotplug Support
posts)
20. 9 May - 11 May (12 Linux 2.6.6 Released
posts)
21. 10 May - 12 May (62 Linux 2.6.6-mm1 Released
posts)
22. 12 May (2 Status Of InterMezzo In 2.6
posts)
Mailing List Stats For This Week
We looked at 3414 posts in 17606K.
There were 868 different contributors. 438 posted more than once. 161 posted
last week too.
The top posters of the week were:
* 133 posts in 521K by Andrew Morton
* 75 posts in 293K by Greg KH
* 55 posts in 297K by Dmitry Torokhov
* 53 posts in 296K by Chris Wright
* 52 posts in 145K by Christoph Hellwig
* Full Stats
1. Linux 2.4.26-rc2 Released
5 Apr - 15 Apr (6 posts) Archive Link: "Linux 2.4.26-rc2"
Topics: Disks: IDE, FS: XFS, Power Management: ACPI
People: Marcelo Tosatti, Paul Mackerras, Stephen Rothwell
Marcelo Tosatti announced Linux 2.4.26-rc2, saying, "Here goes the second
release candidate. It contains an ACPI update, networking updates, IDE updates,
XFS update, etc." Paul Mackerras replied:
Any chance of getting this patch in before 2.4.26 final?
This patch is needed for compiling 2.4 with recent versions of gcc, such as
the gcc 3.3.3 hammer branch or gcc 3.4. The gcc developers changed the name
of the attribute that indicates that something is actually needed, even
though gcc can't see why, from "__unused__" to "__used__". This patch copes
with that.
The patch is from Stephen Rothwell. He discovered the problem on ppc64, but
in fact it would exist on any architecture.
Marcelo accepted the patch.
2. ReiserFS Version 3 Fixes And Updates
6 Apr - 16 Apr (22 posts) Archive Link: "[PATCH] reiserfs v3 fixes and
features"
Topics: Access Control Lists, FS: ReiserFS, FS: ext2
People: Chris Mason, Hans Reiser, Chris Wright, Marc-Christian Petersen
Chris Mason said:
You can download the set of experimental reiserfs v3 patches from:
ftp://ftp.suse.com/pub/people/mason/patches/reiserfs/2.6.5
Since some of these are in -mm and some are not, there are two series
files. series.linus gives you the patches needed for mainline 2.6.5, and
series.mm gives you the patches needed for 2.6.5-mm1
Most of these are from Jeff Mahoney and I, they include:
bug fixes
logging optimizations
data=ordered support
xattrs
acls
quotas
error messages with device names (based on Oleg's 2.4 patch)
block allocator improvements
Jeff Mahoney's acls and xattrs for reiserfs v3 have been in use in the suse
2.4 kernels and now 2.6 kernels for a while. I've posted for review to
namesys many times, but Hans refuses to consider or read the code. I
renewed my efforts over the last month to talk with him about the code, but
he has ignored it entirely.
His past objections seem to be that he doesn't want new features in v3. The
implementation does not change the disk format in any way (xattrs are
stored as regular files in a hidden directory) and is stable. I believe
reiserfs needs these features in order to stay current in the kernel, so
I'm posting for inclusion in -mm. I'm sending Andrew the following patches
from series.mm:
reiserfs-end-trans-bkl
reiserfs-acl-mknod.diff
reiserfs-xattrs-04
reiserfs-acl-02
reiserfs-trusted-02
reiserfs-selinux-02
reiserfs-xattr-locking-02
reiserfs-quota
permission-reiserfs
reiserfs-warning
(which is everything except the new block allocator code)
The block allocator improvements is our attempt to reduce fragmentation.
The patch defaults to the regular 2.6.5 block allocator, but has options
documented at the top of the patch that allow grouping of blocks by packing
locality or object id. It also has an option to inherit lightly used
packing localities across multiple subdirs, which keeps things closer
together in the tree if you have a bunch of subdirs without much in them.
If anyone is interested in experimenting with the block allocator stuff,
please let me know.
There were a number of replies. Marc-Christian Petersen said he'd be interested
in experimenting with the block allocator stuff, and Chris expanded:
There are a few different sides to the block allocator work.
1. don't ruin what the current allocator is good at (desktop esp). A
sequential tree read of freshly copied data is really fast right now.
It could be a little better with some metadata readahead, I'm still
trying to safely revive that code.
The patch tries to keep performance for a full tree read by with the -o
packing_groups option. The basic idea is to reuse bitmap groups for new
subdirectories until the bitmap group is full. This is done by checking
to see how full a given part of the btree is.
2. Improve the fragmentation under multiple writers. With 20 writers, the
default allocator breaks down, fragmenting badly (20% or so). The patch
with -o alloc=skip_busy:dirid_groups on makes things sane (3%).
So, to help test, you need some way of measuring fragmentation and a
whole bunch of benchmarks. I like fibmap: (http://
www.informatik.uni-frankfurt.de/~loizides/reiserfs/fibmap.html)
But Andrew has a fragmentation tool in the ext2 cvs I think.
Elsewhere, Chris Wright offered a patch, and Chris M. replied, "you'll need
some language at the top of the patch giving Hans the copyright on it." But
Hans Reiser said that this statement "was very imprecise. www.namesys.com/
legalese.html (http://www.namesys.com/legalese.html) describes it better. If
you give me the courtesy of allowing me to license reiserfs contributions to
third parties (in addition to the usual GPL license), it is appreciated, and it
allows me to sometimes earn badly needed funds for reiser4."
3. Page Attribute Table Support In 2.6
12 Apr - 21 Apr (20 posts) Archive Link: "PAT support"
Topics: Framebuffer, Ioctls
People: Terence Ripperda, Andi Kleen, Eric W. Biederman
Terence Ripperda said:
quite a while back, I sent out email about adding Page Attribute Table
support to the kernel (http://www.kerneltraffic.org/kernel-traffic/
kt20030616_219.html#3).
At the time, the concern was being able to mark remapped i/o pages Write
Combined in the case that we ran out of MTRRs to do so. This was mainly for
the agp aperture and framebuffer. With PCI Express systems coming out, the
need changes slightly. PCI Express does not have a centralized aperture
like agp that can be marked WC. Instead, individual system pages of memory
need to be marked WC via PAT in the page tables. This significantly
increases the need for PAT support under linux to maintain high performance
levels on PCI Express systems.
I thought the best approach would be to handle some of the original
feedback in the code before I came backto ping lkml. I discussed things a
little offline with Andi Kleen. He suggested I focus on the simpler i/o
regions first, then come back to handling main memory once I had that done
and had gotten feedback on it.
I've worked on a mechanism (cachemap) to track what type of caching a
region of memory is currently mapped as. when a new region of memory is
mapped, cachemap is queried to make sure the new region's caching type
matches the old type (or is compatible with the old type). if the cachemap
query succeeds, it's safe to map the new i/o region, otherwise it's not
safe.
for the first pass, I focused on testing ioremap. so the cachemap queries
are only made from ioremap. I also added code to have the mtrr code call
the cachemap code (in this case, it's a report rather than a query to
indicate the mapping's already made). I've made a few test runs on systems
here, and it seems to work fairly well.
this current patch includes the original PAT support and the new cachemap
mechanism. note that the cachemap mechanism does not actually change any
caching attributes, it only keeps track of the attributes and tests
regions. I think the end idea would be that drivers would use the normal
ioremap/change_page_attr/remap_page_range mechanisms like they already do,
and these mechanisms would in turn use cachemap to make sure there's no
conflicts. I'm completely open to how any specific details should work, and
any changes needed to be made.
Andi Kleen said Terence's work looked good for a start, and would be worth
testing, but still needed some cleanups. He added:
As for an interface - i still think it would be cleaner to just call it
from change_page_attr(). Then other users only need to call a single
function. But that's easily changed.
To make it really useful I think we need ioremap_wrcomb() and support in
the bus/pci mmap function (the PCI layer already has ioctls for this, they
are just ignored on i386 right). Then the X server could start using it.
Without any users the testing coverage would be probably not too good, but
it needs some testing in the real world before it could be merged first.
Maybe it could be simply hooked into the AGP driver and into some DRM
driver. Then people would start testing it at least.
Eric W. Biederman replied:
This would also be extremely useful on machines with large amounts of
memory, for write-back mappings. With large amounts but odd sized entries
it becomes extremely tricky to map all of the memory using mtrrs.
Last I looked the common remedy was to using overlapping mtrrs which the
kernel does not understand and that make it impossible for X to setup
write-combining MTRRs.
The memory map on x86 shows no hope of getting simpler and mtrrs are
getting continually less good at being able to scale so getting PAT support
of some kind in the kernel would be extremely good.
Andi agreed, saying:
I already had vendors complaining about this. But for this it will need
some more work - the MTRRs need to be fully converted to PAT and then
disabled (because MTRRs have higher priority than PAT). Doing so is a lot
more risky than what Terrence's patch does currently though. But longer
term we will need it.
Also it will still need to handle overlapping ranges. I suppose it will
need some simple rules like: converting from UC to WC is always ok.
Andi and Terence went back-and-forth for awhile, with Terence producing updated
patches; and the thread petered out.
4. Linux 2.6.5-mm5 Released
12 Apr - 19 Apr (23 posts) Archive Link: "2.6.5-mm5"
People: Andrew Morton
Andrew Morton announced 2.6.5-mm5, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.5/
2.6.5-mm5/
+ More CPU scheduler work. Hopefully this kernel will now address the
regressions that a few people have noted on certain workloads. We
appear to be getting close.
+ Various fixes, cleanups and code shrinkages.
5. Obfuscating Data For Security At The DM Level
13 Apr - 23 Apr (31 posts) Archive Link: "Using compression before encryption
in device-mapper"
People: Timothy Miller, Guillaume Lac?te, J?rn Engel
Guillaume Lac?te wanted to use compression at the dm (device mapper) level,
because compression would reduce data-redundancy, which would in turn (he
supposed) make it more difficult for attackers to interpret the system in order
to gain a foothold. Timothy Miller suggested that, "If you're compressing only
for the sake of obfuscation, then don't really try to save any space. Use a
fast compression algorithm which doesn't necessarily do a great job." But
Guillaume replied:
Actually I do _not_ want to do that. The reason for that is that I want to
add yet another layer before compression, which would interleave real data
with random bytes. These random bytes are not drawn uniformly but rather
drawn as to make the distribution on huffman trees (and thus on the
encodings) uniform. This ensures that in order to decode my real data, an
attacker has to decode the random data first; but since all _compressed_
random sequences are made equi-probable, there is (hopefully) no better way
for him to do this than brute force. This is the idea I have (successfully
?) implemented in http://jsam.sourceforge.net .
Thus I still want to "compress" my data even if its size grows.
The problem I encounter however is that if forcibly allocating more space
than required (e.g. 9 plain blocks every 8 compressed blocks) I will need
padding. However padding is generally unwise cryptographically speaking ...
J?rn Engel elsewhere offered his suggestions, adding that he thought a lot of
users would be interested in something like this. In Guillaume' original
posting, he'd said "I plan to systematically allocate 2 sectors per real sector
(space efficiency is _not_ my aim, growing entropy per bit is) and to use a
trivial dynamic huffman compression algorithm. Is this solution (which means
having half less space than physically available) acceptable ?" J?rn replied
now:
Makes sense. One of the zlib developers actually calculated the maximum
expansion when zlib-compressing data, so you could even get away with more
than 50% net size, but that makes the code more complicated. Your call.
Performance should not be a big issue, as encryption is a performance
killer anyway.
Whether it is acceptable depends on the user. Make it optional and let the
user decide.
Guillaume replied, "Oops ! I thought it was possible to guarantee with the
Huffman encoding (which is more basic than Lempev-Zif) that the compressed data
use no more than 1 bit for every byte (i.e. 12,5% more space)." J?rn said this
could be true, but he'd like to see the proof. Guillaume replied:
I was referring to the paper by Jeffrey Scott Vitter "Design and Analysis
of Dynamic Huffman Codes" (accessible through http://acm.org). It defines a
refinement of the well-known dynamic Huffman algorithm by Faller, Gallager
and Knuth such that the encoded length will use at most _one_ more bit per
encoded letter than the optimal two-pass Huffman algorithm (it is also
shown that the FGK algorithm an use twice the optimal length + on more bit
per letter).
My conclusion comes from the fact that for every text, the optimal two-pass
huffman encoding can _not_ be longer than the native encoding (apart from
the dictionnary encoding).
Actually I plan to implement the easier FGK algorithm first - if the whole
matter makes sense.
J?rn thought this made sense; he wished him luck, and suggested, "You should
try to use the crypto/ infrastructure in the kernel, add the compression
algorithm there and use it through the normal interface." Other folks continued
the technical discussion with Guillaume, which petered out shortly. However,
some time later J?rn posted again with some new objections to Guillaume's whole
idea. They argued the technical merits briefly, but the discussion dropped off
the list before any real conclusion could be formed.
6. Toward Variable, Concurrent Page Sizes
13 Apr - 15 Apr (15 posts) Archive Link: "hugetlb demand paging patch part [0
/3]"
People: Kenneth W. Chen, Arjan van de Ven, David Gibson, Martin J. Bligh
Kenneth W. Chen said:
In addition to the hugetlb commit handling that we've been working on off
the list, Ray Bryant of SGI and I are also working on demand paging for
hugetlb page. Here are our final version that has been heavily tested on
ia64 and x86. I've broken the patch into 3 pieces so it's easier to read/
review, etc.
1. hugetlb_fix_pte.patch - with demand paging, we can not unconditionally
assume valid pmd/pte. Fix it up in arch specific huge_pge_offset() and
have all caller check the return value.
2. hugetlb_demand_generic.patch - this handles bulk of hugetlb demand
paging for generic portion of the kernel. I've put hugetlb fault
handler in mm/hugetlbpage.c since the fault handler is *exactly* the
same for all arch, but that requires opening up huge_pte_alloc() and
set_huge_pte() functions in each arch. If people object where it should
live. It takes me less than a minute to delete the common code and
replicate it in each of the 5 arch that supports hugetlb. Just let me
know if that's the case.
3. hugetlb_demand_arch.patch - this adds additional arch specific fixes
for x84 and ia64 when generic demand paging is turned on. Also bulk of
the patch is to clean up with functions that no longer needed.
Some caveats: I don't have sh and sparc64 hardware to test. But hugetlb
code in these two arch looked like a triplet twin of x86 code. So I'm
pretty sure it will work right out of box. I've monkeyed around with ppc64
code and after a while I realized it should be left for the experts. I'm
sure there are plenty ppc64 developers out there that can get it done in no
time.
Patches relative to linux-2.6.5-mm4 and on top of hugetlb overcommit
handling patch posted by Andy Whitcroft.
Arjan van de Ven objected very strongly to the direction Kenneth's and Ray
Bryant's work was going. He said, "If you're going to make the kernel deal with
different, concurrent page sizes then please do it for real. Or alternatively
leave hugetlb to be the kludge/hack it is right now. Anything inbetween is the
road to madness..." There were several counters to this. Andy Whitcroft said
the work was really 2.7 material anyway, and as such was perfectly valid.
Martin J. Bligh said that Kenneth's and Ray's work could be seen as a first
(and good) step toward what Arjan's idea of full support. And David Gibson also
said, "bear in mind that in a number of ways these patches actually simplify
the hugetlb code, although I think most of that is not inherently related to
making the hugepage allocation on-demand rather than prefaulted. Nonetheless,
doing the demand allocation is actually really easy. Even if you add COW as
well, which these patches don't, it doesn't actually make the hack any worse
than it was already, but it does make it more useful."
7. Linux 2.6.5-mm6 Released
14 Apr - 18 Apr (7 posts) Archive Link: "2.6.5-mm6"
Topics: FS: ext3, SMP
People: Andrew Morton
Andrew Morton announced Linux 2.6.5-mm6, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.5/
2.6.5-mm6/
+ Added the first cut of the ext3 "reservation" code - it improves layout
of ext3 files, especially on SMP hardware.
+ Various small fixes and cleanups
8. SATA Going Into 2.4.27
15 Apr - 17 Apr (27 posts) Archive Link: "SATA support merge in 2.4.27"
Topics: Disks: IDE, Serial ATA
People: Marcelo Tosatti, Stefan Smietanowski, Andre Hedrick, Jeff Garzik
Serial ATA, although relatively new, has been moving very quickly. This week,
Marcelo Tosatti said:
Jeff Garzik sent me a SATA update to be merged in 2.4.x.
A lot of new boxes are shipping with SATA-only disks, and its pretty bad to
not have a "stable" series without such industry-standard support.
This is the last feature to be merged on 2.4.x, and only because its quite
necessary.
Any oppositions?
Stefan Smietanowski thought that this was a good idea, given that 2.4 would
still be the primary Linux kernel used by folks for quite some time, while 2.6
continued to stablize and become the standard.
Elsewhere, Andre Hedrick accused Marcelo of being bad, on the grounds that he
should have included SATA in 2.4.16 long before. Marcelo replied, "SATA 1.0 was
posted by Jeff in the past, but I preferred to hold it for a while (to make
sure its more stable). It now seems Jeff is more confident with it." There was
more acrimonious argument, and the thread petered out. Clearly, Marcelo's 2.4
decisions will continue to be controversial until 2.6 fully takes over.
9. Automatic Bug Checker Reveals Some Kernel Holes
16 Apr - 20 Apr (19 posts) Archive Link: "[CHECKER] Probable security holes
in 2.6.5"
Topics: Security
People: Ken Ashcraft, Andrea Arcangeli, Chris Wright, Jeff Garzik
Ken Ashcraft posted a long list of potential kernel bugs, explaining:
I'm working at a company called Coverity where we are building an
industrial strength derivative of the MC Checker system created at
Stanford. I've written a static analysis checker that looks for places
where the kernel gets a scalar from the user and then uses it without
performing bounds checks. For example, if a driver copies in an integer
from the user and then uses that integer as an array index or the length
argument to memcpy, the user can cause a buffer overflow.
The error reports below are sorted roughly by their severity. The last 12
of the errors are fairly minor because they are either protected by a
capable() check or the scalar is only 8 bits. I also consider it to be a
minor error to pass an unchecked value to kmalloc(). I realize that kmalloc
() will fail when asked for more than 128k, but it may not be appropriate
to allow the user to allocate that much memory. All of these minor errors
will be marked by [MINOR] and/or [CAPABLE] tags in the error report.
Chris Wright confirmed a bunch of Ken's bug reports, and posted patches; some
of which went directly into the 2.4 and 2.6 kernel trees via Jeff Garzik, and
some of which were first modified by folks like Andrea Arcangeli. Chris also
disconfirmed several other of Ken's reports, before the thread ended.
10. Linux 2.6.6-rc1-mm1 Released; Some Developer Disconnect
18 Apr - 21 Apr (30 posts) Archive Link: "2.6.6-rc1-mm1"
Topics: SMP, Virtual Memory
People: Andrew Morton, William Lee Irwin III, Paul Jackson
Andrew Morton announced Linux 2.6.6-rc1-mm1, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6-rc1/
2.6.6-rc1-mm1/
+ All of the anonmm rmap work is now merged up. No pte chains.
+ Various cleanups and fixups, as usual.
+ The list of external bk trees is getting a little short, due to
problems at bkbits.net. The ones which are here are not necessarily
very up-to-date with the various development trees.
William Lee Irwin III was dismayed to see that some of his recent cpumask fixes
had failed to make it into this release. He said, "What do I have to do to get
the bare minimal correctness fixes in this area propagated to mainline? The
important aspect of these is that they're pertinent to small SMP systems."
Andrew replied:
I confess to being moderately exhasperated at the amount of talk and
patching going on in the bitmap and cpumask areas. So when your patch
floated past with a terse description which was bristling with ifs, buts
and maybes I decided to take a pass.
If you want to send it again, cc'ing your co-conspirators and imparting
some confidence that this darned thing is actually meandering toward a
conclusion, please feel free ;)
William wrote back, CCing Paul Jackson, and saying, "What I believe I have sent
is the bare minimum change, with no cleanups or semantic changes. If you could
review and/or send approval or the like that would be very helpful for the
users of small SMP systems who are affected by the bug(s) you reported."
Unfortunately, at around this point it turned out that Paul, whose bug reports
had inspired William's work, disagreed that the bugs needed to be fixed, or
even that they affected small SMP systems. So that the subthread came to an
abrupt halt as the two of them went to private email to sort it out.
11. PS/2 Mouse Fixes; Targus Idiosyncrasy Workarounds
19 Apr - 20 Apr (7 posts) Archive Link: "[PATCH] psmouse fixes for 2.6.5"
Topics: Hot-Plugging
People: Kim Holviala, Dmitry Torokhov
Kim Holviala said:
Some fixes for PS/2 mice:
+ fixed hotplugging (real reset of device instead of softreset)
+ support for Targus Scroller mice (from my last weeks patch)
+ extended protocol probing fixed
The major change is that the probing of extended protocols is now changed
to be more configurable. Previously the driver probed for all the protocols
it knew about and stopped when it found one that the mouse accepted. This
didn't work with a bunch of mice so now you can choose the protocols which
are to be probed.
In current 2.6.5 the parameter "proto=imps" probes all protocols up to ImPS
/2. The patch changes this so that "proto=imps" ONLY probes for ImPS/2 and
if that fails uses regular PS/2. Similarly "proto=ps2pp,genps,exps" probes
for Logitech, Genius and Intellimouse Expolorer and if none found uses the
bare PS/2.
Dmitry Torokhov was confused why explicit support for Targus mice was needed,
since it seemed to him that the protocol mask could be set correctly by
default. But Kim replied, "Targus mice misuse the normal PS/2 protocol so that
they can sneak through command-filtering PS/2 ports (like on my Digital HiNote
2000). Basically they output very strange but valid traffic when the wheel is
moved. Anyway, this is Linux, and I'd rather force people to turn it on
explicitly rather than take the risk of breaking some valid PS/2 device which
might theoretically output the same stuff." Dmitry said that even so, folks
like distributions would have to turn on that option anyway, in order to be
sure to support the maximum number of systems; and so it should be on by
default regardless. Kim agreed with this, and Dmitry added, "Btw, I am also
trying to work in that area, might be beneficial if we combine our efforts. I
just posted bunch of changes on the list, you may also grab them with 'bk pull
bk://dtor.bkbits.net/input'" They agreed to talk more privately, and the thread
ended.
12. Documentation On Building External Modules
20 Apr - 21 Apr (4 posts) Archive Link: "[RFC] kbuild: Documentation - how to
build external modules"
Topics: Backward Compatibility, Kernel Build System
People: Sam Ravnborg
On the documentation front, Sam Ravnborg posted a rough and incomplete HOWTO on
building external modules:
Building external modules
kbuild offers functionality to build external modules, with the
prerequisite that there is a pre-built kernel avialable with full source. A
subset of the targets available when building the kernel is available when
building an external module.
Building the module
The command looks like his:
make -C <path to kernel src> M=$PWD
For the above command to succeed the kernel must have been built with
modules enabled.
To install the modules just being built:
make -C <path to kernel src> M=$PWD modules_install
More complex examples later, the above should get you going in most cases.
Available targets:
make -C $KDIR M=$PWD
Same as if 'modules' was specified. See description of modules target
below.
make -C $KDIR M=$PWD modules
Will build the module(s) located in current directory. All output files
will be located in the same directory as the module source. No attemps
are made to update the kernel source, and it is expected that a
successfully make has been executed for the kernel.
make -C $KDIR M=$PWD modules_install
Install the external module(s)
make -C $KDIR M=$PWD clean
Remove all generated files in current directory
Available options:
make -C $KDIR
Used to specify where to find the kernel source. '$KDIR' represent the
directory where the kernel source is. Make will actually change
directory to the specified directory when executed.
make -C $KDIR M=$PWD
M= is used to tell kbuild that an external module is being built. The
option given to M= is the directory where the external module is
located. When an external module is being built only a subset of the
usual targets are avialable.
make -C $KDIR SUBDIRS=$PWD
Same as M=. The SUBDIRS= syntax is kept for backwards compatibility.
make -C $KDIR M=$PWD help
help will list the available target when building external modules.
A more advanced example
This example shows a setup where a distribution has wisely decided to
separate kernel source and output files:
Kernel src:
/usr/src/linux-<kernel-version>/
Output from a kernel compile, including .config:
/lib/modules/linux-<kernel-version>/build/
External module to be compiled:
/home/user/module/src/
To compile the module located in the directory above use the following
command:
cd /home/user/module/src
make -C /usr/src/linux-<kernel-version> \
O=/lib/modules/linux-<kernel-version>/build \
M=$PWD
Then to install the module use the following command:
make -C /usr/src/linux-<kernel-version> \
O=/lib/modules/linux-<kernel-version>/build \
M=$PWD modules_install
The above are rather long commands, and the following chapter lists a few
tricks to make it all easier.
Tricks to make it easy
TODO: .... This need to be rewritten......
A make line with several parameters becomes tiresome and errorprone and
what follows here is a little trick to make it possible to build a module
only using a single 'make' command.
Create a makefile named 'Makefile' with the following content:
---> Makefile:
all:
$(MAKE) -C /home/sam/src/kernel/v2.6 M=$(PWD) \
$(filter-out all,$(MAKECMDGOALS))
obj-m := module.o
---> End of Makefile
When make is invoked it will see the all: rule, and simply call make again
with the right parameters.
If a driver is being developed that is targeted for inclusion in the main
kernel, an idea is to seperate out the all: rule to a Makefile nemed
+makefile (lower capital m) like this:
---> makefile
all:
$(MAKE) -f Makefile -C /home/sam/src/kernel/v2.6 \
M=$(PWD) $(MAKECMDGOALS)
---> End of makefile
The kbuild makefile will include only a single statement:
---> Makefile:
obj-m := module.o
---> End of Makefile
When executing make, it looks for a file named makefile, before a file
named Makefile. Therefor make will pick up the file named with lower
capital 'm'.
Prepare the kernel for building external modules
When building external modules the kernel is expected to be prepared. This
includes the precense of certain binaries, the kernel configuration and the
symlink to include/asm. To do this a convinient target is made:
make modules_prepare
For a typical distribution this would look like the follwoing:
make modules_prepare O=/lib/modules/linux-<kernel version>/build
TODO: Fill out the following chapters
Module versioning
Local include files
CLFAGS := include ...
Binary only .o files
Use _shipped ...
13. Linux 2.6.6-rc2 Released
20 Apr (1 post) Archive Link: "Linux 2.6.6-rc2"
People: Linus Torvalds
Linus Torvalds announced 2.6.6-rc2, saying, "Most of the _bulk_ of the rc1->rc2
changes is in the MIPS update, but there's a number of merges here too, notably
some networking, firewire, irda and bluetooth updates. Oh, and the regularly
scheduled network driver updates too, of course.."
14. Identifying Reverted Patches Based On Changelog 'Cset exclude' Entry
20 Apr (6 posts) Archive Link: "matching "Cset exclude" changelog entries to
the changelog entries they revert."
Topics: Version Control
People: Zack Brown, Andy Isaacson
I (Zack Brown) was rooting around in the kernel changelogs, and noticed
occassionally some entries including text like this:
Cset exclude: [email protected]|ChangeSet|20020403195622
It was clear that this meant that a previous patch had been reverted; but there
was no indication of how to identify that patch in the changelog. I posted to
the linux-kernel mailing list, saying:
I count 79 "Cset exclude" changelog entries since 2.5.4-pre1. Is there any
way to identify the changelog entry they revert?
for instance, "Cset exclude: [email protected]|ChangeSet|20020403195622" is in
2.5.8-pre2, as the full text of the changelog entry.
Without a way to identify the particular entry being reverted, I can't rely
on the fact that a particular changelog entry represents what actually went
into the kernel.
I realize there is almost certainly no way to directly deduce which
changelog entry is referenced by a particular 'Cset exclude' entry. But
maybe there is some *indirect* way, perhaps a website somewhere that tracks
this info?
Andy Isaacson said the following BitKeeper command would give me what I needed:
bk changes -r'[email protected]|ChangeSet|20020403195622'
For non-BitKeeper users, he also added:
you can construct a working URL by appending the cset key to
http://linux.bkbits.net:8080/linux-2.5/cset@
like so:
http://linux.bkbits.net:8080/linux-2.5/cset@[email protected]|ChangeSet|
20020403195622
I tried this, and it worked.
15. High Resolution Timers Update For 2.6
20 Apr - 23 Apr (2 posts) Archive Link: "[ANNOUNCE] New high resolution time
patch for 2.6.5 kernel"
Topics: POSIX, Real-Time
People: George Anzinger, Mark Gross
George Anzinger said:
The High Resolution Timers patch for the 2.6.5 kernel has just been posted
on sourceforge.
This patch provides an extension to the POSIX clocks and timers to define
two new high resolution clocks (CLOCK_REALTIME_HR and CLOCK_MONOTONIC_HR).
The resolution of these clocks can be set at CONFIGURE time, with the
default being 10 micro seconds. The high res clocks can be used with
clock_nanosleep() as well as with the POSIX timers.
This version uses apic timers to obtain much better accuracy and
simplicity.
Mark Gross replied happily, "Wow! for my systems this works really well! The
jitter is significantly reduced. It would be cool if this could find its way
into a major developmet tree."
16. Kernel Mailing List Marked As Spammer By SpamCop
20 Apr - 22 Apr (13 posts) Archive Link: "vger.kernel.org is listed by
spamcop"
People: Rik van Riel, Matti Aarnio, Miles Bader, Timothy Miller, Erik Mouw, Jan
De Luyck
Jan De Luyck noticed to his dismay, that vger.kernel.org, the server hosting
the linux-kernel mailing list, had been tagged as a spammer by SpamCop (http://
www.spamcop.net/) . He asked that something be done to get the server off of
their spammers list, so linux-kernel emails could be distributed to users with
SpamCop-filtered email. Rik van Riel replied:
Ask your mail server admin. The only people who need to take action are the
ones stupid enough to use spamcop's blocklist for outright mail blocking.
The spamcop site even says that their list probably shouldn't be used for
outright blocking.
Matti Aarnio also replied to Jan, saying:
Reading SPAMCOP pages I think they are most unwilling to make any
exceptions. Per this document:
http://www.spamcop.net/fom-serve/cache/298.html
The only way to handle this is to have smarter people, who are always
vigilant enough to look deeply into the message headers and do realize that
some spam has leaked thru VGER's lists. They may report those _ONLY_ to
VGER's postmaster (several people), who can (to an extent) add keyword
based filters to Majordomo.
Any single less savvy person receiving the list could still accidentally
get VGER again listed in a number of spam-block lists.
Another would be to run the lists in fully CLOSED mode, which would still
let a bunch of viruses thru... (filters are mostly biting on those already,
though.) But it would be most nasty mode in other forms...
Miles Bader replied:
I'm confused -- the spamcopy info page you listed implies that hosts are
listed if they are an _open relay_, which is a completely different thing
from `spam leaking though VGER's lists.'
If VGER actually is an open relay, that's very bad, but presumably
something easily solved by the machine's maintainers. Some spam getting
through to VGER list recipients, on the other hand, is just annoying (and
certainly shouldn't be the cause of any blacklisting).
The spamcop report page seems to say that the listings are due to user
reports; could the real problem be clueless users who don't understand the
difference above?
Does anyone have a better idea of what's actually going on?
Erik Mouw did some checking, and confirmed that vger.kernel.org was most
definitely not an open relay. Rik added:
While most of the spamcop administrators seem pretty smart, their system
definitely is vulnerable to the "Garbage In, Garbage Out" principle.
I'm certain than vger got listed on spamcop due to linux-kernel subscribers
reporting to spamcop some of the spam that leaked onto lkml, through
Matti's strict filters.
I wouldn't be surprised if some of those same users were now complaining
they couldn't get their linux-kernel email. ;)
In my opinion, there are only two types of anti-spam lists that can be
responsibly used:
+ lists run by people smart enough to recognise that they make mistakes
and are willing to correct them whenever they happen
+ lists run in an entirely automated fashion, with no human input
whatsoever -- but only when the software is administrated by people
willing to correct problems that happen
Lists that take the philosophy of "sorry that was our mistake, but we're
still not going to make an exception" probably aren't the right lists to
use if you care about your email.
The discussion continued briefly, and eventually Timothy Miller said, "It
appears that we've been de-listed from SpamCop, probably because I, amongst
certainly countless others, complained to them about it. Perhaps they will be
smart and permanently whitelist vger.kernel.org."
Indeed, vger.kernel.org's IP number is no longer listed (http://www.spamcop.net
/w3m?action=blcheck&ip=67.72.78.212) as a spammer at SpamCop.
17. Linux 2.6.6-rc2-mm1 Released
21 Apr - 24 Apr (10 posts) Archive Link: "2.6.6-rc2-mm1"
Topics: Framebuffer
People: Andrew Morton
Andrew Morton announced Linux 2.6.6-rc2-mm1, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6-rc2/
2.6.6-rc2-mm1/
+ Several framebuffer driver fixes. Please test.
+ Input driver fixes, cleanups, features.
+ Dropped the signal race fixes. See if we can find a better way to fix
this.
18. ELSA (Enhanced Linux System Accounting) Project Begun
22 Apr (1 post) Archive Link: "[Announce] Enhanced Linux System Accounting -
ELSA project"
Topics: BSD, Version Control
People: Guillaume Thouvenin
Guillaume Thouvenin said:
We are pleased to announce a new project about accounting on Linux. This
project is called ELSA (Enhanced Linux System Accounting) and a description
follows. Any comments, ideas or participation are welcome.
ELSA (Enhanced Linux System Accounting) announce http://
elsa.sourceforge.net.
The goal of accounting is to collect and report the use of various system
resources by applications. Informations, like process time, CPU usage,
connect time or disk space usage, provides data that helps the system to
adjust the use of resources between processes.
The current BSD-like process accounting that already exists in Linux
collects informations on individual users or groups of users. The ELSA
project aims to improve and extend the monitoring of resources with
different criteria like groups of processes. Another target for this
project is to give Linux an homogeneous set of commands for all kinds of
accounting (memory, CPU and I/O).
Here is the state of the art about ELSA (Enhanced Linux System Accounting).
+ Concerning the documentation:
o All drafts can be access at http://sourceforge.net/docman/?group_id
=105806
+ Concerning the code:
o The structure "bank" allowing to manage groups of process is based
on Linux kernel 2.6.5
o A test program is provided making it possible to validate our
implementation.
o Currently, these structures can be handled by a user via the system
call elsa(). A functional patch is provided on the web site http://
sourceforge.net/project/showfiles.php?group_id=105806
+ Concerning the CVS:
o The module structure is a little tricky because we have created
some useless repositories. Anyway, there are 4 modules which are:
# modifs_kernel: contains modified files for a Linux kernel
2.6.5. We don't support versions 2.4.x for the moment but if
it's needed, it's easy to port.
# tests: a program that allows to test the "bank" and a basic
Makefile to build the test.
# drafts: contains the documents relating to ELSA.
# scripts: Scripts can manage an environment of work. It just
copies files between environments. It's more for a personal
use.
o There are also two unused modules which are:
# elsa
# Documentation
19. libata Updates, Including Hotplug Support
24 Apr (2 posts) Archive Link: "[sata] libata update"
Topics: Hot-Plugging, Serial ATA, Version Control
People: Jeff Garzik
Jeff Garzik said:
I'm slowly and (somewhat) quietly cleaning up the libata internals. This is
allow several interesting features to appear in rapid succession: hotplug,
random taskfile submission (SMART!), and ATAPI.
This is the first step in cleaning up the internals. Nothing terribly
interesting for existing users, except for one key change:
Promise SATA driver has been split. Promise TX2/TX4 SATA remains in
"sata_promise". The very-different Promise SX4 support is now found in a
new driver "sata_sx4". Promise users, please test and make sure I didn't
break anything. It seems to work on my Promise SATA h/w.
Linux 2.6.x patch and changelog:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/libata/
2.6.6-rc2-bk3-libata1.patch.bz2
http://www.kernel.org/pub/linux/kernel/people/jgarzik/libata/
2.6.6-rc2-bk3-libata1.log
Linux 2.4.x patch and changelog:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/libata/
2.4.27-pre1-libata1.patch.bz2
http://www.kernel.org/pub/linux/kernel/people/jgarzik/libata/
2.4.27-pre1-libata1.log
BitKeeper repositories:
http://gkernel.bkbits.net/libata-2.4
http://gkernel.bkbits.net/libata-2.6
J. Ryan Earl was very happy to see this, but there was no discussion.
20. Linux 2.6.6 Released
9 May - 11 May (12 posts) Archive Link: "Linux 2.6.6"
Topics: Disks: IDE, FS: CIFS, FS: FAT, FS: NTFS, FS: XFS
People: Linus Torvalds, David Eger
Linus Torvalds announced Linux 2.6.6, saying:
Ok, there it is (well, the tar-file and patches are still being up-loaded,
but should be there soon).
NTFS, XFS, FAT and CIFS updates. IDE cache-flush at shutdown fixes. ppc,
sparc, s390 and ARM updates (and a few x86-64 fixes).
Holler if I missed anything.
David Eger replied, "The -mm branch has two radeonfb bug fixes; please apply
them. One is BenH's. Mine, which fixes a corruption problem with overlapping
copyarea()'s is below."
21. Linux 2.6.6-mm1 Released
10 May - 12 May (62 posts) Archive Link: "2.6.6-mm1"
Topics: FS: sysfs, Hot-Plugging
People: Andrew Morton
Andrew Morton announced 2.6.6-mm1, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6/
2.6.6-mm1/
+ x86_64 sched-domains support
+ Added the sysfs-backing-store patches
+ A number of patches to shrink/consolidate dentry fields. Needs careful
testing. The relevant diffs are:
d_flags-locking-fix.patch
d_vfs_flags-locking-fix.patch
dentry-shrinkage.patch
dentry-qstr-consolidation.patch
dentry-qstr-consolidation-fix.patch
dentry-d_bucket-fix.patch
dentry-d_flags-consolidation.patch
dentry-layout-tweaks.patch
+ The ia64 CPU hotplug stuff is all here now and doesn't appear to break
anything.
+ Lots of fixes/cleanups/etc.
22. Status Of InterMezzo In 2.6
12 May (2 posts) Archive Link: "Re: [PATCH]InterMezzo Patch against
linux-2.6.6"
Topics: FS: InterMezzo
People: Linus Torvalds, Peter J. Braam
Chen Yang asked Linus Torvalds privately why InterMezzo had been removed from
the 2.6 kernel; and Linux explained on the list:
It's removed because Peter seems to not want to support it any more, and
there didn't seem to be any usage. But if people out there are actually
using it, and somebody wants to support it _and_ it is in good working
state for 2.6.x, we could certainly bring it back to life.
But if you only do this because you want to fix warnings that people have
posted about the sources, and you're not actually seriously using it and
maintaining it, then I'll just apply this patch against plain 2.6.6 (so
that it doesn't go away if somebody wants to resurrect it later).
In other words - holler if you want to seriously support it. We're not
deleting it to be spiteful..
Peter J. Braam replied, "I spoke with Chen and you can just drop InterMezzo
now. It is easy to maintain as a module outside the kernel, we did that for
some years."
We Hope You Enjoy Kernel Traffic
Kernel Traffic is grateful to be developed on a computer donated by Professor
Greg Benson and Professor Allan Cruse in the Department of Computer Science at
the University of San Francisco. This is the same department that invented
FlashMob Computing. Kernel Traffic is hosted by the generous folks at
kernel.org. All pages on this site are copyright their original authors, and
distributed under the terms of the GNU General Public License, version 2.0.