[RFC] "High" and "expanding" ROM prefixes, and the trouble with large ROMs

Hi all,

Though many people use gPXE very successfully as a second-stage PXE
loader, or load it from a storage device of one type or another, the
original use of Etherboot was as a burnable ROM, and that's still an
essential feature for us to have. As gPXE gains more optional
subsystems (cf. wireless) that necessitate large amounts of code,
there become more and more use cases that would love to use gPXE in a
ROM, if only it weren't so darned *big*. I'd like to propose, with
code, a couple variations on the theme of gPXE's ROM prefix that allow
it to be used with larger ROMs, at the expense of some degree of
compatibility. These will probably never replace the current ROM
prefix, but they'll probably wind up eventually as options in mainline
if they prove stable and useful, so I want to make sure they get as
much testing and as many comments from interested parties as possible.

Since most of the readership of this list is probably not intimately
familiar with the operation of gPXE's existing ROM prefix, I'll go
into some technical detail.

gPXE in option ROM form is very much subject to the limitations of
legacy PC architecture. In the 1MB of base memory that real-mode
applications like the BIOS can access, all option ROMs must fit
between addresses 0xC0000 and 0xE0000 - a total of 128 kilobytes. With
the average video card's option ROM taking up about 64kB, most systems
that would use gPXE don't have all that much space for it. As more
features become available that people would like to use, 64kB starts
looking more and more cramped. To be clear, the size of a gPXE that
can perform some baseline of tasks hasn't increased much if at all
lately, as we're very careful with code size; but adding a new
subsystem obviously makes the ROM image larger for those who choose to
use it. While one could chainload a larger gPXE image supporting
advanced scripting or a new download protocol, link-layer code
(wireless, Infiniband) really has to be in the ROM for it to be of any
use.

Recent BIOSes supporting version 3.0 of the PCI BIOS specification
provide one part of a solution, in that they don't load the gPXE ROM
to the 0xC0000-0xE0000 option ROM space right away. Instead, they load
us somewhere in base memory (which is not as space-constrained) and
then let us copy our runtime image to option ROM space ourselves,
possibly shrinking that image in the process. This capability is
useless without an older BIOS feature, PMM (POST Memory Manager), that
allows gPXE to allocate a block of high memory to store the bulk of
its code, leaving the cramped option ROM space untouched except for a
~2kB stub. Currently, a machine that supports PCI3 without PMM will be
just as ROM-constrained as in the legacy case; one that supports PMM
without PCI3 allows gPXE to shrink itself to make room for ROMs loaded
after it, but we still have to fit in the option ROM space at the
start.

Almost any machine you might find these days claims to support PMM; in
fact, many vendor PXE ROMs will refuse to boot on a machine that
doesn't support it. The trouble is that gPXE is more demanding than
most clients of the BIOS's PMM services. In order to avoid issues with
the A20 line, it needs a block aligned to an even megabyte in memory;
if the BIOS doesn't start at 2MB by default (which would be the
sensible thing to do), gPXE may have to increase its requested
allocation to 2MB in order to get 2MB alignment. Some BIOSes simply
refuse to hand out allocations that large; I'm aware of AMIBIOS as a
particular offender in this area.

The first prefix extension I wrote, the "high ROM" (.hrom), is a very
simple one, and only affects a very specific class of machines: those
that support PMM but refuse to hand out an allocation we can use. It
adds a fail-safe mechanism: if we can't get a good PMM allocation, we
blindly use a block starting at 4MB without allocating it. While this
seems dangerous (and is in theory), it seems to work well in practice.
The existing ROM prefix uses 4MB as a scratch space in the absence of
PMM, but only after it is executed as a boot device; .hrom requires
that the 4MB area be preserved between ROM initialization and boot
execution, so there is the potential for conflict with other add-in
cards (especially other gPXE cards) that also had this bright idea.
That's why I'm hoping for as much testing as possible. :-) .hrom only
provides a tangible benefit on PCI3 systems; on those systems, it
increases the maximum ROM size to a BIOS-dependent value that is
usually at least 75kB.

The second prefix extension, the "expanding ROM" (.xrom), is more
radical. It only works on real PCI add-in cards - not ISA and (more
importantly) not a NIC integrated into the motherboard - but on those
cards it provides a radical increase in possible ROM size. It
advertises itself in headers as a small stub ROM, about 3kB, so the
BIOS only loads that much of it and only takes up option ROM space
with that much of it, even on pre-PCI3 systems. During initialization,
the PCI card's ROM resource is mapped into memory and copied, in full,
into a PMM-allocated buffer (using the .hrom 4MB failsafe if PMM can't
allocate properly). This provides for no upper limit on ROM size
except the size of the flash device, which often can go up to 128kB or
even beyond. Again, because it's difficult to predict how this
behavior will interact with the strange combinations of hardware out
there, testing would be greatly appreciated.

If you have a gPXE git checkout and would like to help test, you can
merge these prefixes in from the staging repository:
  git remote add staging git://git.etherboot.org/scm/gpxe-staging.git
  git fetch staging
  git merge staging/xrom    # contains both .xrom and .hrom;
staging/hrom has just .hrom
and create ROMs using e.g. "make bin/10ec8139.xrom". Functionality of
normal .rom compilations is not changed at all by the modifications.
(To get back to a pristine git checkout, git reset --hard
origin/master.)

In addition to testing, I welcome questions and comments about the
approach these prefixes take. This stuff can be rather arcane, and I
don't pretend to know I've got it right; all I know is that it's
worked in the testing I've done, and the theory *seems* sound. More
eyeballs can hopefully make that a stronger assertion :-)

Thanks,

-- Josh

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference