Editor Ender Rules

I'm working on implementing rich text editing for Instantbird so we can 
send HTML messages. When I input "enter" it appends a <br> to the 
document, then sends the message.  I want to (in some way) change the 
rules set [1] to only insert a <br> for Shift+Enter, so I can observe 
when an enter is pressed and send the message (which works fine). [2] 
contains a statement implying this can be done, I believe, in:

> Every instantiation of an editor has it's own rules object.
> nsTextEditor and nsHTMLEditor both have an InitRules() function which
> handles this. Until we make rules plugin at the runtime level, you
> will have to settle for plugging them in programmatically by altering
> the InitRules() method of nsHTMLEditor to use your rules system
> instead.

This seems to imply it can be done, but I have no idea how to go about 
doing this.  If possible we could patch the Mozilla editor component 
before it's included in our build, but I'd prefer to do it in JavaScript 
(and we try to patch the toolkit as little as possible).

If someone has an idea of how I could work around this in an easier way 
I'd be interested, but a lot of the documentation seems poorly documented.

Also, is there more up-to-date documentation about editor interfaces and 
components? In particular using them from a XUL application (as opposed 
from a website)?

-Patrick

[1] http://www.mozilla.org/editor/rules.html
[2] http://www.mozilla.org/editor/writing-rules.html