kgdb replacing newline with CRLF in custom query response

Omar Sandoval <[email protected]> Thu, 7 Nov 2024 14:23:26 -0800
Newsgroups org.kernel.vger.linux-debuggers,org.kernel.vger.linux-serial
Message-ID <Zy093jVKPs9gSVx2@telecaster>
Hi everyone,

Amal is working on adding a custom query packet to kgdb for getting the
kernel's vmcoreinfo. The rationale and details are available here:
https://github.com/osandov/drgn/wiki/GDB-Remote-Protocol-proposal:-linux.vmcoreinfo-query-packet

vmcoreinfo is about 3kB, so we were hoping to avoid hex-encoding the
response and doubling the time it takes to transmit over a slow serial
connection. Instead, we were hoping to use the escaped binary format,
which escapes the characters #$}* and leaves other bytes untouched.

We ran into a problem, though: vmcoreinfo contains newline characters,
which the serial core replaces with CRLF; see commit c7d44a02ac60
("serial_core: Commonalize crlf when working w/ a non open console
port"). This effectively corrupts the data and causes a checksum
mismatch.

We'd love some input on how to work around this, especially from the
kgdb maintainers. Here are a few options, in descending order of my
preference:

1. Disable the LF -> CRLF replacement while sending binary data.
2. Escape the newlines using some other custom scheme.
3. Give up and hex-encode the response.

Any thoughts?

Thanks,
Omar