Re: xz meltdown/Lasse Collin

On Sat, 2024-04-13 at 20:11 -0400, Theodore Ts'o wrote:
> On Sat, Apr 13, 2024 at 10:12:44AM -0400, James Bottomley wrote:
> > 
> > I gave an idea for that above.  But the first key step is pro-
> > active identification.  The MO of the burnout attack was to try to
> > make the attacker the single help resource; people in this position
> > tend not to know they need to ask for other help.  So the first
> > step has to be pro-active in looking for them (which is also useful
> > for providing a risk report about the entire ecosystem).
> 
> It's important not to over-index on the specifics of how the xz-utils
> attack was carried out.  Yes, this time the attack was carried out by
> trying to identify someone who was close to being burned out.  It
> also involved generated files and autotools.  But that's not the only
> way such an attack could be carried out.

I'm not saying it is. I'm saying now we've seen an attack vector we
aren't defending, we need to investigate strengthening our response
along it. I mean there are potentially good things the choices point
to: like nothing was committed to the repo because the attackers feared
that would attract more scrutiny, so they went for something indirect.

> If we're talking about a nation-state with a vast amount of resources
> and patience, the next time the attack might involve someone just
> becoming a trusted contributor and contributing years of good work
> and reviews before trying to submit a change which introduces some
> other kind of back door or bug.

I don't think we are talking a nation state: they tend to be way better
resourced and more impatient.  So at least for technology they'd go for
corruption rather than sleeper infiltration.  This looks more like a
blackhat entity challenge.  But regardless a nation state corruption
attack could also come at us along the same initial vector.

> Perhaps the next attack will involve getting an agent hired by a
> major AI chip vendor, and the backdoor will be hidden inside the
> proprietary binaries distributed by that AI chip vendor to download
> firmware into that company's GPU.  (And if these propietary drivers
> and binaries are used by a large number of hyperscale cloud vendors,
> a backdoor into that GPU driver or the GPU's support binaries would
> be.... devastating.)

Ideally I'd like it if there were someone watching all the doors for
the next one, but our particular job here is to close this stable door
to prevent the next horse from bolting again.

> That's not to say that we shouldn't find ways to better support
> https://xkcd.com/2347.

That's the wrong lesson: xz is just one of a number of compression
tools and it isn't the compression tool du jour, it's now an also ran.
The library that reached sshd just happened to pull it in for
compatibility.  So our problem isn't with essential, it's with
peripheral but still present.

If you try to think like the attacker: they were looking for an ssh
backdoor (they already had the payload, they were looking for
delivery), so they first charted all the indirect dependencies of sshd
in the target system then they identified a bunch of potentially
vulnerable projects and then gamed what to do with each before lighting
on xz to invest their time in.  In this scenario, I think the fact that
that the indirect dependency was distro patched rather than upstream
was somewhat accidental.

>   Or that we interrogate whether autotools is the best long-term
> solution, or how to come up with a better solution without going down
> the systemd approach of "we only care about Linux and all other OS's
> are Not Our PRoblem).  Or that we should have tools that look for
> differences between binaries built for distributions versus built out
> of the git tree, perhaps by looking for behavioral or
> performacne differences.

I think there is considerable focus on proveability, reproduceability
and transparency in builds, which will eventually fix this issue
without our needing to do anything, yes.

> But let's not over-focus on the burnout attack, or the "only activate
> if the build infrastructure is Debian or Fedora" attributes of what
> happened this time around.  It is very likely that the next attempt
> to subvert software supply chain will be quite different.

I think it will come along a similar vector.  Probably corruption of a
disaffected maintainer rather than burnout, but we should still try to
fix that.  Beyond the kernel trying to rein in our indirect
dependencies (and the code they can execute without ever being used by
the target binary) would also be a good thing.

James