Re: Proposal -- Interpretation of DFSG on Artificial Intelligence (AI) Models

Matthias Urlichs <[email protected]> writes:

> The fact remains that our builders will be unable to reproduce the
> resulting network, for well-known practical reasons. Thus we
> mostly-have-to-trust the original publisher that their network has been
> built as documented (or even "documented" given the status of gnubg). In
> practice this is not a problem for a Backgammon engine, or even for
> Tesseract because any serious use case supports, if not requires, human
> verification of the result — but how sure can I be that a LLM intended
> for home automation doesn't contain an Open Sesame backdoor that unlocks
> my *home*'s back door?

Right, this is a known attack in the security literature with some
research behind it already. See, for example:

    https://arxiv.org/abs/2204.06974

We could get some protection by retraining the model from the base
training data and substituting our constructed model for the
upstream-provided one, but (a) that puts a lot more weight on our ability
to rebuild the model than just verification, and (b) I would not assume
it's impossible to hide backdoor construction in the training data either,
particularly if the training data is voluminous. See, for example:

    https://nisos.com/research/building-trustworthy-ai/

and that's just the first of many links that I found in a quick search.

Obviously there are a bunch of use cases for these things that will never
involve adversarial data, and not everything needs to be robust in order
to be included in Debian, but it's one of the things to be thinking about
if accepting even our current status quo position of including some ML
models in Debian main without being able to verify the model construction.

LLMs in particular are nascent techology with novel security flaws that
researchers are only starting to explore. I think the chances are high
that their security will get much, much worse before it gets better. It is
one of the many reasons why I am generally an LLM skeptic.

-- 
Russ Allbery ([email protected])              <https://www.eyrie.org/~eagle/>