Re: Dealing with LLMs in IETF discussions draft

Hi Rob,

Thanks for the comments. You're right that "humans originate" is too strong
as stated: if a model produces a proof no one has found, that is new
knowledge in a meaningful sense, not simply regurgitation.

The distinction I was trying to draw is between novel outputs and sustained
innovation. Many current AI breakthroughs are new solutions within an
existing conceptual framework, while historical innovations often involve
creating new concepts or redefining the problem space itself. Current LLMs
can occasionally produce such insights, but they do not yet appear to have
a sustained autonomous loop for innovation — setting goals, accumulating
experience, learning from failures, and evolving their conceptual models
over time. (I would concede that this gap is narrowing: RL-based systems
already iterate within closed loops, although the goals and evaluation
criteria are still largely defined externally.)

This does not mean AI will not innovate in the future. I believe future AGI
systems will likely become autonomous innovators, which will have
significant implications for fields like IETF. The key question is not
whether AI can generate something new, but whether it can continuously
create, evaluate, and evolve new concepts.

On the broader question of AI risk, I take the concerns seriously. But I do
not think the answer is to stop building. The challenge is to create the
conditions, while we still can, for increasingly capable AI systems to
interact with human society in a reliable and beneficial way. Standards,
protocols, and institutions are part of that foundation — and IETF is one
of the places where this future is being shaped.

Best regards,
Chong

Rob Wilton (rwilton) <[email protected]> 于 2026年8月7日周五 下午4:42写道：

> Hi Chong,
>
> I'm not convinced of the "human's originate" argument, at least not for
> the recent generations of frontier models.  I think that it is easy to
> assume that because LLMs are just token prediction engjnes that means they
> cannot originate new ideas or concepts, but only regurgitate what is
> already there, but I don't think that is true.
>
> E.g., this link (
> https://www.forbes.com/sites/jonmarkman/2026/08/03/openais-astra-solved-10-decades-old-math-problems-for-just-2000/
> ) is about how LLMs were used to cheaply solve maths problems that no
> human mathematicians have been able to solve.  There must surely be some
> element of an LLM creating new knowledge as part of forming these proofs
> even if the individual steps of the proofs happen to build on other known
> work.
>
> As a complete tangent to the discussion, there is also this book:
> https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_Dies, where
> the authors claim that if anyone builds AIs (using current techniques and
> little actual control of their behaviour) that have greater than human
> intelligence then it is a 100% guaranteed certainty to wipe out humanity.
> Since I believe that their proposed mitigation steps are not likely I hope
> they are wrong ;-)
>
> Kind regards,
> Rob
>
>
> *From: *chong feng <[email protected]>
> *Date: *Friday, 7 August 2026 at 07:42
> *To: *Nathanael Ritz <[email protected]>
> *Cc: *Stephen Farrell <[email protected]>;
> IETF-Discussion <[email protected]>
> *Subject: *Re: Dealing with LLMs in IETF discussions draft
>
> Hi Nathanael,
>
> Thanks for the thoughtful comments. I think you raised an important
> distinction that is easy to overlook: the difference between knowledge
> that is new to an individual and knowledge that is new to humanity.
>
> I agree that LLMs can introduce concepts, connections, or perspectives
> that are new to the human user. However, I am not sure this is
> fundamentally different from other sources of intellectual input —
> reading a paper, a book, an encyclopedia, or discussing with an
> expert.
>
> Let me give a concrete example from my own work. AIN started from a
> question I did not know the answer to: how can AI agents collaborate
> with each other at scale? I asked an LLM and learned about existing
> approaches. That information was useful, but I identified a
> fundamental bottleneck: manually configured relationships between
> agents would not scale. This led me to ask a different question: why
> can't agents discover each other through something analogous to
> routing, not based on network addresses but on semantic intent?
>
> The LLM helped me explore the existing landscape and validate the
> direction. But it did not identify the bottleneck, make the
> cross-domain analogy with IP routing, or create the abstraction of
> semantic addressing. Those steps required human understanding of the
> problem space and the ability to connect concepts across domains.
>
> This is what I mean by "humans originate." The origin is not about
> where information first entered my mind. It is about who performs the
> abstraction, integration, and judgement required to turn information
> into a new conceptual framework.
>
> This distinction matters for IETF. The final contributor needs to
> understand, evaluate, and stand behind the technical claims being
> made. If someone submits a draft they cannot explain under questioning
> — whether it was produced with AI, a ghostwriter, or copied from
> existing material — the community will discover that through review.
> The important question is not tracing the provenance of every idea,
> but whether the author truly understands and takes responsibility for
> the work.
>
> Best,
> Chong
>
> Nathanael Ritz <[email protected]> 于2026年8月7日周五 11:09写道：
> >
> > Comments inline
> >
> > On Thu, 6 Aug 2026 at 20:24, Jay Daley <[email protected]> wrote:
> >>
> >> Hi Stephen, Chong
> >>
> >> > We just pushed a -01 version of the draft [1] that lists
> >> > (most of) the arguments raised on the list as that may be
> >> > a more useful starting point for a discussion on a new
> >> > list.
> >>
> >> Thanks for this.
> >>
> >> The one big thing the document does not tackle is the assertion that
> where the AI introduces something the human did not know about, then the
> human will inevitably trade off their own understanding of that for the
> acceleration of output that AIs enable.
> >>
> >>
> >> In addition, the hypothesis of section 4 of the document relies on two
> pillars that I think obscure the real issues:
> >>
> >> 1. The claim that only humans can create new knowledge, but not AIs,
> and that is what is being introduced by the human into the discourse:
> >>
> >> > Genuine innovation --- the creation of new conceptual territory
> rather than more efficient mapping of existing terrain --- remains a human
> capacity.
> >>
> >>
> >> > The symmetry is clean: humans originate, AI executes. Humans open new
> territory; AI operates efficiently within it.
> >>
> >>
> >> I doubt very much if every message sent with the use of an AI is
> creating new knowledge rather than applying existing knowledge and would be
> surprised if it happens at all.  Engineering is, by definition, the
> application of existing knowledge to new problems.  It is more likely that
> authors who think this is happening are mistaking knowledge that is new to
> them as knowledge that is new to humanity.
> >>
> >> We could always test it by asking an AI "is this message introducing
> any genuine innovation or is it just applying existing knowledge?"
> >>
> >> 2.  The claim that the human being the decision maker somehow shapes
> what is said:
> >>
> >> > the AI is functioning as a thinking partner within a bounded space,
> not as an originator. The human remains the decision-maker about what to
> accept and what to discard. What falls outside this paradigm is delegating
> the thinking itself: asking AI what position to take, what arguments to
> make, or what conclusions to draw
> >>
> >> The key point that is being obscured there is that the AI will
> inevitably be introducing that human to concepts that are new to that human
> and if they are incorporated in any way, either directly or by
> linkage/comparison, then the AI is responsible for that innovation not the
> human.  This is exactly the same as if I were to sit down with an expert in
> a particular field and discuss my proposed email with them and then ask
> them to draft if for me.
> >>
> >
> > Ultimately, I don't think many of us can claim to know much beyond what
> others have imparted to us anyway. Even the self-taught autodidacts surely
> started by reading through something of personal interest that someone else
> first discussed, and then experimenting from there.
> >
> > In this case, I honestly don't see the situation as dramatically
> different from an informal web search. We have not and do not typically
> cite every signal piece of knowledge we share that's new to us despite
> having access to web search, Wikipedia and/or say TV Tropes for figurative
> eons.
> >
> > And much like I personally often saw Wikipedia as a useful jumping off
> point, I'm sure we know plenty of people who treat a two-minute read of a
> Wikipedia article as having 'done their research' enough to speak
> confidently on a subject they don't really have the means to understand yet.
> >
> > What's different, perhaps, is an LLM's conversational nature, which
> allows for much faster traversal of a topic. This, combined with AI's
> unguarded propensity to make stuff up while speaking as confidently as a
> genuine expert (who might otherwise know when to say "I don't know")
> creates some genuine risks when neither party in the exchange has enough
> information to make informed decisions about what was just reviewed.
> >
> > But where someone learns something genuinely new to them and then
> composes it with another existing system to introduce some productive novel
> effect?
> >
> >>
> >> This conversation has made it clear to me that this is about so much
> more than someone putting their name to the output and taking
> responsibility for it, because it seems as if the boundary between what the
> AI introduced and what the human introduced is hard for the human to
> self-identify.
> >
> >
> > In that case, while deeply fascinating to me personally, I am not sure
> why it would matter so much if it was AI, Wikipedia, a book or a magazine
> that first made them wise to the subject matter.
> >
> > Cheers,
> > Nathanael
> >
> >>
> >>
> >> Jay
> >>
> >>
> >> --
> >> Jay Daley
> >> [email protected]
> >> www.ietf.org
> >>
> >>
> >>
> >>
>
>