Re: Hallucinated citations are polluting the scientific literature. What can be done?

On Mon, 18 May 2026 10:46:11 -0400, Jonathan Lamothe
<[email protected]> wrote:

>Steve Hayes <[email protected]> writes:
>
><snip>
>> On Tue, 12 May 2026 02:45:03 +0100, Lawfare Review
>> <[email protected]> wrote:
>> LLM bots can be useful tools but they are not AI, and are not
>> reliable.
>
>I have never seen an actual definition if what AI *is*.  Infind it to be
>a rather useless term fornthat reason.
>
>> They are not intelligent or sentient (though the programmers
>> of some of them try to make them appear so). They do not "understand"
>> what they are fed, or what they spit out. Their use in education
>> should be limited to what they are good at, and one needs education
>> apart form LLM bots to be able to discern what they are good at and
>> what they are not good at. If students use them to write essays, they
>> will not learn that discernment.
>
>I don't knownthat there's an argument to be made that they're
>particularly good at *anything*.

They are useful as a kind of super search engine.

They can gather information from all over the web and present it in
digest form. Of couyrse this information is only as accurate as the
sources it gathers it from, and needs to be checked, but it generally
saves a lot of time. 

>
>> My friend who submitted the journal article to Claude is a nuclear
>> physicist, but the article he got it to write was in my field, not
>> his, and he rather naively trusted what Claude spat out. If it had
>> been in his field, his bullshit detectors would have been better
>> equipped to deal with it. 
>>
>> In the age of so-called AI, educators need to give serious thought to
>> better ways of honing students' bullshit detectors. 
>
>I usually tell people the following about LLMs:
>
>Ask it questions about a subject you know well.  See how long it takes
>it to tell you sonething inaccurate.  Ask yourself if you'd have caught
>the mistake if you didn't have the knowledge you do.  Finally, ask
>yourself: do you still trust the LLM?

In addition to the journal article I referred to, I've been trying it
out on fiction, since I've seen ads from people who claim that if you
use their tools you can publish dozens if not hundreds of novels under
your own name in a fraction of the time it would take to write them. 

We fed it two chapters of a novel I have written but not published,
and asked it to complete it. Chapter 3 was a quite entertaining
extrapolation from the first two, but after that it began to
deteriorate. 

I was impressed with its ability to generate authentic names for new
characters it introduced (better than some human authoers have done.
It had accurate informationa bvout some aspects of the setting, but
the plot was trying to make a mystery out of something trivial and
improbable, and people behaved and responded to things in ways
unlikely because of the setting. 

A human author understands human emotions and can make a better guess
at human reactions. LLms have no empathy, so can't do this. Someone
says something that most people would see as insulting, but the bot
writes that the hearer took comfort from it:

"I don't feel brave."
"You don't look it either."

LLMs can be no better than the masterial they work with. They are good
with factual material, but sometimes have difficulty in distinguishing
between facts and factoids. Of course humans have the same difficulty:
as you say, ask yourself if you'd avoid the same mistake if you didn't
have the knowledge that you do. 








-- 
Steve Hayes from Tshwane, South Africa
Web:  http://www.khanya.org.za/stevesig.htm
Blog: http://methodius.blogspot.com
E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk