Re: Hallucinated citations are polluting the scientific literature. What can be done?

On Tue, 12 May 2026 02:45:03 +0100, Lawfare Review
<[email protected]> wrote:

>Earlier this year, computer scientist Guillaume Cabanac received a 
>notification from Google Scholar that one of his publications had 
>been cited in a paper published in the International Dental Journal1. 
>That was unexpected, because his research on spotting fabricated 
>papers doesn’t typically intersect with dentistry. “I was very 
>surprised to see that I couldn’t recognize my own reference,” 
>says Cabanac, who is based at the University of Toulouse in France.
>
>https://www.nature.com/articles/d41586-026-00969-z

A friend of mine got an LLM bot (Claude) to write an academic paper,
which he then sent to me. I read it as I would if I have been asked to
do a peer review for a journal, and he gave that feedback to the bot,
and sent the revised paper to a journal, which, unsurprisingly (to me
at any rate) rejected it. 

He also got Claude to produce a bibliography of works relevant to the
topic, which did turn out to be quite useful, though it did need
careful checking for the avoidance of hallucinations, as described
above. It gave several bogus urls. It appeared that such bots could be
a useful supplement to (but not a replacement for) the work of
reference librarians. 

I then asked my friend to test Claude's generative ability with
fiction -- got him to submit to it the first two chapters of an
unpublished novel I had written, and complete it, so I could then
compare the result with what I had actually written. The first couple
of chapters it produced were quite entertaining, but after that it
began to go off the rails. It was a children's book, and it began to
have child characters talking and behaving like adults. The plot was
thin, and turned on a complex legal point that was difficult for
adults to follow, and would probably have bored any child reader out
of their skull. It had the setting switching back and forth between
spring and autummn, with lyrical descriptions of spring blossoms in
one chapter, and falling leaves the next.

My friend fed my comments to Claude and has sent me back a revised
text. I haven't read it yet. 

LLM bots can be useful tools but they are not AI, and are not
reliable. They are not intelligent or sentient (though the programmers
of some of them try to make them appear so). They do not "understand"
what they are fed, or what they spit out. Their use in education
should be limited to what they are good at, and one needs education
apart form LLM bots to be able to discern what they are good at and
what they are not good at. If students use them to write essays, they
will not learn that discernment.

My friend who submitted the journal article to Claude is a nuclear
physicist, but the article he got it to write was in my field, not
his, and he rather naively trusted what Claude spat out. If it had
been in his field, his bullshit detectors would have been better
equipped to deal with it. 

In the age of so-called AI, educators need to give serious thought to
better ways of honing students' bullshit detectors. 



-- 
Stephen Hayes, Author of The Year of the Dragon
Sample or purchase The Year of the Dragon: 
https://www.smashwords.com/books/view/907935
Web site: http://www.khanya.org.za/stevesig.htm
Blog: http://methodius.blogspot.com
E-mail: [email protected] or if you use Gmail [email protected]