Re: Amazing work on bug fixing

Thank you, Viktor.
Yes, it cannot be stressed enough that it would be extremely dangerous to
blindly trust the LLM's explanations and proposed fixes.
During my bug-fixing work over the past weeks, there have been almost no
cases where the LLM immediately came up with the correct answer. Without
knowledge of Maxima's internals, it would be very difficult to decide
what's a good idea and what isn't.

Viktor T. Toth <[email protected]> schrieb am Sa., 20. Juni 2026, 05:11:

> Allow me to also express appreciation for your debugging efforts, David,
> but also for the manner in which you use an LLM: Not as an "agent" but as a
> capable assistant, retaining control over the process. I have been using
> LLMs in a similar manner in the past three and a half years (through the
> APIs, mostly, less often through the Web) and it's been an incredible
> productivity booster. Agentic use, "vibe programming," can produce
> unpleasant surprises.
>
>
> Viktor
>
>
>
>
> On 2026-06-19 12:49, David Scherfgen via Maxima-discuss wrote:
>
> If the affected module's source code file is less than ~100 KB, I don't
> put too much work into filtering and paste it completely.
>
> Gemini (and other comparable LLMs) already have a decent "baked-in"
> understanding of the basic functions like SIMPLIFYA, supposedly because
> lots of publicly available discussions (mailing list, big reports, the code
> itself) were contained in the training data. You can try asking your model
> about how exactly SIMPLIFYA works and what its arguments X and Y mean, and
> instruct it not to use any external sources. The answer should be quite
> good.
>
> I will do some more tests regarding pasting code into the prompt and
> uploading it as an attachment.
>
> When I asked Gemini itself about this question (paste vs. upload), it told
> me that a hybrid approach may be best. Upload the (zipped) complete
> codebase, but also include highly relevant functions directly in the prompt
> to steer the model's attention towards them.
>
> All this is really exciting!
>
>
> Stavros Macrakis <[email protected]> schrieb am Fr., 19. Juni 2026,
> 18:15:
>
>> Thanks again!
>>
>> I do have a paid subscription, and am able to set 3.1 Pro / Extended in
>> the UI.
>>
>> It seems that you manually select the relevant sections of Maxima's
>> source code. How big a context do you include -- dozens of functions,
>> hundreds of lines? Presumably the source "background" functions like
>> add/simplus/simplifya/... isn't needed in many cases, because it guesses at
>> the functionality based on the function name?
>>
>> I have tried command-line Claude on whole directories (smaller than
>> Maxima), and it seems to do a pretty good job of "understanding" what the
>> code is trying to do and even spontaneously noticing bugs (without bug
>> reports). And then I can ask about particular bits of functionality without
>> specifying which source file implements it. Is your experience that giving
>> it too much context consumes too many tokens, or just doesn't work well?
>>
>>               -s
>>
>> On Fri, Jun 19, 2026 at 11:53 AM David Scherfgen <
>> [email protected]> wrote:
>>
>>> I use the web version. The model can be chosen in the drop-down menu. I
>>> have a paid subscription. It's possible that without, you can't effectively
>>> use Extended Thinking because it will quickly reach your allowed limit.
>>>
>>> You can upload files and attach them to the conversation, but I found
>>> that it gives better results when pasting the source code directly into the
>>> prompt. I'll have to test that again, though.
>>>
>>> Stavros Macrakis <[email protected]> schrieb am Fr., 19. Juni 2026,
>>> 17:22:
>>>
>>>> Thank you very much! That is a lot of great detail.
>>>>
>>>> Do you use Gemini on the command line, the Web interface (More uploads
>>>> / Import code directly from Sourceforge, or local git repository?), or via
>>>> some IDE (Google AI Studio?)? Do you have a GEMINI.md file? Is that where
>>>> you configure Extended Thinking?
>>>>
>>>> Thanks again!!
>>>>
>>>>               -s
>>>>
>>>> On Fri, Jun 19, 2026 at 11:15 AM David Scherfgen <
>>>> [email protected]> wrote:
>>>>
>>>>> Thanks, Stavros!
>>>>>
>>>>> When the "Extended Thinking" mode of Gemini 3.1 Pro was launched, I
>>>>> decided to give it a try, after previous attempts with Gemini 3.1 Pro in
>>>>> normal mode as well as ChatGPT and Claude weren't very successful.
>>>>> Extended Thinking is really a breakthrough.
>>>>>
>>>>> I usually write the prompt as follows:
>>>>>
>>>>> We're debugging the Maxima Computer Algebra System written in Lisp.
>>>>>> There is a bug in the *[module name]* module that can be reproduced
>>>>>> as follows:
>>>>>> ----------
>>>>>> *[Code example]*
>>>>>> ----------
>>>>>> As you can see, *[actual behavior]*, while we would expect *[correct
>>>>>> behavior]*.
>>>>>>
>>>>> *[Any notable observations, e.g. similar cases that work correctly]*
>>>>>
>>>>> ----------
>>>>>> Here's a trace:
>>>>>> *[Trace of important functions or stack trace in case of a Lisp
>>>>>> error]*
>>>>>> ----------
>>>>>> Your task is to thoroughly investigate this problem, explain in
>>>>>> detail every step that causes it, and find a fix for it. Ideally, the fix
>>>>>> should be minimally invasive.
>>>>>> ----------
>>>>>> Relevant sections of Maxima's source code:
>>>>>> ----------
>>>>>> *[Source code]*
>>>>>>
>>>>>
>>>>> When pasting source code:
>>>>>
>>>>>    - Strip away everything that's clearly not involved in the problem.
>>>>>    - Ideally, include every function that gets called during the
>>>>>    reproduction of the bug.
>>>>>    - Be patient, as pasting into Gemini's prompt textbox can
>>>>>    sometimes take a lot of time (the browser may even ask you if you want to
>>>>>    wait or close the page - wait).
>>>>>
>>>>> Now Gemini will think for a while.
>>>>> When it comes back with an answer:
>>>>>
>>>>>    - Make sure that the selected model is still "Gemini 3.1 Pro
>>>>>    (Extended)". It eats your quota really quickly (there's a limit that resets
>>>>>    every 5 hours + a weekly limit), and then it will fall back to a simpler
>>>>>    model, which almost never comes up with anything useful. You can also
>>>>>    notice by how quickly it responds.
>>>>>    - Be *extremely critical*. LLMs are incredibly good at selling,
>>>>>    they will make it sound like it's 100% sure that they're right.
>>>>>    - If you find the answer convincing, test the proposed fix against
>>>>>    the problem and the test suite.
>>>>>
>>>>> If it's an obvious case where you can clearly see that the fix is
>>>>> correct, you're done.
>>>>> If it's a more difficult case, let Gemini regenerate the answer a few
>>>>> times to help you decide. See if it always finds the same fix or different
>>>>> ones.
>>>>>
>>>>> In case you get mostly incorrect fixes:
>>>>>
>>>>>    - *Don't continue the conversation for too long*, as it will
>>>>>    eventually forget the first part of it, which contains the highly important
>>>>>    source code. LLMs have a limited context window. If the conversation gets
>>>>>    too long, it starts forgetting things said earlier.
>>>>>    - Instead, edit your initial prompt and include a section like:
>>>>>    "You previously proposed the following fixes, which didn't work: [...]".
>>>>>    Include a short description for every proposed fix and why it wasn't
>>>>>    working (e.g. it broke certain tests). If editing the initial prompt is not
>>>>>    possible because you have sent follow-up prompts already, you can "fork"
>>>>>    the conversation after the first reply, and then you will be able to edit
>>>>>    the initial prompt in the forked conversation.
>>>>>
>>>>> I hope this helps!
>>>>>
>>>>> Best regards
>>>>> David Scherfgen
>>>>>
>>>>> Am Fr., 19. Juni 2026 um 16:19 Uhr schrieb Stavros Macrakis <
>>>>> [email protected]>:
>>>>>
>>>>>> David,
>>>>>>
>>>>>> Thank you very much for contributing so many bug fixes to Maxima!
>>>>>>
>>>>>> I think you've mentioned that you use an LLM to help you out. Could
>>>>>> you tell us more about your setup and what you've learned about working
>>>>>> effectively with it?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>           -s
>>>>>>
>>>>>>
>
> _______________________________________________
> Maxima-discuss mailing [email protected]://lists.sourceforge.net/lists/listinfo/maxima-discuss
>
>

_______________________________________________
Maxima-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/maxima-discuss