Re: [MAINTAINERS SUMMIT] Other LLM-related topics - tags, newcomers, etc

On Sun, 19 Jul 2026 11:01:03 +0200
Mauro Carvalho Chehab <[email protected]> wrote:

> On Thu, 16 Jul 2026 22:27:54 -0400
> "Theodore Tso" <[email protected]> wrote:
> 
> > I guess we could try running Sashiko using ollama-mlx on a Macbook
> > with 128GB, and see how it works, but my assumption is that the answer
> > is "not well" --- which is why I really want to look at fine-tuning
> > one of these smaller models first.  
> 
> I'm currently attempting to run Sashiko locally with ollama:
> 	https://github.com/sashiko-dev/sashiko/issues/200
> 
> It sounds that a patch is needed for it to work. I'm working on it
> at this RFC (I'm using LLM to generate Rust code):
> 
> 	https://github.com/sashiko-dev/sashiko/pull/338

Sashiko merged today the Ollama patches from my PR. It should now be
easier to test running it on your local machine for the ones that
prefer running it locally.

On my tests so far (on a cheap AMD rDNA4 GPU with 16GB VRAM), it 
doesn't take too much time to review one or two patchs (something like 
10-20 minutes per patch, using a model with thinking). 

Considering such preliminary results, that's probably not good enough
for real reviews, but, as technology advances, maybe we might have
something doable to use locally in the next few years.

I'm now running some tests on some machines I have access do do more
GPU testing. They have high end old GPUs. My main goal is to see how 
it performs there. 

The major problem I got is that those models sometimes don't produce
proper JSON output. Such errors reduced when I increased the context
and token size, but we're limited to the maximum context size that
such models accept. The models I tested had either 128k or 262k
max size. There are some open-weight models with bigger ones.
For instance, mistral3 has 393216 max. There is one model I have
at ollama (but I didn't test yet) that has 1MB max context size:

Model: Maoyue/mistral-nemo-instruct-2407:latest
-----------------------------------------------
  family: llama
  parameter_size: 12.2B
  quantization_level: Q4_K_M
  architecture: llama
  finetune: Instruct
  license: apache-2.0
  parameter_count: 12247782400
  context_length: 1024000
  model: gpt2
  modified_at: 2026-07-08
  parameters: 
        stop                           "[INST]"
        stop                           "[/INST]"
  capabilities: completion, tools

but it is gpt2, so probably not good enough.

Thanks,
Mauro