Re: Stop false review statements

On Sun, 17 May 2026 12:42:12 -0700
Roman Gushchin <[email protected]> wrote:

> =EF=BB=BF
> > On May 17, 2026, at 11:57=E2=80=AFAM, Theodore Tso <[email protected]> wrot=
e:
> > =EF=BB=BFOn Sun, May 17, 2026 at 11:17:06AM -0700, Roman Gushchin wrote=
: =20
> >>=20
> >> I actually tried to run it with ollama on my
> >> personal framework 13. Adding nominal support is trivial, but the
> >> whole thing is not really useful: I can get maybe few hundreds
> >> tokens per second using a quantified model with reduced quality; an
> >> average sashiko review is consuming 3.5 millions tokens (with Gemini
> >> 3.1 pro, it=E2=80=99s also model-dependent). =20
> >=20
> > I'm curious.  What hardware and LLM model were you using?  A few
> > hundred tokens per second seems surprising high.  My initial
> > research[1] showes that an M5 Max Macbook Pro costing 5 or 6 kilobucks
> > can do 31.6 tokens/second on a 27B 4-bit Quanitized model (Qwen 3.5). =
=20
>=20
> I=E2=80=99ve framework 13 with amd 7840u. I=E2=80=99ve tried several mode=
ls both on cpu and gpu.=20
> Sorry, it was a couple of months ago and I don=E2=80=99t remember all the=
 details, so I won=E2=80=99t=20
> claim any specific numbers, but as I remember the best numbers were aroun=
d=20
> a hundred tokens per second. In any case it=E2=80=99s few orders of magni=
tude slower than
>  what is realistically required.
>=20
> If someone has a powerful hardware and is willing to benchmark sashiko wi=
th open-source
> models, I=E2=80=99m very interested in results.

If you add the patch you used with ollama somewhere, I can try
running here and do some benchmarks - that is assuming that=20
it won't try to run 3.5 millions of tokens.


>=20
> > [1] https://www.reddit.com/r/LocalLLaMA/comments/1rzkw4x/m5_max_128g_pe=
rformance_tests_i_just_got_my_new/
> >=20
> > The model matters of course.  With Gemma 3 27B and a 6-bit
> > quantization, it's 21 tokens/s, and with Deepseek R1 8B Q6_K, it's
> > 72.8 tokens/second.  But unless you're using a really low-end model,
> > or a really expensive, splufty hardware platform, I haven't seen
> > reports of hundreds of tokens per second on hardware costing a
> > reasonable amount of memory.  (I'll set aside the question of whether
> > spending $6k for a fully spec'ed out M5 Max Macbook Pro, or $15k for a
> > fully spec'ed out M3 Ultra Mac Studio is "reasonable".)
> >=20
> > As a result I'm not entirely sure how realistic it is to do reviews
> > using "free" (you still have to pay $$$ for the hardware) local,
> > open-weight LLM's if an average review requires around 3.5 million
> > tokens. =20
>=20
> Fully agree. But it might change in few years, things are moving quickly.


Thanks,
Mauro