Re: Full subsystem codebase scan for identifying pre-existing issues?

Hi Joanne,

On Thu, Jul 30, 2026 at 05:34:10PM -0700, Joanne Koong wrote:
> Hi Derek,
> 
> On Wed, Jul 29, 2026 at 6:43 PM Derek Barbosa <[email protected]> wrote:
> >
> > Hi,
> >
> > On Wed, Jul 29, 2026 at 01:34:10PM -0700, Joanne Koong wrote:
> > > Hi,
> > >
> > > Is there a way for Sashiko to run a full scan of a subsystem/tree to
> > > identify all the pre-existing issues so that they can be fixed
> > > proactively?
> >
> > At the moment, no. At least nothing that I am aware of :)
> >
> > There have been some small conversations about such functionality, but nothing
> > has really taken shape yet. One of the concerns was trying to validate the sheer
> > number of potential-false-positives, and how to present such information in an
> 
> Gotcha, thanks for the context :) That makes sense, that sounds tough
> balancing potential false positives.

We're actively working on trying to get more data (such as semcode) to act as
some sort of backing "evidence" of a finding. That, in theory, should at least
filter out the totally bogus claims. But it is still all a WIP.

> 
> Just thinking out loud here... if those false positives are currently
> getting reported/flagged by Sashiko on contributor submissions
> anyways, maybe it'd still be helpful for them to show up in the
> subsystem codebase scan so maintainers can course-correct the rules
> for them in Sashiko (and hopefully save contributors not as familiar
> with the subsystem some confusion :)).

I see what you mean. I think there may be a way to reach a "good enough"
solution in the interim: https://github.com/sashiko-dev/sashiko/issues/385

> 
> > actionable manner. IIRC syzbot/syzkaller's way of storing such data was used as
> > a comparison point here.
> >
> > FWIW, Chris Mason has a repository for a tool that runs such scans on codebases
> > -- paired with tools like semcode. His prompts are the same ones that we
> > frequently sync back to Sashiko. It won't solve the other point you made, but it
> 
> Awesome! I'm a huge fan of Chris's work. I wasn't sure if there were
> differences in the analysis between Sashiko and Chris's tools, but
> it's useful to know that the prompts are synced from Chris's repo.
> Anecdotally, when I ran a couple of my series through Chris's AI
> systems a couple months ago, there were some things it pointed out
> that Sashiko didn't and vice versa, but I think that's probably more
> due to the underlying LLM being used (eg claude vs gemini).

Right. From what I've seen, different harnesses (claude code, opencode,
gemini-cli, etc) have their own set of underlying prompts and housekeeping logic
that result in an interesting set of permutations when you throw custom prompts
and tool calls in the mix. Sashiko would be an example of a harness (one
tailored to work in a more "distributed" fashion) that calls out to Gemini in
the current "prod" instance).

FWIW I have also seen considerable differences in said prompts across different
harnesses, etc.

> 
> > may be worth looking into. [0][1]
> >
> > > Currently, they're flagged on patch reviews, which can be
> > > a bit frustrating for contributors who didn't introduce the problem,
> > > and adds a bit of noise for maintainers/reviewers looking at the
> > > Sashiko report.
> >
> > Unfortunately, you aren't the first to report a problem with these.
> >
> > Currently, in the 11th stage instruction in the code, pre-existing issues are
> > propagated in the report like so:
> >
> > <snip>
> >
> > CRITICAL RULE: If a finding is flagged as pre-existing (`\"preexisting\":
> > true`), you MUST explicitly state in your inline comment that this issue is
> > pre-existing and was not introduced by the patch under review. Use phrasing like
> > \"This isn't a bug introduced by this patch, but...\" or \"This is a
> > pre-existing issue, but...\" to start the comment.
> >
> > </snip>
> >
> > Instead of them replying inline with "This isn't a bug introduced by this patch,
> > but by..." or "This is a pre-existing issue, but does..." how would you like to
> > see this information presented/conveyed?
> 
> I (like Andrew) also find the pre-existing issue reporting really
> useful and definitely wouldn't want to lose them. Maybe something that
> could help is if they were marked even more obviously as pre-existing
> issues? (eg prefaced with a bold "PRE-EXISTING ISSUE - " or displayed
> in a different font color or enclosed within parentheses). I usually
> look at the web ui and sometimes when glancing at a series (eg [1]),
> it would be faster to differentiate between bugs I caused vs
> pre-existing ones if they were visually marked differently. This is
> not a big deal at all though. Thanks for your work on Sashiko!

I'll take note of that and try to run an experiment or two with your
suggestions. Changing prompts can feel like alchemy at times! :)

> 
> Thanks,
> Joanne
> 
> [1] https://sashiko.dev/#/patchset/20260715174305.336261-1-joannelkoong%40gmail.com
> 

Thanks for the suggestions,

-- 
Derek <[email protected]>