Re: Structured Changelogs for ELPA packages

Jean Louis <[email protected]> writes:

> On 2026-08-08 04:07, Andrew Hyatt wrote:
>> Richard Stallman <[email protected]> writes:
>> 
>>>> I know this will be controversial, but if we want to maintain that
>>>> changelogs are for humans and not machines, the solution is to use
>>>> an LLM to produce a standardized changelog from the freeform
>>>> changelog.
>>> It is beyond radical to proclaim something weird, and with
>>> drawbacks, as "the" solution. Why ignore the solution that we have
>>> always made to work? That is, to adopt a format and follow it.
>>  I don't think my statement was weird or radical, but perhaps there
>> was a misunderstanding. Let me attempt to clarify.
>>  Mandating a format is a perfectly good solution, but the
>> disadvantage
>> is that formats have to be standardized. Everyone has to conform.
>>  If you instead really want to accept any human-readable changelog
>> format (what has historically been allowed), with the advantage that
>> no one has to change what they are doing, yet still transform or
>> process it in a structured manner, then an LLM is the appropriate
>> solution to that problem.
>
> Hmm. I must add something to this. Why do you generalize all the
> millions of LLMs into single "LLM". What does it practically mean?
>
> Which LLM do you use?
> How do you run it?
>
> To me it is contradictory to say that "formats have to be
> standardized" yet you insist that LLM is appropriate solution to that
> problem.
>
> That is maybe your isolated feeling while using one or some LLMs.
>
> Knowing that LLMs may output such a nonsense, there is absolutely no
> guarantee that all of the LLMs that are downloaded on my hard drives,
> all of the 452 of them, would give the same results, not even correct
> output, for changelogs.
>
> So which model exactly do you mean that could "standardize" it?
>
> What percentage of users can run such model on their computer
> practically?
>
> Is such model really free?
>
> How about some tests:
>
> model      : /mnt/data/LLM/quantized/Apertus-8B-Instruct-2509-Q6_K.gguf
> CRITICAL FORMATTING RULES:
> 1. Header MUST be: YYYY-MM-DD  Author Name  <[email protected]> (with
> TWO spaces between each element)
> 2. Entries MUST start with a tab, then an asterisk: '   * '
> 3. Each entry MUST include: filename.ext (function_name): Description
> 4. Function names MUST be written in FULL (no abbreviations)
> 5. Separat ... (truncated)
>         * 2026-08-08 Your Name <[email protected]> (1 tab)
>       data_processor.py: Added caching mechanism for JSON data loading.
>       data_processor.py: Improved data validation in load_data method.
>       data_processor.py: Enhanced processing of values with optional
>       multiplier.
>       data_processor.py: Updated statistics calculation to include
>       count, sum,
>         avg, max and min.
>
> - Header is on the wrong line and has wrong format
> - The asterisk * should NOT be on the header line
> - The header should be the FIRST line, not indented
>
> See:
> https://www.gnu.org/prep/standards/html_node/Style-of-Change-Logs.html
>
> Next test:
>
> model      : /mnt/maxtor/LLM/quantized/Hermes-4-14B.Q6_K.gguf
>
> CRITICAL FORMATTING RULES:
> 1. Header MUST be: YYYY-MM-DD  Author Name  <[email protected]> (with
> TWO spaces between each element)
> 2. Entries MUST start with a tab, then an asterisk: '   * '
> 3. Each entry MUST include: filename.ext (function_name): Description
> 4. Function names MUST be written in FULL (no abbreviations)
> 5. Separat ... (truncated)
> 2026-08-08  Your Name  <[email protected]>
>
>         * data_processor.py (DataProcessor.__init__,
>           DataProcessor.load_data,
>           DataProcessor._validate_data, DataProcessor.process_values,
>           DataProcessor.get_statistics): New implementation of an
>           advanced data
>           processing utility with caching and validation. The class
>           now includes
>           methods for loading JSON data from files or cache,
>           validating required keys
>           in the data structure, processing values with a multiplier,
>           and calculating
>           statistics such as count, sum, average, max, and min.
>         * data_processor.py (main): Added a main function
>           demonstrating usage of the
>           DataProcessor class by loading sample data, processing it,
>           and printing
>           statistics.
>
> - Header format: 2026-08-08 Your Name <[email protected]> - Perfect
> - Grouping too many functions on one line. According to GNU standards,
>   you should NOT group unrelated functions like this. Each function
>   should have its own entry with specific changes.
> - The description is too generic "New implementation of an advanced
>   data processing utility with caching and validation" while every
>   function needs specific description of what changed.
> - Missing blank line between unrelated changes
>
> Next test:
>
> model      : /mnt/nvme0n1/LLM/quantized/LFM2-24B-A2B-Q4_K_M.gguf
> 2026-08-08 Your Name  [email protected]
>         * data_processing.py: Added caching mechanism for loaded data
>           to improve performance and reduce redundant file reads
>         * data_processing.py: Implemented validation of JSON structure
>           ensuring required fields are present before processing
>         * data_processing.py: Enhanced value processing with optional
>           multiplier parameter for flexible scaling operations
>
> - Specific per-function changes are not there
> - Function names in parentheses missing
> - proper grouping isn't there
>
> But I get tired proving that LLMs are not reliable.
>
> Next test:
>
> CRITICAL FORMATTING RULES:
> 1. Header MUST be: YYYY-MM-DD  Author Name  <[email protected]> (with
> TWO spaces between each element)
> 2. Entries MUST start with a tab, then an asterisk: '   * '
> 3. Each entry MUST include: filename.ext (function_name): Description
> 4. Function names MUST be written in FULL (no abbreviations)
> 5. Separat ... (truncated)
> 2023-03-15  Your Name  <[email protected]>
>
>         * DataProcessor.py: Added _validate_data method to ensure
>           required keys and data structure in loaded JSON files.
>         * DataProcessor.py: Updated load_data method to use
>           _validate_data before caching and returning data.
>         * DataProcessor.py: Modified process_values to accept an
>           optional multiplier parameter for value processing.
>         * main.py: Updated to demonstrate the usage of the new
>           multiplier parameter in process_values.\n
>
> - header is good
> - Missing function names in parentheses, should be DataProcessor.py
>   (DataProcessor._validate_data): New method...
> - wrong filename, Script uses data_processor.py (lowercase with
>   underscore), Model output uses DataProcessor.py (camelcase) and
>  main.py
> - Since entries are all in the same file, should be grouped
>
> I am not exhaustive, just showing few examples.
>
> CONCLUSION: LLMs cannot be trusted to generate correctly formatted GNU
> ChangeLogs; they are probabilistic text generators, not
> format-enforcement engines.

It is not clear to me what you are testing, but IIUC you are generating
changelogs?  That is not what we are talking about, we are talking about
taking some arbitrary changelog and extracting a section out of it (or
just the line number range, if that would make it easier).  Can you
repeat your experiment, along with some information about how long this
takes and some system resource usage info, on some sample NEWS files
from ELPA?  If you have issues deciding, take NEWS.org from Compat:

https://gitweb.git.savannah.gnu.org/gitweb/?p=elpa/gnu.git;a=blob_plain;f=NEWS.org;hb=refs/heads/externals/compat