Re: [PATCH 2/4] scripts: add "scan" mode to process-advisories.sh to find more backports

On 21/08/26 02:40, Rolf Eike Beer wrote:
> This will look through the stable branches to find additional backports of 
> the fix for the given advisory.
> 
> Signed-off-by: Rolf Eike Beer <[email protected]>

Describe this new option and the workflow change on the advisories/README.

> ---
>  scripts/process-advisories.sh | 34 ++++++++++++++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/process-advisories.sh b/scripts/process-advisories.sh
> index a520fab5e6..00a8cd9416 100755
> --- a/scripts/process-advisories.sh
> +++ b/scripts/process-advisories.sh
> @@ -25,7 +25,7 @@ command=$1
>  
>  usage () {
>      cat >&2 <<EOF
> -usage: $0 {update|news}
> +usage: $0 {update|news|scan}
>  EOF
>      exit 1
>  }
> @@ -33,7 +33,7 @@ EOF
>  command="$1"
>  
>  case "$command" in
> -    update|news)
> +    update|news|scan)
>      ;;
>      *)
>          usage
> @@ -69,6 +69,36 @@ advisories_update() {
>      done
>  }
>  
> +advisories_scan() {
> +  advisory=$1
> +
> +  if [ -z $1 ]; then
> +    echo "Usage: $0 update GLIBC-SA-YYYY-NNNN"
> +    exit 1
> +  fi
> +
> +  advisory_file=advisories/$advisory
> +
> +  FIX=$(sed -nr '/^Fix-Commit: /s/Fix-Commit: *([0-9a-fA-F]+) *\(2\.[0-9]+\).*/\1/p' $advisory_file)

This breaks if an advisory has two master fix commits, FIX becomes a 
newline-separated list, and git treats the embedded newline as pattern OR.

For instance:

$ sed -nr '/^Fix-Commit: /s/Fix-Commit: *([0-9a-fA-F]+) *\(2\.[0-9]+\).*/\1/p' advisories/GLIBC-SA-2024-0006
b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa

And thus:

$ ./scripts/process-advisories.sh scan GLIBC-SA-2024-0006
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.31
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.32
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.33
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.34
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.35
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.36
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.37
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.38
Multiple matches for backport of b048a482f088e53144d26a61c390bed0210f49f2
7835b00dbce53c3c87bbbb1754a95fb5e58187aa found in branch 2.39

I think you need to check each fix like:

  for fix in ${FIX}; do
    BACKPORT=($(git log ..origin/release/2.${n}/master --grep "cherry picked from commit ${fix}"   --format=%H))
    [...]
  done

Also, I think we should accept both bare *and( release-annotated master commits:

  # Master fix commits is either not yet annotated with a release version or
  # annotated with a plain "(2.NN)".  Backport entries carry a "(2.NN-MMM)"
  # subscript and must not be scanned for.
  FIX=$(sed -nr 's/^Fix-Commit: *([0-9a-fA-F]+) *(\(2\.[0-9]+\))? *$/\1/p' $advisory_file)

> +
> +  if [ -z "${FIX}" ]; then
> +    echo "No Fix-Commit found in $advisory_file"
> +    exit 1
> +  fi
> +
> +  for n in $(seq 20 44); do

This would require to update this file on each release. We can assume that
release branch will always follow the release/2*./master pattern and query
the available one as:

  branches=$(git for-each-ref --format='%(refname:short)' \
            'refs/remotes/origin/release/2.*/master')

And iterate as:

  for branch in ${branches}; do

> +    BACKPORT=($(git log ..origin/release/2.${n}/master --grep "cherry picked from commit ${FIX}"   --format=%H))

The return might be empty depending on current tree status, I think
it would be better to specify the script should always scan master
(where the fix always land first):

  BACKPORT=($(git log origin/master..${branch} --grep "cherry picked from commit ${fix}"   --format=%H))

> +    if [ ${#BACKPORT[@]} -eq 0 ]; then
> +      continue
> +    fi
> +    if [ ${#BACKPORT[@]} -ne 1 ]; then
> +      echo "Multiple matches for backport of ${FIX} found in branch 2.${n}"
> +      continue
> +    fi
> +    grep -q "^Fix-Commit: ${BACKPORT[0]} " $advisory_file || echo "Fix-Commit: ${BACKPORT[0]} ($(get_rel ${BACKPORT[0]}))" >> $advisory_file

The extra space after the pattern only matches lines where something follows the
hash. And Advisory files contain Fix-Commit lines in two shapes:

* Fix-Commit: 2ae9446c... (2.38-74) — the post-update shape and it has the expected
  space.

* Fix-Commit: 2ae9446c... - there is no trailing space to match, so grep reports 
  "not found".

We need to handle both cases:

  grep -qE "^Fix-Commit: ${BACKPORT[0]}( |\$)" $advisory_file || echo "Fix-Commit: ${BACKPORT[0]} ($(get_rel ${BACKPORT[0]}))" >> $advisory_file

> +  done
> +}
> +
>  advisories_news() {
>    rel=$(get_rel "HEAD")
>    for f in $(grep -l "^Fix-Commit: .* ($rel)$" advisories/*); do