[PATCH RFC 8/8] github: refactor into reusable actions and workflows
Daniel Gomez <[email protected]>
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <[email protected]> |
From: Daniel Gomez <[email protected]> Refactor the .github/ CI support into reusable actions and workflows. This allows to use the new actions not only internally in kdevops but also in other projects such linux-modules-kpd and similar. Keeping the ci logic in one place and allowing other projects to get the latest changes. CI supports the following: - Split workflow in setup, test, archive and cleanup actions - Fixed kernel: v6.15 on linux tree. This is to speed up builds relying on the same kernel. We can update when needed or timely basis - Fixed workflows: blktests_nvme and xfs_reflink_4k. We want to validate kdevops workflows and blktests with nvme section/profile has been added on top of the current fstests/xfs reflink 4k - 2 CI modes: A. kdevops selftest/ci validation. A single test execution for the given workflow (e.g. generic/003 for xfs). This has it's own commit format. B. kdevops full workflow. This is to validate a specific kernel on a given test suite. It's supported on manual trigger as well as on schedule. Add automatic runner assignment based on ci_workflow names to enable parallel execution on dedicated runners: - Parallelize GitHub workflow execution on dedicated runners based on CI_WORKFLOW names (using tags). E.g.: * blktests workflows → [self-hosted, Linux, X64, blktests] * xfs workflows → [self-hosted, Linux, X64, xfs] - Set timeout 120 minutes for CI B mode. - Disable fail-fast to matrix strategy in both push.yml and schedule.yml workflows to prevent cancellation of successful jobs when other matrix jobs fail. Example: this allows blktests_nvme to continue running even if xfs_reflink_4k fails Signed-off-by: Daniel Gomez <[email protected]> --- .github/actions/archive/action.yml | 28 ++++ .github/actions/cleanup/action.yml | 24 ++++ .github/actions/setup/action.yml | 187 ++++++++++++++++++++++++ .github/actions/test/action.yml | 213 +++++++++++++++++++++++++++ .github/workflows/destroy.yml | 93 ++++++++++++ .github/workflows/docker-tests.yml | 2 +- .github/workflows/fstests.yml | 98 ------------- .github/workflows/main.yml | 97 +++++++++++++ .github/workflows/manual.yml | 122 ++++++++++++++++ .github/workflows/push.yml | 29 ++++ .github/workflows/schedule.yml | 54 +++++++ scripts/generate_ci_commit_message.sh | 263 ++++++++++++++++++++++++++++++++++ scripts/github_output.sh | 9 ++ scripts/korg-releases.py | 94 ++++++++++++ 14 files changed, 1214 insertions(+), 99 deletions(-) diff --git a/.github/actions/archive/action.yml b/.github/actions/archive/action.yml new file mode 100644 index 00000000..6efe20fc --- /dev/null +++ b/.github/actions/archive/action.yml @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Archive results +description: Archive kdevops results in https://github.com/linux-kdevops/kdevops-results-archive.git +inputs: + ci_workflow: + required: false + type: string + default: 'demo' + dir: + description: 'Directory' + required: true + default: 'workdir' + +runs: + using: "composite" + steps: + - name: Get systemd journal files + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + make journal-dump + + - name: Build our kdevops archive results + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + make ci-archive CI_WORKFLOW="${{ inputs.ci_workflow }}" diff --git a/.github/actions/cleanup/action.yml b/.github/actions/cleanup/action.yml new file mode 100644 index 00000000..18a8656a --- /dev/null +++ b/.github/actions/cleanup/action.yml @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Cleanup kdevops VMs +description: Destroy VMs and cleanup workspace + +inputs: + dir: + description: 'Directory' + required: true + default: 'workdir' + +runs: + using: "composite" + steps: + - name: Run kdevops make destroy + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + make destroy + + - name: Remove working-directory + shell: bash + run: | + rm --recursive --force --verbose ${{ inputs.dir }} diff --git a/.github/actions/setup/action.yml b/.github/actions/setup/action.yml new file mode 100644 index 00000000..407c8ac3 --- /dev/null +++ b/.github/actions/setup/action.yml @@ -0,0 +1,187 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Setup kdevops +description: Setup kdevops workspace + +inputs: + ci_workflow: + required: false + type: string + default: 'demo' + dir: + description: 'Directory' + required: true + default: 'workdir' + kernel_tree: + required: false + type: string + default: 'linux' + kernel_ref: + required: false + type: string + default: 'master' + +runs: + using: "composite" + steps: + - name: Create workspace directory + shell: bash + run: | + set -euxo pipefail + rm --recursive --force --verbose ${{ inputs.dir }} + mkdir --parent --verbose ${{ inputs.dir }} + + - name: Configure git + shell: bash + run: | + set -euxo pipefail + git config --global --add safe.directory '*' + git config --global user.name "kdevops" + git config --global user.email "[email protected]" + + - name: Checkout kdevops + working-directory: ${{ inputs.dir }} + shell: bash + run: | + set -euxo pipefail + git clone https://github.com/dkruces/kdevops.git --branch main kdevops + + - name: Checkout custom branch with delta on kdevops/linux + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + LINUX_TREE="/mirror/${{ inputs.kernel_tree }}.git" + LINUX_TREE_REF="${{ inputs.kernel_ref }}" + git clone $LINUX_TREE linux + cd linux + git checkout $LINUX_TREE_REF + git log -1 + + - name: Make sure our repo kdevops defconfig exists + id: defconfig + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + KDEVOPS_DEFCONFIG=${{ inputs.ci_workflow }} + + if [[ ! -f defconfigs/$KDEVOPS_DEFCONFIG ]]; then + echo "Missing defconfig: defconfigs/$KDEVOPS_DEFCONFIG" + exit 1 + fi + + "${{ github.workspace }}/scripts/github_output.sh" \ + KDEVOPS_DEFCONFIG "$KDEVOPS_DEFCONFIG" + + - name: Initialize CI metadata for kdevops-results-archive for linux + id: metadata + working-directory: ${{ inputs.dir }}/kdevops/linux + shell: bash + run: | + set -euxo pipefail + echo "${{ inputs.kernel_tree }}" > ../ci.trigger + # Get the kdevops commit subject, not kernel commit + cd "${{ github.workspace }}" && git log -1 --pretty=format:"%s" > "${{ inputs.dir }}/kdevops/ci.subject" + cd - > /dev/null + echo "${{ inputs.kernel_ref }}" > ../ci.ref + + RELEVANT_GIT_TAG=$(cat ../ci.ref) + RELEVANT_GIT_REF=$(git rev-parse --short=12 $RELEVANT_GIT_TAG) + + "${{ github.workspace }}/scripts/github_output.sh" \ + LINUX_GIT_REF "$RELEVANT_GIT_REF" + "${{ github.workspace }}/scripts/github_output.sh" \ + LINUX_GIT_TAG "$RELEVANT_GIT_TAG" + + # Start out pessimistic + echo "unknown" > ../ci.result + echo "Nothing to write home about." > ../ci.commit_extra + + - name: Run kdevops make defconfig-repo + working-directory: ${{ inputs.dir }}/kdevops + env: + LINUX_GIT_TAG: ${{ steps.metadata.outputs.LINUX_GIT_TAG }} + LINUX_GIT_REF: ${{ steps.metadata.outputs.LINUX_GIT_REF }} + KDEVOPS_DEFCONFIG: ${{ steps.defconfig.outputs.KDEVOPS_DEFCONFIG }} + shell: bash + run: | + set -euxo pipefail + LINUX_TREE="/mirror/${{ inputs.kernel_tree }}.git" + LINUX_TREE_REF="$LINUX_GIT_TAG" + + # We make the compromise here to use a relevant git tag for the + # host prefix so that folks can easily tell what exact kernel tree + # is being tested by using the relevant git ref. That is, if you + # pushed a tree with the .github/ directory as the top of the tree, + # that commit will not be used, we'll use the last one as that is + # the relevant git ref we want to annotate a test for. + # + # The compromise here is that we expect no two same identical tests + # on the same self-hosted server. We could extend this with something + # like github.run_id but its not yet clear if we can have kdevops + # hosts with a bundled prefix ID like that ref-runid-testname. Tests + # have been done with the full lenght sha1sum though and we know that + # does work. + KDEVOPS_HOSTS_PREFIX="kci-$LINUX_GIT_REF" + + echo "Going to use defconfig-$KDEVOPS_DEFCONFIG" + + echo "Linux tree: $LINUX_TREE" + echo "Linux trigger ref: $LINUX_TREE_REF" + echo "Linux tag: $LINUX_GIT_TAG" + echo "Runner ID: ${{ github.run_id }}" + echo "kdevops host prefix: $KDEVOPS_HOSTS_PREFIX" + echo "kdevops defconfig: defconfig-$KDEVOPS_DEFCONFIG" + + # Customize KMOD_TIMEOUT when required + KMOD_TIMEOUT_ARG= + if [[ "$(hostname)" == *smc111* && \ + "$KDEVOPS_DEFCONFIG" == "linux-modules-kpd" ]]; then + KMOD_TIMEOUT_ARG="KMOD_TIMEOUT=222" + fi + + KDEVOPS_ARGS="\ + KDEVOPS_HOSTS_PREFIX=$KDEVOPS_HOSTS_PREFIX \ + LINUX_TREE=$LINUX_TREE \ + LINUX_TREE_REF=$LINUX_TREE_REF \ + ${KMOD_TIMEOUT_ARG} \ + defconfig-$KDEVOPS_DEFCONFIG" + echo "Going to run:" + echo "make $KDEVOPS_ARGS" + + make $KDEVOPS_ARGS + ./scripts/kconfig/merge_config.sh \ + -n .config \ + defconfigs/configs/diy.config \ + defconfigs/configs/ci.config + + - name: Run kdevops make + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + make -j$(nproc) + + - name: Run kdevops make bringup + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + ls -ld linux + make destroy + make bringup + + - name: Build linux and boot test nodes on test kernel + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + make linux + + - name: Build required ci tests + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + make ci-build-test CI_WORKFLOW=${{ inputs.ci_workflow }} diff --git a/.github/actions/test/action.yml b/.github/actions/test/action.yml new file mode 100644 index 00000000..a9022b4c --- /dev/null +++ b/.github/actions/test/action.yml @@ -0,0 +1,213 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Setup kdevops +description: Setup kdevops workspace + +inputs: + dir: + description: 'Directory' + required: true + default: 'workdir' + ci_workflow: + required: false + type: string + default: 'demo' + test_mode: + description: 'Testing mode' + required: false + default: 'full-testing' + tests: + description: 'Custom test to run (for kdevops-validation mode only)' + required: false + default: '' + +runs: + using: "composite" + steps: + - name: Run CI tests + id: ci_test + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + + # Create start time for duration calculation + echo "$(date +%s)" > ci.start_time + + # Handle test mode and tests parameters + if [[ -n "${{ inputs.tests }}" ]]; then + # Custom test specified for kdevops-validation mode + TESTS="${{ inputs.tests }}" + elif [[ "${{ inputs.test_mode }}" == "kdevops-validation" ]]; then + # Auto-assign appropriate test based on workflow + case "${{ inputs.ci_workflow }}" in + *fstests*|*xfs*|*btrfs*|*ext4*|*tmpfs*) + TESTS="generic/003" + ;; + blktests*) + TESTS="block/003" + ;; + selftests*|*modules*|*kmod*|*firmware*|*mm*) + TESTS="kmod/test_001" + ;; + *) + TESTS="generic/003" + ;; + esac + else + # Full testing mode - no TESTS variable + TESTS="" + fi + + # Export TESTS for current step and save to GitHub output for later steps + export TESTS="$TESTS" + "${{ github.workspace }}/scripts/github_output.sh" TESTS "$TESTS" + + make ci-test CI_WORKFLOW="${{ inputs.ci_workflow }}" + + - name: Generate workflow results path + id: setpath + shell: bash + run: | + set -euxo pipefail + + case "${{ inputs.ci_workflow }}" in + blktests*) wpath="workflows/blktests" ;; + *btrfs*) wpath="workflows/fstests" ;; + *ext4*) wpath="workflows/fstests" ;; + tmpfs*) wpath="workflows/fstests" ;; + *xfs*) wpath="workflows/fstests" ;; + *) wpath="workflows/selftests" ;; + esac + + "${{ github.workspace }}/scripts/github_output.sh" wpath "$wpath" + + - name: Generate CI commit info with workflow-specific logic + working-directory: ${{ inputs.dir }}/kdevops + shell: bash + run: | + set -euxo pipefail + + wpath="${{ steps.setpath.outputs.wpath }}" + + # Workflow-specific result collection + case "${{ inputs.ci_workflow }}" in + *xfs*|*btrfs*|*ext4*|tmpfs*|*fstests*) + # fstests workflows: Use xunit_results.txt for rich summary + echo "Collecting fstests results..." + if find "$wpath/results/last-run" -name "xunit_results.txt" -type f -exec cat {} \; > ci.commit_extra 2>/dev/null; then + echo "Found xunit_results.txt" + else + echo "No xunit_results.txt found, using fallback..." + echo "Kernel tests results:" > ci.commit_extra + find "$wpath/results/last-run/" -name '*.dmesg.log' -exec tail -n 1 {} + >> ci.commit_extra 2>/dev/null || true + fi + + # fstests success detection + if ! grep -E "failures, [1-9]|errors, [1-9]" ci.commit_extra >/dev/null 2>&1; then + echo "ok" > ci.result + else + echo "not ok" > ci.result + fi + ;; + blktests*) + # blktests workflows: Collect individual test results + + echo "Kernel tests results:" > ci.commit_extra + + # Collect test results from last-run directory + if [ -d "$wpath/results/last-run" ]; then + echo -e "\nBlktests summary:" >> ci.commit_extra + + # Count total tests by unique test names (not file extensions) + # Get base test names by removing extensions and extracting just the test name + test_names=$(find "$wpath/results/last-run" -type f -name "*" | \ + grep -E '/[^/]*$' | \ + sed 's|.*/||' | \ + sed 's/\.\(full\|out\|runtime\|dmesg\)$//' | \ + sort -u) + total_tests=$(echo "$test_names" | grep -v '^$' | wc -l) + + bad_files=$(find "$wpath/results/last-run" -name "*.out.bad" -type f) + if [[ -n "$bad_files" ]]; then + failed_tests=$(echo "$bad_files" | wc -l) + else + failed_tests=0 + fi + + passed_tests=$((total_tests - failed_tests)) + + echo "Tests run: $total_tests, Passed: $passed_tests, Failed: $failed_tests" >> ci.commit_extra + + # List failed tests if any + if [ $failed_tests -gt 0 ]; then + echo -e "\nFailed tests:" >> ci.commit_extra + find "$wpath/results/last-run" -name "*.out.bad" -type f | sed 's|.*/\([^/]*\)\.out\.bad$|\1|' | sort >> ci.commit_extra 2>/dev/null || true + fi + + # Show sample test status files for passed tests + if [ $passed_tests -gt 0 ]; then + echo -e "\nSample passed test:" >> ci.commit_extra + sample_files=$(find "$wpath/results/last-run" -type f -name "*" ! -name "*.out.bad" ! -name "*.dmesg") + sample_file=$(echo "$sample_files" | head -1) + if [ -n "$sample_file" ] && [ -f "$sample_file" ]; then + cat "$sample_file" >> ci.commit_extra || echo "Failed to read sample file" >> ci.commit_extra + else + echo "No valid sample file found" >> ci.commit_extra + fi + fi + else + echo -e "\nNo blktests results found in $wpath/results/last-run" >> ci.commit_extra + fi + + # blktests success detection - look for .out.bad files (failures) + bad_check=$(find "$wpath/results/last-run" -name "*.out.bad" -type f | head -1) + + if [ -n "$bad_check" ]; then + echo "not ok" > ci.result + else + echo "ok" > ci.result + fi + ;; + *selftests*|*modules*|*mm*|*firmware*) + # selftests workflows: Use userspace.log + echo "Kernel tests results:" > ci.commit_extra + find "$wpath/results/last-run/" -name '*.userspace.log' -exec cat {} \; >> ci.commit_extra 2>/dev/null || true + + # selftests success detection + if grep -q "passed\|PASS" ci.commit_extra && ! grep -q "FAIL\|failed\|ERROR" ci.commit_extra; then + echo "ok" > ci.result + else + echo "not ok" > ci.result + fi + ;; + *) + # Default/unknown workflows: Generic approach + echo "Kernel tests results:" > ci.commit_extra + find "$wpath/results/last-run/" -name '*.dmesg.log' -exec tail -n 1 {} + >> ci.commit_extra 2>/dev/null || true + echo -e "\n\nUserspace test results:" >> ci.commit_extra + find "$wpath/results/last-run/" -name '*.userspace.log' -exec tail -n 1 {} + >> ci.commit_extra 2>/dev/null || true + + if grep -i -q "fail" ci.commit_extra; then + echo "not ok" > ci.result + else + echo "ok" > ci.result + fi + ;; + esac + + # Set environment variables for the commit message script + export CI_WORKFLOW="${{ inputs.ci_workflow }}" + export KERNEL_TREE="${{ inputs.kernel_tree || 'linux' }}" + + # Get TESTS from GitHub output (determines kdevops validation vs full testing) + TESTS="${{ steps.ci_test.outputs.TESTS }}" + if [[ -n "${TESTS:-}" ]]; then + export TESTS="$TESTS" + fi + + # Generate enhanced commit message using our script + ./scripts/generate_ci_commit_message.sh > ci.commit_message_enhanced + + # Keep the original ci.commit_extra for backward compatibility + # The archive action will use ci.commit_message_enhanced if available diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml new file mode 100644 index 00000000..6689452b --- /dev/null +++ b/.github/workflows/destroy.yml @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Run kdevops Destroyer - Manual + +on: + workflow_dispatch: + inputs: + ci_workflow: + description: "CI Workflow" + required: true + default: 'blktests_nvme' + type: choice + options: + - blktests + - blktests_block + - blktests_loop + - blktests_meta + - blktests_nbd + - blktests_nvme + - blktests_nvmemp + - blktests_scsi + - blktests_srp + - blktests_zbd + - lbs-xfs + - lbs-xfs-bdev-large-nvme + - lbs-xfs-bdev-nvme + - lbs-xfs-small + - xfs + - xfs_crc + - xfs_crc_logdev + - xfs_crc_rtdev + - xfs_crc_rtdev_extsize_28k + - xfs_crc_rtdev_extsize_64k + - xfs_nocrc + - xfs_nocrc_16k_4ks + - xfs_nocrc_1k + - xfs_nocrc_2k + - xfs_nocrc_32_4ks + - xfs_nocrc_4k + - xfs_nocrc_512 + - xfs_nocrc_64_4ks + - xfs_nocrc_8k_4ks + - xfs_nocrc_lbs + - xfs_reflink + - xfs_reflink_1024 + - xfs_reflink_16k_4ks + - xfs_reflink_2k + - xfs_reflink_32k_4ks + - xfs_reflink_4k + - xfs_reflink_4k_8ks + - xfs_reflink_64k_4ks + - xfs_reflink_dir_bsize_8k + - xfs_reflink_lbs + - xfs_reflink_logdev + - xfs_reflink_normapbt + - xfs_reflink_nrext64 + - xfs_reflink_stripe_len + - xfs-soak + host_prefix: + description: "Host prefix" + required: true + default: 'debian13' + type: string + +jobs: + destroyer: + name: Destroy guests + runs-on: [self-hosted] + steps: + - name: Create workdir + run: | + rm --recursive --force --verbose ${{ inputs.ci_workflow }} + mkdir --parent --verbose ${{ inputs.ci_workflow }} + + - name: Checkout kdevops + run: | + cd ${{ inputs.ci_workflow }} + git clone https://github.com/dkruces/kdevops.git --branch ci-workflow kdevops + + - name: Run kdevops make defconfig-repo + run: | + set -euxo pipefail + + sudo virsh list --all + + cd ${{ inputs.ci_workflow }}/kdevops + make \ + KDEVOPS_HOSTS_PREFIX="${{ inputs.host_prefix }}" \ + defconfig-${{ inputs.ci_workflow }} + + make destroy + + sudo virsh list --all diff --git a/.github/workflows/docker-tests.yml b/.github/workflows/docker-tests.yml index c0e0d03d..33d67f09 100644 --- a/.github/workflows/docker-tests.yml +++ b/.github/workflows/docker-tests.yml @@ -1,4 +1,4 @@ -name: Setup kdevops +name: kdevops Docker Tests on: push: diff --git a/.github/workflows/fstests.yml b/.github/workflows/fstests.yml deleted file mode 100644 index e13978c7..00000000 --- a/.github/workflows/fstests.yml +++ /dev/null @@ -1,98 +0,0 @@ -name: Run kdevops on self-hosted runner - -on: - push: - branches: - - '**' - pull_request: - branches: - - '**' - workflow_dispatch: # Add this for manual triggering of the workflow - -jobs: - run-kdevops: - name: Run kdevops CI - runs-on: [self-hosted, Linux, X64] - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set CI metadata for kdevops-results-archive - run: | - echo "$(basename ${{ github.repository }})" > ci.trigger - git log -1 --pretty=format:"%s" > ci.subject - # Start out pessimistic - echo "not ok" > ci.result - echo "Nothing to write home about." > ci.commit_extra - - - name: Set kdevops path - run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV - - - name: Configure git - run: | - git config --global --add safe.directory '*' - git config --global user.name "kdevops" - git config --global user.email "[email protected]" - - - name: Run kdevops make defconfig-repo - run: | - KDEVOPS_TREE_REF="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}" - SHORT_PREFIX="$(echo ${KDEVOPS_TREE_REF:0:12})" - make KDEVOPS_HOSTS_PREFIX="$SHORT_PREFIX" \ - ANSIBLE_CFG_CALLBACK_PLUGIN="debug" \ - LINUX_TREE_REF="v6.15" \ - defconfig-xfs_reflink_4k - - - name: Run kdevops make - run: | - make V=1 -j$(nproc) - - - name: Run kdevops make bringup - run: | - make V=1 bringup - - - name: Build linux and boot test nodes on test kernel - run: | - make V=1 linux - - - name: Build fstests - run: | - make V=1 fstests - - - name: Run just one fstest to verify we tests and test collection works - run: | - make V=1 fstests-baseline TESTS=generic/003 - echo "ok" > ci.result - find workflows/fstests/results/last-run -name xunit_results.txt -type f -exec cat {} \; > ci.commit_extra || true - if ! grep -E "failures, [1-9]|errors, [1-9]" ci.commit_extra; then - echo "ok" > ci.result - fi - - - name: Get systemd journal files - if: always() # This ensures the step runs even if previous steps failed - run: | - make V=1 journal-dump - - - name: Start SSH Agent - if: always() # Ensure this step runs even if previous steps failed - uses: webfactory/[email protected] - with: - ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - - - name: Build our kdevops archive results - if: always() # This ensures the step runs even if previous steps failed - run: | - make V=1 ci-archive - - - name: Upload our kdevops results archive - if: always() # This ensures the step runs even if previous steps failed - uses: actions/upload-artifact@v4 - with: - name: kdevops-ci-results - path: ${{ env.KDEVOPS_PATH }}/archive/*.zip - - # Ensure make destroy always runs, even on failure - - name: Run kdevops make destroy - if: always() # This ensures the step runs even if previous steps failed - run: | - make V=1 destroy diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml new file mode 100644 index 00000000..35423403 --- /dev/null +++ b/.github/workflows/main.yml @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Run kdevops CI Workflow - Reusable + +on: + workflow_call: + inputs: + ci_workflow: + description: "CI Workflow" + required: true + default: 'blktests_nvme' + type: string + kernel_ref: + description: "Linux tree git reference (branch/tag/commit-id)" + required: true + default: 'master' + type: string + kernel_tree: + description: "Linux kernel tree to use" + required: true + default: 'linux' + type: string + test_mode: + description: 'Testing mode' + required: false + default: 'full-testing' + type: string + tests: + description: 'Custom test to run (for kdevops-validation mode only)' + required: false + default: '' + type: string + +jobs: + setup: + name: Setup kdevops workspace + runs-on: ${{ contains(inputs.ci_workflow, 'blktests') && fromJSON('["self-hosted", "Linux", "X64", "blktests"]') || contains(inputs.ci_workflow, 'tmpfs') && fromJSON('["self-hosted", "Linux", "X64", "tmpfs"]') || contains(inputs.ci_workflow, 'xfs') && fromJSON('["self-hosted", "Linux", "X64", "xfs"]') || contains(inputs.ci_workflow, 'btrfs') && fromJSON('["self-hosted", "Linux", "X64", "btrfs"]') || contains(inputs.ci_workflow, 'ext4') && fromJSON('["self-hosted", "Linux", "X64", "ext4"]') || contains(inputs.ci_workflow, 'selftests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'seltests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'modules') && fromJSON('["self-hosted", "Linux", "X64", "modules"]') || contains(inputs.ci_workflow, 'firmware') && fromJSON('["self-hosted", "Linux", "X64", "firmware"]') || contains(inputs.ci_workflow, 'mm') && fromJSON('["self-hosted", "Linux", "X64", "mm"]') || fromJSON('["self-hosted"]') }} + steps: + - name: Checkout dkruces/kdevops + uses: actions/checkout@v4 + with: + clean: false + + - name: kdevops setup + uses: ./.github/actions/setup + with: + ci_workflow: ${{ inputs.ci_workflow }} + dir: ${{ inputs.ci_workflow }} + kernel_ref: ${{ inputs.kernel_ref }} + kernel_tree: ${{ inputs.kernel_tree }} + + test: + name: Run kdevops ci-test + runs-on: ${{ contains(inputs.ci_workflow, 'blktests') && fromJSON('["self-hosted", "Linux", "X64", "blktests"]') || contains(inputs.ci_workflow, 'tmpfs') && fromJSON('["self-hosted", "Linux", "X64", "tmpfs"]') || contains(inputs.ci_workflow, 'xfs') && fromJSON('["self-hosted", "Linux", "X64", "xfs"]') || contains(inputs.ci_workflow, 'btrfs') && fromJSON('["self-hosted", "Linux", "X64", "btrfs"]') || contains(inputs.ci_workflow, 'ext4') && fromJSON('["self-hosted", "Linux", "X64", "ext4"]') || contains(inputs.ci_workflow, 'selftests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'seltests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'modules') && fromJSON('["self-hosted", "Linux", "X64", "modules"]') || contains(inputs.ci_workflow, 'firmware') && fromJSON('["self-hosted", "Linux", "X64", "firmware"]') || contains(inputs.ci_workflow, 'mm') && fromJSON('["self-hosted", "Linux", "X64", "mm"]') || fromJSON('["self-hosted"]') }} + needs: [setup] + timeout-minutes: 120 + steps: + - name: kdevops ci-test + uses: ./.github/actions/test + with: + ci_workflow: ${{ inputs.ci_workflow }} + dir: ${{ inputs.ci_workflow }} + test_mode: ${{ inputs.test_mode }} + tests: ${{ inputs.tests }} + + archive: + name: Archive kdevops + runs-on: ${{ contains(inputs.ci_workflow, 'blktests') && fromJSON('["self-hosted", "Linux", "X64", "blktests"]') || contains(inputs.ci_workflow, 'tmpfs') && fromJSON('["self-hosted", "Linux", "X64", "tmpfs"]') || contains(inputs.ci_workflow, 'xfs') && fromJSON('["self-hosted", "Linux", "X64", "xfs"]') || contains(inputs.ci_workflow, 'btrfs') && fromJSON('["self-hosted", "Linux", "X64", "btrfs"]') || contains(inputs.ci_workflow, 'ext4') && fromJSON('["self-hosted", "Linux", "X64", "ext4"]') || contains(inputs.ci_workflow, 'selftests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'seltests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'modules') && fromJSON('["self-hosted", "Linux", "X64", "modules"]') || contains(inputs.ci_workflow, 'firmware') && fromJSON('["self-hosted", "Linux", "X64", "firmware"]') || contains(inputs.ci_workflow, 'mm') && fromJSON('["self-hosted", "Linux", "X64", "mm"]') || fromJSON('["self-hosted"]') }} + needs: [setup, test] + steps: + - name: Start SSH Agent + uses: webfactory/[email protected] + with: + ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} + + - name: Archive ci-test results + uses: ./.github/actions/archive + with: + ci_workflow: ${{ inputs.ci_workflow }} + dir: ${{ inputs.ci_workflow }} + + - name: Upload our kdevops results archive + uses: actions/upload-artifact@v4 + with: + name: kdevops-ci-results + path: ${{ inputs.ci_workflow }}/kdevops/archive/*.zip + + cleanup: + name: Cleanup kdevops workspace + runs-on: ${{ contains(inputs.ci_workflow, 'blktests') && fromJSON('["self-hosted", "Linux", "X64", "blktests"]') || contains(inputs.ci_workflow, 'tmpfs') && fromJSON('["self-hosted", "Linux", "X64", "tmpfs"]') || contains(inputs.ci_workflow, 'xfs') && fromJSON('["self-hosted", "Linux", "X64", "xfs"]') || contains(inputs.ci_workflow, 'btrfs') && fromJSON('["self-hosted", "Linux", "X64", "btrfs"]') || contains(inputs.ci_workflow, 'ext4') && fromJSON('["self-hosted", "Linux", "X64", "ext4"]') || contains(inputs.ci_workflow, 'selftests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'seltests') && fromJSON('["self-hosted", "Linux", "X64", "selftests"]') || contains(inputs.ci_workflow, 'modules') && fromJSON('["self-hosted", "Linux", "X64", "modules"]') || contains(inputs.ci_workflow, 'firmware') && fromJSON('["self-hosted", "Linux", "X64", "firmware"]') || contains(inputs.ci_workflow, 'mm') && fromJSON('["self-hosted", "Linux", "X64", "mm"]') || fromJSON('["self-hosted"]') }} + needs: [setup, test, archive] + if: always() + steps: + - name: kdevops cleanup + uses: ./.github/actions/cleanup + with: + dir: ${{ inputs.ci_workflow }} diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml new file mode 100644 index 00000000..30a035c8 --- /dev/null +++ b/.github/workflows/manual.yml @@ -0,0 +1,122 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Run kdevops CI Workflow - Manual + +on: + workflow_dispatch: + inputs: + ci_workflow: + description: "CI Workflow" + required: true + default: 'blktests_nvme' + type: choice + options: + - blktests + - blktests_block + - blktests_loop + - blktests_meta + - blktests_nbd + - blktests_nvme + - blktests_nvmemp + - blktests_scsi + - blktests_srp + - blktests_zbd + - lbs-xfs + - lbs-xfs-bdev-large-nvme + - lbs-xfs-bdev-nvme + - lbs-xfs-small + - xfs + - xfs_crc + - xfs_crc_logdev + - xfs_crc_rtdev + - xfs_crc_rtdev_extsize_28k + - xfs_crc_rtdev_extsize_64k + - xfs_nocrc + - xfs_nocrc_16k_4ks + - xfs_nocrc_1k + - xfs_nocrc_2k + - xfs_nocrc_32_4ks + - xfs_nocrc_4k + - xfs_nocrc_512 + - xfs_nocrc_64_4ks + - xfs_nocrc_8k_4ks + - xfs_nocrc_lbs + - xfs_reflink + - xfs_reflink_1024 + - xfs_reflink_16k_4ks + - xfs_reflink_2k + - xfs_reflink_32k_4ks + - xfs_reflink_4k + - xfs_reflink_4k_8ks + - xfs_reflink_64k_4ks + - xfs_reflink_dir_bsize_8k + - xfs_reflink_lbs + - xfs_reflink_logdev + - xfs_reflink_normapbt + - xfs_reflink_nrext64 + - xfs_reflink_stripe_len + - xfs-soak + kernel_tree: + description: "Linux kernel tree to use" + required: true + default: 'linux' + type: choice + options: + - linux + - linux-next + - linux-stable + kernel_ref: + description: "Linux tree git reference (branch/tag/commit-id)" + required: true + default: 'master' + type: string + test_mode: + description: 'Testing mode' + required: false + default: 'full-testing' + type: choice + options: + - 'full-testing' # Complete test suite + - 'kdevops-validation' # Single test for framework validation + tests: + description: 'Custom test to run (for kdevops-validation mode only)' + required: false + type: string + default: '' + +jobs: + check_ref: + name: Check Linux kernel Git Reference + runs-on: [self-hosted] + steps: + - name: Checkout repo + uses: actions/checkout@v4 + + - name: Check kernel_ref exists + id: check_kernel_ref + run: | + set -euxo pipefail + + ref="${{ github.event.inputs.kernel_ref }}" + tree="${{ github.event.inputs.kernel_tree }}" + mirror="/mirror/${tree}.git" + ls_remote="$(git ls-remote "$mirror" "refs/*/${ref}" || true)" + contains_branch="$(git -C "$mirror" branch --contains "${ref}" || true)" + contains_tag="$(git -C "$mirror" branch --contains "${ref}" || true)" + + if [ -z "$ls_remote" ] && [ -z "$contains_branch" ] && [ -z "$contains_tag" ]; then + echo "Linux kernel ${ref} does not exist." + exit 1 + fi + + manual: + name: Manual kdevops CI + needs: [check_ref] + uses: ./.github/workflows/main.yml + secrets: inherit + with: + ci_workflow: ${{ inputs.ci_workflow }} + kernel_ref: ${{ inputs.kernel_ref }} + kernel_tree: ${{ inputs.kernel_tree }} + test_mode: ${{ inputs.test_mode }} + tests: ${{ inputs.tests }} diff --git a/.github/workflows/push.yml b/.github/workflows/push.yml new file mode 100644 index 00000000..561e873e --- /dev/null +++ b/.github/workflows/push.yml @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Run kdevops CI Workflow - Push/PR + +on: + push: + branches: + - '**' + pull_request: + branches: + - '**' + +jobs: + push-pr: + name: Push/PR kdevops CI + uses: ./.github/workflows/main.yml + secrets: inherit + strategy: + fail-fast: false # Don't cancel other jobs if one fails + matrix: + ci_workflow: + - blktests_nvme + - xfs_reflink_4k + with: + ci_workflow: ${{ matrix.ci_workflow }} + kernel_ref: 'v6.15' + kernel_tree: 'linux' + test_mode: 'kdevops-validation' + tests: '' diff --git a/.github/workflows/schedule.yml b/.github/workflows/schedule.yml new file mode 100644 index 00000000..8f4589b4 --- /dev/null +++ b/.github/workflows/schedule.yml @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: GPL-2.0 +--- +name: Run kdevops CI Workflow - Schedule + +on: + schedule: + - cron: '0 14 * * *' + + workflow_dispatch: + +jobs: + check_ref: + name: Check Linux kernel Git Reference + outputs: + kernel_ref: ${{ steps.check_kernel_ref.outputs.kernel_ref }} + kernel_tree: ${{ steps.check_kernel_ref.outputs.kernel_tree }} + runs-on: [self-hosted] + steps: + - name: Checkout repo + uses: actions/checkout@v4 + + - name: Generate kernel_ref + id: check_kernel_ref + run: | + set -euxo pipefail + day_of_week=$(date +%u) + + kernel_ref=$(./scripts/korg-releases.py --moniker linux-next) + kernel_tree=linux-next + if [ "$day_of_week" -eq 1 ]; then + kernel_ref=$(./scripts/korg-releases.py --moniker mainline) + kernel_tree=linux + fi + + "${{ github.workspace }}/scripts/github_output.sh" kernel_ref "$kernel_ref" + "${{ github.workspace }}/scripts/github_output.sh" kernel_tree "$kernel_tree" + + schedule: + name: Scheduled kdevops CI + needs: [check_ref] + uses: ./.github/workflows/main.yml + secrets: inherit + strategy: + fail-fast: false # Don't cancel other jobs if one fails + matrix: + ci_workflow: + - blktests + - linux-mm-kpd + - linux-modules-kpd + - tmpfs + with: + ci_workflow: ${{ matrix.ci_workflow }} + kernel_ref: ${{ needs.check_ref.outputs.kernel_ref }} + kernel_tree: ${{ needs.check_ref.outputs.kernel_tree }} diff --git a/scripts/generate_ci_commit_message.sh b/scripts/generate_ci_commit_message.sh new file mode 100755 index 00000000..8be2c0b3 --- /dev/null +++ b/scripts/generate_ci_commit_message.sh @@ -0,0 +1,263 @@ +#!/bin/bash +# SPDX-License-Identifier: copyleft-next-0.3.1 + +# Generate enhanced CI commit message for kdevops-results-archive +# This script creates commit messages following the kdevops CI format specification + +set -euo pipefail + +# Default values +CI_WORKFLOW="${CI_WORKFLOW:-demo}" +KERNEL_TREE="${KERNEL_TREE:-linux}" +TESTS_PARAM="${TESTS:-${LIMIT_TESTS:-}}" +DURATION="${DURATION:-unknown}" + +# Input files (expected to be created by GitHub Actions) +CI_COMMIT_EXTRA="${CI_COMMIT_EXTRA:-ci.commit_extra}" +CI_RESULT="${CI_RESULT:-ci.result}" +CI_START_TIME="${CI_START_TIME:-ci.start_time}" +CI_REF="${CI_REF:-ci.ref}" +CI_TRIGGER="${CI_TRIGGER:-ci.trigger}" + +# Helper function to wrap long commit subjects at 72 chars +wrap_commit_subject() { + local subject="$1" + if [ ${#subject} -le 68 ]; then # 68 to account for quotes + echo "\"$subject\"" + else + echo "\"$subject\"" | fold -s -w 68 | sed '2,$s/^/ /' + fi +} + +# Calculate duration if start time available +calculate_duration() { + if [ -f "$CI_START_TIME" ]; then + local start_time=$(cat "$CI_START_TIME" 2>/dev/null || echo $(date +%s)) + local end_time=$(date +%s) + local duration_sec=$((end_time - start_time)) + local minutes=$((duration_sec / 60)) + local seconds=$((duration_sec % 60)) + + if [ $minutes -gt 0 ]; then + echo "${minutes}m ${seconds}s" + else + echo "${seconds}s" + fi + else + echo "unknown" + fi +} + +# Determine scope and header format with enhanced context-aware detection +determine_scope() { + # Simple detection: if TESTS is set, it's kdevops validation + if [[ -n "${TESTS:-}" ]] || [[ -n "${LIMIT_TESTS:-}" ]] || [[ -n "${TESTS_PARAM:-}" ]]; then + echo "kdevops" + else + echo "tests" + fi +} + +# Get kernel information +get_kernel_info() { + # Try to get git describe from linux directory, fallback gracefully + local kernel_describe="unknown" + local kernel_hash="unknown" + local kernel_subject="unknown commit" + + if [ -d "linux/.git" ]; then + kernel_describe=$(cd linux && git describe --tags --always --dirty 2>/dev/null || echo "unknown") + kernel_hash=$(cd linux && git rev-parse --short=12 HEAD 2>/dev/null || echo "unknown") + kernel_subject=$(cd linux && git log -1 --pretty=format:"%s" 2>/dev/null || echo "unknown commit") + fi + + echo "$kernel_describe|$kernel_hash|$kernel_subject" +} + +# Get kdevops information (from kdevops directory context) +get_kdevops_info() { + # This will be run from the kdevops directory + local kdevops_hash=$(git rev-parse --short=12 HEAD 2>/dev/null || echo "unknown") + local kdevops_subject=$(git log -1 --pretty=format:"%s" 2>/dev/null || echo "unknown commit") + + echo "$kdevops_hash|$kdevops_subject" +} + +# Read test results +get_test_results() { + local result_content="" + local test_result="unknown" + + if [ -f "$CI_COMMIT_EXTRA" ]; then + result_content=$(cat "$CI_COMMIT_EXTRA") + else + result_content="No test results available." + fi + + if [ -f "$CI_RESULT" ]; then + test_result=$(cat "$CI_RESULT" | tr -d '\n' | tr -d '\r') + fi + + echo "$test_result|$result_content" +} + +# Determine workflow type for result formatting +get_workflow_type() { + case "$CI_WORKFLOW" in + *xfs*|*btrfs*|*ext4*|*tmpfs*|*lbs-xfs*) echo "fstests" ;; + blktests*) echo "blktests" ;; + selftests*|*firmware*|*modules*|*mm*) echo "selftests" ;; + ai_*) echo "ai" ;; + mmtests_*) echo "mmtests" ;; + ltp_*) echo "ltp" ;; + fio-tests*|fio_*) echo "fio-tests" ;; + minio_*) echo "minio" ;; + *) echo "other" ;; + esac +} + +# Generate the enhanced commit message +generate_commit_message() { + local scope=$(determine_scope) + local workflow_type=$(get_workflow_type) + local duration=$(calculate_duration) + + # Get kernel tree and ref from metadata files + local actual_kernel_tree="$KERNEL_TREE" + local actual_kernel_ref="unknown" + + if [ -f "$CI_TRIGGER" ]; then + actual_kernel_tree=$(cat "$CI_TRIGGER" | tr -d '\n' | tr -d '\r') + fi + + if [ -f "$CI_REF" ]; then + actual_kernel_ref=$(cat "$CI_REF" | tr -d '\n' | tr -d '\r') + fi + + # Get kernel info (this assumes we're in the kernel directory context) + local kernel_info=$(get_kernel_info) + local kernel_describe=$(echo "$kernel_info" | cut -d'|' -f1) + local kernel_hash=$(echo "$kernel_info" | cut -d'|' -f2) + local kernel_subject=$(echo "$kernel_info" | cut -d'|' -f3) + + # Use metadata ref if available, fallback to git describe + # Ensure we use the most accurate kernel version consistently + if [ "$actual_kernel_ref" != "unknown" ] && [ "$actual_kernel_ref" != "" ]; then + kernel_describe="$actual_kernel_ref" + fi + + # Debug kernel version information + + # Get kdevops info (this assumes we can access kdevops directory) + local kdevops_info=$(get_kdevops_info) + local kdevops_hash=$(echo "$kdevops_info" | cut -d'|' -f1) + local kdevops_subject=$(echo "$kdevops_info" | cut -d'|' -f2) + + # Get test results + local results_info=$(get_test_results) + local test_result=$(echo "$results_info" | cut -d'|' -f1) + local result_content=$(echo "$results_info" | cut -d'|' -f2-) + + # Build header + local header="" + if [ "$scope" = "kdevops" ]; then + # kdevops-ci validation format: focus on kdevops commit being tested + header="kdevops-ci: $CI_WORKFLOW: $kdevops_hash $kdevops_subject" + else + # Full test suite format: include PASS/FAIL status + local status="PASS" + if [ "$test_result" = "not ok" ] || [ "$test_result" = "fail" ]; then + status="FAIL" + fi + header="$CI_WORKFLOW ($actual_kernel_tree $kernel_describe): $status" + fi + + # Build scope description + local scope_desc="" + if [ "$scope" = "kdevops" ]; then + local test_param="${TESTS:-${LIMIT_TESTS:-${TESTS_PARAM:-}}}" + if [[ -n "$test_param" ]]; then + scope_desc="kdevops validation (single test: $test_param)" + else + scope_desc="kdevops validation" + fi + else + scope_desc="full test suite" + fi + + # Wrap kernel commit subject + local wrapped_kernel_subject=$(wrap_commit_subject "$kernel_subject") + local wrapped_kdevops_subject=$(wrap_commit_subject "$kdevops_subject") + + # Generate metadata line with 72-char handling + local metadata_line="workflow: $CI_WORKFLOW | tree: $actual_kernel_tree | ref: $kernel_describe | scope: $scope | result: $test_result" + local metadata_output="" + if [ ${#metadata_line} -gt 72 ]; then + metadata_output="workflow: $CI_WORKFLOW | tree: $actual_kernel_tree"$'\n'"ref: $kernel_describe | scope: $scope | result: $test_result" + else + metadata_output="$metadata_line" + fi + + # Build kernel validation info + local kernel_info_line="" + if [ "$actual_kernel_ref" != "unknown" ] && [ "$kernel_describe" != "$actual_kernel_ref" ]; then + # Show both requested and actual if they differ + kernel_info_line=" Kernel: $actual_kernel_ref (requested) → $kernel_describe (actual)" + else + # Show just the kernel version if they match or no metadata + kernel_info_line=" Kernel: $kernel_describe" + fi + + # Build the complete commit message based on scope + if [ "$scope" = "kdevops" ]; then + # kdevops-ci validation format + cat << EOF +$header + +BUILD INFO: + kdevops: $kdevops_hash ($wrapped_kdevops_subject) + Workflow: $CI_WORKFLOW + Test: ${TESTS_PARAM:-"complete suite"} +$kernel_info_line + Duration: $duration + +EXECUTION RESULTS: +$result_content + +METADATA: +workflow: $CI_WORKFLOW | scope: kdevops | test: ${TESTS_PARAM:-"complete suite"} | requested: $actual_kernel_ref | actual: $kernel_describe | result: $test_result +EOF + else + # Full test suite format + cat << EOF +$header + +BUILD INFO: +$kernel_info_line ($wrapped_kernel_subject) + Workflow: $CI_WORKFLOW + kdevops: $kdevops_hash ($wrapped_kdevops_subject) + Scope: $scope_desc + Duration: $duration + +EXECUTION RESULTS: +$result_content + +METADATA: +$metadata_output | requested: $actual_kernel_ref | actual: $kernel_describe +EOF + fi +} + +# Main execution +main() { + # Validate required environment + if [ -z "$CI_WORKFLOW" ]; then + echo "Error: CI_WORKFLOW environment variable is required" >&2 + exit 1 + fi + + generate_commit_message +} + +# Run main function +main "$@" \ No newline at end of file diff --git a/scripts/github_output.sh b/scripts/github_output.sh new file mode 100755 index 00000000..711306c3 --- /dev/null +++ b/scripts/github_output.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# +# Usage: ./github_output.sh key value +set -euxo pipefail + +key="$1" +value="$2" + +echo "$key=$value" >> "$GITHUB_OUTPUT" diff --git a/scripts/korg-releases.py b/scripts/korg-releases.py new file mode 100755 index 00000000..de494327 --- /dev/null +++ b/scripts/korg-releases.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: copyleft-next-0.3.1 +import sys +import logging +import argparse +import json +import urllib.request +import socket +import re + + +def parser(): + parser = argparse.ArgumentParser(description="kernel.org/releases.json checker") + parser.add_argument("--debug", action="store_true", help="debug") + parser.add_argument( + "--moniker", + help="moniker (mainline, stable, longterm or linux-next)", + required=True, + ) + parser.add_argument( + "--pname", + help="project name for User-Agent request", + default="kdevops", + ) + parser.add_argument( + "--pversion", + help="project version for User-Agent request", + default="5.0.2", + ) + return parser + + +def _check_connection(host, port, timeout=2): + try: + with socket.create_connection((host, port), timeout): + logging.debug(f"Connection to {host} on port {port} succeeded!") + return True + except (socket.timeout, socket.error) as e: + logging.debug(f"Connection to {host} on port {port} failed: {e}") + return False + + +def kreleases(args) -> None: + """Get the latest kernel releases from kernel.org/releases.json""" + + reflist = [] + if _check_connection("kernel.org", 80): + _url = "https://www.kernel.org/releases.json" + req = urllib.request.Request( + _url, + headers={ + "User-Agent": f"{args.pname}/{args.pversion} ([email protected])" + }, + ) + with urllib.request.urlopen(req) as url: + data = json.load(url) + + for release in data["releases"]: + if release["moniker"] == args.moniker: + # Check if release.json is aa.bb.cc type + if re.compile(r"^\d+\.\d+(\.\d+|-rc\d+)?$").match( + release["version"] + ): + reflist.append("v" + release["version"]) + else: + reflist.append(release["version"]) + + logging.debug(f"{reflist}") + for r in reflist: + print(r) + + +def main() -> None: + """Kconfig choice generator for git refereces""" + log = logging.getLogger() + log.setLevel(logging.INFO) + p = parser() + args, _ = p.parse_known_args() + if args.debug: + log.setLevel(logging.DEBUG) + + kreleases(args) + + +if __name__ == "__main__": + ret = 0 + try: + main() + except Exception: + ret = 1 + import traceback + + traceback.print_exc() + sys.exit(ret) -- 2.50.1