[PATCH v2 0/4] declared hosts support
Luis Chamberlain <[email protected]>
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <[email protected]> |
This v2 adresses Chuck's feedback to make the include just use the workflow name. This scales better. While at it, I decided to ensure to only enable workflows which are tested, so far that is none, so they all just need to be disabled. But I'll add one next which goes tested. The data partition stuff is modified so to be skipped as well. We expect the user to have set that up, they can howeer specify a custom path for that. The new workflow is minio WARP testing. Since we already used minio setup via docker, we take that from the milvus ai role, and generalize it to be shared. If folks want to try this: https://github.com/linux-kdevops/kdevops/tree/20250830-lambda-minio And how you can easily test this on an existing server which you already have ssh access to: make defconfig-minio-warp-xfs DECLARE_HOSTS=foo WARP_DEVICE=/dev/nvme4n1 make make minio make minio-warp See the resutls on workflows/minio/results/ That will configure the device with xfs with 4k block size. To test with 16k block size you can use: make defconfig-minio-warp-xfs-16k DECLARE_HOSTS=foo WARP_DEVICE=/dev/nvme4n1 Luis Chamberlain (4): gen_hosts: use kdevops_workflow_name directly for template selection declared_hosts: add support for pre-existing infrastructure Makefile: add missing extra_vars.yaml dependencies minio: add MinIO Warp S3 benchmarking with declared hosts support .gitignore | 2 + Makefile | 12 +- defconfigs/minio-warp | 52 ++ defconfigs/minio-warp-ab | 41 + defconfigs/minio-warp-btrfs | 35 + defconfigs/minio-warp-declared-hosts | 56 ++ defconfigs/minio-warp-multifs | 74 ++ defconfigs/minio-warp-storage | 65 ++ defconfigs/minio-warp-xfs | 35 + defconfigs/minio-warp-xfs-16k | 65 ++ defconfigs/minio-warp-xfs-lbs | 65 ++ defconfigs/xfs_reflink_16k_4ks_declared | 19 + kconfigs/Kconfig.bringup | 9 +- kconfigs/Kconfig.declared_hosts | 81 ++ kconfigs/workflows/Kconfig | 43 + kconfigs/workflows/Kconfig.data_partition | 11 + playbooks/minio.yml | 53 ++ playbooks/roles/ai_setup/tasks/main.yml | 40 +- .../create_data_partition/tasks/main.yml | 23 + playbooks/roles/devconfig/defaults/main.yml | 2 + playbooks/roles/devconfig/tasks/main.yml | 25 + playbooks/roles/gen_hosts/defaults/main.yml | 1 + playbooks/roles/gen_hosts/tasks/main.yml | 319 ++----- playbooks/roles/gen_hosts/templates/hosts.j2 | 242 +---- .../roles/gen_hosts/templates/workflows/ai.j2 | 99 ++ .../gen_hosts/templates/workflows/blktests.j2 | 58 ++ .../gen_hosts/templates/workflows/cxl.j2 | 7 + .../templates/workflows/declared-hosts.j2 | 239 +++++ .../templates/workflows/fio-tests.j2 | 38 + .../gen_hosts/templates/workflows/fstests.j2 | 72 ++ .../gen_hosts/templates/workflows/gitr.j2 | 41 + .../gen_hosts/templates/workflows/ltp.j2 | 41 + .../gen_hosts/templates/workflows/minio.j2 | 173 ++++ .../gen_hosts/templates/workflows/mix.j2 | 62 ++ .../gen_hosts/templates/workflows/mmtests.j2 | 77 ++ .../gen_hosts/templates/workflows/nfstest.j2 | 41 + .../gen_hosts/templates/workflows/pynfs.j2 | 7 + .../templates/workflows/reboot-limit.j2 | 33 + .../templates/workflows/selftests.j2 | 53 ++ .../gen_hosts/templates/workflows/sysbench.j2 | 53 ++ playbooks/roles/gen_nodes/defaults/main.yml | 1 + playbooks/roles/gen_nodes/tasks/main.yml | 132 ++- .../linux-mirror/python/start-mirroring.py | 4 +- playbooks/roles/minio_destroy/tasks/main.yml | 34 + playbooks/roles/minio_install/tasks/main.yml | 61 ++ playbooks/roles/minio_results/tasks/main.yml | 86 ++ playbooks/roles/minio_setup/defaults/main.yml | 16 + playbooks/roles/minio_setup/tasks/main.yml | 100 ++ .../roles/minio_uninstall/tasks/main.yml | 17 + playbooks/roles/minio_warp_run/tasks/main.yml | 249 +++++ .../templates/warp_config.json.j2 | 14 + workflows/Makefile | 4 + workflows/ai/Makefile | 3 - workflows/blktests/Makefile | 3 - workflows/cxl/Makefile | 2 - workflows/demos/reboot-limit/Kconfig | 5 + workflows/fio-tests/Makefile | 3 - workflows/fstests/Makefile | 3 - workflows/gitr/Makefile | 3 - workflows/linux/Makefile | 1 - workflows/ltp/Makefile | 3 - workflows/minio/Kconfig | 23 + workflows/minio/Kconfig.docker | 66 ++ workflows/minio/Kconfig.storage | 364 ++++++++ workflows/minio/Kconfig.warp | 141 +++ workflows/minio/Makefile | 76 ++ .../minio/scripts/analyze_warp_results.py | 858 ++++++++++++++++++ .../minio/scripts/generate_warp_report.py | 404 +++++++++ .../minio/scripts/run_benchmark_suite.sh | 116 +++ workflows/mmtests/Makefile | 3 - workflows/nfstest/Makefile | 3 - workflows/pynfs/Makefile | 3 - workflows/selftests/Makefile | 3 - workflows/sysbench/Makefile | 3 - 74 files changed, 4705 insertions(+), 566 deletions(-) create mode 100644 defconfigs/minio-warp create mode 100644 defconfigs/minio-warp-ab create mode 100644 defconfigs/minio-warp-btrfs create mode 100644 defconfigs/minio-warp-declared-hosts create mode 100644 defconfigs/minio-warp-multifs create mode 100644 defconfigs/minio-warp-storage create mode 100644 defconfigs/minio-warp-xfs create mode 100644 defconfigs/minio-warp-xfs-16k create mode 100644 defconfigs/minio-warp-xfs-lbs create mode 100644 defconfigs/xfs_reflink_16k_4ks_declared create mode 100644 kconfigs/Kconfig.declared_hosts create mode 100644 playbooks/minio.yml create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/minio.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mix.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2 create mode 100644 playbooks/roles/minio_destroy/tasks/main.yml create mode 100644 playbooks/roles/minio_install/tasks/main.yml create mode 100644 playbooks/roles/minio_results/tasks/main.yml create mode 100644 playbooks/roles/minio_setup/defaults/main.yml create mode 100644 playbooks/roles/minio_setup/tasks/main.yml create mode 100644 playbooks/roles/minio_uninstall/tasks/main.yml create mode 100644 playbooks/roles/minio_warp_run/tasks/main.yml create mode 100644 playbooks/roles/minio_warp_run/templates/warp_config.json.j2 create mode 100644 workflows/minio/Kconfig create mode 100644 workflows/minio/Kconfig.docker create mode 100644 workflows/minio/Kconfig.storage create mode 100644 workflows/minio/Kconfig.warp create mode 100644 workflows/minio/Makefile create mode 100755 workflows/minio/scripts/analyze_warp_results.py create mode 100755 workflows/minio/scripts/generate_warp_report.py create mode 100755 workflows/minio/scripts/run_benchmark_suite.sh -- 2.50.1