[PATCH v2 2/4] workflows: bootlinux: add comprehensive ccache support
Daniel Gomez <[email protected]>
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <[email protected]> |
From: Daniel Gomez <[email protected]> Add ccache (compiler cache) support to accelerate kernel builds with both GCC and Clang toolchains. Implementation provides flexible configuration options while maintaining simplicity for common use cases. Key features: - System-wide ccache configuration (default) for immediate use - kdevops-managed configuration for project isolation - Automatic package installation across all supported distros - Modular environment variable composition for clean integration - LLVM documentation compliance (CC="ccache clang") - Seamless integration with reproducible builds and existing features Configuration: - Cache size specified in GiB (digits only, e.g., 10 = 10 GiB) - Uses automatic Kconfig→Ansible integration via .extra_vars_auto.yaml - Template-based configuration management with TOPDIR_PATH isolation - Organized in dedicated ccache submenu for clean UI Technical implementation: - Uses modular bootlinux_make_params for community.general.make module - Uses modular bootlinux_build_environment for shell tasks - Automatic compiler detection (ccache gcc vs ccache clang) - Enhanced debug output for build command validation Generated-by: Claude AI Signed-off-by: Daniel Gomez <[email protected]> --- playbooks/roles/bootlinux/defaults/main.yml | 6 ++ playbooks/roles/bootlinux/tasks/build/9p.yml | 5 +- playbooks/roles/bootlinux/tasks/ccache.yml | 37 ++++++++++ .../bootlinux/tasks/install-deps/debian/main.yml | 1 + .../bootlinux/tasks/install-deps/redhat/main.yml | 1 + .../bootlinux/tasks/install-deps/suse/main.yml | 1 + .../tasks/install-minimal-deps/debian/main.yml | 1 + .../tasks/install-minimal-deps/redhat/main.yml | 2 + .../tasks/install-minimal-deps/suse/main.yml | 1 + playbooks/roles/bootlinux/tasks/main.yml | 9 +++ playbooks/roles/bootlinux/templates/ccache.conf.j2 | 27 ++++++++ workflows/linux/Kconfig | 78 ++++++++++++++++++++++ 12 files changed, 168 insertions(+), 1 deletion(-) diff --git a/playbooks/roles/bootlinux/defaults/main.yml b/playbooks/roles/bootlinux/defaults/main.yml index 2d2eb798..97adec30 100644 --- a/playbooks/roles/bootlinux/defaults/main.yml +++ b/playbooks/roles/bootlinux/defaults/main.yml @@ -37,6 +37,8 @@ target_linux_make_cmd: "{{ make }} -j{{ ansible_processor_vcpus }}" bootlinux_make_params: >- {{ ({} | combine( + {'CC': 'ccache clang' if bootlinux_compiler_clang|default(false)|bool else 'ccache gcc'} if bootlinux_ccache|default(false)|bool else {} + ) | combine( { 'KBUILD_BUILD_TIMESTAMP': '', 'KBUILD_BUILD_USER': 'kdevops', @@ -49,6 +51,10 @@ bootlinux_make_params: >- bootlinux_build_environment: >- {{ ({} | combine( + {'CCACHE_CONFIGPATH': topdir_path + '/.ccache/ccache.conf'} if (bootlinux_ccache|default(false)|bool and bootlinux_ccache_kdevops_managed|default(false)|bool) else {} + ) | combine( + {'CC': 'ccache clang' if bootlinux_compiler_clang|default(false)|bool else 'ccache gcc'} if bootlinux_ccache|default(false)|bool else {} + ) | combine( { 'KBUILD_BUILD_TIMESTAMP': '', 'KBUILD_BUILD_USER': 'kdevops', diff --git a/playbooks/roles/bootlinux/tasks/build/9p.yml b/playbooks/roles/bootlinux/tasks/build/9p.yml index 17b19cb3..6f03ed9a 100644 --- a/playbooks/roles/bootlinux/tasks/build/9p.yml +++ b/playbooks/roles/bootlinux/tasks/build/9p.yml @@ -164,8 +164,10 @@ build_jobs: {{ nproc_9p.stdout }} bootlinux_make_params: {{ bootlinux_make_params }} bootlinux_build_environment: {{ bootlinux_build_environment }} + bootlinux_ccache: {{ bootlinux_ccache|default(false) }} bootlinux_compiler_clang: {{ bootlinux_compiler_clang|default(false) }} bootlinux_reproducible_builds: {{ bootlinux_reproducible_builds|default(false) }} + bootlinux_ccache_kdevops_managed: {{ bootlinux_ccache_kdevops_managed|default(false) }} run_once: true delegate_to: localhost vars: @@ -178,7 +180,8 @@ echo "CC=${CC:-'not set'}" echo "PATH=${PATH}" which gcc || echo "gcc not found" - env | grep -E '^(CC|PATH)' | sort + which ccache || echo "ccache not found" + env | grep -E '^(CC|CCACHE|PATH)' | sort environment: "{{ bootlinux_build_environment }}" register: cc_test_result run_once: true diff --git a/playbooks/roles/bootlinux/tasks/ccache.yml b/playbooks/roles/bootlinux/tasks/ccache.yml new file mode 100644 index 00000000..38b2c034 --- /dev/null +++ b/playbooks/roles/bootlinux/tasks/ccache.yml @@ -0,0 +1,37 @@ +--- +# ccache configuration and setup + +- name: Create kdevops ccache directory structure + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: '0755' + loop: + - "{{ topdir_path }}/.ccache" + - "{{ bootlinux_ccache_dir }}" + when: + - bootlinux_ccache|default(false)|bool + - bootlinux_ccache_kdevops_managed|default(false)|bool + +- name: Generate kdevops-managed ccache configuration + ansible.builtin.template: + src: ccache.conf.j2 + dest: "{{ topdir_path }}/.ccache/ccache.conf" + mode: '0644' + when: + - bootlinux_ccache|default(false)|bool + - bootlinux_ccache_kdevops_managed|default(false)|bool + +- name: Display ccache configuration info + ansible.builtin.debug: + msg: | + ccache enabled: {{ bootlinux_ccache|default(false)|bool }} + ccache mode: {{ 'kdevops-managed' if bootlinux_ccache_kdevops_managed|default(false)|bool else 'system-wide' }} + {% if bootlinux_ccache_kdevops_managed|default(false)|bool %} + Config file: {{ topdir_path }}/.ccache/ccache.conf + Cache directory: {{ bootlinux_ccache_dir }} + Max size: {{ bootlinux_ccache_max_size }} GiB + {% endif %} + when: bootlinux_ccache|default(false)|bool + vars: + ansible_callback_diy_runner_on_ok_msg: "{{ ansible_callback_diy.result.output.msg }}" diff --git a/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml b/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml index 4edf8e1d..e5911029 100644 --- a/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml +++ b/playbooks/roles/bootlinux/tasks/install-deps/debian/main.yml @@ -39,4 +39,5 @@ - zstd - libncurses-dev - b4 + - ccache state: present diff --git a/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml b/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml index c7b18cd5..7f1955bf 100644 --- a/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml +++ b/playbooks/roles/bootlinux/tasks/install-deps/redhat/main.yml @@ -52,6 +52,7 @@ - dwarves - userspace-rcu - zstd + - ccache - name: Install btrfs-progs become: true diff --git a/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml b/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml index 7f0cf249..5c9f0e3b 100644 --- a/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml +++ b/playbooks/roles/bootlinux/tasks/install-deps/suse/main.yml @@ -28,4 +28,5 @@ - portmap - hwinfo - open-iscsi + - ccache disable_recommends: false diff --git a/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml b/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml index fab4998a..cc05a44a 100644 --- a/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml +++ b/playbooks/roles/bootlinux/tasks/install-minimal-deps/debian/main.yml @@ -24,4 +24,5 @@ - make - gcc - kmod + - ccache state: present diff --git a/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml b/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml index 50651ab1..950dbfe1 100644 --- a/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml +++ b/playbooks/roles/bootlinux/tasks/install-minimal-deps/redhat/main.yml @@ -10,6 +10,7 @@ - make - gcc - kmod + - ccache state: present when: - ansible_facts['distribution_major_version']|int < 8 @@ -22,6 +23,7 @@ - make - gcc - kmod + - ccache state: present when: - ansible_facts['distribution_major_version']|int >= 8 diff --git a/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml b/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml index a4cc27bb..d8f75240 100644 --- a/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml +++ b/playbooks/roles/bootlinux/tasks/install-minimal-deps/suse/main.yml @@ -10,4 +10,5 @@ - make - gcc - kmod-compat + - ccache state: present diff --git a/playbooks/roles/bootlinux/tasks/main.yml b/playbooks/roles/bootlinux/tasks/main.yml index 4110d596..eff6fb6b 100644 --- a/playbooks/roles/bootlinux/tasks/main.yml +++ b/playbooks/roles/bootlinux/tasks/main.yml @@ -32,6 +32,15 @@ - bootlinux_9p|bool - not workflow_linux_packaged|bool +# Setup ccache configuration +- name: Setup ccache configuration + ansible.builtin.import_tasks: + file: ccache.yml + when: + - bootlinux_ccache|default(false)|bool + - not workflow_linux_packaged|bool + tags: ['ccache'] + # We do this regardless of what distro you use - name: Install b4 become: true diff --git a/playbooks/roles/bootlinux/templates/ccache.conf.j2 b/playbooks/roles/bootlinux/templates/ccache.conf.j2 new file mode 100644 index 00000000..8c0a7bf9 --- /dev/null +++ b/playbooks/roles/bootlinux/templates/ccache.conf.j2 @@ -0,0 +1,27 @@ +# kdevops-managed ccache configuration +# Generated automatically - do not edit manually + +# Cache settings +cache_dir = {{ bootlinux_ccache_dir }} +max_size = {{ bootlinux_ccache_max_size }}.0 GiB + +# Performance settings +compression = {{ bootlinux_ccache_compression | lower }} +stats = true + +# Build reproducibility - ensure deterministic compilation +compiler_check = mtime +direct_mode = true +hash_dir = true + +# Debugging (disabled by default for performance) +debug = false + +# File handling +file_clone = false +hard_link = false +inode_cache = true + +# Security +umask = +read_only = false diff --git a/workflows/linux/Kconfig b/workflows/linux/Kconfig index 897cb00f..cbfa8a3c 100644 --- a/workflows/linux/Kconfig +++ b/workflows/linux/Kconfig @@ -188,6 +188,84 @@ config BOOTLINUX_REPRODUCIBLE_BUILDS Recommended for CI/CD environments and when build reproducibility is required. Compatible with both GCC and Clang toolchains. +menu "ccache configuration" + +config BOOTLINUX_CCACHE + bool "Enable ccache for faster kernel builds" + output yaml + default n + help + Enable ccache (compiler cache) to accelerate repeated kernel + builds by caching compilation results. Particularly effective + for incremental builds and CI/CD environments. + + When enabled with Clang (LLVM=1), uses CC="ccache clang". + When enabled with GCC, uses CC="ccache gcc". + + Requires ccache package to be installed on build nodes. + +choice + prompt "ccache configuration mode" + depends on BOOTLINUX_CCACHE + default BOOTLINUX_CCACHE_SYSTEM_WIDE + help + Choose how ccache configuration is managed. + +config BOOTLINUX_CCACHE_SYSTEM_WIDE + bool "Use system-wide ccache configuration" + output yaml + help + Use the system's existing ccache configuration + (~/.cache/ccache or system defaults). Allows manual + tuning via "ccache --set-config" commands. + +config BOOTLINUX_CCACHE_KDEVOPS_MANAGED + bool "kdevops-managed ccache configuration" + output yaml + help + kdevops generates and manages a project-specific ccache + configuration file using TOPDIR_PATH. This ensures + isolation from system-wide ccache settings and allows + project-specific tuning. + + Configuration location: $(TOPDIR_PATH)/.ccache/ccache.conf + +endchoice + +if BOOTLINUX_CCACHE_KDEVOPS_MANAGED + +config BOOTLINUX_CCACHE_MAX_SIZE + int "Maximum cache size (GiB)" + output yaml + default 10 + help + Maximum size for the ccache in GiB. Common values: + - 5 (default ccache setting) + - 10 (recommended for kernel development) + - 20 (large projects/multiple kernels) + - 50 (high-performance development environments) + +config BOOTLINUX_CCACHE_DIR + string "Cache directory" + output yaml + default "$(TOPDIR_PATH)/.ccache/cache" + help + Directory where ccache stores cached compilation results. + Using TOPDIR_PATH ensures project isolation. + +config BOOTLINUX_CCACHE_COMPRESSION + bool "Enable compression" + output yaml + default y + help + Enable compression of cached files to save disk space. + Slightly increases CPU usage but significantly reduces + storage requirements. + +endif # BOOTLINUX_CCACHE_KDEVOPS_MANAGED + +endmenu + choice prompt "Type of development version of Linux to use" default BOOTLINUX_LINUS if !BOOTLINUX_TREE_SET_BY_CLI && !BOOTLINUX_TREE_REF_SET_BY_CLI -- 2.50.1