[gcc(refs/users/meissner/heads/work252-float)] Add conversions between __bfloat16 and float/double.
Michael Meissner via Gcc-cvs <[email protected]>
| Newsgroups | gmane.comp.gcc.cvs |
|---|---|
| Message-ID | <[email protected]> |
https://gcc.gnu.org/g:3ac9f7f2d7cea1037f18634010a99bc003fa4bd2 commit 3ac9f7f2d7cea1037f18634010a99bc003fa4bd2 Author: Michael Meissner <[email protected]> Date: Fri Jul 24 00:54:31 2026 -0400 Add conversions between __bfloat16 and float/double. This patch provides conversions between __bfloat16 and float/double scalars on power10 and power11 systems. This patch also adds support for converting between _Float16 and __bfloat16 types. The previous times these patches were submitted, they did not have this conversion between the 16-bit floating point formats. Unlike the support for _Float16, there is not a single instruction to convert between a __bfloat16 and float/double scalar value on the power10. For normal conversions between a __bfloat16 scalar, we use the vector instruction xvcvbf16d to convert a vector of the even BFmode elements. We use splat words to build the vector because we don't use the odd elements. To convert a __bfloat16 scalar to double and then store it, GCC will generate: lxsihzx 0,0,4 Load up BFmode variable xxspltw 0,0,1 Create splat vector of even elements xvcvbf16spn 0,0 Convert to a V4SF vector xscvspdpn 0,0 Convert element 0 to DFmode memory format stfd 0,0(3) Store the value If we are doing a conversion to float and then storing the float value directly, we can optimize this by just shifting the __bfloat16 value left 16-bits which gives the float memory format. Gcc would generate: To convert a __bfloat16 scalar to float and then store it, GCC will generate: lhz 2,0(4) Load value into a GPR slwi 2,2,16 Shift the value to form at SFmode value stw 2,0(3) Store it To convert a scalar float/double to __bfloat16, we need to use the vector xvcvspbf16 instruction to do the conversion in order to properly round the float/double value. GCC will generation. xscvdpsp 0,0 Convert float scalar to float memory format xvcvspbf16 0,0 Convert vector float to vector __bfloat16 I have committed all of the patches in my backlog (dense math registers, other -mcpu=future instructions, random bug fixes, support for _Float16 and __bfloat16, and optimizations for vector logical operations on power10/power11) into the IBM vendor branch: vendors/ibm/gcc-17-future 2026-07-24 Michael Meissner <[email protected]> gcc/ * config/rs6000/float16.md (FP16_HW): Add BFmode. (VFP16_HW): New mode iterator. (cvt_fp16_to_v4sf): New mode attribute. (cvt_v4sf_to_fp16): Likewise. (FP16_VECTOR4): Likewise. (UNSPEC_BF_SHIFT_LEFT_16BIT): New unspec. (UNSPEC_XXSPLTW_FP16): Likewise. (UNSPEC_XVCVSPBF16_BF): Likewise. (UNSPEC_CVT_V4SF_TO_FP16): Likewise. (extendbf<mode>2): New insns to convert between BFmode and SFmode/DFmode. (xscvdpspn_sf): Likewise. (xscvspdpn_sf): Likewise. (convert_bf_to_sf_store): New insn for converting BFmdoe to SFmode and then storing it. (shift_bf_16bits): Likewise. (trunc<mode>bf): New insns to convert SFmode/DFmode to BFmode. (vsx_xscvdpspn_sf): Likewise. (cvt_fp16_to_v4sf_<mode): Likewise. (cvt_fp16_to_v4sf_<mode>_le): Likewise. (cvt_fp16_to_v4sf_<mode>_be): Likewise. (cvt_v4sf_to_fp16_<mode>): Likewise. (dup_<mode>_to_v4s): Likewise. (xxspltw_<mode>): Likewise. (xvcvbf16spn_bf): Likewise. (xvcvspbf16_bf): Likewise. (trunchfbf2): New insn to convert BFmode to HFmode. (trunchfbf2_mem): Optimize converting BFmode from memory to HFmode. (extendbfhf2): New insn to convert HFmode to BFmode. (extendbfhf2_mem): Optimize converting HFmode from memory to BFmode. * config/rs6000/rs6000-c.cc (rs6000_target_modify_macros): Define __BFLOAT16_HW__ if we have hardware support for __bfloat16. * config/rs6000/rs6000.cc (rs6000_init_hard_regno_mode_ok): Mark that we use VSX arithmetic support for V8BFmode if we are a power10 or later. Diff: --- gcc/config/rs6000/float16.md | 392 +++++++++++++++++++++++++++++++++++++++++- gcc/config/rs6000/rs6000-c.cc | 3 + gcc/config/rs6000/rs6000.cc | 3 + 3 files changed, 397 insertions(+), 1 deletion(-) diff --git a/gcc/config/rs6000/float16.md b/gcc/config/rs6000/float16.md index 7c66c527e8d8..fece52794011 100644 --- a/gcc/config/rs6000/float16.md +++ b/gcc/config/rs6000/float16.md @@ -28,7 +28,25 @@ ;; Mode iterator for 16-bit floating point modes on machines with ;; hardware support both as a scalar and as a vector. -(define_mode_iterator FP16_HW [(HF "TARGET_FLOAT16_HW")]) +(define_mode_iterator FP16_HW [(BF "TARGET_BFLOAT16_HW") + (HF "TARGET_FLOAT16_HW")]) + +(define_mode_iterator VFP16_HW [(V8BF "TARGET_BFLOAT16_HW") + (V8HF "TARGET_FLOAT16_HW")]) + +;; Mode attribute giving the instruction to convert the even +;; V8HFmode or V8BFmode elements to V4SFmode +(define_mode_attr cvt_fp16_to_v4sf [(BF "xvcvbf16spn") + (HF "xvcvhpsp") + (V8BF "xvcvbf16spn") + (V8HF "xvcvhpsp")]) + +;; Mode attribute giving the instruction to convert the V4SFmode +;; elements to the even V8HFmode or V8BFmode elements +(define_mode_attr cvt_v4sf_to_fp16 [(BF "xvcvspbf16") + (HF "xvcvsphp") + (V8BF "xvcvspbf16") + (V8HF "xvcvsphp")]) ;; Mode attribute giving the vector mode for a 16-bit floating point ;; scalar in both upper and lower case. @@ -37,6 +55,20 @@ (define_mode_attr fp16_vector8 [(BF "v8bf") (HF "v8hf")]) + +;; Mode attribute giving the vector mode with 4 16-bit floating point +;; elements given a scalar or 8 element vector. +(define_mode_attr FP16_VECTOR4 [(BF "V4BF") + (HF "V4HF") + (V8BF "V4BF") + (V8HF "V4HF")]) + +;; UNSPEC constants +(define_c_enum "unspec" + [UNSPEC_BF_SHIFT_LEFT_16BIT + UNSPEC_XXSPLTW_FP16 + UNSPEC_XVCVSPBF16_BF + UNSPEC_CVT_V4SF_TO_FP16]) ;; _Float16 and __bfloat16 moves (define_expand "mov<mode>" @@ -179,3 +211,361 @@ "TARGET_FLOAT16_HW" "xscvdphp %x0,%x1" [(set_attr "type" "fpsimple")]) + +;; Convert BFmode to SFmode/DFmode. +;; 3 instructions are generated: +;; XXSPLTW -- duplicate BFmode into all even elements +;; XVCVBF16SPN -- convert even BFmode elements to SFmode +;; XSCVSPNDP -- convert memory format of SFmode to DFmode. +(define_insn_and_split "extendbf<mode>2" + [(set (match_operand:SFDF 0 "vsx_register_operand" "=wa") + (float_extend:SFDF + (match_operand:BF 1 "vsx_register_operand" "v"))) + (clobber (match_scratch:V4SF 2 "=wa"))] + "TARGET_BFLOAT16_HW" + "#" + "&& 1" + [(pc)] +{ + rtx op0 = operands[0]; + rtx op1 = operands[1]; + rtx op2 = operands[2]; + + if (GET_CODE (op2) == SCRATCH) + op2 = gen_reg_rtx (V4SFmode); + + /* XXSPLTW -- duplicate BFmode element into all of the even elements. We + can use splat words because we won't be using the odd elements */ + emit_insn (gen_xxspltw_bf (op2, op1)); + + /* XVCVBF16SPN -- convert even V8BFmode elements to V4SFmode. */ + rtx op2_v8bf = gen_lowpart (V8BFmode, op2); + emit_insn (gen_cvt_fp16_to_v4sf_v8bf (op2, op2_v8bf)); + + /* XSCVSPNDP -- convert single V4SFmode element to DFmode. */ + emit_insn (GET_MODE (op0) == SFmode + ? gen_xscvspdpn_sf (op0, op2) + : gen_vsx_xscvspdpn (op0, op2)); + + DONE; +} + [(set_attr "type" "fpsimple") + (set_attr "length" "12")]) + +;; Convert a SFmode scalar represented as DFmode to elements 0 and 1 of +;; V4SFmode. +(define_insn "xscvdpspn_sf" + [(set (match_operand:V4SF 0 "vsx_register_operand" "=wa") + (unspec:V4SF [(match_operand:SF 1 "vsx_register_operand" "wa")] + UNSPEC_VSX_CVSPDP))] + "VECTOR_UNIT_VSX_P (SFmode)" + "xscvdpspn %x0,%x1" + [(set_attr "type" "fp")]) + +;; Convert element 0 of a V4SFmode to scalar SFmode (which on the +;; PowerPC uses the DFmode encoding). +(define_insn "xscvspdpn_sf" + [(set (match_operand:SF 0 "vsx_register_operand" "=wa") + (unspec:SF [(match_operand:V4SF 1 "vsx_register_operand" "wa")] + UNSPEC_VSX_CVSPDPN))] + "TARGET_XSCVSPDPN" + "xscvspdpn %x0,%x1" + [(set_attr "type" "fp")]) + +;; Optimize storing the conversion of BFmode to SFmode by shifting the +;; BFmode left 16 bits. +(define_insn_and_split "*convert_bf_to_sf_store" + [(set (match_operand:SF 0 "memory_operand" "=m") + (float_extend:SF + (match_operand:BF 1 "int_reg_operand" "r"))) + (clobber (match_scratch:SF 2 "=r"))] + "TARGET_FLOAT16" + "#" + "&& 1" + [(set (match_dup 2) + (unspec:SF [(match_dup 1)] UNSPEC_BF_SHIFT_LEFT_16BIT)) + (set (match_dup 0) + (match_dup 2))] +{ + if (GET_CODE (operands[2]) == SCRATCH) + operands[2] = gen_reg_rtx (SFmode); +} + [(set_attr "length" "8") + (set_attr "type" "store")]) + +;; Shfit a BFmode left 16 bits getting a SFmode memory value in the GPR +(define_insn "*shift_bf_16bits" + [(set (match_operand:SF 0 "int_reg_operand" "=r") + (unspec:SF + [(match_operand:BF 1 "int_reg_operand" "r")] + UNSPEC_BF_SHIFT_LEFT_16BIT))] + "TARGET_FLOAT16" + "slwi %0,%1,16" + [(set_attr "type" "shift")]) + +;; Convert SFmode/DFmode to BFmode. +;; 2 instructions are generated: +;; XSCVDPSPN -- convert SFmode/DFmode scalar to V4SFmode +;; XVCVSPBF16 -- convert V4SFmode to even V8BFmode + +(define_insn_and_split "trunc<mode>bf2" + [(set (match_operand:BF 0 "vsx_register_operand" "=wa") + (float_truncate:BF + (match_operand:SFDF 1 "vsx_register_operand" "wa"))) + (clobber (match_scratch:V4SF 2 "=wa"))] + "TARGET_BFLOAT16_HW" + "#" + "&& 1" + [(pc)] +{ + rtx op0 = operands[0]; + rtx op1 = operands[1]; + rtx op2 = operands[2]; + + if (GET_CODE (op2) == SCRATCH) + op2 = gen_reg_rtx (V4SFmode); + + emit_insn (GET_MODE (op1) == SFmode + ? gen_xscvdpspn_sf (op2, op1) + : gen_vsx_xscvdpspn (op2, op1)); + + emit_insn (gen_xvcvspbf16_bf (op0, op2)); + DONE; +} + [(set_attr "type" "fpsimple")]) + +(define_insn "vsx_xscvdpspn_sf" + [(set (match_operand:V4SF 0 "vsx_register_operand" "=wa") + (unspec:V4SF [(match_operand:SF 1 "vsx_register_operand" "wa")] + UNSPEC_VSX_CVDPSPN))] + "TARGET_XSCVDPSPN" + "xscvdpspn %x0,%x1" + [(set_attr "type" "fp")]) + + +;; Convert the even elements of a vector 16-bit floating point to +;; V4SFmode. Deal with little endian vs. big endian element ordering +;; in identifying which elements are converted. + +(define_expand "cvt_fp16_to_v4sf_<mode>" + [(set (match_operand:V4SF 0 "vsx_register_operand") + (float_extend:V4SF + (vec_select:<FP16_VECTOR4> + (match_operand:VFP16_HW 1 "vsx_register_operand") + (parallel [(match_dup 2) + (match_dup 3) + (match_dup 4) + (match_dup 5)]))))] + "" +{ + int endian_adjust = WORDS_BIG_ENDIAN ? 0 : 1; + operands[2] = GEN_INT (0 + endian_adjust); + operands[3] = GEN_INT (2 + endian_adjust); + operands[4] = GEN_INT (4 + endian_adjust); + operands[5] = GEN_INT (6 + endian_adjust); +}) + +(define_insn "*cvt_fp16_to_v4sf_<mode>_le" + [(set (match_operand:V4SF 0 "vsx_register_operand") + (float_extend:V4SF + (vec_select:<FP16_VECTOR4> + (match_operand:VFP16_HW 1 "vsx_register_operand") + (parallel [(const_int 1) + (const_int 3) + (const_int 5) + (const_int 7)]))))] + "!WORDS_BIG_ENDIAN" + "<cvt_fp16_to_v4sf> %x0,%x1" + [(set_attr "type" "vecfloat")]) + +(define_insn "*cvt_fp16_to_v4sf_<mode>_be" + [(set (match_operand:V4SF 0 "vsx_register_operand") + (float_extend:V4SF + (vec_select:<FP16_VECTOR4> + (match_operand:VFP16_HW 1 "vsx_register_operand") + (parallel [(const_int 0) + (const_int 2) + (const_int 4) + (const_int 6)]))))] + "WORDS_BIG_ENDIAN" + "<cvt_fp16_to_v4sf> %x0,%x1" + [(set_attr "type" "vecfloat")]) + +;; Convert a V4SFmode vector a 16-bit floating point value. We only +;; care about the 2nd element. This is used to convert BFmode to +;; HFmode or HFmode to BFmode. + +(define_insn "cvt_v4sf_to_fp16_<mode>" + [(set (match_operand:FP16_HW 0 "vsx_register_operand") + (unspec:FP16_HW [(match_operand:V4SF 1 "vsx_register_operand")] + UNSPEC_CVT_V4SF_TO_FP16))] + "" + "<cvt_v4sf_to_fp16> %x0,%x1" + [(set_attr "type" "vecfloat")]) + +;; Duplicate and convert a 16-bit floating point scalar to V4SFmode. + +(define_insn_and_split "*dup_<mode>_to_v4sf" + [(set (match_operand:V4SF 0 "vsx_register_operand" "=wa") + (vec_duplicate:V4SF + (float_extend:SF + (match_operand:FP16_HW 1 "vsx_register_operand" "wa"))))] + "" + "#" + "&& 1" + [(pc)] +{ + rtx op0 = operands[0]; + rtx op1 = operands[1]; + rtx op0_vfp16 = gen_lowpart (<FP16_VECTOR8>mode, op0); + + emit_insn (gen_xxspltw_<mode> (op0, op1)); + emit_insn (gen_cvt_fp16_to_v4sf_<fp16_vector8> (op0, op0_vfp16)); + DONE; +} + [(set_attr "length" "8") + (set_attr "type" "vecperm")]) + +;; Duplicate a HF/BF value so it can be used for xvcvhpspn/xvcvbf16spn. +;; Because xvcvhpspn/xvcvbf16spn only uses the even elements, we can +;; use xxspltw instead of vspltw. This has the advantage that the +;; register allocator can use any of the 64 VSX registers instead of +;; being limited to the 32 Altivec registers that VSPLTH would require. + +(define_insn "xxspltw_<mode>" + [(set (match_operand:V4SF 0 "vsx_register_operand" "=wa") + (unspec:V4SF [(match_operand:FP16_HW 1 "vsx_register_operand" "wa")] + UNSPEC_XXSPLTW_FP16))] + "" + "xxspltw %x0,%x1,1" + [(set_attr "type" "vecperm")]) + +;; Convert a V4SFmode vector to a 16-bit floating point scalar. We +;; only care about the 2nd V4SFmode element, which is the element we +;; converted the 16-bit scalar (4th element) to V4SFmode to do the +;; operation, and converted it back. + +(define_insn "xvcvspbf16_bf" + [(set (match_operand:BF 0 "vsx_register_operand" "=wa") + (unspec:BF [(match_operand:V4SF 1 "vsx_register_operand" "wa")] + UNSPEC_XVCVSPBF16_BF))] + "TARGET_BFLOAT16_HW" + "xvcvspbf16 %x0,%x1" + [(set_attr "type" "vecfloat")]) + + +;; Convert HFmode to BFmode +;; The instructions generated are: +;; XXSPLTIW tmp1,r1 +;; XVCVHPSP tmp1,tmp1 +;; XVCVSPHPN r0,tmp1 + +(define_insn_and_split "trunchfbf2" + [(set (match_operand:BF 0 "vsx_register_operand" "=wa") + (float_truncate:BF + (match_operand:HF 1 "vsx_register_operand" "wa"))) + (clobber (match_scratch:V4SF 2 "=wa"))] + "TARGET_BFLOAT16_HW" + "#" + "&& 1" + [(pc)] +{ + rtx tmp_v4sf = operands[2]; + + if (GET_CODE (tmp_v4sf) == SCRATCH) + tmp_v4sf = gen_reg_rtx (V4SFmode); + + rtx tmp_v8hf = gen_lowpart (V8HFmode, tmp_v4sf); + + emit_insn (gen_xxspltw_hf (tmp_v4sf, operands[1])); + emit_insn (gen_cvt_fp16_to_v4sf_v8hf (tmp_v4sf, tmp_v8hf)); + emit_insn (gen_cvt_v4sf_to_fp16_bf (operands[0], tmp_v4sf)); + DONE; +} + [(set_attr "length" "12") + (set_attr "type" "vecfloat")]) + +;; Optimize converting HFmode to BFmode if the BFmode is coming from +;; memory. We can eliminate the XXSPLTW instruction since the load +;; zeros out the other elements. + +(define_insn_and_split "*trunchfbf2_mem" + [(set (match_operand:BF 0 "vsx_register_operand" "=wa") + (float_truncate:BF + (match_operand:HF 1 "memory_operand" "Z"))) + (clobber (match_scratch:V4SF 2 "=wa"))] + "TARGET_BFLOAT16_HW" + "#" + "&& reload_completed" + [(pc)] +{ + rtx tmp_v4sf = operands[2]; + unsigned int r = reg_or_subregno (tmp_v4sf); + rtx tmp_v8hf = gen_rtx_REG (V8HFmode, r); + rtx tmp_hf = gen_rtx_REG (HFmode, r); + + emit_move_insn (tmp_hf, operands[1]); + emit_insn (gen_cvt_fp16_to_v4sf_v8hf (tmp_v4sf, tmp_v8hf)); + emit_insn (gen_cvt_v4sf_to_fp16_bf (operands[0], tmp_v4sf)); + DONE; +} + [(set_attr "length" "12") + (set_attr "type" "vecfloat")]) + +;; Convert BFmode to HFmode +;; The instructions generated are: +;; XXSPLTIW tmp1,r1 +;; XVCVBF16SP tmp1,tmp1 +;; XVCVSPBF16N r0,tmp1 + +(define_insn_and_split "expandbfhf2" + [(set (match_operand:HF 0 "vsx_register_operand" "=wa") + (float_extend:HF + (match_operand:BF 1 "vsx_register_operand" "wa"))) + (clobber (match_scratch:V4SF 2 "=wa"))] + "TARGET_BFLOAT16_HW" + "#" + "&& 1" + [(pc)] +{ + rtx tmp_v4sf = operands[2]; + + if (GET_CODE (tmp_v4sf) == SCRATCH) + tmp_v4sf = gen_reg_rtx (V4SFmode); + + rtx tmp_v8bf = gen_lowpart (V8BFmode, tmp_v4sf); + + emit_insn (gen_xxspltw_bf (tmp_v4sf, operands[1])); + emit_insn (gen_cvt_fp16_to_v4sf_v8bf (tmp_v4sf, tmp_v8bf)); + emit_insn (gen_cvt_v4sf_to_fp16_hf (operands[0], tmp_v4sf)); + DONE; +} + [(set_attr "length" "12") + (set_attr "type" "vecfloat")]) + +;; Optimize converting BFmode to HFmode if the HFmode is coming from +;; memory. We can eliminate the XXSPLTW instruction since the load +;; zeros out the other elements. + +(define_insn_and_split "*extendbfhf2_mem" + [(set (match_operand:HF 0 "vsx_register_operand" "=wa") + (float_extend:HF + (match_operand:BF 1 "memory_operand" "Z"))) + (clobber (match_scratch:V4SF 2 "=wa"))] + "TARGET_BFLOAT16_HW" + "#" + "&& reload_completed" + [(pc)] +{ + rtx tmp_v4sf = operands[2]; + unsigned int r = reg_or_subregno (tmp_v4sf); + rtx tmp_v8bf = gen_rtx_REG (V8BFmode, r); + rtx tmp_bf = gen_rtx_REG (BFmode, r); + + emit_move_insn (tmp_bf, operands[1]); + emit_insn (gen_cvt_fp16_to_v4sf_v8bf (tmp_v4sf, tmp_v8bf)); + emit_insn (gen_cvt_v4sf_to_fp16_hf (operands[0], tmp_v4sf)); + DONE; +} + [(set_attr "length" "12") + (set_attr "type" "vecfloat")]) diff --git a/gcc/config/rs6000/rs6000-c.cc b/gcc/config/rs6000/rs6000-c.cc index e8eecabefce8..32b09d935ae0 100644 --- a/gcc/config/rs6000/rs6000-c.cc +++ b/gcc/config/rs6000/rs6000-c.cc @@ -591,6 +591,9 @@ rs6000_target_modify_macros (bool define_p, HOST_WIDE_INT flags) if ((flags & OPTION_MASK_P9_VECTOR) != 0) rs6000_define_or_undefine_macro (define_p, "__FLOAT16_HW__"); + + if ((flags & OPTION_MASK_POWER10) != 0) + rs6000_define_or_undefine_macro (define_p, "__BFLOAT16_HW__"); } /* Tell the user if we are targeting CELL. */ if (rs6000_cpu == PROCESSOR_CELL) diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc index d3f4b8c5002e..b8e262f59c73 100644 --- a/gcc/config/rs6000/rs6000.cc +++ b/gcc/config/rs6000/rs6000.cc @@ -3003,6 +3003,9 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p) if (TARGET_P9_VECTOR) rs6000_vector_unit[V8HFmode] = VECTOR_VSX; + + if (TARGET_POWER10) + rs6000_vector_unit[V8BFmode] = VECTOR_VSX; } /* DFmode, see if we want to use the VSX unit. Memory is handled