Reland [llvm] Add support for llvm IR atomicrmw fminimum/fmaximum instructions #137701
+679
−73
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
llvmbot
added
backend:AArch64
backend:AMDGPU
llvm:globalisel
llvm:instcombine
llvm:support
llvm:SelectionDAG
|
@llvm/pr-subscribers-llvm-transforms @llvm/pr-subscribers-backend-aarch64 Author: Jonathan Thackray (jthackray) ChangesThis patch adds support for LLVM IR atomicrmw These mirror the Future changes will allow this LLVM IR to be lowered to specialised assembler instructions on suitable targets, such as AArch64. Patch is 65.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/137701.diff 38 Files Affected:
diff --git a/llvm/docs/GlobalISel/GenericOpcode.rst b/llvm/docs/GlobalISel/GenericOpcode.rst
index 5291b42d80870..987d19e2f6ce1 100644
--- a/llvm/docs/GlobalISel/GenericOpcode.rst
+++ b/llvm/docs/GlobalISel/GenericOpcode.rst
@@ -922,7 +922,8 @@ operands.
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX,
G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX,
- G_ATOMICRMW_FMIN, G_ATOMICRMW_UINC_WRAP,
+ G_ATOMICRMW_FMIN, G_ATOMICRMW_FMAXIMUM,
+ G_ATOMICRMW_FMINIMUM, G_ATOMICRMW_UINC_WRAP,
G_ATOMICRMW_UDEC_WRAP, G_ATOMICRMW_USUB_COND,
G_ATOMICRMW_USUB_SAT
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5bd1d29487139..70c8c4d68443e 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -11598,6 +11598,8 @@ operation. The operation must be one of the following keywords:
- fsub
- fmax
- fmin
+- fmaximum
+- fminimum
- uinc_wrap
- udec_wrap
- usub_cond
@@ -11607,7 +11609,7 @@ For most of these operations, the type of '<value>' must be an integer
type whose bit width is a power of two greater than or equal to eight
and less than or equal to a target-specific size limit. For xchg, this
may also be a floating point or a pointer type with the same size constraints
-as integers. For fadd/fsub/fmax/fmin, this must be a floating-point
+as integers. For fadd/fsub/fmax/fmin/fmaximum/fminimum, this must be a floating-point
or fixed vector of floating-point type. The type of the '``<pointer>``'
operand must be a pointer to that type. If the ``atomicrmw`` is marked
as ``volatile``, then the optimizer is not allowed to modify the
@@ -11648,8 +11650,10 @@ operation argument:
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
-- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
-- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
+- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*` intrinsic)
+- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*` intrinsic)
+- fmaximum: ``*ptr = maximum(*ptr, val)`` (match the `llvm.maximum.*` intrinsic)
+- fminimum: ``*ptr = minimum(*ptr, val)`` (match the `llvm.minimum.*` intrinsic)
- uinc_wrap: ``*ptr = (*ptr u>= val) ? 0 : (*ptr + 1)`` (increment value with wraparound to zero when incremented above input value)
- udec_wrap: ``*ptr = ((*ptr == 0) || (*ptr u> val)) ? val : (*ptr - 1)`` (decrement with wraparound to input value when decremented below zero).
- usub_cond: ``*ptr = (*ptr u>= val) ? *ptr - val : *ptr`` (subtract only if no unsigned overflow).
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 9e2e63cffdf82..f4906f7cbf0eb 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -72,6 +72,9 @@ Changes to LLVM infrastructure
themselves (i.e., the `TargetIntrinsicInfo` class).
* Fix Microsoft demangling of string literals to be stricter
(#GH129970))
+* Added the support for ``fmaximum`` and ``fminimum`` in ``atomicrmw`` instruction. The
+ comparison is expected to match the behavior of ``llvm.maximum.*`` and
+ ``llvm.minimum.*`` respectively.
Changes to building LLVM
------------------------
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index 596531c93a949..6857944e6875f 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -393,6 +393,12 @@ typedef enum {
LLVMAtomicRMWBinOpUSubCond, /**<Subtracts the value only if no unsigned
overflow */
LLVMAtomicRMWBinOpUSubSat, /**<Subtracts the value, clamping to zero */
+ LLVMAtomicRMWBinOpFMaximum, /**< Sets the value if it's greater than the
+ original using an floating point comparison and
+ return the old one */
+ LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the
+ original using an floating point comparison and
+ return the old one */
} LLVMAtomicRMWBinOp;
typedef enum {
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index a8f9c71781701..c7e4bdf3ff811 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -276,6 +276,8 @@ enum Kind {
kw_umin,
kw_fmax,
kw_fmin,
+ kw_fmaximum,
+ kw_fminimum,
kw_uinc_wrap,
kw_udec_wrap,
kw_usub_cond,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 92b6e68d9d0a7..9317e6aa793de 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -504,7 +504,9 @@ enum RMWOperations {
RMW_UINC_WRAP = 15,
RMW_UDEC_WRAP = 16,
RMW_USUB_COND = 17,
- RMW_USUB_SAT = 18
+ RMW_USUB_SAT = 18,
+ RMW_FMAXIMUM = 19,
+ RMW_FMINIMUM = 20,
};
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
index f9dcbeb370bef..d5f621a74b6e7 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
@@ -1663,6 +1663,42 @@ class MachineIRBuilder {
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
MachineMemOperand &MMO);
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAXIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point maximum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMINIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point minimum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_USUB_COND Addr, Val, MMO`.
///
/// Atomically replace the value at \p Addr with the original value minus \p
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index 5d9e96e3b5286..1042318343987 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -1376,6 +1376,8 @@ enum NodeType {
ATOMIC_LOAD_FSUB,
ATOMIC_LOAD_FMAX,
ATOMIC_LOAD_FMIN,
+ ATOMIC_LOAD_FMAXIMUM,
+ ATOMIC_LOAD_FMINIMUM,
ATOMIC_LOAD_UINC_WRAP,
ATOMIC_LOAD_UDEC_WRAP,
ATOMIC_LOAD_USUB_COND,
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index b279ca90be9e8..568c5e8155314 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1517,6 +1517,8 @@ class MemSDNode : public SDNode {
case ISD::ATOMIC_LOAD_FSUB:
case ISD::ATOMIC_LOAD_FMAX:
case ISD::ATOMIC_LOAD_FMIN:
+ case ISD::ATOMIC_LOAD_FMAXIMUM:
+ case ISD::ATOMIC_LOAD_FMINIMUM:
case ISD::ATOMIC_LOAD_UINC_WRAP:
case ISD::ATOMIC_LOAD_UDEC_WRAP:
case ISD::ATOMIC_LOAD_USUB_COND:
@@ -1603,6 +1605,8 @@ class AtomicSDNode : public MemSDNode {
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMAXIMUM ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMINIMUM ||
N->getOpcode() == ISD::ATOMIC_LOAD_UINC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_UDEC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index 95f0ef875fc07..c164f76eb335b 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -751,6 +751,14 @@ class AtomicRMWInst : public Instruction {
/// \p minnum matches the behavior of \p llvm.minnum.*.
FMin,
+ /// *p = maximum(old, v)
+ /// \p maximum matches the behavior of \p llvm.maximum.*.
+ FMaximum,
+
+ /// *p = minimum(old, v)
+ /// \p minimum matches the behavior of \p llvm.minimum.*.
+ FMinimum,
+
/// Increment one up to a maximum value.
/// *p = (old u>= v) ? 0 : (old + 1)
UIncWrap,
@@ -812,6 +820,8 @@ class AtomicRMWInst : public Instruction {
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
return true;
default:
return false;
diff --git a/llvm/include/llvm/Support/TargetOpcodes.def b/llvm/include/llvm/Support/TargetOpcodes.def
index 5ef3707b81fe9..771c318da817d 100644
--- a/llvm/include/llvm/Support/TargetOpcodes.def
+++ b/llvm/include/llvm/Support/TargetOpcodes.def
@@ -426,6 +426,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAXIMUM)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMINIMUM)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UINC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UDEC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_USUB_COND)
diff --git a/llvm/include/llvm/Target/GenericOpcodes.td b/llvm/include/llvm/Target/GenericOpcodes.td
index e134bab61bf63..c041c3cdfca5b 100644
--- a/llvm/include/llvm/Target/GenericOpcodes.td
+++ b/llvm/include/llvm/Target/GenericOpcodes.td
@@ -1351,6 +1351,8 @@ def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAXIMUM : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMINIMUM : G_ATOMICRMW_OP;
def G_ATOMICRMW_UINC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_UDEC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_USUB_COND : G_ATOMICRMW_OP;
diff --git a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
index c8c0eeb57099a..cb55a6fb112c5 100644
--- a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
+++ b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
@@ -266,6 +266,8 @@ def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
+def : GINodeEquiv<G_ATOMICRMW_FMAXIMUM, atomic_load_fmaximum>;
+def : GINodeEquiv<G_ATOMICRMW_FMINIMUM, atomic_load_fminimum>;
def : GINodeEquiv<G_ATOMICRMW_UINC_WRAP, atomic_load_uinc_wrap>;
def : GINodeEquiv<G_ATOMICRMW_UDEC_WRAP, atomic_load_udec_wrap>;
def : GINodeEquiv<G_ATOMICRMW_USUB_COND, atomic_load_usub_cond>;
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 3515a7da71075..b28a8b118de7a 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -781,6 +781,10 @@ def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmaximum : SDNode<"ISD::ATOMIC_LOAD_FMAXIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fminimum : SDNode<"ISD::ATOMIC_LOAD_FMINIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_uinc_wrap : SDNode<"ISD::ATOMIC_LOAD_UINC_WRAP", SDTAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_udec_wrap : SDNode<"ISD::ATOMIC_LOAD_UDEC_WRAP", SDTAtomic2,
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 4d25b12c9ab06..ce813e1d7b1c4 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -749,6 +749,8 @@ lltok::Kind LLLexer::LexIdentifier() {
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
+ KEYWORD(fmaximum);
+ KEYWORD(fminimum);
KEYWORD(uinc_wrap);
KEYWORD(udec_wrap);
KEYWORD(usub_cond);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 370d124dc42b4..956364af95625 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -8626,6 +8626,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
Operation = AtomicRMWInst::FMin;
IsFP = true;
break;
+ case lltok::kw_fmaximum:
+ Operation = AtomicRMWInst::FMaximum;
+ IsFP = true;
+ break;
+ case lltok::kw_fminimum:
+ Operation = AtomicRMWInst::FMinimum;
+ IsFP = true;
+ break;
}
Lex.Lex(); // Eat the operation.
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 1d7aa189026a5..0e75c44333af5 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1356,6 +1356,10 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
+ case bitc::RMW_FMAXIMUM:
+ return AtomicRMWInst::FMaximum;
+ case bitc::RMW_FMINIMUM:
+ return AtomicRMWInst::FMinimum;
case bitc::RMW_UINC_WRAP:
return AtomicRMWInst::UIncWrap;
case bitc::RMW_UDEC_WRAP:
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 73bed85c65b3d..0893cf7aec8c0 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -678,6 +678,10 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
+ case AtomicRMWInst::FMaximum:
+ return bitc::RMW_FMAXIMUM;
+ case AtomicRMWInst::FMinimum:
+ return bitc::RMW_FMINIMUM;
case AtomicRMWInst::UIncWrap:
return bitc::RMW_UINC_WRAP;
case AtomicRMWInst::UDecWrap:
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index a3e9700fa3089..c376de877ac7d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -931,6 +931,8 @@ static Value *performMaskedAtomicOp(AtomicRMWInst::BinOp Op,
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMin:
case AtomicRMWInst::FMax:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::UIncWrap:
case AtomicRMWInst::UDecWrap:
case AtomicRMWInst::USubCond:
@@ -1819,6 +1821,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::UMin:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub:
case AtomicRMWInst::UIncWrap:
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 5a4ee12d730cc..a4069ad896a92 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3435,6 +3435,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
case AtomicRMWInst::FMin:
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
break;
+ case AtomicRMWInst::FMaximum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
Opcode = TargetOpcode::G_ATOMICRMW_UINC_WRAP;
break;
diff --git a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
index 47e3351bb33d7..121d7e80251c7 100644
--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -1157,6 +1157,22 @@ MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
MMO);
}
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAXIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMINIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
MachineInstrBuilder
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
return buildInstr(TargetOpcode::G_FENCE)
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index 8cc434a49afa2..b79d77da61369 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -9101,6 +9101,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
Opcode == ISD::ATOMIC_LOAD_UMAX || Opcode == ISD::ATOMIC_LOAD_FADD ||
Opcode == ISD::ATOMIC_LOAD_FSUB || Opcode == ISD::ATOMIC_LOAD_FMAX ||
Opcode == ISD::ATOMIC_LOAD_FMIN ||
+ Opcode == ISD::ATOMIC_LOAD_FMINIMUM ||
+ Opcode == ISD::ATOMIC_LOAD_FMAXIMUM ||
Opcode == ISD::ATOMIC_LOAD_UINC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_UDEC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 25b47838e2c17..d1f5ed0d0b9ea 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5081,6 +5081,12 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
+ case AtomicRMWInst::FMaximum:
+ NT = ISD::ATOMIC_LOAD_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ NT = ISD::ATOMIC_LOAD_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
NT = ISD::ATOMIC_...
[truncated]
|
|
@llvm/pr-subscribers-backend-amdgpu Author: Jonathan Thackray (jthackray) ChangesThis patch adds support for LLVM IR atomicrmw These mirror the Future changes will allow this LLVM IR to be lowered to specialised assembler instructions on suitable targets, such as AArch64. Patch is 65.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/137701.diff 38 Files Affected:
diff --git a/llvm/docs/GlobalISel/GenericOpcode.rst b/llvm/docs/GlobalISel/GenericOpcode.rst
index 5291b42d80870..987d19e2f6ce1 100644
--- a/llvm/docs/GlobalISel/GenericOpcode.rst
+++ b/llvm/docs/GlobalISel/GenericOpcode.rst
@@ -922,7 +922,8 @@ operands.
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX,
G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX,
- G_ATOMICRMW_FMIN, G_ATOMICRMW_UINC_WRAP,
+ G_ATOMICRMW_FMIN, G_ATOMICRMW_FMAXIMUM,
+ G_ATOMICRMW_FMINIMUM, G_ATOMICRMW_UINC_WRAP,
G_ATOMICRMW_UDEC_WRAP, G_ATOMICRMW_USUB_COND,
G_ATOMICRMW_USUB_SAT
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5bd1d29487139..70c8c4d68443e 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -11598,6 +11598,8 @@ operation. The operation must be one of the following keywords:
- fsub
- fmax
- fmin
+- fmaximum
+- fminimum
- uinc_wrap
- udec_wrap
- usub_cond
@@ -11607,7 +11609,7 @@ For most of these operations, the type of '<value>' must be an integer
type whose bit width is a power of two greater than or equal to eight
and less than or equal to a target-specific size limit. For xchg, this
may also be a floating point or a pointer type with the same size constraints
-as integers. For fadd/fsub/fmax/fmin, this must be a floating-point
+as integers. For fadd/fsub/fmax/fmin/fmaximum/fminimum, this must be a floating-point
or fixed vector of floating-point type. The type of the '``<pointer>``'
operand must be a pointer to that type. If the ``atomicrmw`` is marked
as ``volatile``, then the optimizer is not allowed to modify the
@@ -11648,8 +11650,10 @@ operation argument:
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
-- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
-- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
+- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*` intrinsic)
+- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*` intrinsic)
+- fmaximum: ``*ptr = maximum(*ptr, val)`` (match the `llvm.maximum.*` intrinsic)
+- fminimum: ``*ptr = minimum(*ptr, val)`` (match the `llvm.minimum.*` intrinsic)
- uinc_wrap: ``*ptr = (*ptr u>= val) ? 0 : (*ptr + 1)`` (increment value with wraparound to zero when incremented above input value)
- udec_wrap: ``*ptr = ((*ptr == 0) || (*ptr u> val)) ? val : (*ptr - 1)`` (decrement with wraparound to input value when decremented below zero).
- usub_cond: ``*ptr = (*ptr u>= val) ? *ptr - val : *ptr`` (subtract only if no unsigned overflow).
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 9e2e63cffdf82..f4906f7cbf0eb 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -72,6 +72,9 @@ Changes to LLVM infrastructure
themselves (i.e., the `TargetIntrinsicInfo` class).
* Fix Microsoft demangling of string literals to be stricter
(#GH129970))
+* Added the support for ``fmaximum`` and ``fminimum`` in ``atomicrmw`` instruction. The
+ comparison is expected to match the behavior of ``llvm.maximum.*`` and
+ ``llvm.minimum.*`` respectively.
Changes to building LLVM
------------------------
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index 596531c93a949..6857944e6875f 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -393,6 +393,12 @@ typedef enum {
LLVMAtomicRMWBinOpUSubCond, /**<Subtracts the value only if no unsigned
overflow */
LLVMAtomicRMWBinOpUSubSat, /**<Subtracts the value, clamping to zero */
+ LLVMAtomicRMWBinOpFMaximum, /**< Sets the value if it's greater than the
+ original using an floating point comparison and
+ return the old one */
+ LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the
+ original using an floating point comparison and
+ return the old one */
} LLVMAtomicRMWBinOp;
typedef enum {
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index a8f9c71781701..c7e4bdf3ff811 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -276,6 +276,8 @@ enum Kind {
kw_umin,
kw_fmax,
kw_fmin,
+ kw_fmaximum,
+ kw_fminimum,
kw_uinc_wrap,
kw_udec_wrap,
kw_usub_cond,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 92b6e68d9d0a7..9317e6aa793de 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -504,7 +504,9 @@ enum RMWOperations {
RMW_UINC_WRAP = 15,
RMW_UDEC_WRAP = 16,
RMW_USUB_COND = 17,
- RMW_USUB_SAT = 18
+ RMW_USUB_SAT = 18,
+ RMW_FMAXIMUM = 19,
+ RMW_FMINIMUM = 20,
};
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
index f9dcbeb370bef..d5f621a74b6e7 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
@@ -1663,6 +1663,42 @@ class MachineIRBuilder {
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
MachineMemOperand &MMO);
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAXIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point maximum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMINIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point minimum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_USUB_COND Addr, Val, MMO`.
///
/// Atomically replace the value at \p Addr with the original value minus \p
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index 5d9e96e3b5286..1042318343987 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -1376,6 +1376,8 @@ enum NodeType {
ATOMIC_LOAD_FSUB,
ATOMIC_LOAD_FMAX,
ATOMIC_LOAD_FMIN,
+ ATOMIC_LOAD_FMAXIMUM,
+ ATOMIC_LOAD_FMINIMUM,
ATOMIC_LOAD_UINC_WRAP,
ATOMIC_LOAD_UDEC_WRAP,
ATOMIC_LOAD_USUB_COND,
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index b279ca90be9e8..568c5e8155314 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1517,6 +1517,8 @@ class MemSDNode : public SDNode {
case ISD::ATOMIC_LOAD_FSUB:
case ISD::ATOMIC_LOAD_FMAX:
case ISD::ATOMIC_LOAD_FMIN:
+ case ISD::ATOMIC_LOAD_FMAXIMUM:
+ case ISD::ATOMIC_LOAD_FMINIMUM:
case ISD::ATOMIC_LOAD_UINC_WRAP:
case ISD::ATOMIC_LOAD_UDEC_WRAP:
case ISD::ATOMIC_LOAD_USUB_COND:
@@ -1603,6 +1605,8 @@ class AtomicSDNode : public MemSDNode {
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMAXIMUM ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMINIMUM ||
N->getOpcode() == ISD::ATOMIC_LOAD_UINC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_UDEC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index 95f0ef875fc07..c164f76eb335b 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -751,6 +751,14 @@ class AtomicRMWInst : public Instruction {
/// \p minnum matches the behavior of \p llvm.minnum.*.
FMin,
+ /// *p = maximum(old, v)
+ /// \p maximum matches the behavior of \p llvm.maximum.*.
+ FMaximum,
+
+ /// *p = minimum(old, v)
+ /// \p minimum matches the behavior of \p llvm.minimum.*.
+ FMinimum,
+
/// Increment one up to a maximum value.
/// *p = (old u>= v) ? 0 : (old + 1)
UIncWrap,
@@ -812,6 +820,8 @@ class AtomicRMWInst : public Instruction {
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
return true;
default:
return false;
diff --git a/llvm/include/llvm/Support/TargetOpcodes.def b/llvm/include/llvm/Support/TargetOpcodes.def
index 5ef3707b81fe9..771c318da817d 100644
--- a/llvm/include/llvm/Support/TargetOpcodes.def
+++ b/llvm/include/llvm/Support/TargetOpcodes.def
@@ -426,6 +426,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAXIMUM)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMINIMUM)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UINC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UDEC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_USUB_COND)
diff --git a/llvm/include/llvm/Target/GenericOpcodes.td b/llvm/include/llvm/Target/GenericOpcodes.td
index e134bab61bf63..c041c3cdfca5b 100644
--- a/llvm/include/llvm/Target/GenericOpcodes.td
+++ b/llvm/include/llvm/Target/GenericOpcodes.td
@@ -1351,6 +1351,8 @@ def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAXIMUM : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMINIMUM : G_ATOMICRMW_OP;
def G_ATOMICRMW_UINC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_UDEC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_USUB_COND : G_ATOMICRMW_OP;
diff --git a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
index c8c0eeb57099a..cb55a6fb112c5 100644
--- a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
+++ b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
@@ -266,6 +266,8 @@ def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
+def : GINodeEquiv<G_ATOMICRMW_FMAXIMUM, atomic_load_fmaximum>;
+def : GINodeEquiv<G_ATOMICRMW_FMINIMUM, atomic_load_fminimum>;
def : GINodeEquiv<G_ATOMICRMW_UINC_WRAP, atomic_load_uinc_wrap>;
def : GINodeEquiv<G_ATOMICRMW_UDEC_WRAP, atomic_load_udec_wrap>;
def : GINodeEquiv<G_ATOMICRMW_USUB_COND, atomic_load_usub_cond>;
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 3515a7da71075..b28a8b118de7a 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -781,6 +781,10 @@ def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmaximum : SDNode<"ISD::ATOMIC_LOAD_FMAXIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fminimum : SDNode<"ISD::ATOMIC_LOAD_FMINIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_uinc_wrap : SDNode<"ISD::ATOMIC_LOAD_UINC_WRAP", SDTAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_udec_wrap : SDNode<"ISD::ATOMIC_LOAD_UDEC_WRAP", SDTAtomic2,
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 4d25b12c9ab06..ce813e1d7b1c4 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -749,6 +749,8 @@ lltok::Kind LLLexer::LexIdentifier() {
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
+ KEYWORD(fmaximum);
+ KEYWORD(fminimum);
KEYWORD(uinc_wrap);
KEYWORD(udec_wrap);
KEYWORD(usub_cond);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 370d124dc42b4..956364af95625 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -8626,6 +8626,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
Operation = AtomicRMWInst::FMin;
IsFP = true;
break;
+ case lltok::kw_fmaximum:
+ Operation = AtomicRMWInst::FMaximum;
+ IsFP = true;
+ break;
+ case lltok::kw_fminimum:
+ Operation = AtomicRMWInst::FMinimum;
+ IsFP = true;
+ break;
}
Lex.Lex(); // Eat the operation.
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 1d7aa189026a5..0e75c44333af5 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1356,6 +1356,10 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
+ case bitc::RMW_FMAXIMUM:
+ return AtomicRMWInst::FMaximum;
+ case bitc::RMW_FMINIMUM:
+ return AtomicRMWInst::FMinimum;
case bitc::RMW_UINC_WRAP:
return AtomicRMWInst::UIncWrap;
case bitc::RMW_UDEC_WRAP:
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 73bed85c65b3d..0893cf7aec8c0 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -678,6 +678,10 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
+ case AtomicRMWInst::FMaximum:
+ return bitc::RMW_FMAXIMUM;
+ case AtomicRMWInst::FMinimum:
+ return bitc::RMW_FMINIMUM;
case AtomicRMWInst::UIncWrap:
return bitc::RMW_UINC_WRAP;
case AtomicRMWInst::UDecWrap:
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index a3e9700fa3089..c376de877ac7d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -931,6 +931,8 @@ static Value *performMaskedAtomicOp(AtomicRMWInst::BinOp Op,
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMin:
case AtomicRMWInst::FMax:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::UIncWrap:
case AtomicRMWInst::UDecWrap:
case AtomicRMWInst::USubCond:
@@ -1819,6 +1821,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::UMin:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub:
case AtomicRMWInst::UIncWrap:
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 5a4ee12d730cc..a4069ad896a92 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3435,6 +3435,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
case AtomicRMWInst::FMin:
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
break;
+ case AtomicRMWInst::FMaximum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
Opcode = TargetOpcode::G_ATOMICRMW_UINC_WRAP;
break;
diff --git a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
index 47e3351bb33d7..121d7e80251c7 100644
--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -1157,6 +1157,22 @@ MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
MMO);
}
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAXIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMINIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
MachineInstrBuilder
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
return buildInstr(TargetOpcode::G_FENCE)
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index 8cc434a49afa2..b79d77da61369 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -9101,6 +9101,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
Opcode == ISD::ATOMIC_LOAD_UMAX || Opcode == ISD::ATOMIC_LOAD_FADD ||
Opcode == ISD::ATOMIC_LOAD_FSUB || Opcode == ISD::ATOMIC_LOAD_FMAX ||
Opcode == ISD::ATOMIC_LOAD_FMIN ||
+ Opcode == ISD::ATOMIC_LOAD_FMINIMUM ||
+ Opcode == ISD::ATOMIC_LOAD_FMAXIMUM ||
Opcode == ISD::ATOMIC_LOAD_UINC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_UDEC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 25b47838e2c17..d1f5ed0d0b9ea 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5081,6 +5081,12 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
+ case AtomicRMWInst::FMaximum:
+ NT = ISD::ATOMIC_LOAD_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ NT = ISD::ATOMIC_LOAD_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
NT = ISD::ATOMIC_...
[truncated]
|
|
@llvm/pr-subscribers-llvm-globalisel Author: Jonathan Thackray (jthackray) ChangesThis patch adds support for LLVM IR atomicrmw These mirror the Future changes will allow this LLVM IR to be lowered to specialised assembler instructions on suitable targets, such as AArch64. Patch is 65.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/137701.diff 38 Files Affected:
diff --git a/llvm/docs/GlobalISel/GenericOpcode.rst b/llvm/docs/GlobalISel/GenericOpcode.rst
index 5291b42d80870..987d19e2f6ce1 100644
--- a/llvm/docs/GlobalISel/GenericOpcode.rst
+++ b/llvm/docs/GlobalISel/GenericOpcode.rst
@@ -922,7 +922,8 @@ operands.
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX,
G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX,
- G_ATOMICRMW_FMIN, G_ATOMICRMW_UINC_WRAP,
+ G_ATOMICRMW_FMIN, G_ATOMICRMW_FMAXIMUM,
+ G_ATOMICRMW_FMINIMUM, G_ATOMICRMW_UINC_WRAP,
G_ATOMICRMW_UDEC_WRAP, G_ATOMICRMW_USUB_COND,
G_ATOMICRMW_USUB_SAT
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5bd1d29487139..70c8c4d68443e 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -11598,6 +11598,8 @@ operation. The operation must be one of the following keywords:
- fsub
- fmax
- fmin
+- fmaximum
+- fminimum
- uinc_wrap
- udec_wrap
- usub_cond
@@ -11607,7 +11609,7 @@ For most of these operations, the type of '<value>' must be an integer
type whose bit width is a power of two greater than or equal to eight
and less than or equal to a target-specific size limit. For xchg, this
may also be a floating point or a pointer type with the same size constraints
-as integers. For fadd/fsub/fmax/fmin, this must be a floating-point
+as integers. For fadd/fsub/fmax/fmin/fmaximum/fminimum, this must be a floating-point
or fixed vector of floating-point type. The type of the '``<pointer>``'
operand must be a pointer to that type. If the ``atomicrmw`` is marked
as ``volatile``, then the optimizer is not allowed to modify the
@@ -11648,8 +11650,10 @@ operation argument:
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
-- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
-- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
+- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*` intrinsic)
+- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*` intrinsic)
+- fmaximum: ``*ptr = maximum(*ptr, val)`` (match the `llvm.maximum.*` intrinsic)
+- fminimum: ``*ptr = minimum(*ptr, val)`` (match the `llvm.minimum.*` intrinsic)
- uinc_wrap: ``*ptr = (*ptr u>= val) ? 0 : (*ptr + 1)`` (increment value with wraparound to zero when incremented above input value)
- udec_wrap: ``*ptr = ((*ptr == 0) || (*ptr u> val)) ? val : (*ptr - 1)`` (decrement with wraparound to input value when decremented below zero).
- usub_cond: ``*ptr = (*ptr u>= val) ? *ptr - val : *ptr`` (subtract only if no unsigned overflow).
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 9e2e63cffdf82..f4906f7cbf0eb 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -72,6 +72,9 @@ Changes to LLVM infrastructure
themselves (i.e., the `TargetIntrinsicInfo` class).
* Fix Microsoft demangling of string literals to be stricter
(#GH129970))
+* Added the support for ``fmaximum`` and ``fminimum`` in ``atomicrmw`` instruction. The
+ comparison is expected to match the behavior of ``llvm.maximum.*`` and
+ ``llvm.minimum.*`` respectively.
Changes to building LLVM
------------------------
diff --git a/llvm/include/llvm-c/Core.h b/llvm/include/llvm-c/Core.h
index 596531c93a949..6857944e6875f 100644
--- a/llvm/include/llvm-c/Core.h
+++ b/llvm/include/llvm-c/Core.h
@@ -393,6 +393,12 @@ typedef enum {
LLVMAtomicRMWBinOpUSubCond, /**<Subtracts the value only if no unsigned
overflow */
LLVMAtomicRMWBinOpUSubSat, /**<Subtracts the value, clamping to zero */
+ LLVMAtomicRMWBinOpFMaximum, /**< Sets the value if it's greater than the
+ original using an floating point comparison and
+ return the old one */
+ LLVMAtomicRMWBinOpFMinimum, /**< Sets the value if it's smaller than the
+ original using an floating point comparison and
+ return the old one */
} LLVMAtomicRMWBinOp;
typedef enum {
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index a8f9c71781701..c7e4bdf3ff811 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -276,6 +276,8 @@ enum Kind {
kw_umin,
kw_fmax,
kw_fmin,
+ kw_fmaximum,
+ kw_fminimum,
kw_uinc_wrap,
kw_udec_wrap,
kw_usub_cond,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 92b6e68d9d0a7..9317e6aa793de 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -504,7 +504,9 @@ enum RMWOperations {
RMW_UINC_WRAP = 15,
RMW_UDEC_WRAP = 16,
RMW_USUB_COND = 17,
- RMW_USUB_SAT = 18
+ RMW_USUB_SAT = 18,
+ RMW_FMAXIMUM = 19,
+ RMW_FMINIMUM = 20,
};
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
index f9dcbeb370bef..d5f621a74b6e7 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h
@@ -1663,6 +1663,42 @@ class MachineIRBuilder {
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
MachineMemOperand &MMO);
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAXIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point maximum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
+ /// Build and insert `OldValRes<def> = G_ATOMICRMW_FMINIMUM Addr, Val, MMO`.
+ ///
+ /// Atomically replace the value at \p Addr with the floating point minimum of
+ /// \p Val and the original value. Puts the original value from \p Addr in \p
+ /// OldValRes.
+ ///
+ /// \pre setBasicBlock or setMI must have been called.
+ /// \pre \p OldValRes must be a generic virtual register.
+ /// \pre \p Addr must be a generic virtual register with pointer type.
+ /// \pre \p OldValRes, and \p Val must be generic virtual registers of the
+ /// same type.
+ ///
+ /// \return a MachineInstrBuilder for the newly created instruction.
+ MachineInstrBuilder buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr,
+ const SrcOp &Val,
+ MachineMemOperand &MMO);
+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_USUB_COND Addr, Val, MMO`.
///
/// Atomically replace the value at \p Addr with the original value minus \p
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index 5d9e96e3b5286..1042318343987 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -1376,6 +1376,8 @@ enum NodeType {
ATOMIC_LOAD_FSUB,
ATOMIC_LOAD_FMAX,
ATOMIC_LOAD_FMIN,
+ ATOMIC_LOAD_FMAXIMUM,
+ ATOMIC_LOAD_FMINIMUM,
ATOMIC_LOAD_UINC_WRAP,
ATOMIC_LOAD_UDEC_WRAP,
ATOMIC_LOAD_USUB_COND,
diff --git a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
index b279ca90be9e8..568c5e8155314 100644
--- a/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
+++ b/llvm/include/llvm/CodeGen/SelectionDAGNodes.h
@@ -1517,6 +1517,8 @@ class MemSDNode : public SDNode {
case ISD::ATOMIC_LOAD_FSUB:
case ISD::ATOMIC_LOAD_FMAX:
case ISD::ATOMIC_LOAD_FMIN:
+ case ISD::ATOMIC_LOAD_FMAXIMUM:
+ case ISD::ATOMIC_LOAD_FMINIMUM:
case ISD::ATOMIC_LOAD_UINC_WRAP:
case ISD::ATOMIC_LOAD_UDEC_WRAP:
case ISD::ATOMIC_LOAD_USUB_COND:
@@ -1603,6 +1605,8 @@ class AtomicSDNode : public MemSDNode {
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMAXIMUM ||
+ N->getOpcode() == ISD::ATOMIC_LOAD_FMINIMUM ||
N->getOpcode() == ISD::ATOMIC_LOAD_UINC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_UDEC_WRAP ||
N->getOpcode() == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/include/llvm/IR/Instructions.h b/llvm/include/llvm/IR/Instructions.h
index 95f0ef875fc07..c164f76eb335b 100644
--- a/llvm/include/llvm/IR/Instructions.h
+++ b/llvm/include/llvm/IR/Instructions.h
@@ -751,6 +751,14 @@ class AtomicRMWInst : public Instruction {
/// \p minnum matches the behavior of \p llvm.minnum.*.
FMin,
+ /// *p = maximum(old, v)
+ /// \p maximum matches the behavior of \p llvm.maximum.*.
+ FMaximum,
+
+ /// *p = minimum(old, v)
+ /// \p minimum matches the behavior of \p llvm.minimum.*.
+ FMinimum,
+
/// Increment one up to a maximum value.
/// *p = (old u>= v) ? 0 : (old + 1)
UIncWrap,
@@ -812,6 +820,8 @@ class AtomicRMWInst : public Instruction {
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
return true;
default:
return false;
diff --git a/llvm/include/llvm/Support/TargetOpcodes.def b/llvm/include/llvm/Support/TargetOpcodes.def
index 5ef3707b81fe9..771c318da817d 100644
--- a/llvm/include/llvm/Support/TargetOpcodes.def
+++ b/llvm/include/llvm/Support/TargetOpcodes.def
@@ -426,6 +426,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAXIMUM)
+HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMINIMUM)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UINC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UDEC_WRAP)
HANDLE_TARGET_OPCODE(G_ATOMICRMW_USUB_COND)
diff --git a/llvm/include/llvm/Target/GenericOpcodes.td b/llvm/include/llvm/Target/GenericOpcodes.td
index e134bab61bf63..c041c3cdfca5b 100644
--- a/llvm/include/llvm/Target/GenericOpcodes.td
+++ b/llvm/include/llvm/Target/GenericOpcodes.td
@@ -1351,6 +1351,8 @@ def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMAXIMUM : G_ATOMICRMW_OP;
+def G_ATOMICRMW_FMINIMUM : G_ATOMICRMW_OP;
def G_ATOMICRMW_UINC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_UDEC_WRAP : G_ATOMICRMW_OP;
def G_ATOMICRMW_USUB_COND : G_ATOMICRMW_OP;
diff --git a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
index c8c0eeb57099a..cb55a6fb112c5 100644
--- a/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
+++ b/llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td
@@ -266,6 +266,8 @@ def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
+def : GINodeEquiv<G_ATOMICRMW_FMAXIMUM, atomic_load_fmaximum>;
+def : GINodeEquiv<G_ATOMICRMW_FMINIMUM, atomic_load_fminimum>;
def : GINodeEquiv<G_ATOMICRMW_UINC_WRAP, atomic_load_uinc_wrap>;
def : GINodeEquiv<G_ATOMICRMW_UDEC_WRAP, atomic_load_udec_wrap>;
def : GINodeEquiv<G_ATOMICRMW_USUB_COND, atomic_load_usub_cond>;
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 3515a7da71075..b28a8b118de7a 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -781,6 +781,10 @@ def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fmaximum : SDNode<"ISD::ATOMIC_LOAD_FMAXIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
+def atomic_load_fminimum : SDNode<"ISD::ATOMIC_LOAD_FMINIMUM", SDTFPAtomic2,
+ [SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_uinc_wrap : SDNode<"ISD::ATOMIC_LOAD_UINC_WRAP", SDTAtomic2,
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
def atomic_load_udec_wrap : SDNode<"ISD::ATOMIC_LOAD_UDEC_WRAP", SDTAtomic2,
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 4d25b12c9ab06..ce813e1d7b1c4 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -749,6 +749,8 @@ lltok::Kind LLLexer::LexIdentifier() {
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
+ KEYWORD(fmaximum);
+ KEYWORD(fminimum);
KEYWORD(uinc_wrap);
KEYWORD(udec_wrap);
KEYWORD(usub_cond);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 370d124dc42b4..956364af95625 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -8626,6 +8626,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
Operation = AtomicRMWInst::FMin;
IsFP = true;
break;
+ case lltok::kw_fmaximum:
+ Operation = AtomicRMWInst::FMaximum;
+ IsFP = true;
+ break;
+ case lltok::kw_fminimum:
+ Operation = AtomicRMWInst::FMinimum;
+ IsFP = true;
+ break;
}
Lex.Lex(); // Eat the operation.
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 1d7aa189026a5..0e75c44333af5 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -1356,6 +1356,10 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
+ case bitc::RMW_FMAXIMUM:
+ return AtomicRMWInst::FMaximum;
+ case bitc::RMW_FMINIMUM:
+ return AtomicRMWInst::FMinimum;
case bitc::RMW_UINC_WRAP:
return AtomicRMWInst::UIncWrap;
case bitc::RMW_UDEC_WRAP:
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 73bed85c65b3d..0893cf7aec8c0 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -678,6 +678,10 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
+ case AtomicRMWInst::FMaximum:
+ return bitc::RMW_FMAXIMUM;
+ case AtomicRMWInst::FMinimum:
+ return bitc::RMW_FMINIMUM;
case AtomicRMWInst::UIncWrap:
return bitc::RMW_UINC_WRAP;
case AtomicRMWInst::UDecWrap:
diff --git a/llvm/lib/CodeGen/AtomicExpandPass.cpp b/llvm/lib/CodeGen/AtomicExpandPass.cpp
index a3e9700fa3089..c376de877ac7d 100644
--- a/llvm/lib/CodeGen/AtomicExpandPass.cpp
+++ b/llvm/lib/CodeGen/AtomicExpandPass.cpp
@@ -931,6 +931,8 @@ static Value *performMaskedAtomicOp(AtomicRMWInst::BinOp Op,
case AtomicRMWInst::FSub:
case AtomicRMWInst::FMin:
case AtomicRMWInst::FMax:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::UIncWrap:
case AtomicRMWInst::UDecWrap:
case AtomicRMWInst::USubCond:
@@ -1819,6 +1821,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
case AtomicRMWInst::UMin:
case AtomicRMWInst::FMax:
case AtomicRMWInst::FMin:
+ case AtomicRMWInst::FMaximum:
+ case AtomicRMWInst::FMinimum:
case AtomicRMWInst::FAdd:
case AtomicRMWInst::FSub:
case AtomicRMWInst::UIncWrap:
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 5a4ee12d730cc..a4069ad896a92 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -3435,6 +3435,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
case AtomicRMWInst::FMin:
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
break;
+ case AtomicRMWInst::FMaximum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ Opcode = TargetOpcode::G_ATOMICRMW_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
Opcode = TargetOpcode::G_ATOMICRMW_UINC_WRAP;
break;
diff --git a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
index 47e3351bb33d7..121d7e80251c7 100644
--- a/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp
@@ -1157,6 +1157,22 @@ MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
MMO);
}
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMaximum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAXIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
+MachineInstrBuilder
+MachineIRBuilder::buildAtomicRMWFMinimum(const DstOp &OldValRes,
+ const SrcOp &Addr, const SrcOp &Val,
+ MachineMemOperand &MMO) {
+ return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMINIMUM, OldValRes, Addr,
+ Val, MMO);
+}
+
MachineInstrBuilder
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
return buildInstr(TargetOpcode::G_FENCE)
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
index 8cc434a49afa2..b79d77da61369 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp
@@ -9101,6 +9101,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
Opcode == ISD::ATOMIC_LOAD_UMAX || Opcode == ISD::ATOMIC_LOAD_FADD ||
Opcode == ISD::ATOMIC_LOAD_FSUB || Opcode == ISD::ATOMIC_LOAD_FMAX ||
Opcode == ISD::ATOMIC_LOAD_FMIN ||
+ Opcode == ISD::ATOMIC_LOAD_FMINIMUM ||
+ Opcode == ISD::ATOMIC_LOAD_FMAXIMUM ||
Opcode == ISD::ATOMIC_LOAD_UINC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_UDEC_WRAP ||
Opcode == ISD::ATOMIC_LOAD_USUB_COND ||
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 25b47838e2c17..d1f5ed0d0b9ea 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -5081,6 +5081,12 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
+ case AtomicRMWInst::FMaximum:
+ NT = ISD::ATOMIC_LOAD_FMAXIMUM;
+ break;
+ case AtomicRMWInst::FMinimum:
+ NT = ISD::ATOMIC_LOAD_FMINIMUM;
+ break;
case AtomicRMWInst::UIncWrap:
NT = ISD::ATOMIC_...
[truncated]
|
|
The only change different from the original (#136759, now reverted) is updating landed just before I merged, which caused a test failure in my PR when I merged (and I hadn't rebased since Friday). |
arsenm
approved these changes
Apr 29, 2025
Pierre-vh
approved these changes
Apr 29, 2025
…tructions (#136759) This patch adds support for LLVM IR atomicrmw `fmaximum` and `fminimum` instructions. These mirror the `llvm.maximum.*` and `llvm.minimum.*` instructions, but are atomic and use IEEE754 2019 handling for NaNs, which is different to `fmax` and `fmin`. See: https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic for more details. Future changes will allow this LLVM IR to be lowered to specialised assembler instructions on suitable targets, such as AArch64.
jthackray
force-pushed
the
users/jthackray/new-atomics
branch
from
50efa5b to
92ea098
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds support for LLVM IR atomicrmw
fmaximumandfminimuminstructions.These mirror the
llvm.maximum.*andllvm.minimum.*instructions, but are atomic and use IEEE754 2019 handling for NaNs, which is different tofmaxandfmin. See:https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic
for more details.
Future changes will allow this LLVM IR to be lowered to specialised assembler instructions on suitable targets, such as AArch64.