LLVM: llvm::LoopVectorizationCostModel Class Reference (original) (raw)

LoopVectorizationCostModel - estimates the expected speedups due to vectorization. More...

LoopVectorizationCostModel - estimates the expected speedups due to vectorization.

In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.

Definition at line 966 of file LoopVectorize.cpp.

◆ InstWidening

Decision that was taken during cost calculation for memory instruction.

Definition at line 1126 of file LoopVectorize.cpp.

◆ blockNeedsPredicationForAnyReason()

◆ calculateRegisterUsage()

Returns

Returns information about the register usages of the loop for the given vectorization factors.

Definition at line 5133 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::LoopBlocksDFS::beginRPO(), collectInLoopReductions(), collectUniformsAndScalars(), llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), End, llvm::LoopBlocksDFS::endRPO(), llvm::SmallPtrSetImpl< PtrType >::erase(), llvm::VectorType::get(), llvm::ElementCount::getFixed(), llvm::TargetTransformInfo::getRegisterClassForType(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::TargetTransformInfo::getRegUsageForType(), I, Idx, llvm::SetVector< T, Vector, Set, N >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), isScalarAfterVectorization(), llvm::Type::isTokenTy(), llvm::VectorType::isValidElementType(), llvm::ElementCount::isVector(), LI, llvm::List, LLVM_DEBUG, llvm::LoopVectorizationCostModel::RegisterUsage::LoopInvariantRegs, llvm::make_range(), llvm::LoopVectorizationCostModel::RegisterUsage::MaxLocalUsers, llvm::LoopBlocksDFS::perform(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::RegUsage, llvm::ArrayRef< T >::size(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::size(), llvm::SmallPtrSetImplBase::size(), llvm::SmallVectorBase< Size_T >::size(), TheLoop, ToRemove, TTI, ValuesToIgnore, and VecValuesToIgnore.

Referenced by selectInterleaveCount().

◆ canTruncateToMinimalBitwidth()

◆ canVectorizeReductions()

◆ collectElementTypesForWidening()

Collect all element types in the loop for which widening is needed.

Definition at line 4805 of file LoopVectorize.cpp.

References assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), ElementTypesInLoop, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::LoopVectorizationLegality::isReductionVariable(), Legal, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, TheLoop, useOrderedReductions(), and ValuesToIgnore.

Referenced by llvm::LoopVectorizationPlanner::plan(), and processLoopInVPlanNativePath().

◆ collectInLoopReductions()

Split reductions into those that happen in the loop, and those that happen outside.

In loop reductions are collected into InLoopReductions.

Definition at line 6981 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::SmallVectorBase< Size_T >::empty(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::RecurrenceDescriptor::getReductionOpChain(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::SmallPtrSetImpl< PtrType >::insert(), Legal, LLVM_DEBUG, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, Reduction, TheLoop, and useOrderedReductions().

Referenced by calculateRegisterUsage(), and llvm::LoopVectorizationPlanner::plan().

◆ collectInstsToScalarize()

Collects the instructions to scalarize for each predicated instruction in the loop.

Definition at line 5355 of file LoopVectorize.cpp.

References _, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::begin(), blockNeedsPredicationForAnyReason(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::clear(), CM_Scalarize, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarAfterVectorization(), isScalarWithPredication(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), llvm::predecessors(), and TheLoop.

Referenced by llvm::LoopVectorizationPlanner::plan(), and selectUserVectorizationFactor().

◆ collectUniformsAndScalars()

◆ collectValuesToIgnore()

Collect values we want to ignore in the cost model.

Definition at line 6824 of file LoopVectorize.cpp.

References _, AC, llvm::all_of(), llvm::any_of(), llvm::SmallVectorImpl< T >::append(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallPtrSetImpl< PtrType >::begin(), llvm::LoopBlocksDFS::beginRPO(), llvm::CodeMetrics::collectEphemeralValues(), llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::SmallPtrSetImpl< PtrType >::end(), llvm::LoopBlocksDFS::endRPO(), llvm::RecurrenceDescriptor::getCastInsts(), llvm::InductionDescriptor::getCastInsts(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::getInductionVars(), getInterleavedAccessGroup(), llvm::getLoadStorePointerOperand(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::BasicBlock::getSingleSuccessor(), I, isAccessInterleaved(), IsEmptyBlock(), llvm::LoopVectorizationLegality::isInvariantAddressOfReduction(), Legal, LI, llvm::make_range(), llvm::LoopBlocksDFS::perform(), llvm::BasicBlock::phis(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), Reduction, requiresScalarEpilogue(), llvm::reverse(), llvm::SmallVectorBase< Size_T >::size(), TheLoop, TLI, ValuesToIgnore, VecValuesToIgnore, and llvm::wouldInstructionBeTriviallyDead().

Referenced by llvm::LoopVectorizationPlanner::plan().

◆ computeMaxVF()

Returns

An upper bound for the vectorization factors (both fixed and scalable). If the factors are 0, vectorization and interleaving should be avoided up front.

Definition at line 3929 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), llvm::CM_ScalarEpilogueAllowed, llvm::CM_ScalarEpilogueNotAllowedLowTripLoop, llvm::CM_ScalarEpilogueNotAllowedOptSize, llvm::CM_ScalarEpilogueNotAllowedUsePredicate, llvm::CM_ScalarEpilogueNotNeededUsePredicate, llvm::DataWithEVL, llvm::dbgs(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::FixedScalableVFPair::FixedVF, foldTailByMasking(), llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getConstant(), llvm::LoopBase< BlockT, LoopT >::getExitingBlock(), llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), getMaxVScale(), llvm::ScalarEvolution::getMinusOne(), llvm::FixedScalableVFPair::getNone(), llvm::ScalarEvolution::getOne(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::Type::getScalarSizeInBits(), llvm::PredicatedScalarEvolution::getSE(), llvm::PredicatedScalarEvolution::getSmallConstantMaxTripCount(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::PredicatedScalarEvolution::getSymbolicMaxBackedgeTakenCount(), getTailFoldingStyle(), llvm::SCEV::getType(), llvm::ScalarEvolution::getURemExpr(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::TargetTransformInfo::hasBranchDivergence(), llvm::CmpInst::ICMP_EQ, InterleaveInfo, llvm::InterleavedAccessInfo::invalidateGroupsRequiringScalarEpilogue(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isNonZero(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::TargetTransformInfo::isVScaleKnownToBeAPowerOfTwo(), llvm::SCEV::isZero(), Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, ORE, PSE, llvm::reportVectorizationFailure(), runtimeChecksRequired(), llvm::FixedScalableVFPair::ScalableVF, setTailFoldingStyles(), TheFunction, TheLoop, and useMaskedInterleavedAccesses().

Referenced by llvm::LoopVectorizationPlanner::plan().

◆ expectedCost()

Returns the expected execution cost.

The unit of the cost does not matter because we use the 'cost' units to compare different vector widths. The cost that is returned is not normalized by the factor width.

Definition at line 5520 of file LoopVectorize.cpp.

References addFullyUnrolledInstructionsToIgnore(), llvm::LoopVectorizationLegality::blockNeedsPredication(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::CallingConv::C, llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), foldTailByMasking(), ForceTargetInstructionCost, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::LoopVectorizationLegality::getInductionVars(), getInstructionCost(), llvm::getReciprocalPredBlockProb(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantTripCount(), I, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isFixed(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), Legal, LLVM_DEBUG, PSE, TheLoop, ValuesToIgnore, and VecValuesToIgnore.

Referenced by llvm::LoopVectorizationPlanner::computeBestVF(), selectInterleaveCount(), and selectUserVectorizationFactor().

◆ foldTailByMasking()

◆ foldTailWithEVL()

◆ getCallWideningDecision()

◆ getDivRemSpeculationCost()

Return the costs for our two available strategies for lowering a div/rem operation which requires speculating at least one lane.

First result is for scalarization (will be invalid for scalable vectors); second is for the safe-divisor strategy.

Definition at line 3312 of file LoopVectorize.cpp.

References assert(), llvm::CmpInst::BAD_ICMP_PREDICATE, CostKind, llvm::TargetTransformInfo::getArithmeticInstrCost(), llvm::TargetTransformInfo::getCFInstrCost(), llvm::TargetTransformInfo::getCmpSelInstrCost(), llvm::Type::getInt1Ty(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::TargetTransformInfo::getOperandInfo(), llvm::getReciprocalPredBlockProb(), I, llvm::LoopVectorizationLegality::isInvariant(), llvm::isSafeToSpeculativelyExecute(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::TargetTransformInfo::OperandValueInfo::Kind, Legal, llvm::TargetTransformInfo::OK_AnyValue, llvm::TargetTransformInfo::OK_UniformValue, Operands, and llvm::toVectorTy().

Referenced by getInstructionCost(), and isScalarWithPredication().

◆ getInstructionCost()

Returns the execution time cost of an instruction for a given vector width.

Vector width of one means scalar.

Definition at line 6385 of file LoopVectorize.cpp.

References llvm::all_of(), assert(), llvm::CmpInst::BAD_ICMP_PREDICATE, canTruncateToMinimalBitwidth(), CM_GatherScatter, CM_Interleave, CM_IntrinsicCall, CM_Scalarize, CM_Unknown, CM_VectorCall, CM_Widen, CM_Widen_Reverse, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::contains(), llvm::LoopBase< BlockT, LoopT >::contains(), CostKind, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), foldTailWithEVL(), llvm::TargetTransformInfo::GatherScatter, llvm::IntegerType::get(), llvm::VectorType::get(), llvm::APInt::getAllOnes(), llvm::TargetTransformInfo::getArithmeticInstrCost(), llvm::TargetTransformInfo::getCastInstrCost(), llvm::TargetTransformInfo::getCFInstrCost(), llvm::TargetTransformInfo::getCmpSelInstrCost(), llvm::Type::getContext(), getDivRemSpeculationCost(), llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::getHistogramInfo(), llvm::TargetTransformInfo::getInstructionCost(), getInstructionCost(), llvm::Type::getInt1Ty(), llvm::TargetTransformInfo::getIntrinsicInstrCost(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::getLoadStoreType(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::TargetTransformInfo::getNumberOfParts(), llvm::TargetTransformInfo::getOperandInfo(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), llvm::LoadInst::getPointerOperandType(), getReductionPatternCost(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getScalarizationOverhead(), llvm::Type::getScalarSizeInBits(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::TargetTransformInfo::getShuffleCost(), llvm::BranchInst::getSuccessor(), llvm::Value::getType(), getVectorCallCost(), llvm::Type::getVoidTy(), getWideningCost(), getWideningDecision(), I, llvm::CmpInst::ICMP_EQ, Info, llvm::TargetTransformInfo::Interleave, llvm::RecurrenceDescriptor::isAnyOfRecurrenceKind(), llvm::BranchInst::isConditional(), isDivRemScalarWithPredication(), llvm::LoopVectorizationLegality::isFixedOrderRecurrence(), isInLoopReduction(), llvm::ScalarEvolution::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::SCEV::isOne(), isOptimizableIVTruncate(), isPredicatedInst(), isProfitableToScalarize(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarAfterVectorization(), llvm::ScalarEvolution::isSCEVable(), isUniformAfterVectorization(), llvm::ElementCount::isVector(), llvm::Type::isVectorTy(), llvm::TargetTransformInfo::OperandValueInfo::Kind, Legal, llvm_unreachable, llvm::llvm_unreachable_internal(), llvm::HistogramInfo::Load, llvm::PatternMatch::m_LogicalAnd(), llvm::PatternMatch::m_LogicalOr(), llvm::PatternMatch::m_Value(), llvm::TargetTransformInfo::Masked, llvm::PatternMatch::match(), llvm::TargetTransformInfo::None, llvm::TargetTransformInfo::Normal, llvm::TargetTransformInfo::OK_AnyValue, llvm::TargetTransformInfo::OK_UniformValue, Operands, PSE, RetTy, llvm::TargetTransformInfo::Reversed, RHS, shouldConsiderInvariant(), llvm::TargetTransformInfo::SK_Splice, llvm::TargetTransformInfo::TCC_Free, TheLoop, TLI, and llvm::toVectorTy().

Referenced by expectedCost(), getInstructionCost(), and llvm::VPCostContext::getLegacyCost().

◆ getInterleavedAccessGroup()

◆ getMaxSafeElements()

Return maximum safe number of elements to be processed per vector iteration, which do not prevent store-load forwarding and are safe with regard to the memory dependencies.

Required for EVL-based VPlans to correctly calculate AVL (application vector length) as min(remaining AVL, MaxSafeElements). TODO: need to consider adjusting cost model to use this value as a vectorization factor for EVL-based vectorization.

Definition at line 1477 of file LoopVectorize.cpp.

◆ getMinimalBitwidths()

Returns

The smallest bitwidth each instruction can be represented with. The vector equivalents of these instructions should be truncated to this type.

Definition at line 1064 of file LoopVectorize.cpp.

◆ getReductionPatternCost()

Return the cost of instructions in an inloop reduction pattern, if I is part of that pattern.

Definition at line 5771 of file LoopVectorize.cpp.

References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::at(), CostKind, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::SmallPtrSetImplBase::empty(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::FMulAdd, llvm::VectorType::get(), llvm::TargetTransformInfo::getArithmeticInstrCost(), llvm::TargetTransformInfo::getArithmeticReductionCost(), llvm::TargetTransformInfo::getCastInstrCost(), llvm::TargetTransformInfo::getExtendedReductionCost(), llvm::RecurrenceDescriptor::getFastMathFlags(), llvm::Type::getIntegerBitWidth(), llvm::TargetTransformInfo::getMinMaxReductionCost(), llvm::getMinMaxReductionIntrinsicOp(), llvm::TargetTransformInfo::getMulAccReductionCost(), llvm::Instruction::getOpcode(), llvm::RecurrenceDescriptor::getOpcode(), llvm::User::getOperand(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::Value::getType(), llvm::Value::hasOneUser(), I, llvm::Loop::isLoopInvariant(), llvm::RecurrenceDescriptor::isMinMaxRecurrenceKind(), llvm::ElementCount::isScalar(), llvm::InstructionCost::isValid(), Legal, llvm::PatternMatch::m_Instruction(), llvm::PatternMatch::m_Mul(), llvm::PatternMatch::m_OneUse(), llvm::PatternMatch::m_Value(), llvm::PatternMatch::m_ZExtOrSExt(), llvm::PatternMatch::match(), llvm::TargetTransformInfo::None, TheLoop, useOrderedReductions(), and llvm::Instruction::user_back().

Referenced by getInstructionCost(), getVectorCallCost(), and setVectorizedCallDecision().

◆ getSmallestAndWidestTypes()

Returns

The size (in bits) of the smallest and widest types in the code that needs to be vectorized. We ignore values that remain scalar such as 64 bit loop indices.

Definition at line 4774 of file LoopVectorize.cpp.

References DL, ElementTypesInLoop, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::Function::getDataLayout(), llvm::RecurrenceDescriptor::getMinWidthCastToRecurrenceTypeInBits(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::Type::getScalarSizeInBits(), Legal, and TheFunction.

Referenced by determineVPlanVF().

◆ getTailFoldingStyle()

◆ getVectorCallCost()

Estimate cost of a call instruction CI if it were vectorized with factor VF.

Return the cost of the instruction, including scalarization overhead if it's needed.

Definition at line 2829 of file LoopVectorize.cpp.

References llvm::CallBase::args(), CostKind, llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), getReductionPatternCost(), llvm::Value::getType(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::RecurrenceDescriptor::isFMulAddIntrinsic(), llvm::ElementCount::isScalar(), RetTy, and TLI.

Referenced by getInstructionCost().

◆ getVectorIntrinsicCost()

Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF.

Return the cost of the instruction, including scalarization overhead if it's needed.

Definition at line 2863 of file LoopVectorize.cpp.

References llvm::CallBase::args(), Arguments, assert(), CostKind, llvm::CallBase::getCalledFunction(), llvm::Function::getFunctionType(), llvm::TargetTransformInfo::getIntrinsicInstrCost(), llvm::Value::getType(), llvm::getVectorIntrinsicIDForCall(), maybeVectorizeType(), llvm::FunctionType::param_begin(), llvm::FunctionType::param_end(), RetTy, and TLI.

Referenced by getVectorCallCost(), and setVectorizedCallDecision().

◆ getWideningCost()

◆ getWideningDecision()

Return the cost model decision for the given instruction I and vector width VF.

Return CM_Unknown if this instruction did not pass through the cost modeling.

Definition at line 1176 of file LoopVectorize.cpp.

References assert(), CM_Unknown, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), I, llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::ElementCount::isVector(), and TheLoop.

Referenced by getInstructionCost(), interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().

◆ hasPredStores()

◆ interleavedAccessCanBeWidened()

Returns true if I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles.

Definition at line 3375 of file LoopVectorize.cpp.

References assert(), blockNeedsPredicationForAnyReason(), CM_Unknown, DL, getInterleavedAccessGroup(), llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), getWideningDecision(), hasIrregularType(), I, Idx, isAccessInterleaved(), llvm::TargetTransformInfo::isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedStore(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), isScalarEpilogueAllowed(), Legal, and useMaskedInterleavedAccesses().

Referenced by setCostBasedWideningDecision().

◆ invalidateCostModelingDecisions()

◆ isAccessInterleaved()

◆ isDivRemScalarWithPredication()

◆ isEpilogueVectorizationProfitable()

◆ isInLoopReduction()

◆ isLegalGatherOrScatter()

◆ isLegalMaskedLoad()

◆ isLegalMaskedStore()

◆ isOptimizableIVTruncate()

◆ isPredicatedInst()

Returns true if I is an instruction that needs to be predicated at runtime.

The result is independent of the predication mechanism. Superset of instructions that return true for isScalarWithPredication.

Definition at line 3261 of file LoopVectorize.cpp.

References assert(), llvm::LoopVectorizationLegality::blockNeedsPredication(), blockNeedsPredicationForAnyReason(), llvm::getLoadStorePointerOperand(), I, llvm::LoopVectorizationLegality::isInvariant(), llvm::Loop::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::isSafeToSpeculativelyExecute(), Legal, llvm_unreachable, and TheLoop.

Referenced by getInstructionCost(), llvm::VPRecipeBuilder::handleReplication(), isScalarWithPredication(), and shouldConsiderInvariant().

◆ isProfitableToScalarize()

◆ isScalarAfterVectorization()

◆ isScalarEpilogueAllowed()

◆ isScalarWithPredication()

Returns true if I is an instruction which requires predication and for which our chosen predication strategy is scalarization (i.e.

we don't have an alternate strategy such as masking available). VF is the vectorization factor that will be used to vectorize I.

Definition at line 3219 of file LoopVectorize.cpp.

References CM_Scalarize, llvm::VectorType::get(), getDivRemSpeculationCost(), llvm::getLoadStoreAlignment(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, isDivRemScalarWithPredication(), llvm::TargetTransformInfo::isLegalMaskedGather(), isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedScatter(), isLegalMaskedStore(), isPredicatedInst(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), and Ptr.

Referenced by collectInstsToScalarize(), memoryInstructionCanBeWidened(), and setCostBasedWideningDecision().

◆ isUniformAfterVectorization()

◆ memoryInstructionCanBeWidened()

◆ requiresScalarEpilogue() [1/2]

Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop.

Definition at line 1363 of file LoopVectorize.cpp.

References llvm::dbgs(), EnableEarlyExitVectorization, llvm::LoopBase< BlockT, LoopT >::getExitingBlock(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopVectorizationLegality::hasUncountableEarlyExit(), InterleaveInfo, isScalarEpilogueAllowed(), Legal, LLVM_DEBUG, llvm::InterleavedAccessInfo::requiresScalarEpilogue(), and TheLoop.

Referenced by collectValuesToIgnore(), requiresScalarEpilogue(), and selectInterleaveCount().

◆ requiresScalarEpilogue() [2/2]

◆ runtimeChecksRequired()

Returns

True if runtime checks are required for vectorization, and false otherwise.

Definition at line 3708 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::LoopVectorizationLegality::getLAI(), llvm::PredicatedScalarEvolution::getPredicate(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::LoopAccessInfo::getSymbolicStrides(), llvm::SCEVPredicate::isAlwaysTrue(), Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, ORE, PSE, llvm::reportVectorizationFailure(), and TheLoop.

Referenced by computeMaxVF().

◆ selectInterleaveCount()

Returns

The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

Definition at line 4849 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::bit_floor(), llvm::LoopBase< BlockT, LoopT >::blocks(), calculateRegisterUsage(), llvm::dbgs(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::TargetTransformInfo::enableAggressiveInterleaving(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, expectedCost(), F, foldTailWithEVL(), ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, getEstimatedRuntimeVF(), llvm::LoopBase< BlockT, LoopT >::getLoopDepth(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), llvm::TargetTransformInfo::getNumberOfRegisters(), llvm::LoopVectorizationLegality::getNumLoads(), llvm::LoopVectorizationLegality::getNumStores(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), getSmallBestKnownTC(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::InstructionCost::getValue(), llvm::LoopVectorizationLegality::hasUncountableEarlyExit(), llvm::LoopVectorizationLegality::isSafeForAnyVectorWidth(), llvm::ElementCount::isScalar(), isScalarEpilogueAllowed(), llvm::InstructionCost::isValid(), llvm::ElementCount::isVector(), Legal, LLVM_DEBUG, MaxNestedScalarReductionIC, llvm::RuntimePointerChecking::Need, PSE, Reduction, requiresScalarEpilogue(), SmallLoopCost, and TheLoop.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ selectUserVectorizationFactor()

◆ setCallWideningDecision()

◆ setCostBasedWideningDecision()

Memory access instruction may be vectorized in more than one way.

Form of instruction after vectorization depends on cost. This function takes cost-based decisions for Load/Store instructions and collects them in a map. This decisions map is used for building the lists of loop-uniform and loop-scalar instructions. The calculated cost is saved with widening decision in order to avoid redundant calculations.

Definition at line 6012 of file LoopVectorize.cpp.

References llvm::append_range(), assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), CM_GatherScatter, CM_Interleave, CM_Scalarize, CM_Unknown, CM_Widen, CM_Widen_Reverse, llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallVectorBase< Size_T >::empty(), foldTailByMasking(), llvm::ElementCount::getFixed(), getInterleavedAccessGroup(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), getWideningDecision(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), interleavedAccessCanBeWidened(), isAccessInterleaved(), llvm::LoopVectorizationLegality::isConsecutivePtr(), isLegalGatherOrScatter(), llvm::Loop::isLoopInvariant(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarWithPredication(), llvm::LoopVectorizationLegality::isUniformMemOp(), Legal, memoryInstructionCanBeWidened(), llvm::SmallVectorImpl< T >::pop_back_val(), llvm::TargetTransformInfo::prefersVectorizedAddressing(), Ptr, llvm::SmallVectorTemplateBase< T, bool >::push_back(), setWideningDecision(), and TheLoop.

Referenced by collectUniformsAndScalars().

◆ setTailFoldingStyles()

Selects and saves TailFoldingStyle for 2 options - if IV update may overflow or not.

Parameters

Definition at line 1418 of file LoopVectorize.cpp.

References assert(), llvm::LoopVectorizationLegality::canFoldTailByMasking(), llvm::DataWithEVL, llvm::DataWithoutLaneMask, llvm::dbgs(), llvm::SmallPtrSetImplBase::empty(), llvm::EnableVPlanNativePath, ForceTailFoldingStyle, llvm::LoopVectorizationLegality::getFixedOrderRecurrences(), llvm::TargetTransformInfo::getPreferredTailFoldingStyle(), llvm::TargetTransformInfo::hasActiveVectorLength(), Legal, LLVM_DEBUG, and llvm::None.

Referenced by computeMaxVF().

◆ setVectorizedCallDecision()

A call may be vectorized in different ways depending on whether we have vectorized variants available and whether the target supports masking.

This function analyzes all calls in the function at the supplied VF, makes a decision based on the costs of available options, and stores that decision in a map for use in planning and plan execution.

Definition at line 6199 of file LoopVectorize.cpp.

References llvm::CallBase::args(), assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), CM_IntrinsicCall, CM_Scalarize, CM_VectorCall, CostKind, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::VectorType::get(), llvm::SCEVConstant::getAPInt(), llvm::CallBase::getArgOperand(), llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::Type::getContext(), llvm::Module::getFunction(), llvm::Function::getFunctionType(), llvm::Type::getInt1Ty(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VFDatabase::getMappings(), llvm::Instruction::getModule(), llvm::VFInfo::getParamIndexForOptionalMask(), getReductionPatternCost(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::APInt::getSExtValue(), llvm::TargetTransformInfo::getShuffleCost(), llvm::Value::getType(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::GlobalPredicate, I, Info, llvm::RecurrenceDescriptor::isFMulAddIntrinsic(), llvm::ScalarEvolution::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::CallBase::isNoBuiltin(), llvm::ElementCount::isScalar(), isUniformAfterVectorization(), llvm::ElementCount::isVector(), Legal, llvm::Intrinsic::not_intrinsic, llvm::OMP_Linear, llvm::OMP_Uniform, PSE, llvm::SmallVectorTemplateBase< T, bool >::push_back(), RetTy, setCallWideningDecision(), llvm::TargetTransformInfo::SK_Broadcast, TheLoop, TLI, llvm::toVectorTy(), and llvm::Vector.

Referenced by collectUniformsAndScalars().

◆ setWideningDecision() [1/2]

Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF.

Broadcast this decicion to all instructions inside the group. When interleaving, the cost will only be assigned one instruction, the insert position. For other cases, add the appropriate fraction of the total cost to each instruction. This ensures accurate costs are used, even if the insert position instruction is not used.

Definition at line 1147 of file LoopVectorize.cpp.

References assert(), CM_Interleave, llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::InterleaveGroup< InstTy >::getMember(), llvm::InterleaveGroup< InstTy >::getNumMembers(), I, Idx, and llvm::ElementCount::isVector().

◆ setWideningDecision() [2/2]

◆ shouldConsiderInvariant()

Returns true if Op should be considered invariant and if it is trivially hoistable.

Definition at line 6371 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::LoopBase< BlockT, LoopT >::contains(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::isInvariant(), isPredicatedInst(), Legal, shouldConsiderInvariant(), and TheLoop.

Referenced by getInstructionCost(), and shouldConsiderInvariant().

◆ useOrderedReductions()

◆ usePredicatedReductionSelect()

◆ LoopVectorizationPlanner

◆ AC

◆ CostKind

◆ DB

◆ ElementTypesInLoop

SmallPtrSet<Type *, 16> llvm::LoopVectorizationCostModel::ElementTypesInLoop

◆ Hints

◆ InterleaveInfo

◆ Legal

Vectorization legality.

Definition at line 1758 of file LoopVectorize.cpp.

Referenced by blockNeedsPredicationForAnyReason(), canVectorizeReductions(), collectElementTypesForWidening(), collectInLoopReductions(), collectValuesToIgnore(), computeMaxVF(), expectedCost(), getDivRemSpeculationCost(), getInstructionCost(), getReductionPatternCost(), getSmallestAndWidestTypes(), interleavedAccessCanBeWidened(), isLegalMaskedLoad(), isLegalMaskedStore(), isOptimizableIVTruncate(), isPredicatedInst(), memoryInstructionCanBeWidened(), requiresScalarEpilogue(), runtimeChecksRequired(), selectInterleaveCount(), setCostBasedWideningDecision(), setTailFoldingStyles(), setVectorizedCallDecision(), and shouldConsiderInvariant().

◆ LI

LoopInfo* llvm::LoopVectorizationCostModel::LI

◆ ORE

◆ PSE

◆ TheFunction

◆ TheLoop

Loop* llvm::LoopVectorizationCostModel::TheLoop

The loop that we evaluate.

Definition at line 1749 of file LoopVectorize.cpp.

Referenced by calculateRegisterUsage(), collectElementTypesForWidening(), collectInLoopReductions(), collectInstsToScalarize(), collectValuesToIgnore(), computeMaxVF(), expectedCost(), getInstructionCost(), getReductionPatternCost(), getWideningDecision(), isEpilogueVectorizationProfitable(), isPredicatedInst(), isProfitableToScalarize(), isScalarAfterVectorization(), isUniformAfterVectorization(), requiresScalarEpilogue(), runtimeChecksRequired(), selectInterleaveCount(), setCostBasedWideningDecision(), setVectorizedCallDecision(), and shouldConsiderInvariant().

◆ TLI

◆ TTI

◆ ValuesToIgnore

◆ VecValuesToIgnore

The documentation for this class was generated from the following file:

• lib/Transforms/Vectorize/LoopVectorize.cpp