LLVM: llvm::InnerLoopVectorizer Class Reference (original) (raw)

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Public Member Functions
	InnerLoopVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetLibraryInfo TLI, const TargetTransformInfo TTI, AssumptionCache AC, OptimizationRemarkEmitter ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality LVL, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks, VPlan &Plan)

virtual	~InnerLoopVectorizer ()=default

virtual BasicBlock *	createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)
	Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

void	fixVectorizedLoop (VPTransformState &State)
	Fix the vectorized code, taking care of header phi's, and more.

bool	areSafetyChecksAdded ()

void	scalarizeInstruction (const Instruction Instr, VPReplicateRecipe RepRecipe, const VPLane &Lane, VPTransformState &State)
	A helper function to scalarize a single Instruction in the innermost loop.

void	fixNonInductionPHIs (VPTransformState &State)
	Fix the non-induction PHIs in Plan.

Value *	getTripCount () const
	Returns the original loop trip count.

void	setTripCount (Value *TC)
	Used to set the trip count after ILV's construction and after the preheader block has been executed.

Value *	getInductionAdditionalBypassValue (PHINode *OrigPhi) const
	induction header phi.

BasicBlock *	getAdditionalBypassBlock () const
	Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Protected Member Functions
void	sinkScalarOperands (Instruction *PredInst)
	Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.

Value *	getOrCreateVectorTripCount (BasicBlock *InsertBlock)
	Returns (and creates if needed) the trip count of the widened loop.

void	emitIterationCountCheck (BasicBlock *Bypass)
	Emit a bypass check to see if the vector trip count is zero, including if it overflows.

BasicBlock *	emitSCEVChecks (BasicBlock *Bypass)
	Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

BasicBlock *	emitMemRuntimeChecks (BasicBlock *Bypass)
	Emit bypass checks to check any memory assumptions we may have made.

void	createVectorLoopSkeleton (StringRef Prefix)
	Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader.

void	createInductionAdditionalBypassValues (const SCEV2ValueTy &ExpandedSCEVs, Value *MainVectorTripCount)
	Create and record the values for induction variables to resume coming from the additional bypass block.

virtual void	printDebugTracesAtStart ()
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

virtual void	printDebugTracesAtEnd ()

void	introduceCheckBlockInVPlan (BasicBlock *CheckIRBB)
	Introduces a new VPIRBasicBlock for CheckIRBB to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader.

Protected Attributes
Loop *	OrigLoop
	The original loop.

PredicatedScalarEvolution &	PSE
	A wrapper around ScalarEvolution used to add runtime SCEV checks.

LoopInfo *	LI
	Loop Info.

DominatorTree *	DT
	Dominator Tree.

const TargetLibraryInfo *	TLI
	Target Library Info.

const TargetTransformInfo *	TTI
	Target Transform Info.

AssumptionCache *	AC
	Assumption Cache.

OptimizationRemarkEmitter *	ORE
	Interface to emit optimization remarks.

ElementCount	VF
	The vectorization SIMD factor to use.

ElementCount	MinProfitableTripCount

unsigned	UF
	The vectorization unroll factor to use.

IRBuilder	Builder
	The builder that we use.

BasicBlock *	LoopVectorPreHeader
	The vector-loop preheader.

BasicBlock *	LoopScalarPreHeader
	The scalar-loop preheader.

BasicBlock *	LoopMiddleBlock
	Middle Block between the vector and the scalar.

SmallVector< BasicBlock *, 4 >	LoopBypassBlocks
	A list of all bypass blocks. The first block is the entry of the loop.

SmallVector< Instruction *, 4 >	PredicatedInstructions
	Store instructions that were predicated.

Value *	TripCount = nullptr
	Trip count of the original loop.

Value *	VectorTripCount = nullptr
	Trip count of the widened loop (TripCount - TripCount % (VF*UF))

LoopVectorizationLegality *	Legal
	The legality analysis.

LoopVectorizationCostModel *	Cost
	The profitablity analysis.

bool	AddedSafetyChecks = false

BlockFrequencyInfo *	BFI
	BFI and PSI are used to check for profile guided size optimizations.

ProfileSummaryInfo *	PSI

bool	OptForSizeBasedOnProfile

GeneratedRTChecks &	RTChecks
	Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

DenseMap< PHINode , Value >	Induction2AdditionalBypassValue
	Mapping of induction phis to their additional bypass values.

BasicBlock *	AdditionalBypassBlock = nullptr
	The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

VPlan &	Plan

VPBlockBase *	VectorPHVPB
	The vector preheader block of Plan, used as target for check blocks introduced during skeleton creation.

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

It inserts an epilogue loop for handling loops that don't have iteration counts that are known to be a multiple of the vectorization factor.
It handles the code generation for reduction variables.
Scalarization (implementation using scalars) of un-vectorizable instructions. InnerLoopVectorizer does not perform any vectorization-legality checks, and relies on the caller to check for the different legality aspects. The InnerLoopVectorizer relies on the LoopVectorizationLegality class to provide information about the induction and reduction variables that were found to a given vectorization factor.

Definition at line 466 of file LoopVectorize.cpp.

llvm::InnerLoopVectorizer::InnerLoopVectorizer ( Loop * OrigLoop, PredicatedScalarEvolution & PSE, LoopInfo * LI, DominatorTree * DT, const TargetLibraryInfo * TLI, const TargetTransformInfo * TTI, AssumptionCache * AC, OptimizationRemarkEmitter * ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality * LVL, LoopVectorizationCostModel * CM, BlockFrequencyInfo * BFI, ProfileSummaryInfo * PSI, GeneratedRTChecks & RTChecks, VPlan & Plan )

inline

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )	virtualdefault

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )	inline

◆ createInductionAdditionalBypassValues()

void InnerLoopVectorizer::createInductionAdditionalBypassValues ( const SCEV2ValueTy & ExpandedSCEVs, Value * MainVectorTripCount )	protected

Create and record the values for induction variables to resume coming from the additional bypass block.

Definition at line 2680 of file LoopVectorize.cpp.

References assert(), emitTransformedIndex(), getAdditionalBypassBlock(), getExpandedStep(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::LoopVectorizationLegality::getPrimaryInduction(), II, Induction2AdditionalBypassValue, Legal, llvm::IRBuilderBase::setFastMathFlags(), and llvm::Value::setName().

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton().

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( const SCEV2ValueTy & ExpandedSCEVs)	virtual

◆ createVectorLoopSkeleton()

void InnerLoopVectorizer::createVectorLoopSkeleton ( StringRef Prefix)	protected

Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader.

Definition at line 2626 of file LoopVectorize.cpp.

References assert(), DT, llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::VPlan::getMiddleBlock(), llvm::VPlan::getScalarPreheader(), llvm::BasicBlock::getTerminator(), llvm::LoopBase< BlockT, LoopT >::getUniqueLatchExitBlock(), llvm::ElementCount::isVector(), LI, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorPreHeader, OrigLoop, Plan, replaceVPBBWithIRVPBB(), llvm::SplitBlock(), and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ emitIterationCountCheck()

void InnerLoopVectorizer::emitIterationCountCheck ( BasicBlock * Bypass)	protected

Emit a bypass check to see if the vector trip count is zero, including if it overflows.

Definition at line 2470 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), Builder, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateBinaryIntrinsic(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::DataAndControlFlowWithoutRuntimeCheck, DT, llvm::IRBuilderBase::getFalse(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::CmpInst::getInversePredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::IRBuilderBase::getTrue(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, introduceCheckBlockInVPlan(), isIndvarOverflowCheckKnownFalse(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isVector(), LHS, LI, LoopBypassBlocks, LoopVectorPreHeader, MinItersBypassWeights, MinProfitableTripCount, llvm::None, OrigLoop, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), PSE, llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::SplitBlock(), UF, and VF.

Referenced by createVectorizedLoopSkeleton().

◆ emitMemRuntimeChecks()

Emit bypass checks to check any memory assumptions we may have made.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 2574 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), DEBUG_TYPE, llvm::OptimizationRemarkEmitter::emit(), llvm::EnableVPlanNativePath, llvm::LoopVectorizeHints::FK_Enabled, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::Loop::getStartLoc(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, ORE, OrigLoop, and RTChecks.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ emitSCEVChecks()

Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 2555 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), llvm::LoopVectorizeHints::FK_Enabled, llvm::BasicBlock::getParent(), llvm::Function::hasOptSize(), introduceCheckBlockInVPlan(), LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, and RTChecks.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ fixNonInductionPHIs()

Fix the non-induction PHIs in Plan.

Definition at line 3011 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::get(), llvm::VPlan::getEntry(), llvm::VPWidenPHIRecipe::getIncomingBlock(), llvm::VPWidenPHIRecipe::getIncomingValue(), llvm::VPUser::getNumOperands(), Idx, P, Plan, llvm::IRBuilderBase::SetInsertPoint(), llvm::vp_depth_first_deep(), and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by fixVectorizedLoop().

◆ fixVectorizedLoop()

Fix the vectorized code, taking care of header phi's, and more.

Definition at line 2884 of file LoopVectorize.cpp.

References llvm::VPTransformState::CFG, cse(), llvm::EnableVPlanNativePath, fixNonInductionPHIs(), llvm::ScalarEvolution::forgetBlockAndLoopDispositions(), llvm::ScalarEvolution::forgetLcssaPhiWithNewPredecessor(), llvm::ScalarEvolution::forgetLoop(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::LoopBase< BlockT, LoopT >::getExitBlocks(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPlan::getVectorLoopRegion(), LI, OrigLoop, llvm::VPTransformState::Plan, PredicatedInstructions, PSE, llvm::setProfileInfoAfterUnrolling(), sinkScalarOperands(), UF, VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getAdditionalBypassBlock()

BasicBlock * llvm::InnerLoopVectorizer::getAdditionalBypassBlock ( ) const	inline

◆ getInductionAdditionalBypassValue()

Value * llvm::InnerLoopVectorizer::getInductionAdditionalBypassValue ( PHINode * OrigPhi) const	inline

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( BasicBlock * InsertBlock)	protected

Returns (and creates if needed) the trip count of the widened loop.

Definition at line 2406 of file LoopVectorize.cpp.

References assert(), Builder, llvm::IRBuilderBase::CreateAdd(), llvm::IRBuilderBase::CreateICmpEQ(), llvm::IRBuilderBase::CreateSelect(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateURem(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::Value::getType(), llvm::isPowerOf2_32(), llvm::ElementCount::isVector(), UF, VectorTripCount, and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and llvm::LoopVectorizationPlanner::executePlan().

◆ getTripCount()

Value * llvm::InnerLoopVectorizer::getTripCount ( ) const	inline

◆ introduceCheckBlockInVPlan()

void InnerLoopVectorizer::introduceCheckBlockInVPlan ( BasicBlock * CheckIRBB)	protected

Introduces a new VPIRBasicBlock for CheckIRBB to Plan between the vector preheader and its predecessor, also connecting the new block to the scalar preheader.

Definition at line 2455 of file LoopVectorize.cpp.

References assert(), llvm::VPBlockUtils::connectBlocks(), llvm::VPlan::createVPIRBasicBlock(), llvm::VPBlockBase::getNumSuccessors(), llvm::VPlan::getScalarPreheader(), llvm::VPBlockBase::getSinglePredecessor(), llvm::VPBlockBase::getSuccessors(), llvm::VPBlockUtils::insertOnEdge(), Plan, llvm::VPBlockBase::swapSuccessors(), and VectorPHVPB.

Referenced by emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), and emitSCEVChecks().

◆ printDebugTracesAtEnd()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtEnd ( )	inlineprotectedvirtual

◆ printDebugTracesAtStart()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtStart ( )	inlineprotectedvirtual

◆ scalarizeInstruction()

A helper function to scalarize a single Instruction in the innermost loop.

Generates a sequence of scalar instances for each lane between MinLane and MaxLane, times each part between MinPart and MaxPart, inclusive. Uses the VPValue operands from RepRecipe instead of Instr's operands.

Definition at line 2347 of file LoopVectorize.cpp.

References AC, llvm::VPTransformState::addNewMetadata(), llvm::all_of(), assert(), llvm::VPTransformState::Builder, DL, llvm::enumerate(), llvm::VPTransformState::get(), llvm::VPLane::getFirstLane(), llvm::VPBlockBase::getParent(), llvm::VPRecipeBase::getParent(), llvm::VPBlockBase::getPlan(), llvm::Value::getType(), llvm::VPlan::getVectorLoopRegion(), I, II, llvm::VPTypeAnalysis::inferScalarType(), llvm::IRBuilderBase::Insert(), llvm::VPRegionBlock::isReplicator(), llvm::vputils::isUniformAfterVectorization(), llvm::VPUser::operands(), PredicatedInstructions, llvm::AssumptionCache::registerAssumption(), llvm::VPTransformState::set(), llvm::VPTransformState::setDebugLocFrom(), llvm::VPRecipeWithIRFlags::setFlags(), llvm::Value::setName(), llvm::User::setOperand(), and llvm::VPTransformState::TypeAnalysis.

Referenced by llvm::VPReplicateRecipe::execute().

◆ setTripCount()

void llvm::InnerLoopVectorizer::setTripCount ( Value * TC)	inline

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction * PredInst)	protected

Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.

Definition at line 2935 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallVectorImpl< T >::clear(), llvm::SetVector< T, Vector, Set, N >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::PHINode::getIncomingValueNumForOperand(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), I, llvm::SetVector< T, Vector, Set, N >::insert(), LI, llvm::User::op_begin(), llvm::User::op_end(), llvm::SetVector< T, Vector, Set, N >::pop_back_val(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

Referenced by fixVectorizedLoop().

◆ LoopVectorizationPlanner

◆ AC

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false	protected

◆ AdditionalBypassBlock

BasicBlock* llvm::InnerLoopVectorizer::AdditionalBypassBlock = nullptr	protected

◆ BFI

◆ Builder

◆ Cost

◆ DT

◆ Induction2AdditionalBypassValue

◆ Legal

◆ LI

◆ LoopBypassBlocks

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock	protected

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader	protected

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader	protected

◆ MinProfitableTripCount

ElementCount llvm::InnerLoopVectorizer::MinProfitableTripCount	protected

◆ OptForSizeBasedOnProfile

bool llvm::InnerLoopVectorizer::OptForSizeBasedOnProfile	protected

◆ ORE

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop	protected

◆ Plan

VPlan& llvm::InnerLoopVectorizer::Plan	protected

◆ PredicatedInstructions

◆ PSE

◆ PSI

◆ RTChecks

GeneratedRTChecks& llvm::InnerLoopVectorizer::RTChecks	protected

◆ TLI

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr	protected

◆ TTI

◆ UF

◆ VectorPHVPB

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr	protected

◆ VF

The documentation for this class was generated from the following file:

lib/Transforms/Vectorize/LoopVectorize.cpp