LLVM: llvm::PeelingModuloScheduleExpander Class Reference (original) (raw)
A reimplementation of ModuloScheduleExpander. More...
#include "[llvm/CodeGen/ModuloSchedule.h](ModuloSchedule%5F8h%5Fsource.html)"
Public Member Functions | |
---|---|
PeelingModuloScheduleExpander (MachineFunction &MF, ModuloSchedule &S, LiveIntervals *LIS) | |
void | expand () |
void | validateAgainstModuloScheduleExpander () |
Runs ModuloScheduleExpander and treats it as a golden input to validate aspects of the code generated by PeelingModuloScheduleExpander. | |
Protected Member Functions | |
---|---|
void | rewriteKernel () |
Converts BB from the original loop body to the rewritten, pipelined steady-state. | |
MachineBasicBlock * | peelKernel (LoopPeelDirection LPD) |
Peels one iteration of the rewritten kernel (BB) in the specified direction. | |
void | filterInstructions (MachineBasicBlock *MB, int MinStage) |
void | moveStageBetweenBlocks (MachineBasicBlock *DestBB, MachineBasicBlock *SourceBB, unsigned Stage) |
void | peelPrologAndEpilogs () |
Peel the kernel forwards and backwards to produce prologs and epilogs, and stitch them together. | |
Register | getEquivalentRegisterIn (Register Reg, MachineBasicBlock *BB) |
All prolog and epilog blocks are clones of the kernel, so any produced register in one block has an corollary in all other blocks. | |
void | rewriteUsesOf (MachineInstr *MI) |
Change all users of MI, if MI is predicated out (LiveStages[MI->getParent()] == false). | |
void | fixupBranches () |
Insert branches between prologs, kernel and epilogs. | |
MachineBasicBlock * | CreateLCSSAExitingBlock () |
Create a poor-man's LCSSA by cloning only the PHIs from the kernel block to a block dominated by all prologs and epilogs. | |
unsigned | getStage (MachineInstr *MI) |
Helper to get the stage of an instruction in the schedule. | |
Register | getPhiCanonicalReg (MachineInstr *CanonicalPhi, MachineInstr *Phi) |
Helper function to find the right canonical register for a phi instruction coming from a peeled out prologue. | |
Protected Attributes | |
---|---|
ModuloSchedule & | Schedule |
MachineFunction & | MF |
const TargetSubtargetInfo & | ST |
MachineRegisterInfo & | MRI |
const TargetInstrInfo * | TII = nullptr |
LiveIntervals * | LIS = nullptr |
MachineBasicBlock * | BB = nullptr |
The original loop block that gets rewritten in-place. | |
MachineBasicBlock * | Preheader = nullptr |
The original loop preheader. | |
SmallVector< MachineBasicBlock *, 4 > | Prologs |
All prolog and epilog blocks. | |
SmallVector< MachineBasicBlock *, 4 > | Epilogs |
DenseMap< MachineBasicBlock *, BitVector > | LiveStages |
For every block, the stages that are produced. | |
DenseMap< MachineBasicBlock *, BitVector > | AvailableStages |
For every block, the stages that are available. | |
DenseMap< MachineInstr *, unsigned > | PhiNodeLoopIteration |
When peeling the epilogue keep track of the distance between the phi nodes and the kernel. | |
DenseMap< MachineInstr *, MachineInstr * > | CanonicalMIs |
CanonicalMIs and BlockMIs form a bidirectional map between any of the loop kernel clones. | |
DenseMap< std::pair< MachineBasicBlock *, MachineInstr * >, MachineInstr * > | BlockMIs |
std::deque< MachineBasicBlock * > | PeeledFront |
State passed from peelKernel to peelPrologAndEpilogs(). | |
std::deque< MachineBasicBlock * > | PeeledBack |
SmallVector< MachineInstr *, 4 > | IllegalPhisToDelete |
Illegal phis that need to be deleted once we re-link stages. | |
std::unique_ptr< TargetInstrInfo::PipelinerLoopInfo > | LoopInfo |
Target loop info before kernel peeling. | |
A reimplementation of ModuloScheduleExpander.
It works by generating a standalone kernel loop and peeling out the prologs and epilogs.
Definition at line 283 of file ModuloSchedule.h.
◆ CreateLCSSAExitingBlock()
MachineBasicBlock * PeelingModuloScheduleExpander::CreateLCSSAExitingBlock ( ) | protected |
---|
Create a poor-man's LCSSA by cloning only the PHIs from the kernel block to a block dominated by all prologs and epilogs.
This allows us to treat the loop exiting block as any other kernel clone.
Definition at line 1863 of file ModuloSchedule.cpp.
References llvm::MachineInstrBuilder::addMBB(), llvm::MachineInstrBuilder::addReg(), llvm::MachineBasicBlock::addSuccessor(), llvm::TargetInstrInfo::analyzeBranch(), assert(), BB, BlockMIs, llvm::BuildMI(), CanonicalMIs, Cond, llvm::MachineFunction::CreateMachineBasicBlock(), llvm::MachineRegisterInfo::createVirtualRegister(), llvm::MCInstrInfo::get(), llvm::MachineBasicBlock::getBasicBlock(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::MachineBasicBlock::getParent(), llvm::MachineRegisterInfo::getRegClass(), llvm::MachineRegisterInfo::getTargetRegisterInfo(), llvm::MachineFunction::insert(), llvm::TargetInstrInfo::insertBranch(), llvm::TargetInstrInfo::insertUnconditionalBranch(), MF, MI, MRI, llvm::MachineBasicBlock::phis(), llvm::TargetInstrInfo::removeBranch(), llvm::MachineBasicBlock::replaceSuccessor(), llvm::MachineBasicBlock::succ_begin(), TBB, TII, llvm::MachineRegisterInfo::use_instructions(), and Uses.
Referenced by peelPrologAndEpilogs().
◆ expand()
void PeelingModuloScheduleExpander::expand | ( | ) |
---|
Definition at line 2009 of file ModuloSchedule.cpp.
References llvm::TargetInstrInfo::analyzeLoopForPipelining(), assert(), BB, llvm::ModuloSchedule::dump(), fixupBranches(), llvm::ModuloSchedule::getLoop(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::MachineLoop::getTopBlock(), LLVM_DEBUG, peelPrologAndEpilogs(), Preheader, rewriteKernel(), Schedule, and TII.
Referenced by llvm::SwingSchedulerDAG::schedule().
◆ filterInstructions()
void PeelingModuloScheduleExpander::filterInstructions ( MachineBasicBlock * MB, int MinStage ) | protected |
---|
Definition at line 1614 of file ModuloSchedule.cpp.
References assert(), llvm::SmallVectorImpl< T >::emplace_back(), getEquivalentRegisterIn(), llvm::MachineBasicBlock::getFirstInstrTerminator(), llvm::MachineBasicBlock::getFirstNonPHI(), llvm::MachineInstrBuilder::getReg(), getStage(), llvm::MachineRegisterInfo::getTargetRegisterInfo(), I, LIS, MI, MRI, llvm::LiveIntervals::RemoveMachineInstrFromMaps(), llvm::MachineRegisterInfo::use_instructions(), and UseMI.
Referenced by peelPrologAndEpilogs().
◆ fixupBranches()
void PeelingModuloScheduleExpander::fixupBranches ( ) | protected |
---|
Insert branches between prologs, kernel and epilogs.
Definition at line 1957 of file ModuloSchedule.cpp.
References Cond, llvm::dbgs(), Epilog, Epilogs, llvm::ModuloSchedule::getNumStages(), llvm::TargetInstrInfo::insertBranch(), llvm::TargetInstrInfo::insertUnconditionalBranch(), LLVM_DEBUG, P, llvm::MachineBasicBlock::phis(), Prolog, Prologs, llvm::TargetInstrInfo::removeBranch(), Schedule, and TII.
Referenced by expand().
◆ getEquivalentRegisterIn()
◆ getPhiCanonicalReg()
Helper function to find the right canonical register for a phi instruction coming from a peeled out prologue.
Definition at line 1729 of file ModuloSchedule.cpp.
References assert(), llvm::MachineOperand::getMBB(), llvm::MachineInstr::getNumOperands(), llvm::MachineInstr::getOperand(), llvm::MachineInstr::getParent(), llvm::MachineOperand::getReg(), llvm::MachineRegisterInfo::getVRegDef(), I, llvm::MachineInstr::isPHI(), MRI, PhiNodeLoopIteration, and std::swap().
Referenced by peelPrologAndEpilogs().
◆ getStage()
◆ moveStageBetweenBlocks()
Definition at line 1643 of file ModuloSchedule.cpp.
References llvm::MachineInstrBuilder::addMBB(), llvm::MachineInstrBuilder::addReg(), assert(), BlockMIs, llvm::BuildMI(), CanonicalMIs, llvm::MachineFunction::CloneMachineInstr(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::MachineRegisterInfo::createVirtualRegister(), llvm::MachineBasicBlock::end(), llvm::MCInstrInfo::get(), llvm::MachineBasicBlock::getFirstNonPHI(), llvm::MachineInstr::getOperand(), llvm::MachineOperand::getReg(), llvm::MachineRegisterInfo::getRegClass(), getStage(), llvm::MachineRegisterInfo::getUniqueVRegDef(), llvm::MachineRegisterInfo::getVRegDef(), I, llvm::MachineBasicBlock::insert(), llvm::MachineOperand::isReg(), llvm::make_early_inc_range(), llvm::make_range(), MF, MI, MRI, P, PhiNodeLoopIteration, llvm::MachineBasicBlock::phis(), llvm::MachineBasicBlock::pred_begin(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::MachineRegisterInfo::replaceRegWith(), llvm::MachineOperand::setMBB(), llvm::MachineOperand::setReg(), and TII.
Referenced by peelPrologAndEpilogs().
◆ peelKernel()
Peels one iteration of the rewritten kernel (BB) in the specified direction.
Definition at line 1598 of file ModuloSchedule.cpp.
References BB, llvm::MachineBasicBlock::begin(), BlockMIs, CanonicalMIs, I, llvm::LPD_Front, MRI, PeeledBack, PeeledFront, llvm::PeelSingleBlockLoop(), and TII.
Referenced by peelPrologAndEpilogs().
◆ peelPrologAndEpilogs()
void PeelingModuloScheduleExpander::peelPrologAndEpilogs ( ) | protected |
---|
Peel the kernel forwards and backwards to produce prologs and epilogs, and stitch them together.
Definition at line 1746 of file ModuloSchedule.cpp.
References assert(), AvailableStages, B, BB, Blocks, CanonicalMIs, llvm::SmallVectorImpl< T >::clear(), llvm::copy(), CreateLCSSAExitingBlock(), llvm::MachineOperand::CreateMBB(), llvm::MachineOperand::CreateReg(), Epilogs, filterInstructions(), getEquivalentRegisterIn(), llvm::ModuloSchedule::getNumStages(), getPhiCanonicalReg(), llvm::MachineRegisterInfo::getUniqueVRegDef(), I, IllegalPhisToDelete, llvm::MachineInstr::isPHI(), LIS, LiveStages, llvm::LPD_Back, llvm::LPD_Front, MI, moveStageBetweenBlocks(), MRI, PeeledBack, PeeledFront, peelKernel(), PhiNodeLoopIteration, llvm::MachineBasicBlock::pred_begin(), Prologs, llvm::LiveIntervals::RemoveMachineInstrFromMaps(), llvm::reverse(), rewriteUsesOf(), and Schedule.
Referenced by expand(), and validateAgainstModuloScheduleExpander().
◆ rewriteKernel()
void PeelingModuloScheduleExpander::rewriteKernel ( ) | protected |
---|
◆ rewriteUsesOf()
void PeelingModuloScheduleExpander::rewriteUsesOf ( MachineInstr * MI) | protected |
---|
Change all users of MI, if MI is predicated out (LiveStages[MI->getParent()] == false).
Definition at line 1914 of file ModuloSchedule.cpp.
References assert(), AvailableStages, llvm::SmallVectorImpl< T >::emplace_back(), getEquivalentRegisterIn(), llvm::MachineInstrBuilder::getReg(), llvm::MachineRegisterInfo::getRegClass(), getStage(), llvm::MachineRegisterInfo::getTargetRegisterInfo(), llvm::MachineRegisterInfo::getUniqueVRegDef(), IllegalPhisToDelete, LIS, LiveStages, MI, MRI, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::LiveIntervals::RemoveMachineInstrFromMaps(), llvm::MachineRegisterInfo::replaceRegWith(), llvm::MachineRegisterInfo::setRegClass(), llvm::MachineRegisterInfo::use_instructions(), and UseMI.
Referenced by peelPrologAndEpilogs().
◆ validateAgainstModuloScheduleExpander()
void PeelingModuloScheduleExpander::validateAgainstModuloScheduleExpander | ( | ) |
---|
Runs ModuloScheduleExpander and treats it as a golden input to validate aspects of the code generated by PeelingModuloScheduleExpander.
Definition at line 2021 of file ModuloSchedule.cpp.
References llvm::MachineBasicBlock::addSuccessor(), assert(), BB, llvm::MachineBasicBlock::begin(), llvm::ModuloScheduleExpander::cleanup(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::MachineBasicBlock::end(), llvm::errs(), llvm::ModuloScheduleExpander::expand(), llvm::Failed(), llvm::raw_ostream::flush(), llvm::MachineBasicBlock::getFirstNonPHI(), llvm::ModuloSchedule::getLoop(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::ModuloScheduleExpander::getRewrittenKernel(), llvm::MachineLoop::getTopBlock(), llvm::SmallPtrSetImpl< PtrType >::insert(), LIS, MF, MRI, OS, peelPrologAndEpilogs(), Preheader, llvm::ModuloSchedule::print(), llvm::MachineBasicBlock::print(), llvm::MachineBasicBlock::removeSuccessor(), llvm::report_fatal_error(), and Schedule.
◆ AvailableStages
◆ BB
◆ BlockMIs
◆ CanonicalMIs
◆ Epilogs
◆ IllegalPhisToDelete
◆ LIS
LiveIntervals* llvm::PeelingModuloScheduleExpander::LIS = nullptr | protected |
---|
◆ LiveStages
◆ LoopInfo
◆ MF
◆ MRI
◆ PeeledBack
◆ PeeledFront
◆ PhiNodeLoopIteration
◆ Preheader
◆ Prologs
◆ Schedule
◆ ST
◆ TII
The documentation for this class was generated from the following files:
- include/llvm/CodeGen/ModuloSchedule.h
- lib/CodeGen/ModuloSchedule.cpp