Fennel: ExternalSortExecStreamImpl Class Reference (original) (raw)

• Main Page

• Namespaces

• Classes

• Files

• Directories

• Related Pages

• Alphabetical List

• Class List

• Class Hierarchy

• Class Members

ExternalSortExecStreamImpl implements the ExternalSortExecStream interface. More...

#include <[ExternalSortExecStreamImpl.h](ExternalSortExecStreamImpl%5F8h-source.html)>

Inheritance diagram for ExternalSortExecStreamImpl:

List of all members.

Detailed Description

ExternalSortExecStreamImpl implements the ExternalSortExecStream interface.

Version:

Id

//open/dev/fennel/sorter/ExternalSortExecStreamImpl.h#2

Definition at line 52 of file ExternalSortExecStreamImpl.h.

Constructor & Destructor Documentation

| ExternalSortExecStreamImpl::ExternalSortExecStreamImpl | ( | | ) | [explicit] | | ------------------------------------------------------ | - | | - | ------------ |

Member Function Documentation

| void ExternalSortExecStreamImpl::computeFirstResult | ( | | ) | [private] | | --------------------------------------------------- | - | | - | ----------- |

| void ExternalSortExecStreamImpl::computeFirstResultParallel | ( | | ) | [private] | | ----------------------------------------------------------- | - | | - | ----------- |

Performs enough sorting to be able to start returning results (parallel version).

Definition at line 374 of file ExternalSortExecStreamImpl.cpp.

References EXTSORT_ENDOFDATA, mergeFirstResult(), nParallel, reserveRunLoader(), resultsReady, ThreadPoolBase::start(), ExternalSortRunLoader::startRun(), ThreadPoolBase::stop(), ThreadPool< Task >::submitTask(), threadPool, and unreserveRunLoader().

Referenced by execute().

| void ExternalSortExecStreamImpl::mergeFirstResult | ( | | ) | [private] | | ------------------------------------------------- | - | | - | ----------- |

Performs enough merging to be able to start returning results.

Definition at line 286 of file ExternalSortExecStreamImpl.cpp.

References deleteStoredRunInfo(), nParallel, ExternalSortInfo::nSortMemPages, optimizeRunOrder(), pFinalRunAccessor, pMerger, pOutputWriter, releaseResources(), runLoaders, sortInfo, storedRuns, storeFinalRun, storeRun(), and TRACE_FINE.

Referenced by computeFirstResultParallel(), and execute().

| void ExternalSortExecStreamImpl::optimizeRunOrder | ( | | ) | [private] | | ------------------------------------------------- | - | | - | ----------- |

| void ExternalSortExecStreamImpl::releaseResources | ( | | ) | [private] | | ------------------------------------------------- | - | | - | ----------- |

| void ExternalSortExecStreamImpl::closeImpl | ( | | ) | [private, virtual] | | ------------------------------------------ | - | | - | -------------------- |

Implements ExternalSortExecStream.

Definition at line 69 of file ExternalSortExecStreamImpl.cpp.

References ExternalSortInfo::cbPage, ExternalSortExecStreamParams::descendingKeyColumns, ExternalSortInfo::descendingKeyColumns, ExternalSortExecStreamParams::distinctness, DUP_ALLOW, DUP_DISCARD, DUP_FAIL, ExternalSortExecStreamParams::earlyClose, earlyClose, ExternalSortExecStreamParams::estimatedNumRows, estimatedNumRows, ExternalSortInfo::externalSegmentAccessor, ExternalSortInfo::keyDesc, ExternalSortExecStreamParams::keyProj, ExternalSortInfo::keyProj, ExternalSortInfo::memSegmentAccessor, nParallel, ExternalSortInfo::nSortMemPages, SegmentAccessor::pCacheAccessor, SingleInputExecStream::pInAccessor, ConduitExecStream::prepare(), TupleDescriptor::projectFrom(), SegmentAccessor::pSegment, ExternalSortExecStreamParams::pTempSegment, pTempSegment, resultsReady, sortInfo, ExternalSortExecStreamParams::storeFinalRun, storeFinalRun, and ExternalSortInfo::tupleDesc.

Opens this stream, acquiring any resources needed in order to be able to fetch data.

A precondition is that input streams must already be opened. A stream can be closed and reopened.

Parameters:

Reimplemented from ConduitExecStream.

Definition at line 160 of file ExternalSortExecStreamImpl.cpp.

References nParallel, ExternalSortInfo::nSortMemPages, ExternalSortInfo::nSortMemPagesPerRun, ConduitExecStream::open(), pOutputWriter, releaseResources(), resultsReady, runLoaders, and sortInfo.

Executes this stream.

Parameters:

Returns:

code indicating reason execution ceased

Implements ExecStream.

Definition at line 195 of file ExternalSortExecStreamImpl.cpp.

References ExecStreamGraphImpl::closeProducers(), computeFirstResult(), computeFirstResultParallel(), earlyClose, EXECBUF_EOS, EXECRC_BUF_UNDERFLOW, EXECRC_YIELD, ExecStream::getGraph(), ExecStream::getStreamId(), ExternalSortRunLoader::isStarted(), mergeFirstResult(), nParallel, SingleInputExecStream::pInAccessor, SingleOutputExecStream::pOutAccessor, pOutputWriter, ConduitExecStream::precheckConduitBuffers(), resultsReady, runLoaders, sortRun(), storedRuns, storeFinalRun, and storeRun().

Determines resource requirements for this stream.

Default implementation declares zero resource requirements.

Parameters:

Reimplemented from ExecStream.

Definition at line 109 of file ExternalSortExecStreamImpl.cpp.

References estimatedNumRows, EXEC_RESOURCE_ESTIMATE, EXEC_RESOURCE_UNBOUNDED, ExecStream::getResourceRequirements(), isMAXU(), max(), MAXU, ExternalSortInfo::memSegmentAccessor, ExecStreamResourceQuantity::nCachePages, SingleOutputExecStream::pOutAccessor, SegmentAccessor::pSegment, and sortInfo.

Definition at line 42 of file ConduitExecStream.cpp.

References SingleInputExecStream::pInAccessor, SingleOutputExecStream::pOutAccessor, SingleOutputExecStream::prepare(), and SingleInputExecStream::prepare().

Referenced by prepare(), LcsClusterAppendExecStream::prepare(), LbmNormalizerExecStream::prepare(), LhxAggExecStream::prepare(), FtrsTableWriterExecStream::prepare(), BTreeSearchExecStream::prepare(), BTreeInsertExecStream::prepare(), UncollectExecStream::prepare(), SortedAggExecStream::prepare(), SegBufferReaderExecStream::prepare(), SegBufferExecStream::prepare(), ScratchBufferExecStream::prepare(), ReshapeExecStream::prepare(), DoubleBufferExecStream::prepare(), CopyExecStream::prepare(), CollectExecStream::prepare(), BernoulliSamplingExecStream::prepare(), and CalcExecStream::prepare().

Prepares this stream for execution.

A precondition is that input streams must already be defined and prepared. As an effect of this call, the tuple shape should be defined for all output buffers and remain unchanged for the lifetime of the stream. This method is only ever called once, before the first open. Although this method is virtual, derived classes may choose to define an overloaded version instead with a specialized covariant parameter class.

Parameters:

Definition at line 84 of file ExecStream.cpp.

References ExecStreamGraph::getDynamicParamManager(), SegmentAccessor::pCacheAccessor, ExecStreamParams::pCacheAccessor, ExecStream::pDynamicParamManager, ExecStream::pGraph, ExecStream::pQuotaAccessor, ExecStream::pScratchQuotaAccessor, and ExecStreamParams::scratchAccessor.

Referenced by JavaTransformExecStream::prepare(), SingleOutputExecStream::prepare(), and SingleInputExecStream::prepare().

Definition at line 48 of file SingleOutputExecStream.cpp.

References SingleOutputExecStream::getOutputBufProvision(), SingleOutputExecStreamParams::outputTupleDesc, SingleOutputExecStreamParams::outputTupleFormat, SingleOutputExecStream::pOutAccessor, and ExecStream::prepare().

Referenced by BTreeExecStream::prepare(), FlatFileExecStreamImpl::prepare(), ValuesExecStream::prepare(), MockResourceExecStream::prepare(), MockProducerExecStream::prepare(), ConfluenceExecStream::prepare(), and ConduitExecStream::prepare().

| ExecStreamResult ConduitExecStream::precheckConduitBuffers | ( | | ) | [protected, inherited] | | ----------------------------------------------------------------------------------------------------------------------- | - | | - | ------------------------ |

Checks the state of the input and output buffers.

If input empty, requests production. If input EOS, propagates that to output buffer. If output full, returns EXECRC_OVERFLOW.

Returns:

result of precheck; anything but EXECRC_YIELD indicates that execution should terminate immediately with returned code

Definition at line 61 of file ConduitExecStream.cpp.

References EXECBUF_EMPTY, EXECBUF_EOS, EXECBUF_NONEMPTY, EXECBUF_OVERFLOW, EXECBUF_UNDERFLOW, EXECRC_BUF_OVERFLOW, EXECRC_BUF_UNDERFLOW, EXECRC_EOS, EXECRC_YIELD, SingleInputExecStream::pInAccessor, and SingleOutputExecStream::pOutAccessor.

Referenced by execute(), FtrsTableWriterExecStream::execute(), BTreeSearchUniqueExecStream::execute(), BTreeSearchExecStream::execute(), BTreeInsertExecStream::execute(), UncollectExecStream::execute(), SortedAggExecStream::execute(), ReshapeExecStream::execute(), CopyExecStream::execute(), CollectExecStream::execute(), BernoulliSamplingExecStream::execute(), and CalcExecStream::execute().

| bool ExecStream::canEarlyClose | ( | | ) | [virtual, inherited] | | ------------------------------ | - | | - | ---------------------- |

| ExecStreamId ExecStream::getStreamId | ( | | ) | const [inline, inherited] | | -------------------------------------------------------------------------------------------------- | - | | - | --------------------------- |

Returns:

the identifier for this stream within containing graph

Definition at line 288 of file ExecStream.h.

References ExecStream::id.

Referenced by execute(), SegBufferWriterExecStream::execute(), SegBufferExecStream::execute(), CorrelationJoinExecStream::execute(), BarrierExecStream::execute(), Java_net_sf_farrago_fennel_FennelStorage_tupleStreamTransformFetch(), JavaTransformExecStream::open(), SingleInputExecStream::open(), ConfluenceExecStream::open(), CartesianJoinExecStream::prepare(), ParallelExecStreamScheduler::readStream(), DfsTreeExecStreamScheduler::readStream(), LbmMinusExecStream::restartSubtrahends(), ExecStreamScheduler::tracePostExecution(), ExecStreamScheduler::tracePreExecution(), ExecStreamScheduler::traceStreamBuffers(), and ParallelExecStreamScheduler::tryExecuteTask().

00289 { 00290 return id; 00291 }

Reimplemented in DoubleBufferExecStream, ScratchBufferExecStream, SegBufferExecStream, SegBufferReaderExecStream, SegBufferWriterExecStream, FlatFileExecStreamImpl, BTreeInsertExecStream, BTreeReadExecStream, FtrsTableWriterExecStream, LbmChopperExecStream, LbmSplicerExecStream, LcsClusterAppendExecStream, LcsClusterReplaceExecStream, LcsRowScanBaseExecStream, and LcsRowScanExecStream.

Definition at line 102 of file ExecStream.cpp.

References ExecStreamResourceQuantity::nCachePages, and ExecStreamResourceQuantity::nThreads.

00105 { 00106 minQuantity.nThreads = 0; 00107 minQuantity.nCachePages = 0; 00108 optQuantity = minQuantity; 00109 }

| void ExecStream::setName | ( | std::string const & | | ) | [virtual, inherited] | | ------------------------ | - | ------------------- | | - | ---------------------- |

| std::string const & ExecStream::getName | ( | | ) | const [virtual, inherited] | | --------------------------------------- | - | | - | ---------------------------- |

| bool ExecStream::mayBlock | ( | | ) | const [virtual, inherited] | | ------------------------- | - | | - | ---------------------------- |

Queries whether this stream's implementation may block when execute() is called.

For accurate scheduling, non-blocking implementations are preferred; the scheduler must be aware of the potential for blocking so that it can allocate extra threads accordingly.

Returns:

whether stream may block; default is false

Definition at line 167 of file ExecStream.cpp.

00168 { 00169 return false; 00170 }

| void ExecStream::checkAbort | ( | | ) | const [virtual, inherited] | | --------------------------- | - | | - | ---------------------------- |

Queries the BufferProvision which this stream is capable of when producing tuples.

Returns:

supported model; default is BUFPROV_NONE

Reimplemented in BarrierExecStream, DiffluenceExecStream, DoubleBufferExecStream, MergeExecStream, MockResourceExecStream, ScratchBufferExecStream, SegBufferExecStream, SegBufferReaderExecStream, SegBufferWriterExecStream, SingleOutputExecStream, SplitterExecStream, ValuesExecStream, JavaTransformExecStream, FtrsTableWriterExecStream, and LcsClusterAppendExecStream.

Definition at line 172 of file ExecStream.cpp.

References BUFPROV_NONE.

| bool ClosableObject::isClosed | ( | | ) | const [inline, inherited] | | ----------------------------- | - | | - | --------------------------- |

Returns:

whether the object has been closed

Definition at line 58 of file ClosableObject.h.

| void ClosableObject::close | ( | | ) | [inherited] | | -------------------------- | - | | - | ------------- |

Closes this object, releasing any unallocated resources.

Reimplemented in CollectExecStream, CorrelationJoinExecStream, LcsClusterAppendExecStream, and LcsClusterReplaceExecStream.

Definition at line 39 of file ClosableObject.cpp.

References ClosableObject::closeImpl(), and ClosableObject::needsClose.

Referenced by CacheImpl< PageT, VictimPolicyT >::allocatePages(), LcsRowScanBaseExecStream::closeImpl(), ExecStreamGraphImpl::closeImpl(), FlatFileBuffer::open(), ClosableObjectDestructor::operator()(), and Segment::~Segment().

| bool TraceSource::isTracing | ( | | ) | const [inline, inherited] | | --------------------------- | - | | - | --------------------------- |

| TraceTarget& TraceSource::getTraceTarget | ( | | ) | const [inline, inherited] | | ----------------------------------------------------------------- | - | | - | --------------------------- |

| std::string TraceSource::getTraceSourceName | ( | | ) | const [inline, inherited] | | ------------------------------------------- | - | | - | --------------------------- |

Sets the name of this source.

Useful to construct dynamic names for fine-grained filtering.

Definition at line 136 of file TraceSource.h.

00137 { 00138 name = n; 00139 }

| TraceLevel TraceSource::getMinimumTraceLevel | ( | | ) | const [inline, inherited] | | ------------------------------------------------------------------------------------------------------ | - | | - | --------------------------- |

| void TraceSource::disableTracing | ( | | ) | [inherited] | | -------------------------------- | - | | - | ------------- |

| bool ErrorSource::hasTarget | ( | | ) | const [inline, inherited] | | --------------------------- | - | | - | --------------------------- |

| ErrorTarget& ErrorSource::getErrorTarget | ( | | ) | const [inline, inherited] | | ----------------------------------------------------------------- | - | | - | --------------------------- |

Returns:

the SharedErrorTarget for this source

Definition at line 129 of file ErrorSource.h.

| std::string ErrorSource::getErrorSourceName | ( | | ) | const [inline, inherited] | | ------------------------------------------- | - | | - | --------------------------- |

Gets the name of this source.

Useful to construct nested names for subcomponents that are also ErrorSources.

Returns:

the name

Definition at line 139 of file ErrorSource.h.

00140 { 00141 return name; 00142 }

Sets the name of this source.

Useful to construct dynamic names for fine-grained filtering.

Definition at line 148 of file ErrorSource.h.

00149 { 00150 name = n; 00151 }

| void ErrorSource::disableTarget | ( | | ) | [inherited] | | ------------------------------- | - | | - | ------------- |

Friends And Related Function Documentation

Member Data Documentation

Array of helpers used to load, quicksort, and store runs.

This array has size equal to nParallel. For non-parallel sort (nParallel=1), this means there's just one entry. For parallel sort, this array acts as an availability queue governed by runLoaderAvailable, runLoaderMutex, and the ExternalSortRunLoader::runningParallelTask flag.

Definition at line 79 of file ExternalSortExecStreamImpl.h.

Referenced by computeFirstResult(), execute(), mergeFirstResult(), open(), releaseResources(), and reserveRunLoader().

Definition at line 51 of file SingleInputExecStream.h.

Referenced by SortedAggExecStream::compareGroupByKeys(), computeFirstResult(), execute(), LbmSplicerExecStream::execute(), FtrsTableWriterExecStream::execute(), BTreeSortExecStream::execute(), BTreeSearchUniqueExecStream::execute(), BTreeSearchExecStream::execute(), BTreePrefetchSearchExecStream::execute(), BTreeInsertExecStream::execute(), JavaSinkExecStream::execute(), UncollectExecStream::execute(), SplitterExecStream::execute(), SortedAggExecStream::execute(), SegBufferWriterExecStream::execute(), SegBufferReaderExecStream::execute(), SegBufferExecStream::execute(), ScratchBufferExecStream::execute(), ReshapeExecStream::execute(), MockConsumerExecStream::execute(), DoubleBufferExecStream::execute(), CopyExecStream::execute(), CollectExecStream::execute(), BernoulliSamplingExecStream::execute(), CalcExecStream::execute(), BTreePrefetchSearchExecStream::getNextPageForPrefetch(), LcsClusterReplaceExecStream::getTupleForLoad(), LcsClusterAppendExecStream::getTupleForLoad(), LbmSplicerExecStream::getValidatedTuple(), LcsClusterReplaceExecStream::initTupleLoadParams(), BTreeSearchExecStream::innerSearchLoop(), BTreePrefetchSearchExecStream::innerSearchLoop(), LcsClusterReplaceExecStream::open(), LbmNormalizerExecStream::open(), SingleInputExecStream::open(), SegBufferWriterExecStream::open(), SegBufferExecStream::open(), ScratchBufferExecStream::open(), MockConsumerExecStream::open(), DoubleBufferExecStream::open(), CollectExecStream::open(), LcsClusterAppendExecStream::postProcessTuple(), ConduitExecStream::precheckConduitBuffers(), prepare(), LcsClusterAppendExecStream::prepare(), LbmSplicerExecStream::prepare(), LbmNormalizerExecStream::prepare(), BTreeSearchExecStream::prepare(), BTreeInsertExecStream::prepare(), UncollectExecStream::prepare(), SortedAggExecStream::prepare(), SingleInputExecStream::prepare(), SegBufferReaderExecStream::prepare(), ReshapeExecStream::prepare(), DiffluenceExecStream::prepare(), ConduitExecStream::prepare(), CollectExecStream::prepare(), BernoulliSamplingExecStream::prepare(), CalcExecStream::prepare(), SortedAggExecStream::produce(), BTreeSearchExecStream::readSearchKey(), LhxAggExecStream::setAggComputers(), LhxAggExecStream::setHashInfo(), SingleInputExecStream::setInputBufAccessors(), and JavaSinkExecStream::stuffByteBuffer().

Dataflow graph containing this stream.

Note that we don't use a weak_ptr for this because it needs to be accessed frequently during execution, and the extra locking overhead would be frivolous.

Definition at line 68 of file ExecStream.h.

Referenced by ExecStream::checkAbort(), ExecStream::ExecStream(), CorrelationJoinExecStream::execute(), ExecStream::getGraph(), JavaTransformExecStream::open(), SingleInputExecStream::open(), MergeExecStream::open(), ExecStream::open(), CorrelationJoinExecStream::open(), ConfluenceExecStream::open(), ExecStream::prepare(), CartesianJoinExecStream::prepare(), and LbmMinusExecStream::restartSubtrahends().

The dynamic parameter manager available to this stream.

(Obtained at prepare() time. Keep a shared pointer in case the stream is reassigned to another graph for execution; cf ExecStreamGraph::mergeFrom())

Definition at line 85 of file ExecStream.h.

Referenced by CorrelationJoinExecStream::close(), SegBufferReaderExecStream::closeImpl(), LbmUnionExecStream::execute(), LbmSplicerExecStream::execute(), LbmGeneratorExecStream::execute(), LbmChopperExecStream::execute(), CorrelationJoinExecStream::execute(), BarrierExecStream::execute(), LcsClusterReplaceExecStream::open(), LbmUnionExecStream::open(), LbmSplicerExecStream::open(), LbmGeneratorExecStream::open(), LbmBitOpExecStream::open(), BTreeSearchExecStream::open(), BTreeInsertExecStream::open(), SegBufferWriterExecStream::open(), SegBufferReaderExecStream::open(), NestedLoopJoinExecStream::open(), CorrelationJoinExecStream::open(), ExecStream::prepare(), CalcExecStream::prepare(), NestedLoopJoinExecStream::processLeftInput(), LbmSearchExecStream::reachedTupleLimit(), ReshapeExecStream::readDynamicParams(), SegBufferWriterExecStream::readReaderRefCount(), BTreeSearchExecStream::readSearchKey(), BTreeSearchExecStream::readUpperBoundKey(), LbmSearchExecStream::setAdditionalKeys(), and LbmBitOpExecStream::writeStartRidParamValue().

Definition at line 56 of file SingleOutputExecStream.h.

Referenced by LcsClusterAppendExecStream::compress(), execute(), LcsRowScanExecStream::execute(), LbmUnionExecStream::execute(), LbmNormalizerExecStream::execute(), LbmGeneratorExecStream::execute(), LbmChopperExecStream::execute(), LhxJoinExecStream::execute(), LhxAggExecStream::execute(), FtrsTableWriterExecStream::execute(), BTreeSortExecStream::execute(), BTreeSearchUniqueExecStream::execute(), BTreeScanExecStream::execute(), BTreePrefetchSearchExecStream::execute(), BTreeInsertExecStream::execute(), FlatFileExecStreamImpl::execute(), ValuesExecStream::execute(), UncollectExecStream::execute(), SortedAggExecStream::execute(), SegBufferReaderExecStream::execute(), ScratchBufferExecStream::execute(), ReshapeExecStream::execute(), MockResourceExecStream::execute(), MockProducerExecStream::execute(), MergeExecStream::execute(), DoubleBufferExecStream::execute(), CorrelationJoinExecStream::execute(), CopyExecStream::execute(), CollectExecStream::execute(), CartesianJoinExecStream::execute(), BernoulliSamplingExecStream::execute(), BarrierExecStream::execute(), CalcExecStream::execute(), LbmGeneratorExecStream::flushEntry(), MockProducerExecStream::getProducedRowCount(), getResourceRequirements(), BTreeSearchExecStream::innerFetchLoop(), LbmUnionExecStream::open(), LbmChopperExecStream::open(), LbmBitOpExecStream::open(), SingleOutputExecStream::open(), SegBufferReaderExecStream::open(), SegBufferExecStream::open(), ScratchBufferExecStream::open(), DoubleBufferExecStream::open(), CollectExecStream::open(), SegBufferExecStream::openBufferForRead(), ConduitExecStream::precheckConduitBuffers(), LcsRowScanExecStream::prepare(), LcsRowScanBaseExecStream::prepare(), LcsClusterAppendExecStream::prepare(), LbmUnionExecStream::prepare(), LbmGeneratorExecStream::prepare(), LhxJoinExecStream::prepare(), LhxAggExecStream::prepare(), FtrsTableWriterExecStream::prepare(), FlatFileExecStreamImpl::prepare(), UncollectExecStream::prepare(), SortedAggExecStream::prepare(), SingleOutputExecStream::prepare(), ReshapeExecStream::prepare(), MockResourceExecStream::prepare(), MockProducerExecStream::prepare(), MergeExecStream::prepare(), CorrelationJoinExecStream::prepare(), ConduitExecStream::prepare(), CollectExecStream::prepare(), CartesianJoinExecStream::prepare(), BernoulliSamplingExecStream::prepare(), BarrierExecStream::prepare(), CalcExecStream::prepare(), SortedAggExecStream::produce(), LbmBitOpExecStream::producePendingOutput(), LbmUnionExecStream::produceTuple(), LbmNormalizerExecStream::produceTuple(), LbmMinusExecStream::produceTuple(), LbmChopperExecStream::produceTuple(), LbmBitOpExecStream::produceTuple(), LbmBitOpExecStream::readInput(), LbmMinusExecStream::readMinuendInput(), and SingleOutputExecStream::setOutputBufAccessors().

The documentation for this class was generated from the following files:

• /home/pub/open/dev/fennel/sorter/ExternalSortExecStreamImpl.h
• /home/pub/open/dev/fennel/sorter/ExternalSortExecStreamImpl.cpp

Generated on Mon Jun 22 04:00:31 2009 for Fennel by 1.5.1