SparkContext (Spark 3.5.5 JavaDoc) (original) (raw)
Modifier and Type
Method and Description
void
[addArchive](../../../org/apache/spark/SparkContext.html#addArchive-java.lang.String-)(String path)
:: Experimental :: Add an archive to be downloaded and unpacked with this Spark job on every node.
void
[addFile](../../../org/apache/spark/SparkContext.html#addFile-java.lang.String-)(String path)
Add a file to be downloaded with this Spark job on every node.
void
[addFile](../../../org/apache/spark/SparkContext.html#addFile-java.lang.String-boolean-)(String path, boolean recursive)
Add a file to be downloaded with this Spark job on every node.
void
[addJar](../../../org/apache/spark/SparkContext.html#addJar-java.lang.String-)(String path)
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
void
[addJobTag](../../../org/apache/spark/SparkContext.html#addJobTag-java.lang.String-)(String tag)
Add a tag to be assigned to all the jobs started by this thread.
void
[addSparkListener](../../../org/apache/spark/SparkContext.html#addSparkListener-org.apache.spark.scheduler.SparkListenerInterface-)([SparkListenerInterface](../../../org/apache/spark/scheduler/SparkListenerInterface.html "interface in org.apache.spark.scheduler") listener)
:: DeveloperApi :: Register a listener to receive up-calls from events that happen during execution.
scala.Option<String>
[applicationAttemptId](../../../org/apache/spark/SparkContext.html#applicationAttemptId--)()
String
[applicationId](../../../org/apache/spark/SparkContext.html#applicationId--)()
A unique identifier for the Spark application.
String
[appName](../../../org/apache/spark/SparkContext.html#appName--)()
scala.collection.Seq<String>
[archives](../../../org/apache/spark/SparkContext.html#archives--)()
[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<String,[PortableDataStream](../../../org/apache/spark/input/PortableDataStream.html "class in org.apache.spark.input")>>
[binaryFiles](../../../org/apache/spark/SparkContext.html#binaryFiles-java.lang.String-int-)(String path, int minPartitions)
Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file (useful for binary data)
[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<byte[]>
[binaryRecords](../../../org/apache/spark/SparkContext.html#binaryRecords-java.lang.String-int-org.apache.hadoop.conf.Configuration-)(String path, int recordLength, org.apache.hadoop.conf.Configuration conf)
Load data from a flat binary file, assuming the length of each record is constant.
<T> [Broadcast](../../../org/apache/spark/broadcast/Broadcast.html "class in org.apache.spark.broadcast")<T>
[broadcast](../../../org/apache/spark/SparkContext.html#broadcast-T-scala.reflect.ClassTag-)(T value, scala.reflect.ClassTag<T> evidence$9)
Broadcast a read-only variable to the cluster, returning aBroadcast object for reading it in distributed functions.
void
[cancelAllJobs](../../../org/apache/spark/SparkContext.html#cancelAllJobs--)()
Cancel all jobs that have been scheduled or are running.
void
[cancelJob](../../../org/apache/spark/SparkContext.html#cancelJob-int-)(int jobId)
Cancel a given job if it's scheduled or running.
void
[cancelJob](../../../org/apache/spark/SparkContext.html#cancelJob-int-java.lang.String-)(int jobId, String reason)
Cancel a given job if it's scheduled or running.
void
[cancelJobGroup](../../../org/apache/spark/SparkContext.html#cancelJobGroup-java.lang.String-)(String groupId)
Cancel active jobs for the specified group.
void
[cancelJobsWithTag](../../../org/apache/spark/SparkContext.html#cancelJobsWithTag-java.lang.String-)(String tag)
Cancel active jobs that have the specified tag.
void
[cancelStage](../../../org/apache/spark/SparkContext.html#cancelStage-int-)(int stageId)
Cancel a given stage and all jobs associated with it.
void
[cancelStage](../../../org/apache/spark/SparkContext.html#cancelStage-int-java.lang.String-)(int stageId, String reason)
Cancel a given stage and all jobs associated with it.
void
[clearCallSite](../../../org/apache/spark/SparkContext.html#clearCallSite--)()
Clear the thread-local property for overriding the call sites of actions and RDDs.
void
[clearJobGroup](../../../org/apache/spark/SparkContext.html#clearJobGroup--)()
Clear the current thread's job group ID and its description.
void
[clearJobTags](../../../org/apache/spark/SparkContext.html#clearJobTags--)()
Clear the current thread's job tags.
<T> [CollectionAccumulator](../../../org/apache/spark/util/CollectionAccumulator.html "class in org.apache.spark.util")<T>
[collectionAccumulator](../../../org/apache/spark/SparkContext.html#collectionAccumulator--)()
Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.
<T> [CollectionAccumulator](../../../org/apache/spark/util/CollectionAccumulator.html "class in org.apache.spark.util")<T>
[collectionAccumulator](../../../org/apache/spark/SparkContext.html#collectionAccumulator-java.lang.String-)(String name)
Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.
int
[defaultMinPartitions](../../../org/apache/spark/SparkContext.html#defaultMinPartitions--)()
Default min number of partitions for Hadoop RDDs when not given by user Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2.
int
[defaultParallelism](../../../org/apache/spark/SparkContext.html#defaultParallelism--)()
Default level of parallelism to use when not given by user (e.g.
String
[deployMode](../../../org/apache/spark/SparkContext.html#deployMode--)()
[DoubleAccumulator](../../../org/apache/spark/util/DoubleAccumulator.html "class in org.apache.spark.util")
[doubleAccumulator](../../../org/apache/spark/SparkContext.html#doubleAccumulator--)()
Create and register a double accumulator, which starts with 0 and accumulates inputs by add.
[DoubleAccumulator](../../../org/apache/spark/util/DoubleAccumulator.html "class in org.apache.spark.util")
[doubleAccumulator](../../../org/apache/spark/SparkContext.html#doubleAccumulator-java.lang.String-)(String name)
Create and register a double accumulator, which starts with 0 and accumulates inputs by add.
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[emptyRDD](../../../org/apache/spark/SparkContext.html#emptyRDD-scala.reflect.ClassTag-)(scala.reflect.ClassTag<T> evidence$8)
Get an RDD that has no partitions or elements.
scala.collection.Seq<String>
[files](../../../org/apache/spark/SparkContext.html#files--)()
scala.collection.Seq<[Schedulable](../../../org/apache/spark/scheduler/Schedulable.html "interface in org.apache.spark.scheduler")>
[getAllPools](../../../org/apache/spark/SparkContext.html#getAllPools--)()
:: DeveloperApi :: Return pools for fair scheduler
scala.Option<String>
[getCheckpointDir](../../../org/apache/spark/SparkContext.html#getCheckpointDir--)()
[SparkConf](../../../org/apache/spark/SparkConf.html "class in org.apache.spark")
[getConf](../../../org/apache/spark/SparkContext.html#getConf--)()
Return a copy of this SparkContext's configuration.
scala.collection.Map<String,scala.Tuple2<Object,Object>>
[getExecutorMemoryStatus](../../../org/apache/spark/SparkContext.html#getExecutorMemoryStatus--)()
Return a map from the block manager to the max memory available for caching and the remaining memory available for caching.
scala.collection.immutable.Set<String>
[getJobTags](../../../org/apache/spark/SparkContext.html#getJobTags--)()
Get the tags that are currently set to be assigned to all the jobs started by this thread.
String
[getLocalProperty](../../../org/apache/spark/SparkContext.html#getLocalProperty-java.lang.String-)(String key)
Get a local property set in this thread, or null if it is missing.
static [SparkContext](../../../org/apache/spark/SparkContext.html "class in org.apache.spark")
[getOrCreate](../../../org/apache/spark/SparkContext.html#getOrCreate--)()
This function may be used to get or instantiate a SparkContext and register it as a singleton object.
static [SparkContext](../../../org/apache/spark/SparkContext.html "class in org.apache.spark")
[getOrCreate](../../../org/apache/spark/SparkContext.html#getOrCreate-org.apache.spark.SparkConf-)([SparkConf](../../../org/apache/spark/SparkConf.html "class in org.apache.spark") config)
This function may be used to get or instantiate a SparkContext and register it as a singleton object.
scala.collection.Map<Object,[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<?>>
[getPersistentRDDs](../../../org/apache/spark/SparkContext.html#getPersistentRDDs--)()
Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.
scala.Option<[Schedulable](../../../org/apache/spark/scheduler/Schedulable.html "interface in org.apache.spark.scheduler")>
[getPoolForName](../../../org/apache/spark/SparkContext.html#getPoolForName-java.lang.String-)(String pool)
:: DeveloperApi :: Return the pool associated with the given name, if one exists
[RDDInfo](../../../org/apache/spark/storage/RDDInfo.html "class in org.apache.spark.storage")[]
[getRDDStorageInfo](../../../org/apache/spark/SparkContext.html#getRDDStorageInfo--)()
:: DeveloperApi :: Return information about what RDDs are cached, if they are in mem or on disk, how much space they take, etc.
scala.Enumeration.Value
[getSchedulingMode](../../../org/apache/spark/SparkContext.html#getSchedulingMode--)()
Return current scheduling mode
org.apache.hadoop.conf.Configuration
[hadoopConfiguration](../../../org/apache/spark/SparkContext.html#hadoopConfiguration--)()
A default Hadoop Configuration for the Hadoop code (e.g.
<K,V> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[hadoopFile](../../../org/apache/spark/SparkContext.html#hadoopFile-java.lang.String-java.lang.Class-java.lang.Class-java.lang.Class-int-)(String path, Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>> inputFormatClass, Class<K> keyClass, Class<V> valueClass, int minPartitions)
Get an RDD for a Hadoop file with an arbitrary InputFormat
<K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[hadoopFile](../../../org/apache/spark/SparkContext.html#hadoopFile-java.lang.String-scala.reflect.ClassTag-scala.reflect.ClassTag-scala.reflect.ClassTag-)(String path, scala.reflect.ClassTag<K> km, scala.reflect.ClassTag<V> vm, scala.reflect.ClassTag<F> fm)
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
<K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[hadoopFile](../../../org/apache/spark/SparkContext.html#hadoopFile-java.lang.String-int-scala.reflect.ClassTag-scala.reflect.ClassTag-scala.reflect.ClassTag-)(String path, int minPartitions, scala.reflect.ClassTag<K> km, scala.reflect.ClassTag<V> vm, scala.reflect.ClassTag<F> fm)
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
<K,V> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[hadoopRDD](../../../org/apache/spark/SparkContext.html#hadoopRDD-org.apache.hadoop.mapred.JobConf-java.lang.Class-java.lang.Class-java.lang.Class-int-)(org.apache.hadoop.mapred.JobConf conf, Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>> inputFormatClass, Class<K> keyClass, Class<V> valueClass, int minPartitions)
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other necessary info (e.g.
boolean
[isLocal](../../../org/apache/spark/SparkContext.html#isLocal--)()
boolean
[isStopped](../../../org/apache/spark/SparkContext.html#isStopped--)()
static scala.Option<String>
[jarOfClass](../../../org/apache/spark/SparkContext.html#jarOfClass-java.lang.Class-)(Class<?> cls)
Find the JAR from which a given class was loaded, to make it easy for users to pass their JARs to SparkContext.
static scala.Option<String>
[jarOfObject](../../../org/apache/spark/SparkContext.html#jarOfObject-java.lang.Object-)(Object obj)
Find the JAR that contains the class of a particular object, to make it easy for users to pass their JARs to SparkContext.
scala.collection.Seq<String>
[jars](../../../org/apache/spark/SparkContext.html#jars--)()
boolean
[killExecutor](../../../org/apache/spark/SparkContext.html#killExecutor-java.lang.String-)(String executorId)
:: DeveloperApi :: Request that the cluster manager kill the specified executor.
boolean
[killExecutors](../../../org/apache/spark/SparkContext.html#killExecutors-scala.collection.Seq-)(scala.collection.Seq<String> executorIds)
:: DeveloperApi :: Request that the cluster manager kill the specified executors.
boolean
[killTaskAttempt](../../../org/apache/spark/SparkContext.html#killTaskAttempt-long-boolean-java.lang.String-)(long taskId, boolean interruptThread, String reason)
Kill and reschedule the given task attempt.
scala.collection.Seq<String>
[listArchives](../../../org/apache/spark/SparkContext.html#listArchives--)()
:: Experimental :: Returns a list of archive paths that are added to resources.
scala.collection.Seq<String>
[listFiles](../../../org/apache/spark/SparkContext.html#listFiles--)()
Returns a list of file paths that are added to resources.
scala.collection.Seq<String>
[listJars](../../../org/apache/spark/SparkContext.html#listJars--)()
Returns a list of jar files that are added to resources.
[LongAccumulator](../../../org/apache/spark/util/LongAccumulator.html "class in org.apache.spark.util")
[longAccumulator](../../../org/apache/spark/SparkContext.html#longAccumulator--)()
Create and register a long accumulator, which starts with 0 and accumulates inputs by add.
[LongAccumulator](../../../org/apache/spark/util/LongAccumulator.html "class in org.apache.spark.util")
[longAccumulator](../../../org/apache/spark/SparkContext.html#longAccumulator-java.lang.String-)(String name)
Create and register a long accumulator, which starts with 0 and accumulates inputs by add.
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[makeRDD](../../../org/apache/spark/SparkContext.html#makeRDD-scala.collection.Seq-int-scala.reflect.ClassTag-)(scala.collection.Seq<T> seq, int numSlices, scala.reflect.ClassTag<T> evidence$2)
Distribute a local Scala collection to form an RDD.
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[makeRDD](../../../org/apache/spark/SparkContext.html#makeRDD-scala.collection.Seq-scala.reflect.ClassTag-)(scala.collection.Seq<scala.Tuple2<T,scala.collection.Seq<String>>> seq, scala.reflect.ClassTag<T> evidence$3)
Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object.
String
[master](../../../org/apache/spark/SparkContext.html#master--)()
<K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[newAPIHadoopFile](../../../org/apache/spark/SparkContext.html#newAPIHadoopFile-java.lang.String-java.lang.Class-java.lang.Class-java.lang.Class-org.apache.hadoop.conf.Configuration-)(String path, Class<F> fClass, Class<K> kClass, Class<V> vClass, org.apache.hadoop.conf.Configuration conf)
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
<K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[newAPIHadoopFile](../../../org/apache/spark/SparkContext.html#newAPIHadoopFile-java.lang.String-scala.reflect.ClassTag-scala.reflect.ClassTag-scala.reflect.ClassTag-)(String path, scala.reflect.ClassTag<K> km, scala.reflect.ClassTag<V> vm, scala.reflect.ClassTag<F> fm)
Smarter version of newApiHadoopFile that uses class tags to figure out the classes of keys, values and the org.apache.hadoop.mapreduce.InputFormat (new MapReduce API) so that user don't need to pass them directly.
<K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[newAPIHadoopRDD](../../../org/apache/spark/SparkContext.html#newAPIHadoopRDD-org.apache.hadoop.conf.Configuration-java.lang.Class-java.lang.Class-java.lang.Class-)(org.apache.hadoop.conf.Configuration conf, Class<F> fClass, Class<K> kClass, Class<V> vClass)
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[objectFile](../../../org/apache/spark/SparkContext.html#objectFile-java.lang.String-int-scala.reflect.ClassTag-)(String path, int minPartitions, scala.reflect.ClassTag<T> evidence$4)
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
static void
[org$apache$spark$internal$Logging$$log__$eq](../../../org/apache/spark/SparkContext.html#org:Dapache:Dspark:Dinternal:DLogging:D:Dlog%5F%5F:Deq-org.slf4j.Logger-)(org.slf4j.Logger x$1)
static org.slf4j.Logger
[org$apache$spark$internal$Logging$$log_](../../../org/apache/spark/SparkContext.html#org:Dapache:Dspark:Dinternal:DLogging:D:Dlog%5F--)()
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[parallelize](../../../org/apache/spark/SparkContext.html#parallelize-scala.collection.Seq-int-scala.reflect.ClassTag-)(scala.collection.Seq<T> seq, int numSlices, scala.reflect.ClassTag<T> evidence$1)
Distribute a local Scala collection to form an RDD.
[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<Object>
[range](../../../org/apache/spark/SparkContext.html#range-long-long-long-int-)(long start, long end, long step, int numSlices)
Creates a new RDD[Long] containing elements from start to end(exclusive), increased bystep every element.
void
[register](../../../org/apache/spark/SparkContext.html#register-org.apache.spark.util.AccumulatorV2-)([AccumulatorV2](../../../org/apache/spark/util/AccumulatorV2.html "class in org.apache.spark.util")<?,?> acc)
Register the given accumulator.
void
[register](../../../org/apache/spark/SparkContext.html#register-org.apache.spark.util.AccumulatorV2-java.lang.String-)([AccumulatorV2](../../../org/apache/spark/util/AccumulatorV2.html "class in org.apache.spark.util")<?,?> acc, String name)
Register the given accumulator with given name.
void
[removeJobTag](../../../org/apache/spark/SparkContext.html#removeJobTag-java.lang.String-)(String tag)
Remove a tag previously added to be assigned to all the jobs started by this thread.
void
[removeSparkListener](../../../org/apache/spark/SparkContext.html#removeSparkListener-org.apache.spark.scheduler.SparkListenerInterface-)([SparkListenerInterface](../../../org/apache/spark/scheduler/SparkListenerInterface.html "interface in org.apache.spark.scheduler") listener)
:: DeveloperApi :: Deregister the listener from Spark's listener bus.
boolean
[requestExecutors](../../../org/apache/spark/SparkContext.html#requestExecutors-int-)(int numAdditionalExecutors)
:: DeveloperApi :: Request an additional number of executors from the cluster manager.
boolean
[requestTotalExecutors](../../../org/apache/spark/SparkContext.html#requestTotalExecutors-int-int-scala.collection.immutable.Map-)(int numExecutors, int localityAwareTasks, scala.collection.immutable.Map<String,Object> hostToLocalTaskCount)
Update the cluster manager on our scheduling needs.
scala.collection.Map<String,[ResourceInformation](../../../org/apache/spark/resource/ResourceInformation.html "class in org.apache.spark.resource")>
[resources](../../../org/apache/spark/SparkContext.html#resources--)()
<T,U,R> [PartialResult](../../../org/apache/spark/partial/PartialResult.html "class in org.apache.spark.partial")<R>
[runApproximateJob](../../../org/apache/spark/SparkContext.html#runApproximateJob-org.apache.spark.rdd.RDD-scala.Function2-org.apache.spark.partial.ApproximateEvaluator-long-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function2<[TaskContext](../../../org/apache/spark/TaskContext.html "class in org.apache.spark"),scala.collection.Iterator<T>,U> func,[ApproximateEvaluator](../../../org/apache/spark/partial/ApproximateEvaluator.html "interface in org.apache.spark.partial")<U,R> evaluator, long timeout)
:: DeveloperApi :: Run a job that can return approximate results.
<T,U> Object
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function1-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function1<scala.collection.Iterator<T>,U> func, scala.reflect.ClassTag<U> evidence$15)
Run a job on all partitions in an RDD and return the results in an array.
<T,U> void
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function1-scala.Function2-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function1<scala.collection.Iterator<T>,U> processPartition, scala.Function2<Object,U,scala.runtime.BoxedUnit> resultHandler, scala.reflect.ClassTag<U> evidence$17)
Run a job on all partitions in an RDD and pass the results to a handler function.
<T,U> Object
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function1-scala.collection.Seq-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function1<scala.collection.Iterator<T>,U> func, scala.collection.Seq<Object> partitions, scala.reflect.ClassTag<U> evidence$13)
Run a function on a given set of partitions in an RDD and return the results as an array.
<T,U> Object
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function2-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function2<[TaskContext](../../../org/apache/spark/TaskContext.html "class in org.apache.spark"),scala.collection.Iterator<T>,U> func, scala.reflect.ClassTag<U> evidence$14)
Run a job on all partitions in an RDD and return the results in an array.
<T,U> void
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function2-scala.Function2-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function2<[TaskContext](../../../org/apache/spark/TaskContext.html "class in org.apache.spark"),scala.collection.Iterator<T>,U> processPartition, scala.Function2<Object,U,scala.runtime.BoxedUnit> resultHandler, scala.reflect.ClassTag<U> evidence$16)
Run a job on all partitions in an RDD and pass the results to a handler function.
<T,U> Object
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function2-scala.collection.Seq-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function2<[TaskContext](../../../org/apache/spark/TaskContext.html "class in org.apache.spark"),scala.collection.Iterator<T>,U> func, scala.collection.Seq<Object> partitions, scala.reflect.ClassTag<U> evidence$12)
Run a function on a given set of partitions in an RDD and return the results as an array.
<T,U> void
[runJob](../../../org/apache/spark/SparkContext.html#runJob-org.apache.spark.rdd.RDD-scala.Function2-scala.collection.Seq-scala.Function2-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function2<[TaskContext](../../../org/apache/spark/TaskContext.html "class in org.apache.spark"),scala.collection.Iterator<T>,U> func, scala.collection.Seq<Object> partitions, scala.Function2<Object,U,scala.runtime.BoxedUnit> resultHandler, scala.reflect.ClassTag<U> evidence$11)
Run a function on a given set of partitions in an RDD and pass the results to the given handler function.
<K,V> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[sequenceFile](../../../org/apache/spark/SparkContext.html#sequenceFile-java.lang.String-java.lang.Class-java.lang.Class-)(String path, Class<K> keyClass, Class<V> valueClass)
Get an RDD for a Hadoop SequenceFile with given key and value types.
<K,V> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[sequenceFile](../../../org/apache/spark/SparkContext.html#sequenceFile-java.lang.String-java.lang.Class-java.lang.Class-int-)(String path, Class<K> keyClass, Class<V> valueClass, int minPartitions)
Get an RDD for a Hadoop SequenceFile with given key and value types.
<K,V> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<K,V>>
[sequenceFile](../../../org/apache/spark/SparkContext.html#sequenceFile-java.lang.String-int-scala.reflect.ClassTag-scala.reflect.ClassTag-scala.Function0-scala.Function0-)(String path, int minPartitions, scala.reflect.ClassTag<K> km, scala.reflect.ClassTag<V> vm, scala.Function0<org.apache.spark.WritableConverter<K>> kcf, scala.Function0<org.apache.spark.WritableConverter<V>> vcf)
Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.
void
[setCallSite](../../../org/apache/spark/SparkContext.html#setCallSite-java.lang.String-)(String shortCallSite)
Set the thread-local property for overriding the call sites of actions and RDDs.
void
[setCheckpointDir](../../../org/apache/spark/SparkContext.html#setCheckpointDir-java.lang.String-)(String directory)
Set the directory under which RDDs are going to be checkpointed.
void
[setInterruptOnCancel](../../../org/apache/spark/SparkContext.html#setInterruptOnCancel-boolean-)(boolean interruptOnCancel)
Set the behavior of job cancellation from jobs started in this thread.
void
[setJobDescription](../../../org/apache/spark/SparkContext.html#setJobDescription-java.lang.String-)(String value)
Set a human readable description of the current job.
void
[setJobGroup](../../../org/apache/spark/SparkContext.html#setJobGroup-java.lang.String-java.lang.String-boolean-)(String groupId, String description, boolean interruptOnCancel)
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
void
[setLocalProperty](../../../org/apache/spark/SparkContext.html#setLocalProperty-java.lang.String-java.lang.String-)(String key, String value)
Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.
void
[setLogLevel](../../../org/apache/spark/SparkContext.html#setLogLevel-java.lang.String-)(String logLevel)
Control our logLevel.
String
[sparkUser](../../../org/apache/spark/SparkContext.html#sparkUser--)()
long
[startTime](../../../org/apache/spark/SparkContext.html#startTime--)()
[SparkStatusTracker](../../../org/apache/spark/SparkStatusTracker.html "class in org.apache.spark")
[statusTracker](../../../org/apache/spark/SparkContext.html#statusTracker--)()
void
[stop](../../../org/apache/spark/SparkContext.html#stop--)()
Shut down the SparkContext.
void
[stop](../../../org/apache/spark/SparkContext.html#stop-int-)(int exitCode)
Shut down the SparkContext with exit code that will passed to scheduler backend.
<T,U,R> [SimpleFutureAction](../../../org/apache/spark/SimpleFutureAction.html "class in org.apache.spark")<R>
[submitJob](../../../org/apache/spark/SparkContext.html#submitJob-org.apache.spark.rdd.RDD-scala.Function1-scala.collection.Seq-scala.Function2-scala.Function0-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> rdd, scala.Function1<scala.collection.Iterator<T>,U> processPartition, scala.collection.Seq<Object> partitions, scala.Function2<Object,U,scala.runtime.BoxedUnit> resultHandler, scala.Function0<R> resultFunc)
Submit a job for execution and return a FutureJob holding the result.
[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<String>
[textFile](../../../org/apache/spark/SparkContext.html#textFile-java.lang.String-int-)(String path, int minPartitions)
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
scala.Option<String>
[uiWebUrl](../../../org/apache/spark/SparkContext.html#uiWebUrl--)()
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[union](../../../org/apache/spark/SparkContext.html#union-org.apache.spark.rdd.RDD-scala.collection.Seq-scala.reflect.ClassTag-)([RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> first, scala.collection.Seq<[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>> rest, scala.reflect.ClassTag<T> evidence$7)
Build the union of a list of RDDs passed as variable-length arguments.
<T> [RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>
[union](../../../org/apache/spark/SparkContext.html#union-scala.collection.Seq-scala.reflect.ClassTag-)(scala.collection.Seq<[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T>> rdds, scala.reflect.ClassTag<T> evidence$6)
Build the union of a list of RDDs.
String
[version](../../../org/apache/spark/SparkContext.html#version--)()
The version of Spark on which this application is running.
[RDD](../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<scala.Tuple2<String,String>>
[wholeTextFiles](../../../org/apache/spark/SparkContext.html#wholeTextFiles-java.lang.String-int-)(String path, int minPartitions)
Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.