SparkSession (Spark 3.5.5 JavaDoc) (original) (raw)

Modifier and Type

Method and Description

static [SparkSession](../../../../org/apache/spark/sql/SparkSession.html "class in org.apache.spark.sql")

[active](../../../../org/apache/spark/sql/SparkSession.html#active--)()

Returns the currently active SparkSession, otherwise the default one.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[baseRelationToDataFrame](../../../../org/apache/spark/sql/SparkSession.html#baseRelationToDataFrame-org.apache.spark.sql.sources.BaseRelation-)([BaseRelation](../../../../org/apache/spark/sql/sources/BaseRelation.html "class in org.apache.spark.sql.sources") baseRelation)

Convert a BaseRelation created for external data sources into a DataFrame.

static [SparkSession.Builder](../../../../org/apache/spark/sql/SparkSession.Builder.html "class in org.apache.spark.sql")

[builder](../../../../org/apache/spark/sql/SparkSession.html#builder--)()

[Catalog](../../../../org/apache/spark/sql/catalog/Catalog.html "class in org.apache.spark.sql.catalog")

[catalog](../../../../org/apache/spark/sql/SparkSession.html#catalog--)()

static void

[clearActiveSession](../../../../org/apache/spark/sql/SparkSession.html#clearActiveSession--)()

Clears the active SparkSession for current thread.

static void

[clearDefaultSession](../../../../org/apache/spark/sql/SparkSession.html#clearDefaultSession--)()

Clears the default SparkSession that is returned by the builder.

void

[close](../../../../org/apache/spark/sql/SparkSession.html#close--)()

Synonym for stop().

[RuntimeConfig](../../../../org/apache/spark/sql/RuntimeConfig.html "class in org.apache.spark.sql")

[conf](../../../../org/apache/spark/sql/SparkSession.html#conf--)()

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-org.apache.spark.api.java.JavaRDD-java.lang.Class-)([JavaRDD](../../../../org/apache/spark/api/java/JavaRDD.html "class in org.apache.spark.api.java")<?> rdd, Class<?> beanClass)

Applies a schema to an RDD of Java Beans.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-org.apache.spark.api.java.JavaRDD-org.apache.spark.sql.types.StructType-)([JavaRDD](../../../../org/apache/spark/api/java/JavaRDD.html "class in org.apache.spark.api.java")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")> rowRDD,[StructType](../../../../org/apache/spark/sql/types/StructType.html "class in org.apache.spark.sql.types") schema)

:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-java.util.List-java.lang.Class-)(java.util.List<?> data, Class<?> beanClass)

Applies a schema to a List of Java Beans.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-java.util.List-org.apache.spark.sql.types.StructType-)(java.util.List<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")> rows,[StructType](../../../../org/apache/spark/sql/types/StructType.html "class in org.apache.spark.sql.types") schema)

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-org.apache.spark.rdd.RDD-java.lang.Class-)([RDD](../../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<?> rdd, Class<?> beanClass)

Applies a schema to an RDD of Java Beans.

<A extends scala.Product> [Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-org.apache.spark.rdd.RDD-scala.reflect.api.TypeTags.TypeTag-)([RDD](../../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<A> rdd, scala.reflect.api.TypeTags.TypeTag<A> evidence$2)

Creates a DataFrame from an RDD of Product (e.g.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-org.apache.spark.rdd.RDD-org.apache.spark.sql.types.StructType-)([RDD](../../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")> rowRDD,[StructType](../../../../org/apache/spark/sql/types/StructType.html "class in org.apache.spark.sql.types") schema)

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

<A extends scala.Product> [Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[createDataFrame](../../../../org/apache/spark/sql/SparkSession.html#createDataFrame-scala.collection.Seq-scala.reflect.api.TypeTags.TypeTag-)(scala.collection.Seq<A> data, scala.reflect.api.TypeTags.TypeTag<A> evidence$3)

Creates a DataFrame from a local Seq of Product.

<T> [Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<T>

[createDataset](../../../../org/apache/spark/sql/SparkSession.html#createDataset-java.util.List-org.apache.spark.sql.Encoder-)(java.util.List<T> data,[Encoder](../../../../org/apache/spark/sql/Encoder.html "interface in org.apache.spark.sql")<T> evidence$6)

Creates a Dataset from a java.util.List of a given type.

<T> [Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<T>

[createDataset](../../../../org/apache/spark/sql/SparkSession.html#createDataset-org.apache.spark.rdd.RDD-org.apache.spark.sql.Encoder-)([RDD](../../../../org/apache/spark/rdd/RDD.html "class in org.apache.spark.rdd")<T> data,[Encoder](../../../../org/apache/spark/sql/Encoder.html "interface in org.apache.spark.sql")<T> evidence$5)

Creates a Dataset from an RDD of a given type.

<T> [Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<T>

[createDataset](../../../../org/apache/spark/sql/SparkSession.html#createDataset-scala.collection.Seq-org.apache.spark.sql.Encoder-)(scala.collection.Seq<T> data,[Encoder](../../../../org/apache/spark/sql/Encoder.html "interface in org.apache.spark.sql")<T> evidence$4)

Creates a Dataset from a local Seq of data of a given type.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[emptyDataFrame](../../../../org/apache/spark/sql/SparkSession.html#emptyDataFrame--)()

<T> [Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<T>

[emptyDataset](../../../../org/apache/spark/sql/SparkSession.html#emptyDataset-org.apache.spark.sql.Encoder-)([Encoder](../../../../org/apache/spark/sql/Encoder.html "interface in org.apache.spark.sql")<T> evidence$1)

Creates a new Dataset of type T containing zero elements.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[executeCommand](../../../../org/apache/spark/sql/SparkSession.html#executeCommand-java.lang.String-java.lang.String-scala.collection.immutable.Map-)(String runner, String command, scala.collection.immutable.Map<String,String> options)

Execute an arbitrary string command inside an external execution engine rather than Spark.

[ExperimentalMethods](../../../../org/apache/spark/sql/ExperimentalMethods.html "class in org.apache.spark.sql")

[experimental](../../../../org/apache/spark/sql/SparkSession.html#experimental--)()

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

static scala.Option<[SparkSession](../../../../org/apache/spark/sql/SparkSession.html "class in org.apache.spark.sql")>

[getActiveSession](../../../../org/apache/spark/sql/SparkSession.html#getActiveSession--)()

Returns the active SparkSession for the current thread, returned by the builder.

static scala.Option<[SparkSession](../../../../org/apache/spark/sql/SparkSession.html "class in org.apache.spark.sql")>

[getDefaultSession](../../../../org/apache/spark/sql/SparkSession.html#getDefaultSession--)()

Returns the default SparkSession that is returned by the builder.

[SparkSession.implicits$](../../../../org/apache/spark/sql/SparkSession.implicits$.html "class in org.apache.spark.sql")

[implicits](../../../../org/apache/spark/sql/SparkSession.html#implicits--)()

Accessor for nested Scala object

[ExecutionListenerManager](../../../../org/apache/spark/sql/util/ExecutionListenerManager.html "class in org.apache.spark.sql.util")

[listenerManager](../../../../org/apache/spark/sql/SparkSession.html#listenerManager--)()

An interface to register custom QueryExecutionListeners that listen for execution metrics.

[SparkSession](../../../../org/apache/spark/sql/SparkSession.html "class in org.apache.spark.sql")

[newSession](../../../../org/apache/spark/sql/SparkSession.html#newSession--)()

Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

static void

[org$apache$spark$internal$Logging$$log__$eq](../../../../org/apache/spark/sql/SparkSession.html#org:Dapache:Dspark:Dinternal:DLogging:D:Dlog%5F%5F:Deq-org.slf4j.Logger-)(org.slf4j.Logger x$1)

static org.slf4j.Logger

[org$apache$spark$internal$Logging$$log_](../../../../org/apache/spark/sql/SparkSession.html#org:Dapache:Dspark:Dinternal:DLogging:D:Dlog%5F--)()

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<Long>

[range](../../../../org/apache/spark/sql/SparkSession.html#range-long-)(long end)

Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<Long>

[range](../../../../org/apache/spark/sql/SparkSession.html#range-long-long-)(long start, long end)

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<Long>

[range](../../../../org/apache/spark/sql/SparkSession.html#range-long-long-long-)(long start, long end, long step)

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<Long>

[range](../../../../org/apache/spark/sql/SparkSession.html#range-long-long-long-int-)(long start, long end, long step, int numPartitions)

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.

[DataFrameReader](../../../../org/apache/spark/sql/DataFrameReader.html "class in org.apache.spark.sql")

[read](../../../../org/apache/spark/sql/SparkSession.html#read--)()

Returns a DataFrameReader that can be used to read non-streaming data in as aDataFrame.

[DataStreamReader](../../../../org/apache/spark/sql/streaming/DataStreamReader.html "class in org.apache.spark.sql.streaming")

[readStream](../../../../org/apache/spark/sql/SparkSession.html#readStream--)()

Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.

org.apache.spark.sql.internal.SessionState

[sessionState](../../../../org/apache/spark/sql/SparkSession.html#sessionState--)()

static void

[setActiveSession](../../../../org/apache/spark/sql/SparkSession.html#setActiveSession-org.apache.spark.sql.SparkSession-)([SparkSession](../../../../org/apache/spark/sql/SparkSession.html "class in org.apache.spark.sql") session)

Changes the SparkSession that will be returned in this thread and its children when SparkSession.getOrCreate() is called.

static void

[setDefaultSession](../../../../org/apache/spark/sql/SparkSession.html#setDefaultSession-org.apache.spark.sql.SparkSession-)([SparkSession](../../../../org/apache/spark/sql/SparkSession.html "class in org.apache.spark.sql") session)

Sets the default SparkSession that is returned by the builder.

org.apache.spark.sql.internal.SharedState

[sharedState](../../../../org/apache/spark/sql/SparkSession.html#sharedState--)()

[SparkContext](../../../../org/apache/spark/SparkContext.html "class in org.apache.spark")

[sparkContext](../../../../org/apache/spark/sql/SparkSession.html#sparkContext--)()

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[sql](../../../../org/apache/spark/sql/SparkSession.html#sql-java.lang.String-)(String sqlText)

Executes a SQL query using Spark, returning the result as a DataFrame.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[sql](../../../../org/apache/spark/sql/SparkSession.html#sql-java.lang.String-scala.collection.immutable.Map-)(String sqlText, scala.collection.immutable.Map<String,Object> args)

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[sql](../../../../org/apache/spark/sql/SparkSession.html#sql-java.lang.String-java.util.Map-)(String sqlText, java.util.Map<String,Object> args)

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[sql](../../../../org/apache/spark/sql/SparkSession.html#sql-java.lang.String-java.lang.Object-)(String sqlText, Object args)

Executes a SQL query substituting positional parameters by the given arguments, returning the result as a DataFrame.

[SQLContext](../../../../org/apache/spark/sql/SQLContext.html "class in org.apache.spark.sql")

[sqlContext](../../../../org/apache/spark/sql/SparkSession.html#sqlContext--)()

A wrapped version of this session in the form of a SQLContext, for backward compatibility.

void

[stop](../../../../org/apache/spark/sql/SparkSession.html#stop--)()

Stop the underlying SparkContext.

[StreamingQueryManager](../../../../org/apache/spark/sql/streaming/StreamingQueryManager.html "class in org.apache.spark.sql.streaming")

[streams](../../../../org/apache/spark/sql/SparkSession.html#streams--)()

Returns a StreamingQueryManager that allows managing all theStreamingQuerys active on this.

[Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[Row](../../../../org/apache/spark/sql/Row.html "interface in org.apache.spark.sql")>

[table](../../../../org/apache/spark/sql/SparkSession.html#table-java.lang.String-)(String tableName)

Returns the specified table/view as a DataFrame.

<T> T

[time](../../../../org/apache/spark/sql/SparkSession.html#time-scala.Function0-)(scala.Function0<T> f)

Executes some code block and prints to stdout the time taken to execute the block.

[UDFRegistration](../../../../org/apache/spark/sql/UDFRegistration.html "class in org.apache.spark.sql")

[udf](../../../../org/apache/spark/sql/SparkSession.html#udf--)()

A collection of methods for registering user-defined functions (UDF).

[UDTFRegistration](../../../../org/apache/spark/sql/UDTFRegistration.html "class in org.apache.spark.sql")

[udtf](../../../../org/apache/spark/sql/SparkSession.html#udtf--)()

String

[version](../../../../org/apache/spark/sql/SparkSession.html#version--)()

The version of Spark on which this application is running.