DataStreamWriter (Spark 3.5.5 JavaDoc) (original) (raw)

Modifier and Type

Method and Description

[DataStreamWriter](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html "class in org.apache.spark.sql.streaming")<[T](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html "type parameter in DataStreamWriter")>

[foreach](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#foreach-org.apache.spark.sql.ForeachWriter-)([ForeachWriter](../../../../../org/apache/spark/sql/ForeachWriter.html "class in org.apache.spark.sql")<[T](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html "type parameter in DataStreamWriter")> writer)

Sets the output of the streaming query to be processed using the provided writer object.

[foreachBatch](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#foreachBatch-scala.Function2-)(scala.Function2<[Dataset](../../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[T](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html "type parameter in DataStreamWriter")>,Object,scala.runtime.BoxedUnit> function)

:: Experimental ::

[foreachBatch](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#foreachBatch-org.apache.spark.api.java.function.VoidFunction2-)([VoidFunction2](../../../../../org/apache/spark/api/java/function/VoidFunction2.html "interface in org.apache.spark.api.java.function")<[Dataset](../../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<[T](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html "type parameter in DataStreamWriter")>,Long> function)

:: Experimental ::

[format](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#format-java.lang.String-)(String source)

Specifies the underlying output data source.

[option](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#option-java.lang.String-boolean-)(String key, boolean value)

Adds an output option for the underlying data source.

[option](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#option-java.lang.String-double-)(String key, double value)

Adds an output option for the underlying data source.

[option](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#option-java.lang.String-long-)(String key, long value)

Adds an output option for the underlying data source.

[option](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#option-java.lang.String-java.lang.String-)(String key, String value)

Adds an output option for the underlying data source.

[options](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#options-scala.collection.Map-)(scala.collection.Map<String,String> options)

(Scala-specific) Adds output options for the underlying data source.

[options](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#options-java.util.Map-)(java.util.Map<String,String> options)

Adds output options for the underlying data source.

[outputMode](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#outputMode-org.apache.spark.sql.streaming.OutputMode-)([OutputMode](../../../../../org/apache/spark/sql/streaming/OutputMode.html "class in org.apache.spark.sql.streaming") outputMode)

Specifies how data of a streaming DataFrame/Dataset is written to a streaming sink.

[outputMode](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#outputMode-java.lang.String-)(String outputMode)

Specifies how data of a streaming DataFrame/Dataset is written to a streaming sink.

[partitionBy](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#partitionBy-scala.collection.Seq-)(scala.collection.Seq<String> colNames)

Partitions the output by the given columns on the file system.

[partitionBy](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#partitionBy-java.lang.String...-)(String... colNames)

Partitions the output by the given columns on the file system.

[queryName](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#queryName-java.lang.String-)(String queryName)

Specifies the name of the StreamingQuery that can be started with start().

static String

[SOURCE_NAME_CONSOLE](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCE%5FNAME%5FCONSOLE--)()

static String

[SOURCE_NAME_FOREACH_BATCH](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCE%5FNAME%5FFOREACH%5FBATCH--)()

static String

[SOURCE_NAME_FOREACH](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCE%5FNAME%5FFOREACH--)()

static String

[SOURCE_NAME_MEMORY](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCE%5FNAME%5FMEMORY--)()

static String

[SOURCE_NAME_NOOP](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCE%5FNAME%5FNOOP--)()

static String

[SOURCE_NAME_TABLE](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCE%5FNAME%5FTABLE--)()

static scala.collection.Seq<String>

[SOURCES_ALLOW_ONE_TIME_QUERY](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#SOURCES%5FALLOW%5FONE%5FTIME%5FQUERY--)()

[StreamingQuery](../../../../../org/apache/spark/sql/streaming/StreamingQuery.html "interface in org.apache.spark.sql.streaming")

[start](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#start--)()

Starts the execution of the streaming query, which will continually output results to the given path as new data arrives.

[StreamingQuery](../../../../../org/apache/spark/sql/streaming/StreamingQuery.html "interface in org.apache.spark.sql.streaming")

[start](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#start-java.lang.String-)(String path)

Starts the execution of the streaming query, which will continually output results to the given path as new data arrives.

[StreamingQuery](../../../../../org/apache/spark/sql/streaming/StreamingQuery.html "interface in org.apache.spark.sql.streaming")

[toTable](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#toTable-java.lang.String-)(String tableName)

Starts the execution of the streaming query, which will continually output results to the given table as new data arrives.

[trigger](../../../../../org/apache/spark/sql/streaming/DataStreamWriter.html#trigger-org.apache.spark.sql.streaming.Trigger-)([Trigger](../../../../../org/apache/spark/sql/streaming/Trigger.html "class in org.apache.spark.sql.streaming") trigger)

Set the trigger for the stream query.