Pipeline (Spark 3.5.5 JavaDoc) (original) (raw)

• Object

• • org.apache.spark.ml.PipelineStage
  • • org.apache.spark.ml.Estimator<PipelineModel>
      • • org.apache.spark.ml.Pipeline

• All Implemented Interfaces:
  java.io.Serializable, org.apache.spark.internal.Logging, Params, Identifiable, MLWritable

public class Pipeline
extends Estimator<PipelineModel>
implements MLWritable
A simple pipeline, which acts as an estimator. A Pipeline consists of a sequence of stages, each of which is either an Estimator or a Transformer. When Pipeline.fit is called, the stages are executed in order. If a stage is an Estimator, its Estimator.fit method will be called on the input dataset to fit a model. Then the model, which is a transformer, will be used to transform the dataset as the input to the next stage. If a stage is a Transformer, its Transformer.transform method will be called to produce the dataset for the next stage. The fitted model from a Pipeline is a PipelineModel, which consists of fitted models and transformers, corresponding to the pipeline stages. If there are no stages, the pipeline acts as an identity transformer.
See Also:
Serialized Form

• • Nested Class Summary

  Nested Classes

  Modifier and Type	Class and Description
  static class	Pipeline.SharedReadWrite$ Methods for MLReader and MLWriter shared between Pipeline and PipelineModel

     * ### Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging  
     `org.apache.spark.internal.Logging.SparkShellLoggingFilter`  

  • Constructor Summary

  Constructors

  Constructor and Description
  Pipeline()
  Pipeline(String uid)

  • Method Summary

  All Methods Static Methods Instance Methods Concrete Methods

  Modifier and Type	Method and Description
  Pipeline	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
  PipelineModel	fit(Dataset<?> dataset) Fits the pipeline to the input dataset with additional parameters.
  PipelineStage[]	getStages()
  static Pipeline	load(String path)
  static MLReader<Pipeline>	read()
  Pipeline	setStages(PipelineStage[] value)
  Param<PipelineStage[]>	stages() param for pipeline stages
  StructType	transformSchema(StructType schema) Check transform validity and derive the output schema from the input schema.
  String	uid() An immutable unique ID for the object and its derivatives.
  MLWriter	write() Returns an MLWriter instance for this ML instance.

     * ### Methods inherited from class org.apache.spark.ml.[Estimator](../../../../org/apache/spark/ml/Estimator.html "class in org.apache.spark.ml")  
     `[fit](../../../../org/apache/spark/ml/Estimator.html#fit-org.apache.spark.sql.Dataset-org.apache.spark.ml.param.ParamMap-), [fit](../../../../org/apache/spark/ml/Estimator.html#fit-org.apache.spark.sql.Dataset-org.apache.spark.ml.param.ParamPair-org.apache.spark.ml.param.ParamPair...-), [fit](../../../../org/apache/spark/ml/Estimator.html#fit-org.apache.spark.sql.Dataset-org.apache.spark.ml.param.ParamPair-scala.collection.Seq-), [fit](../../../../org/apache/spark/ml/Estimator.html#fit-org.apache.spark.sql.Dataset-scala.collection.Seq-)`  
     * ### Methods inherited from class org.apache.spark.ml.[PipelineStage](../../../../org/apache/spark/ml/PipelineStage.html "class in org.apache.spark.ml")  
     `[params](../../../../org/apache/spark/ml/PipelineStage.html#params--)`  
     * ### Methods inherited from class Object  
     `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`  
     * ### Methods inherited from interface org.apache.spark.ml.util.[MLWritable](../../../../org/apache/spark/ml/util/MLWritable.html "interface in org.apache.spark.ml.util")  
     `[save](../../../../org/apache/spark/ml/util/MLWritable.html#save-java.lang.String-)`  
     * ### Methods inherited from interface org.apache.spark.ml.param.[Params](../../../../org/apache/spark/ml/param/Params.html "interface in org.apache.spark.ml.param")  
     `[clear](../../../../org/apache/spark/ml/param/Params.html#clear-org.apache.spark.ml.param.Param-), [copyValues](../../../../org/apache/spark/ml/param/Params.html#copyValues-T-org.apache.spark.ml.param.ParamMap-), [defaultCopy](../../../../org/apache/spark/ml/param/Params.html#defaultCopy-org.apache.spark.ml.param.ParamMap-), [explainParam](../../../../org/apache/spark/ml/param/Params.html#explainParam-org.apache.spark.ml.param.Param-), [explainParams](../../../../org/apache/spark/ml/param/Params.html#explainParams--), [extractParamMap](../../../../org/apache/spark/ml/param/Params.html#extractParamMap--), [extractParamMap](../../../../org/apache/spark/ml/param/Params.html#extractParamMap-org.apache.spark.ml.param.ParamMap-), [get](../../../../org/apache/spark/ml/param/Params.html#get-org.apache.spark.ml.param.Param-), [getDefault](../../../../org/apache/spark/ml/param/Params.html#getDefault-org.apache.spark.ml.param.Param-), [getOrDefault](../../../../org/apache/spark/ml/param/Params.html#getOrDefault-org.apache.spark.ml.param.Param-), [getParam](../../../../org/apache/spark/ml/param/Params.html#getParam-java.lang.String-), [hasDefault](../../../../org/apache/spark/ml/param/Params.html#hasDefault-org.apache.spark.ml.param.Param-), [hasParam](../../../../org/apache/spark/ml/param/Params.html#hasParam-java.lang.String-), [isDefined](../../../../org/apache/spark/ml/param/Params.html#isDefined-org.apache.spark.ml.param.Param-), [isSet](../../../../org/apache/spark/ml/param/Params.html#isSet-org.apache.spark.ml.param.Param-), [onParamChange](../../../../org/apache/spark/ml/param/Params.html#onParamChange-org.apache.spark.ml.param.Param-), [set](../../../../org/apache/spark/ml/param/Params.html#set-org.apache.spark.ml.param.Param-T-), [set](../../../../org/apache/spark/ml/param/Params.html#set-org.apache.spark.ml.param.ParamPair-), [set](../../../../org/apache/spark/ml/param/Params.html#set-java.lang.String-java.lang.Object-), [setDefault](../../../../org/apache/spark/ml/param/Params.html#setDefault-org.apache.spark.ml.param.Param-T-), [setDefault](../../../../org/apache/spark/ml/param/Params.html#setDefault-scala.collection.Seq-), [shouldOwn](../../../../org/apache/spark/ml/param/Params.html#shouldOwn-org.apache.spark.ml.param.Param-)`  
     * ### Methods inherited from interface org.apache.spark.ml.util.[Identifiable](../../../../org/apache/spark/ml/util/Identifiable.html "interface in org.apache.spark.ml.util")  
     `[toString](../../../../org/apache/spark/ml/util/Identifiable.html#toString--)`  
     * ### Methods inherited from interface org.apache.spark.internal.Logging  
     `$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize`

• • Constructor Detail

     * #### Pipeline  
     public Pipeline(String uid)  
     * #### Pipeline  
     public Pipeline()  

  • Method Detail

    * #### read  
    public static [MLReader](../../../../org/apache/spark/ml/util/MLReader.html "class in org.apache.spark.ml.util")<[Pipeline](../../../../org/apache/spark/ml/Pipeline.html "class in org.apache.spark.ml")> read()  
    * #### load  
    public static [Pipeline](../../../../org/apache/spark/ml/Pipeline.html "class in org.apache.spark.ml") load(String path)  
    * #### uid  
    public String uid()  
    An immutable unique ID for the object and its derivatives.  
    Specified by:  
    `[uid](../../../../org/apache/spark/ml/util/Identifiable.html#uid--)` in interface `[Identifiable](../../../../org/apache/spark/ml/util/Identifiable.html "interface in org.apache.spark.ml.util")`  
    Returns:  
    (undocumented)  
    * #### stages  
    public [Param](../../../../org/apache/spark/ml/param/Param.html "class in org.apache.spark.ml.param")<[PipelineStage](../../../../org/apache/spark/ml/PipelineStage.html "class in org.apache.spark.ml")[]> stages()  
    param for pipeline stages  
    Returns:  
    (undocumented)  
    * #### setStages  
    public [Pipeline](../../../../org/apache/spark/ml/Pipeline.html "class in org.apache.spark.ml") setStages([PipelineStage](../../../../org/apache/spark/ml/PipelineStage.html "class in org.apache.spark.ml")[] value)  
    * #### getStages  
    public [PipelineStage](../../../../org/apache/spark/ml/PipelineStage.html "class in org.apache.spark.ml")[] getStages()  
    * #### fit  
    public [PipelineModel](../../../../org/apache/spark/ml/PipelineModel.html "class in org.apache.spark.ml") fit([Dataset](../../../../org/apache/spark/sql/Dataset.html "class in org.apache.spark.sql")<?> dataset)  
    Fits the pipeline to the input dataset with additional parameters. If a stage is an[Estimator](../../../../org/apache/spark/ml/Estimator.html "class in org.apache.spark.ml"), its `Estimator.fit` method will be called on the input dataset to fit a model. Then the model, which is a transformer, will be used to transform the dataset as the input to the next stage. If a stage is a [Transformer](../../../../org/apache/spark/ml/Transformer.html "class in org.apache.spark.ml"), its `Transformer.transform` method will be called to produce the dataset for the next stage. The fitted model from a [Pipeline](../../../../org/apache/spark/ml/Pipeline.html "class in org.apache.spark.ml") is an[PipelineModel](../../../../org/apache/spark/ml/PipelineModel.html "class in org.apache.spark.ml"), which consists of fitted models and transformers, corresponding to the pipeline stages. If there are no stages, the output model acts as an identity transformer.  
    Specified by:  
    `[fit](../../../../org/apache/spark/ml/Estimator.html#fit-org.apache.spark.sql.Dataset-)` in class `[Estimator](../../../../org/apache/spark/ml/Estimator.html "class in org.apache.spark.ml")<[PipelineModel](../../../../org/apache/spark/ml/PipelineModel.html "class in org.apache.spark.ml")>`  
    Parameters:  
    `dataset` \- input dataset  
    Returns:  
    fitted pipeline  
    * #### copy  
    public [Pipeline](../../../../org/apache/spark/ml/Pipeline.html "class in org.apache.spark.ml") copy([ParamMap](../../../../org/apache/spark/ml/param/ParamMap.html "class in org.apache.spark.ml.param") extra)  
    Description copied from interface: `[Params](../../../../org/apache/spark/ml/param/Params.html#copy-org.apache.spark.ml.param.ParamMap-)`  
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See `defaultCopy()`.  
    Specified by:  
    `[copy](../../../../org/apache/spark/ml/param/Params.html#copy-org.apache.spark.ml.param.ParamMap-)` in interface `[Params](../../../../org/apache/spark/ml/param/Params.html "interface in org.apache.spark.ml.param")`  
    Specified by:  
    `[copy](../../../../org/apache/spark/ml/Estimator.html#copy-org.apache.spark.ml.param.ParamMap-)` in class `[Estimator](../../../../org/apache/spark/ml/Estimator.html "class in org.apache.spark.ml")<[PipelineModel](../../../../org/apache/spark/ml/PipelineModel.html "class in org.apache.spark.ml")>`  
    Parameters:  
    `extra` \- (undocumented)  
    Returns:  
    (undocumented)  
    * #### transformSchema  
    public [StructType](../../../../org/apache/spark/sql/types/StructType.html "class in org.apache.spark.sql.types") transformSchema([StructType](../../../../org/apache/spark/sql/types/StructType.html "class in org.apache.spark.sql.types") schema)  
    Check transform validity and derive the output schema from the input schema.  
     We check validity for interactions between parameters during `transformSchema` and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by `Param.validate()`.  
     Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.  
    Specified by:  
    `[transformSchema](../../../../org/apache/spark/ml/PipelineStage.html#transformSchema-org.apache.spark.sql.types.StructType-)` in class `[PipelineStage](../../../../org/apache/spark/ml/PipelineStage.html "class in org.apache.spark.ml")`  
    Parameters:  
    `schema` \- (undocumented)  
    Returns:  
    (undocumented)  
    * #### write  
    public [MLWriter](../../../../org/apache/spark/ml/util/MLWriter.html "class in org.apache.spark.ml.util") write()  
    Description copied from interface: `[MLWritable](../../../../org/apache/spark/ml/util/MLWritable.html#write--)`  
    Returns an `MLWriter` instance for this ML instance.  
    Specified by:  
    `[write](../../../../org/apache/spark/ml/util/MLWritable.html#write--)` in interface `[MLWritable](../../../../org/apache/spark/ml/util/MLWritable.html "interface in org.apache.spark.ml.util")`  
    Returns:  
    (undocumented)