pyspark.sql.DataFrame.distinct — PySpark 3.5.5 documentation (original) (raw)

DataFrame. distinct() → pyspark.sql.dataframe.DataFrame[source]¶

Returns a new DataFrame containing the distinct rows in this DataFrame.

New in version 1.3.0.

Changed in version 3.4.0: Supports Spark Connect.

Returns

DataFrame

DataFrame with distinct records.

Examples

df = spark.createDataFrame( ... [(14, "Tom"), (23, "Alice"), (23, "Alice")], ["age", "name"])

Return the number of distinct rows in the DataFrame

df.distinct().count() 2