How to sort by column in descending order in Spark SQL ?

How to sort by column in descending order in Spark SQL ?

Asked on November 15, 2018 in Apache-spark.
Add Comment


  • 3 Answer(s)

    This can be done by sorting the column by importing the spark sql functions

    import org.apache.spark.sql.functions._
    df.orderBy(asc("col1"))
    

    Or

    import org.apache.spark.sql.functions._
    df.sort(desc("col1"))
    

    This is by importing sqlContext.implicits._

    import sqlContext.implicits._
    df.orderBy($"col1".desc)
    

    Or

    import sqlContext.implicits._
    df.sort($"col1".desc)
    
    Answered on November 15, 2018.
    Add Comment

    This is an org.apache.spark.sql.DataFrame for sort method:

    df.sort($"col1", $"col2".desc)
    
    

    Note: In this $ and .desc inside sort for the column to sort the results by.

    Answered on November 15, 2018.
    Add Comment

    This is a simplest way by adding the parameter ascending=False:

    df.orderBy("col1", ascending=False).show(10)
    
    

    This could be used as reference:

    http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.DataFrame.orderBy

    Answered on November 15, 2018.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.