Scala Spark DataFrame : dataFrame.select multiple columns given a Sequence of column names

Scala Spark DataFrame : dataFrame.select multiple columns given a Sequence of column names

Asked on January 12, 2019 in Apache-spark.
Add Comment


  • 2 Answer(s)

    The following code will be helpful :

    val columnNames = Seq("col1","col2",....."coln")
     
    // using the string column names:
    val result = dataframe.select(columnNames.head, columnNames.tail: _*)
     
    // or, equivalently, using Column objects:
    val result = dataframe.select(columnNames.map(c => col(c)): _*)
    
    Answered on January 12, 2019.
    Add Comment
    val columnName=Seq("col1","col2",....."coln");

    Since dataFrame.select() expect the sequence of columns as parmas, so since we have sequence of string so convert the sequence of string to the List of Col and convert that list to the sequence so

    columnName.map(name => col(name)):_*
    gives the sequence of column from the sequence of string, and this passed as a parameters to the dataFrame.select()

      val columnName=Seq("col1","col2");
      val DFFiltered = DF.select(columnName.map(name => col(name)):_*)
    
    
    
    Answered on January 13, 2019.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.