Convert date from String to Date format in Dataframes

Convert date from String to Date format in Dataframes

Asked on January 11, 2019 in Apache-spark.
Add Comment


  • 3 Answer(s)

    The solution is with Java SimpleDateFormatto_date can be used.

    TO_DATE(CAST(UNIX_TIMESTAMP(date, 'MM/dd/yyyy') AS TIMESTAMP))
    

    For instance:

    spark.sql("""
    SELECT TO_DATE(CAST(UNIX_TIMESTAMP('08/26/2016', 'MM/dd/yyyy') AS TIMESTAMP)) AS newdate"""
    ).show()
    +----------+
    | dt       |
    +----------+
    |2016-08-26|
    +----------+
    
    Answered on January 11, 2019.
    Add Comment

    Here without the temp table/view and with dataframe functions, The issue can be solved.

    This could work with yyyy-MM-DD.

    For instance:

    val df = sc.parallelize(Seq("2016-08-26")).toDF("Id")
    val df2 = df.withColumn("Timestamp", (col("Id").cast("timestamp")))
    val df3 = df2.withColumn("Date", (col("Id").cast("date")))
     
    df3.printSchema
     
    root
    |-- Id: string (nullable = true)
    |-- Timestamp: timestamp (nullable = true)
    |-- Date: date (nullable = true)
     
    df3.show
    +----------+--------------------+----------+
    | Id       | Timestamp          | Date     |
    +----------+--------------------+----------+
    |2016-08-26|2016-08-26 00:00:...|2016-08-26|
    +----------+--------------------+----------+
    
    Answered on January 11, 2019.
    Add Comment

    Here the main focus is to convert the type of a column in a DataFrame from String to Timestamp,

    And try by using the below code.

    import org.apache.spark.sql.functions.{to_date, to_timestamp}
    
    val modifiedDF = DF.withColumn("Date", to_date($"Date", "MM/dd/yyyy"))
    
    

    You could also use  (I think this is available from Spark 2.x) if you require fine grained timestamp.

    Alternatively for the version Spark 2.x, to_timestamp can be used.

    Answered on January 11, 2019.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.