Cannot Read a file from HDFS using Spark

Cannot Read a file from HDFS using Spark

Asked on December 22, 2018 in Apache-spark.
Add Comment


  • 3 Answer(s)

    This could be the best solution for this issue.

    sc.textFile("hdfs://nn1home:8020/input/war-and-peace.txt")
    
    

    How will we find out nn1home:8020?

    For the above problem simply search for the file core-site.xml and look for xml element fs.defaultFS.

    Answered on December 22, 2018.
    Add Comment

    When we need to use sc.textFile(“hdfs://…”) we want to give the full path(absolute path), in instance that would be “nn1home:8020/..”

    To create it easier, then simply use sc.textFile(“hdfs:/input/war-and-peace.txt”)

     

    Answered on December 22, 2018.
    Add Comment

    The below code will be helpful and it works.

    val textFile = sc.textFile("hdfs://localhost:9000/user/input.txt")
    

    In this we can use  localhost:9000 from hadoop core-site.xml config file’s fs.defaultFS parameter value.

    Answered on December 22, 2018.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.