Cannot Read a file from HDFS using Spark
This could be the best solution for this issue.
How will we find out nn1home:8020?
For the above problem simply search for the file core-site.xml and look for xml element fs.defaultFS.
When we need to use sc.textFile(“hdfs://…”) we want to give the full path(absolute path), in instance that would be “nn1home:8020/..”
To create it easier, then simply use sc.textFile(“hdfs:/input/war-and-peace.txt”)
The below code will be helpful and it works.
val textFile = sc.textFile("hdfs://localhost:9000/user/input.txt")
In this we can use localhost:9000 from hadoop core-site.xml config file’s fs.defaultFS parameter value.