get specific row from spark dataframe

At starting, DataFrames are distributed, needs to be understood, In typical procedural way this cannot be accessed , At first analysis process is done. With the Scala, here recommended to read the Pyspark Documentation, because this contains more details.

Here we can use some methods of the RDD API cause all DataFrames have one RDD as attribute.

For instance

df = sqlContext.createDataFrame([("a", 1), ("b", 2), ("c", 3)], ["letter", "name"])
myIndex = 1
values = (df.rdd.zipWithIndex()
        .filter(lambda ((l, v), i): i == myIndex)
        .map(lambda ((l,v), i): (l, v))
        .collect())
 
print(values[0])
# (u'b', 2)

The getrows() function below should get the specific rows you want.

For completeness, I have written down the full code in order to reproduce the output.

# Create SparkSession
from pyspark.sql import SparkSession
spark = SparkSession.builder.master(‘local’).appName(‘scratch’).getOrCreate()

# Create the dataframe
df = spark.createDataFrame([(“a”, 1), (“b”, 2), (“c”, 3)], [“letter”, “name”])

# Function to get rows at `rownums`
def getrows(df, rownums=None):
return df.rdd.zipWithIndex().filter(lambda x: x[1] in rownums).map(lambda x: x[0])

# Get rows at positions 0 and 2.
getrows(df, rownums=[0, 2]).collect()

# Output:
#> [(Row(letter=’a’, name=1)), (Row(letter=’c’, name=3))

Is there any alternative for df in scala spark data frames. I want to select specific row from a column of spark data frame. for example 100th row in above R equivalent code.

val parquetFileDF = sqlContext.read.parquet("myParquetFule.parquet")

val myRow7th = parquetFileDF.rdd.take(7).last

get specific row from spark dataframe
apache-spark apache-spark-sql
Is there any alternative for df[100, c(“column”)] in scala spark data frames. I want to select specific row from a column of spark data frame. for example 100th row in above R equivalent codeThe getrows() function below should get the specific rows you want.

For completeness, I have written down the full code in order to reproduce the output.

# Create SparkSession
from pyspark.sql import SparkSession
spark = SparkSession.builder.master(‘local’).appName(‘scratch’).getOrCreate()

# Create the dataframe
df = spark.createDataFrame([(“a”, 1), (“b”, 2), (“c”, 3)], [“letter”, “name”])

# Function to get rows at `rownums`
def getrows(df, rownums=None):
return df.rdd.zipWithIndex().filter(lambda x: x[1] in rownums).map(lambda x: x[0])

# Get rows at positions 0 and 2.
getrows(df, rownums=[0, 2]).collect()

# Output:
#> [(Row(letter=’a’, name=1)), (Row(letter=’c’, name=3))]

val index = ss.sparkContext
    .parallelize(Seq((1,1), (2,0), (3,2), (4,0), (5,1)))
    .toDF("ID", "value")

index.where($"value" === 0).select("ID").show()

There is a scala way (if you have a enough memory on working machine):

val arr = df.select("column").rdd.collect
println(arr(100))
If dataframe schema is unknown, and you know actual type of "column" field (for example double), than you can get arr as following:

val arr = df.select($"column".cast("Double")).as[Double].rdd.collect

row. getAs [String](null)

scala> val row = Row(1, "hello")
row: org.apache.spark.sql.Row = [1,hello]

scala> row(1)
res0: Any = hello

scala> row.get(1)
res1: Any = hello

DataFrame — Dataset of Rows with RowEncoder

type DataFrame = Dataset[Row]

data.groupBy('Product_ID).sum('Score)

val df = Seq(("one", 1), ("one", 1), ("two", 1))
  .toDF("word", "count")

scala> df.show
+----+-----+
|word|count|
+----+-----+
| one|    1|
| one|    1|
| two|    1|
+----+-----+

val counted = df.groupBy('word).count

scala> counted.show
+----+-----+
|word|count|
+----+-----+
| two|    1|
| one|    2|
+----+-----+