Techie BabajiLearn to Code. Build Real Skills.
Scala · Concurrency, Testing & Ecosystem

Apache Spark Basics

Apache Spark processes huge datasets across many machines. Scala is Spark’s native language.

Advanced 5 min Lesson 24 of 25Examples use Scala 3 (3.3 LTS or newer) with sbt or scala-cli.
Before you start:CollectionsCase Classes
0/25 in Scala
On this page · 5 min · Advanced

On this page

  1. In simple words
  2. Learning objectives
  3. Explanation
  4. Code example
  5. Step by step
  6. Real-world use
  7. Common mistakes
  8. Best practices
  9. Remember this
  10. Practice exercise
  11. Mini quiz
  12. Related topics
  13. My notes
Reading time
5 min
Difficulty
Advanced
Updated
Oct 30, 2026

Related tutorials

Interview questions

In simple words#

Spark is like dividing a mountain of exam papers among 100 teachers: each grades a pile at the same time, then the results are combined. Your code says what to do; Spark decides how to split the work.

SparkSession
Entry point to Spark.
DataFrame
Distributed table with named columns.
Lazy evaluation
Nothing runs until an action like show or write.

Learning objectives#

  • Create a SparkSession
  • Load and transform DataFrames
  • Run Spark SQL
  • Use typed Datasets

Explanation#

Add spark-sql to build.sbt (Spark 3.5 supports Scala 2.13; Spark 4 also targets Scala 2.13 — use 2.13 for Spark projects).

Transformations (select, filter, groupBy, join) are lazy; actions (show, count, write) trigger execution.

DataFrames can be queried with methods or SQL via createOrReplaceTempView.

Datasets (Dataset[Order]) add compile-time types using case classes.

Code example: Sales by city#

ScalaSales by city Illustrative
// build.sbt: scalaVersion := "2.13.15"
//            libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.3"
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._


object SalesReport {
  case class Order(city: String, product: String, amount: Double)


  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("SalesReport").master("local[*]").getOrCreate()
    import spark.implicits._


    val orders = Seq(
      Order("Pune", "Laptop", 55000), Order("Delhi", "Mouse", 1400),
      Order("Pune", "Keyboard", 2200), Order("Delhi", "Laptop", 52000)).toDS()
    // Real data: spark.read.option("header", "true").option("inferSchema", "true").csv("s3://bucket/orders.csv")


    orders.groupBy($"city")
      .agg(sum($"amount").as("total"), count("*").as("orders"))
      .orderBy(desc("total"))
      .show()


    orders.createOrReplaceTempView("orders")
    spark.sql("SELECT product, SUM(amount) AS revenue FROM orders GROUP BY product ORDER BY revenue DESC").show()


    spark.stop()
  }
}
Expected output
+-----+-------+------+
| city|  total|orders|
+-----+-------+------+
| Pune|57200.0|     2|
|Delhi|53400.0|     2|
+-----+-------+------+

+--------+--------+
| product| revenue|
+--------+--------+
|  Laptop|107000.0|
|Keyboard|  2200.0|
|   Mouse|  1400.0|
+--------+--------+

This example is illustrative. Copy or download it and run it in your own environment; the expected output is shown above.

Step-by-step explanation#

  1. SparkSession starts Spark locally using all cores.
  2. toDS creates a typed Dataset from case classes.
  3. groupBy + agg computes totals; show is the action.
  4. The same data is queried with Spark SQL.

Real-world use case#

Banks and e-commerce companies run nightly Spark jobs on clusters (Databricks, EMR, HDInsight) to build reports and ML features.

Common mistakes#

Calling collect() on huge data

It pulls everything to the driver and crashes it. Use show, take or write.

Scala version mismatch

Spark artifacts are built for specific Scala versions (2.12/2.13). Match scalaVersion.

Best practices#

  • Use built-in functions over UDFs
  • Avoid collect on big data
  • Partition and cache wisely
  • Test transformations on small samples

Remember this#

Transformations are lazy, actions run.

DataFrame = distributed table.

Match Spark’s Scala version.

Practice exercise#

Your task: Filter orders above ₹2,000 and count them.

Show solution
Scala Illustrative
val bigCount = orders.filter($"amount" > 2000).count()

Mini quiz#

Apache Spark Basics quizMultiple choice Question 1 of 1

Which is a Spark action that triggers execution?

Score so far: 0/0

Want more? Practise all Scala MCQs · Scala interview questions

My notes#

Saved in this browser.

Progress is saved in this browser. Sign in to sync.