Apache Spark & Scala

Overview

This course introduces learners to Apache Spark and Scala for big data processing. Starting with an overview of big data and the Hadoop ecosystem, it progresses into Spark’s architecture and components. Learners will explore Scala fundamentals necessary for Spark development, followed by hands-on work with Spark Core including RDDs, transformations, and actions. The curriculum covers Spark SQL for structured data processing and advanced topics such as performance optimization, file formats, and integration with external systems. Practical exercises include real-world projects like employee data pipelines and retail sales analytics.

Objectives

• Understand the core concepts of big data processing and Apache Spark's architecture.
• Develop scalable data processing applications using Spark Core and Spark SQL with Scala.

Key topics

• Apache Spark Fundamentals
• Spark Core with RDDs and DataFrames
• Spark SQL and Data Processing
• Performance Optimization Techniques

Expected outcomes

• Build and execute Spark applications that process large datasets using RDDs and DataFrames.
• Perform data transformations, aggregations, and analytics using Spark SQL and structured streaming.

How the AI Tutor can help

How the Skillama AI Tutor helps with the listed lectures: explanations, follow-up questions, and running or debugging practice code.

Who this is for

This course is for developers and data engineers who want to learn Apache Spark and Scala for big data processing.

Curriculum

1. 1. Introduction to Big Data & Apache Spark

   • Introduction to Big Data
   • Hadoop Ecosystem Overview
   • Why Apache Spark?
   • Spark vs Hadoop MapReduce
   • Spark Components
   • Spark Architecture
   • Spark Cluster Manager
   • Spark Deployment Modes

2. 2. Scala Fundamentals for Spark

   • Scala Basics
   • Variables & Data Types
   • Functions
   • Collections
   • Object-Oriented Programming
   • Functional Programming
   • Lambda Expressions
   • Hands-on: Scala Programming Exercises

3. 3. Spark Core

   • Spark Context
   • Spark Session
   • RDD Introduction
   • Creating RDDs
   • Transformations
   • Actions
   • Lazy Evaluation
   • Caching & Persistence
   • Shared Variables
   • Broadcast Variables
   • Accumulators
   • Hands-on: Word Count
   • Hands-on: Log File Analysis
   • Hands-on: Employee Data Processing

4. 4. Spark SQL

   • DataFrames
   • Datasets
   • Schema Inference
   • Reading CSV, JSON & Parquet
   • SQL Queries
   • Temporary Views
   • Joins
   • Aggregations
   • Window Functions
   • Hands-on: Sales Data Analysis
   • Hands-on: Customer Analytics
   • Hands-on: Product Performance Report

5. 5. Data Processing with Spark

   • Filtering
   • Sorting
   • GroupBy
   • Union
   • Distinct
   • Drop Duplicates
   • Null Handling
   • User Defined Functions (UDF)
   • Hands-on: Employee Payroll Processing
   • Hands-on: Customer Segmentation

6. 6. Spark File Formats

   • CSV
   • JSON
   • Parquet
   • ORC
   • Avro (Introduction)
   • Hands-on: Read & Write Multiple File Formats

7. 7. Spark Performance Optimization

   • Partitioning
   • Repartition vs Coalesce
   • Caching Strategies
   • Broadcast Joins
   • Catalyst Optimizer
   • Tungsten Engine
   • Execution Plans
   • Hands-on: Performance Tuning Exercises

8. 8. Spark Streaming (Introduction)

   • Batch vs Streaming
   • Structured Streaming
   • Reading Streaming Data
   • Processing Real-Time Data
   • Writing Streaming Output
   • Hands-on: Live Log Stream Processing

9. 9. Spark Integration

   • Spark with Hive
   • Spark with HDFS
   • Spark with S3
   • Spark with MySQL
   • Spark with PostgreSQL
   • Hands-on: Import & Export Data
   • Hands-on: Database Connectivity

10. 10. Spark on Cloud

    • Spark on AWS EMR
    • Spark on Databricks
    • Spark on Google Cloud Dataproc
    • Spark on Azure HDInsight (Overview)
    • Hands-on: Execute Spark Jobs on Cloud

11. 11. Real-Time ETL Project

    • Employee Data Pipeline

12. 12. Capstone Project

    • Retail Sales Analytics Platform

13. 13. Mini Project

    • Flight Data Analysis
    • Flight Dataset Processing
    • Delay Analysis
    • Airline-wise Analysis
    • Airport-wise Analysis
    • Cancellation Analysis
    • Report Generation