PySpark

Overview

This course introduces learners to PySpark, the Python API for Apache Spark, enabling them to process and analyze large-scale datasets efficiently. Starting with an overview of big data concepts and Spark architecture, it progresses through core components like RDDs and DataFrames, and explores advanced topics such as structured streaming and performance tuning. Through hands-on exercises and projects, students gain practical experience in building real-world data engineering solutions.

Objectives

• Understand the fundamentals of big data processing using Apache Spark and its Python API, PySpark.
• Develop skills in creating and manipulating Resilient Distributed Datasets (RDDs) and DataFrames for data analysis.
• Learn how to perform advanced transformations, optimize performance, and build end-to-end ETL pipelines using PySpark.

Key topics

• Introduction to Big Data and PySpark
• Spark Core and RDD Operations
• DataFrames and Spark SQL
• Performance Optimization Techniques
• Structured Streaming and ETL Pipelines

Expected outcomes

• Build and execute scalable data processing applications using PySpark for large datasets.
• Design and implement efficient ETL pipelines that handle ingestion, transformation, and loading of structured and unstructured data.
• Apply performance optimization techniques to improve job execution time and resource usage in Spark jobs.

How the AI Tutor can help

How the Skillama AI Tutor helps with the listed lectures: explains the core concepts of PySpark and Spark components, answers follow-up questions on topics like RDDs, DataFrames, and streaming, and runs or debugs practice code for hands-on exercises such as word count and ETL pipelines.

Who this is for

This course is for data engineers, analysts, and developers who want to learn PySpark for big data processing and analytics.

Curriculum

1. 1. Introduction to Big Data & PySpark

   • Introduction to Big Data
   • Hadoop Ecosystem Overview
   • Why Apache Spark?
   • Spark vs Hadoop MapReduce
   • Spark Components
   • Spark Architecture
   • Spark Cluster Manager
   • Spark Deployment Modes
   • Introduction to PySpark
   • Hands on: First PySpark Program

2. 2. Python Fundamentals for PySpark

   • Python Basics
   • Variables & Data Types Operators
   • Operators
   • Conditional Statements
   • Loops
   • Functions
   • Lambda Functions
   • Lists, Tuples, Dictionaries & Sets
   • Exception Handling
   • Hands-on: Python Programming Exercises
   • Hands-on : Data Manipulation Tasks

3. 3. Spark Core

   • SparkContext
   • SparkSession
   • Creating RDDs
   • Parallelize Collections
   • Reading External Files
   • Transformations
   • Actions
   • Lazy Evaluation
   • Caching & Persistence
   • Broadcast Variables
   • Accumulators
   • Hands-on: Word Count Application
   • Hands-on: Log File Analysis
   • Hands-on: Employee Data Processing

4. 4. DataFrames & Spark SQL

   • Introduction to DataFrames
   • Schema Inference
   • Creating DataFrames
   • Reading CSV, JSON, Parquet & ORC Files
   • DataFrame Operations
   • Temporary Views
   • Spark SQL Queries
   • DataFrame vs RDD
   • Hands-on: Customer Data Analysis
   • Hands-on: Sales Report Generation

5. 5. Data Transformation

   • Select & Alias
   • Filter & Where
   • Sort & OrderBy
   • Distinct
   • Drop Duplicates
   • GroupBy & Aggregations
   • Joins
   • Union
   • Handling Null Values
   • Rename Columns
   • Hands-on: Employee Payroll Processing
   • Hands-on: Customer Segmentation

6. 6. Advanced PySpark

   • User Defined Functions (UDF)
   • Pandas UDF
   • Window Functions
   • Ranking Functions
   • Date & Time Functions
   • String Functions
   • Mathematical Functions
   • Hands-on: Sales Dashboard Dataset
   • Hands-on: Employee Performance Analytics

7. 7. File Formats & Data Sources

   • CSV
   • JSON
   • Parquet
   • ORC
   • Avro (Introduction)
   • Delta Lake (Overview)
   • Hands-on: Import & Export Data
   • Hands-on: Multi-format Data Processing

8. 8. Performance Optimization

   • Partitioning
   • Repartition vs Coalesce
   • Cache & Persist
   • Broadcast Join
   • Shuffle Operations
   • Catalyst Optimizer
   • Execution Plan Analysis
   • Best Practices
   • Hands-on: Optimize Large Data Processing Jobs

9. 9. Structured Streaming

   • Introduction to Streaming
   • Batch vs Streaming
   • Structured Streaming
   • Reading Stream Data
   • Stream Transformations
   • Writing Streaming Output
   • Hands-on: Real-Time Log Monitoring
   • Hands-on: Streaming Data Processing

10. 10. ETL Pipeline Development

    • ETL Concepts
    • Data Ingestion
    • Data Cleansing
    • Data Transformation
    • Data Validation
    • Data Loading
    • Logging & Monitoring
    • Hands-on: Build End-to-End ETL Pipeline

11. 11. Capstone Project

    • Enterprise Data Engineering Pipeline

12. 12. Mini Project

    • Bank Transaction Analysis
    • Transaction Data Processing
    • Credit & Debit Analysis
    • Customer-wise Analysis
    • Suspicious Transaction Detection
    • Search Functionality
    • File Storage
    • Report Generation