Spark Streaming - Stream Processing in Lakehouse - PySpark (Udemy.com)
Master Spark Structured Streaming using Python (PySpark) on Azure Databricks Cloud with a end-to-end Project
Created by: Prashant Kumar Pandey
Last updated August 2024
What you will learn
- Real-time Stream Processing Concepts
- Spark Structured Streaming APIs and Architecture
- Working with Streaming Sources and Sinks
- Kafka for Data Engineers
- Working With Kafka Source and Integrating Spark with Kafka
- State-less and State-full Streaming Transformations
- Windowing Aggregates using Spark Stream
- Watermarking and State Cleanup
- Streaming Joins and Aggregation
- Handling Memory Problems with Streaming Joins
- Working with Azure Databricks
- Capstone Project - Streaming application in Lakehouse
Course Description
About the Course
I am creating Apache Spark and Databricks - Stream Processing in Lakehouse using the Python Language and PySpark API. This course will help you understand Real-time Stream processing using Apache Spark and Databricks Cloud and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session-like approach. We will take a live coding approach and explain all the needed concepts.
Capstone Project
This course also includes an End-To-End Capstone project. The project will help you understand the real-life project design, coding, implementation, testing, and CI/CD approach.
Who should take this Course?
I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with those implementing Apache Spark at the ground level.
Spark Version used in the Course.
This Course is using the Apache Spark 3.5. I have tested all the source code and examples used in this Course on Azure Databricks Cloud using Databricks Runtime 14.1.
Instructor Details
- 4.9 Rating
2,251 Reviews
Prashant Kumar Pandey
Prashant Kumar Pandey has spent over 25 years in IT as a developer, architect, consultant, trainer, and mentor, working with international software services organisations on data-centric, Big Data, and AI projects. That production experience — not just theory — is what shapes how he teaches Databricks, Spark, and cloud data engineering today.
He is the founder, lead author, and chief editor of ScholarNest, a skill-development platform offering courses, training, and technical articles since 2018. He also publishes free training content on his YouTube channel, Learning Journal, and writes for a following of 43,000+ data engineering professionals on LinkedIn.
Prashant is a firm believer in lifelong, continuous learning. Alongside his course work, he authors technical books and articles to help IT professionals and students close the gap between the skills they have and the skills the industry actually needs — the same gap this course is built to close.
More courses by Prashant Kumar Pandey
PySpark - Apache Spark Programming for Beginners (2026)
4.7 (17,491 Reviews)
Provider: Udemy
Time: 28.5h
$89.99
Apache Spark - Beyond Basics and Cracking Job Interviews (2024)
4.7 (5,132 Reviews)
Provider: Udemy
Time: 4.1h
$11.99
Apache Spark Programming in Scala for Beginners (2026)
4.4 (3,470 Reviews)
Provider: Udemy
Time: 17.5h
$84.99
Apache Kafka - Real-time Stream Processing (Master Class) (2023)
4.8 (2,925 Reviews)
Provider: Udemy
Time: 11h
$10.99
Python Programming - Foundation Course for Data Engineers (2024)
4.6 (1,417 Reviews)
Provider: Udemy
Time: 4.4h
$11.99
More Hadoop courses
Apache Kafka Series - KSQL on ksqlDB for Stream Processing ! (2026)
4.9 (2,161 Reviews)
Provider: Udemy
Time: 4h
$12.99
Apache Kafka Series - Kafka Streams for Data Processing (2026)
4.8 (6,826 Reviews)
Provider: Udemy
Time: 4.8h
$14.99
Apache Kafka Series - Confluent Schema Registry & REST Proxy (2026)
4.8 (3,794 Reviews)
Provider: Udemy
Time: 4.5h
$94.99
Apache Kafka - Real-time Stream Processing (Master Class) (2023)
4.8 (2,925 Reviews)
Provider: Udemy
Time: 11h
$10.99
Apache Spark 3 & Big Data Essentials in Scala (2024)
4.8 (2,302 Reviews)
Provider: Udemy
Time: 7.3h
$39.99
PySpark - Apache Spark Programming for Beginners (2026)
4.7 (17,491 Reviews)
Provider: Udemy
Time: 28.5h
$89.99
Reviews
By Sri Pal on 11/23/2025
I really enjoyed all your courses and you are one of the best instructors on Udemy. I registered for one course, realized the depth of your knowledge and exteperise, and your way of structuring the lectures to break complex topics - and became a fan. I registered for all your courses! thanks Mr Pandey!
By Deepu M on 10/31/2025
The course covers all the essential topics and provides a good foundation for beginners. However, the project explanations could be more detailed to help learners better connect the concepts. Also, the codebase could use an update to align with the latest Databricks Free Tier offerings. Overall, it’s a valuable course with room for improvement in clarity and relevance.
By Bala Kishore Kotyada on 5/25/2025
Best course for spark streaming. In depth explanations with good examples. Prasanth sir courses are must watch to learn pyspark in general. This is my third course and already enrolled for the next course. I am from mechanical background with zero knowledge in IT. Enrolled in this course for career transition and I must tell you I got good understanding of spark with the help of his courses.
By Anindya Roy on 5/14/2025
Very well structured and all the basic concepts are first covered clearly. Structured streaming is a very complicated thing and spending a whole chapter to understand the problems associated with it and how to solve for it was very useful. This is brilliant conceptualization of how best to teach the subject.
By Siddartha Yalavarthy on 6/26/2024
Been following this tutor from the beginning of my PySpark Journey. One of the very few courses where unit testing was taken seriously instead of just skimming through the main concepts. Very detailed explanation of even the minute topics has been discussed in this course. To the tutor, Kindly reduce the buzzing noise in the videos and the improve the overall Audio quality. 100% would recommend this course.
By Rami Vemula on 6/23/2024
Prashant Kumar is a very knowledgeable instructor. I found the streaming aggregates section and Capstone project very interesting and useful. All the aspects of streaming queries are covered well with clear distinction with the batch processing concepts. This course is going to help me in my next career move as a Data Engineer. However I feel 22.5 hours is lengthy for this course, especially the introduction to Kafka and Old course content can be removed. Also move the Streaming joins content to the actual course catalog. It would be awesome to see this course to be around 12 - 14 hours. Thank you so much for the course.
By Hua Yang on 4/30/2024
Great course!. It is well prepared with clear and detailed explanation on the key concepts of Structured Streaming and its architecture. I agreed with other that this is one of the best courses on Udemy. It is practical. It propels my journey to mastering modern ETL application.
By Martin Kunc on 4/2/2024
Hi Prashant, thank you for preparing the course. It must have take you a significant time. I must say I have especially liked come parts of old course, when you were describing all the streaming concepts without focusing on databricks, or unit testing. These woudl probably deserve courses on their own. I like the entend in which you have covered the streaming concepts and your course is a unique find in this regard. I hope I will see what else you have to share. Thank you again and enjoy your time, Martin
By Abhishek Mani Tripathi on 2/5/2024
Prashant courses are always upto the industry standards which just does not increase the learning curve but also helps in cracking opportunities as per industry standards. Thanks a lot Prashant for these amazing contents , Keep creating more contents on Databricks projects, Big data analytics projects. Thanks again!!!!!!!!!!!!
By Arijit Das on 1/17/2024
This course is amazingly designed and structured. Each topic is designed with in relation to previous topic which is really helpful and understandable. The course helps you in many ways, 1. If you don't know the concept, the course helps you to understand and implement in depth. 2. If you know the concept, it will be your excellent refresher. 3. And in both cases, it will make you knowledgeable to design and implement the Spark Streaming in your projects. 4. Capstone project helps, new people to understand how to put all concepts together to solve real world problems. 5. Also, this course will help you to clear interviews in DE regarding Spark Streaming and 6. It helps you to understand and answer concepts in Databricks certification. Overall, it is a must have course in your bucket if you want to grow more in Spark Streaming. Excepting more such content from Prashant Pandey, on the space of Data Engineering and Databricks Architect.
Quality Score
No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.
Overall Score : 98 / 100










