Data Engineering using Kafka and Spark Structured Streaming (Udemy.com)
A comprehensive Data Engineering course on building streaming pipelines using Kafka and Spark Structured Streaming
Created by: Durga Viswanatha Raju Gadiraju
Last updated December 2024
What you will learn
- Setting up self support lab with Hadoop (HDFS and YARN), Hive, Spark, and Kafka
- Overview of Kafka to build streaming pipelines
- Data Ingestion to Kafka topics using Kafka Connect using File Source
- Data Ingestion to HDFS using Kafka Connect using HDFS 3 Connector Plugin
- Overview of Spark Structured Streaming to process data as part of Streaming Pipelines
- Incremental Data Processing using Spark Structured Streaming using File Source and File Target
- Integration of Kafka and Spark Structured Streaming - Reading Data from Kafka Topics
How it compares
Against the other 388 rated Udemy courses in SQL on CourseDuck:
- Rating4.6 out of 5higher than 69% of them
- Reviews328#248 of 389 by number of reviews
- Length9.6 hourslonger than 70% of them
- Last updatedDecember 2024more recently than 61% of them
Course Description
As part of this course, you will be learning to build streaming pipelines by integrating Kafka and Spark Structured Streaming. Let us go through the details about what is covered in the course.
First of all, we need to have the proper environment to build streaming pipelines using Kafka and Spark Structured Streaming on top of Hadoop or any other distributed file system. As part of the course, you will start with setting up a self-support lab with all the key components such as Hadoop, Hive, Spark, and Kafka on a single node Linux-based system.
Once the environment is set up you will go through the details related to getting started with Kafka. As part of that process, you will create a Kafka topic, produce messages into the topic as well as consume messages from the topic.
You will also learn how to use Kafka Connect to ingest data from web server logs into Kafka topic as well as ingest data from Kafka topic into HDFS as a sink.
Once you understand Kafka from the perspective of Data Ingestion, you will get an overview of some of the key concepts of related Spark Structured Streaming.
After learning Kafka and Spark Structured streaming separately, you will build a streaming pipeline to consume data from Kafka topic using Spark Structured Streaming, then process and write to different targets.
You will also learn how to take care of incremental data processing using Spark Structured Streaming.
Course Outline
Here is a brief outline of the course. You can choose either Cloud9 or GCP to provision a server to set up the environment.
Setting up Environment using AWS Cloud9 or GCP
Setup Single Node Hadoop Cluster
Setup Hive and Spark on top of Single Node Hadoop Cluster
Setup Single Node Kafka Cluster on top of Single Node Hadoop Cluster
Getting Started with Kafka
Data Ingestion using Kafka Connect - Web server log files as a source to Kafka Topic
Data Ingestion using Kafka Connect - Kafka Topic to HDFS a sink
Overview of Spark Structured Streaming
Kafka and Spark Structured Streaming Integration
Incremental Loads using Spark Structured Streaming
Udemy based support
In case you run into technical challenges while taking the course, feel free to raise your concerns using Udemy Messenger. We will make sure that issue is resolved in 48 hours.
Instructor Details
- 4.6 Rating
328 Reviews
Durga Viswanatha Raju Gadiraju
20+ years of experience in executing complex projects using a vast array of technologies including Big Data and the Cloud.
ITVersity, Inc. - is a US-based organization that provides quality training for IT professionals and we have a track record of training hundreds of thousands of professionals globally.
Building an IT career for people with required tools such as high-quality material, labs, live support, etc to upskill and cross-skill is paramount for our organization.
At this time our training offerings are focused on the following areas:
* Application Development using Python and SQL
* Big Data and Business Intelligence
* Cloud
* Datawarehousing, Databases
More courses by Durga Viswanatha Raju Gadiraju
Data Engineering for Beginners: Learn SQL, Python & Spark (2026)
4.3 (8,867 Reviews)
Provider: Udemy
Time: 56h
$119.99
Data Engineering using AWS Data Analytics (2024)
4.5 (3,535 Reviews)
Provider: Udemy
Time: 25.4h
$119.99
Spark SQL and Spark 3 using Scala Hands-On with Labs (2023)
4.4 (3,127 Reviews)
Provider: Udemy
Time: 24.2h
$74.99
Master Apache Spark using Spark SQL and PySpark 3 (2024)
4.3 (2,655 Reviews)
Provider: Udemy
Time: 32.2h
$10.99
Data Engineering using Databricks on AWS and Azure (2024)
4.6 (2,138 Reviews)
Provider: Udemy
Time: 19h
$14.99
Databricks Certified Associate Developer - Apache Spark (2025)
4.0 (2,015 Reviews)
Provider: Udemy
Time: 14.4h
$10.99
More SQL courses
SQL - The Complete Developer's Guide (MySQL, PostgreSQL) (2025)
4.8 (2,153 Reviews)
Provider: Udemy
Time: 19.7h
$94.99
SQL Server High Availability and Disaster Recovery (HA/DR) (2026)
4.8 (1,810 Reviews)
Provider: Udemy
Time: 88.3h
$16.99
70-461 Session 3: Querying Microsoft SQL Server (T-SQL code) (2025)
4.8 (1,094 Reviews)
Provider: Udemy
Time: 3.6h
$12.99
Modern Event-Driven Spring Boot: Kafka & Spring Cloud Stream (2026)
4.8 (494 Reviews)
Provider: Udemy
Time: 13.4h
$10.99
Oracle Database 12c Backup and Recovery using RMAN (2024)
4.8 (428 Reviews)
Provider: Udemy
Time: 20.1h
$34.99
Reviews
By Nicholas Otieno on 5/16/2025
The course is well structured and has good details to learn how to use and set up kafka and spark. I requested itversity for access to their labs and they promptly granted me. I highly recommend the course!
By Maka Krishna Jaswanth on 9/14/2024
Excellent course for learning kafka and spark structured streaming. Concepts have been explained very well. I would highly recommend this course for anyone who is in data engineer or any relevant fields.
By Karan Malhotra on 8/14/2024
This course is really awesome, I would highly recommend this to someone who want to excel in kafka & spark streaming. we get complementary lab access as part of this course which is really useful.
By Hardik Jain on 8/8/2024
Extremely helpful course, I would highly recommend this to someone who want to excel hands-on in Kafka & Spark Streaming. Additionally complementry lab access provides an edge over all the other courses.
By Ramaa on 8/3/2024
Architecture diagrams could be used to explain the placement of various components discussed and how they work together for solving a problem. And what is the main problem it solves, how performant or scalable it is could ve been discussed.
By Anonymized User on 5/16/2024
All in all, a decent course, especially near the end after I got used to the teaching style. Parts that could be better: - the speaker's accent is very strong and also does not get transcribed properly in the captions, which I needed to use because the speaking was not nice to listen to. Also, kept repeating filler words such as 'nothing but' which gets transcribed as well into the captions which is hard to read and not ideal; - teacher spent a lot of time repeating things and jumping from written code, copying and pasting, reading what he had written in jupyter notebook instead of actually teaching, so the presentation was not great and the teaching of concepts suffered as a result; - would have been nice to get a better overview of concepts (instead of repeating same code over and over) and how different parts and levels within each tool (spark, hive, hadoop, kafka, etc) interplay with one another; - would have been nice to have different example code. was cool when he got data from github, but not when he kept re-using generated logs as an example. - several times in the intro and beginning repeatedly said to give 5 stars or quit, which is ridiculous. also means that the ratings are skewed and not fair. I would give the first half of the course a 3 star rating, and 2nd half 4 stars.
By Kris Guru on 2/20/2024
Good course, the reason for 4 stars is because, the content is outdated for one thing, for instance, while creating postrgres image using docker, following the instructions of the video/ material did not work. I had to research quite a lot and then checked the Q&A to find that the version 13 image of Postgres should be used to create the container. There are many such instances where the content feels outdated as these were recorded back in 2021. The other reason for giving 4 stars is the course does not focus much on explaining why the commands are being run or what they mean, instead, the instructor provides the instructions just to run the commands. Other than these issues, the course is pretty good.
By Satyabrata Dey on 1/9/2023
Streaming is really evolving more and more in DE landscape and this course is a really good. The complimentary lab access, keeping in mind, the installation difficulty, is really thoughtful and helps you to start faster. The only improvement may be I can think of, adding some real life streaming architectural patterns for discussion, adding some note on Confluent and ksql maybe. But overall really good.
By Krupa on 1/1/2022
Those who are looking for structured Streaming with kafka, should take this course and good response also i am getting. Thanks DurgaSir jii. Please add the content to sink with any db like Cassandra. already requested
By Arjun Sharma on 12/31/2021
Absolutely loved this course. Instructor is well prepared and able to explain really well. A huge positive is that it gives you access to Free lab where you can do your practices. Highly Recommend. Will prefer for future courses as well. Thanks!
Quality Score
No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.
Overall Score : 92 / 100












