Spark SQL and Spark 3 using Scala Hands-On with Labs (Udemy.com)
A comprehensive course on Spark SQL as well as Data Frame APIs using Scala with complementary lab access
Created by: Durga Viswanatha Raju Gadiraju
Last updated February 2023
What you will learn
- All the HDFS Commands that are relevant to validate files and folders in HDFS.
- Enough Scala to work Data Engineering Projects using Scala as Programming Language
- Spark Dataframe APIs to solve the problems using Dataframe style APIs.
- Basic Transformations such as Projection, Filtering, Total as well as Aggregations by Keys using Spark Dataframe APIs
- Inner as well as outer joins using Spark Data Frame APIs
- Ability to use Spark SQL to solve the problems using SQL style syntax.
- Basic Transformations such as Projection, Filtering, Total as well as Aggregations by Keys using Spark SQL
- Inner as well as outer joins using Spark SQL
- Basic DDL to create and manage tables using Spark SQL
- Basic DML or CRUD Operations using Spark SQL
- Create and Manage Partitioned Tables using Spark SQL
- Manipulating Data using Spark SQL Functions
How it compares
Against the other 150 rated Udemy courses in Hadoop on CourseDuck:
- Rating4.4 out of 5higher than 54% of them
- Reviews3,127#23 of 151 by number of reviews
- Length24.2 hourslonger than 93% of them
- Last updatedFebruary 2023earlier than 52% of them
Course Description
As part of this course, you will learn all the key skills to build Data Engineering Pipelines using Spark SQL and Spark Data Frame APIs using Scala as a Programming language. This course used to be a CCA 175 Spark and Hadoop Developer course for the preparation of the Certification Exam. As of 10/31/2021, the exam is sunset and we have renamed it to Spark SQL and Spark 3 using Scala as it covers industry-relevant topics beyond the scope of certification.
About Data Engineering
Data Engineering is nothing but processing the data depending on our downstream needs. We need to build different pipelines such as Batch Pipelines, Streaming Pipelines, etc as part of Data Engineering. All roles related to Data Processing are consolidated under Data Engineering. Conventionally, they are known as ETL Development, Data Warehouse Development, etc. Apache Spark is evolved as a leading technology to take care of Data Engineering at scale.
I have prepared this course for anyone who would like to transition into a Data Engineer role using Spark (Scala). I myself am a proven Data Engineering Solution Architect with proven experience in designing solutions using Apache Spark.
Let us go through the details about what you will be learning in this course. Keep in mind that the course is created with a lot of hands-on tasks which will give you enough practice using the right tools. Also, there are tons of tasks and exercises to evaluate yourself.
Setup of Single Node Big Data Cluster
Many of you would like to transition to Big Data from Conventional Technologies such as Mainframes, Oracle PL/SQL, etc and you might not have access to Big Data Clusters. It is very important for you set up the environment in the right manner. Don't worry if you do not have the cluster handy, we will guide you through support via Udemy Q&A.
Setup Ubuntu-based AWS Cloud9 Instance with the right configuration
Ensure Docker is setup
Setup Jupyter Lab and other key components
Setup and Validate Hadoop, Hive, YARN, and Spark
Are you feeling a bit overwhelmed about setting up the environment? Don't worry!!! We will provide complementary lab access for up to 2 months. Here are the details.
Training using an interactive environment. You will get 2 weeks of lab access, to begin with. If you like the environment, and acknowledge it by providing a 5* rating and feedback, the lab access will be extended to additional 6 weeks (2 months). Feel free to send an email to support@itversity.com to get complementary lab access. Also, if your employer provides a multi-node environment, we will help you set up the material for the practice as part of the live session. On top of Q&A Support, we also provide required support via live sessions.
A quick recap of Scala
This course requires a decent knowledge of Scala. To make sure you understand Spark from a Data Engineering perspective, we added a module to quickly warm up with Scala. If you are not familiar with Scala, then we suggest you go through relevant courses on Scala as Programming Language.
Data Engineering using Spark SQL
Let us, deep-dive into Spark SQL to understand how it can be used to build Data Engineering Pipelines. Spark with SQL will provide us the ability to leverage distributed computing capabilities of Spark coupled with easy-to-use developer-friendly SQL-style syntax.
Getting Started with Spark SQL
Basic Transformations using Spark SQL
Managing Spark Metastore Tables - Basic DDL and DML
Managing Spark Metastore Tables Tables - DML and Partitioning
Overview of Spark SQL Functions
Windowing Functions using Spark SQL
Data Engineering using Spark Data Frame APIs
Spark Data Frame APIs are an alternative way of building Data Engineering applications at scale leveraging distributed computing capabilities of Spark. Data Engineers from application development backgrounds might prefer Data Frame APIs over Spark SQL to build Data Engineering applications.
Data Processing Overview using Spark Data Frame APIs leveraging Scala as Programming Language
Processing Column Data using Spark Data Frame APIs leveraging Scala as Programming Language
Basic Transformations using Spark Data Frame APIs leveraging Scala as Programming Language - Filtering, Aggregations, and Sorting
Joining Data Sets using Spark Data Frame APIs leveraging Scala as Programming Language
All the demos are given on our state-of-the-art Big Data cluster. You can avail of one-month complimentary lab access by reaching out to support@itversity.com with a Udemy receipt.
Instructor Details
- 4.4 Rating
3,127 Reviews
Durga Viswanatha Raju Gadiraju
20+ years of experience in executing complex projects using a vast array of technologies including Big Data and the Cloud.
ITVersity, Inc. - is a US-based organization that provides quality training for IT professionals and we have a track record of training hundreds of thousands of professionals globally.
Building an IT career for people with required tools such as high-quality material, labs, live support, etc to upskill and cross-skill is paramount for our organization.
At this time our training offerings are focused on the following areas:
* Application Development using Python and SQL
* Big Data and Business Intelligence
* Cloud
* Datawarehousing, Databases
More courses by Durga Viswanatha Raju Gadiraju
Data Engineering for Beginners: Learn SQL, Python & Spark (2026)
4.3 (8,867 Reviews)
Provider: Udemy
Time: 56h
$119.99
Data Engineering using AWS Data Analytics (2024)
4.5 (3,535 Reviews)
Provider: Udemy
Time: 25.4h
$119.99
Master Apache Spark using Spark SQL and PySpark 3 (2024)
4.3 (2,655 Reviews)
Provider: Udemy
Time: 32.2h
$10.99
Data Engineering using Databricks on AWS and Azure (2024)
4.6 (2,138 Reviews)
Provider: Udemy
Time: 19h
$14.99
Databricks Certified Associate Developer - Apache Spark (2025)
4.0 (2,015 Reviews)
Provider: Udemy
Time: 14.4h
$10.99
Linux Fundamentals for IT Professionals using Ubuntu (2026)
4.6 (1,902 Reviews)
Provider: Udemy
Time: 22h
$11.99
More Hadoop courses
Spark Streaming - Stream Processing in Lakehouse - PySpark (2024)
4.9 (2,251 Reviews)
Provider: Udemy
Time: 22.4h
$11.99
Apache Kafka Series - KSQL on ksqlDB for Stream Processing ! (2026)
4.9 (2,161 Reviews)
Provider: Udemy
Time: 4h
$12.99
Apache Kafka Series - Kafka Streams for Data Processing (2026)
4.8 (6,826 Reviews)
Provider: Udemy
Time: 4.8h
$14.99
Apache Kafka Series - Confluent Schema Registry & REST Proxy (2026)
4.8 (3,794 Reviews)
Provider: Udemy
Time: 4.5h
$94.99
Apache Kafka - Real-time Stream Processing (Master Class) (2023)
4.8 (2,925 Reviews)
Provider: Udemy
Time: 11h
$10.99
Apache Spark 3 & Big Data Essentials in Scala (2024)
4.8 (2,302 Reviews)
Provider: Udemy
Time: 7.3h
$39.99
Reviews
By Arindam Pal on 3/22/2021
Excellent course for practice CCA 175 certification exam. It helps me a lot to get used to with the certification environment and solving the questions with in given time for exam. I passed the exam using this course. The only improvement could be now Cloudera certification comes with 9 question which you can answer with API or SQL . Hence RDD solutions kind of no use now.
By Sb on 1/8/2021
One of the most exhaustive material on big data stack . This tutorial surely gives lot confidence . Illustrations & examples cited , reflects practical use cases from IT world .. not just theoretical hello world sort of examples . The author has in depth knowledge of the Big data eco system and I thank him for helping me to venture into big data domain.
By Srikanth Sriram on 7/22/2019
Wonderful Content Sir, Thank you for that. Also i feel this online udemy training is more advantage than a class room training, only small concern is that we may not get immediate help if we are struck somewhere as part of practice, but i hope we can get help from discuss.itversity as part of this course itself. Thank you once again.
By Eric Simonse on 5/3/2019
It is amazing how much work has been put into this. For the last chapters the concept is getting a bit unclear: they have a lot of redundancy from previous chapters but do not contribute that strong (could be shortened by just giving the solution and pointing to previous chapters). The over-all time frame of 32 hours is very long (even if demands for other topics exist, this time frame – to my opinion - should not be extended).
By Subrat Kumar on 1/28/2019
I have finished this course recently. A very good real time examples explained in details. I have set up my own cluster and did practice. I love instructor effort a lot. This course is not for absolute beginner. He must have some prior knowledge into Big Data world and its eco-system. It will be very useful for them who didn't have practice well. Please do have this course into your bucket if you want to clear the exam. I have already purchased few courses from him and finished almost all. Got very nice grip. Subrat Gupta
By Ettione Stuckey on 4/15/2018
I waited until I finished the course to write this review. You cannot properly evaluate a course after one or two lessons. I think Udemy does this, because most people won't finish the course anyway, so they ask for reviews early in the course. Anyway, this course is exactly what I needed. Durga, the instructor is experienced and gives in-depth explanations, and supporting information about coding, and frameworks, etc. I took the CCA-175 before taking this course, about 2 months ago, and I did not pass. This course really clarified some things for me, like solving business cases using core Spark and Scala APIs vs. SQL and Data-frames. I especially like Durga's course over others, because he actually shows you how to write the code in real-time, on-screen, and even makes mistakes, then corrects them. Other courses simply move at light-speed through the material, using code they have already pre-written and debugged, and also without giving proper explanation and context. Fortunately, Durga doesn't do this... This course is almost like being in a real classroom. I am also an instructor, and have delivered technical classroom training on subjects like Linux and Solaris to fortune 500 companies around the country. That being said, I appreciate the way Durga takes his time and clearly explains each topic, as well as telling you and showing you where to find additional information and resources. I have attended real classroom training before taking this course, and sadly I learned more from Durga. I also bought two other Big Data courses on Udemy. One was good, but not as good as Durga's, and the other just plain bombed! If you want to really learn how to code Hadoop, and Spark applications in Scala, especially how to write applications in Spark/Scala - get this course! It's the best one on Udemy! You won't be sorry! FYI: I also took an additional Scala Fundamentals course, and an SQL course while taking this one. I plan to sit for the CCA-175 again by the end of this month. I hope this helps someone.
By Dohn Joe on 3/27/2018
WASTE OF MONEY!! COMPLETELY USELESS INSTRUCTOR!! ALWAYS ADVERTISES HIS CLUSTER!! Too many corrections in the lectures. Keeps changing the code and his statements. In Section 2 Lecture 15 at 6:24 he says after writing some code "If I run like this it will fail. Most likely, let us see". The code executes well and his reply "No, it didn't fail. That's good". Is he literally doing trial and error during the course? Not just once but happens a lot of times during the course. Unprofessional instructor, a complete waste of money. Expect at least some kind of professionalism and confidence when teaching a course online. For anyone thinking about joining the course, DON'T. It's a complete waste of money. They are using this platform to make people sign up for their cluster.
By Anvarroy Siluvaimuthu on 3/1/2018
Anvar: 05/03/2018 Hello Durga I have taken the CCA 175 Ceritification Exam today and Passed successfully with 8 out of 9 questions. This Course is simply great and very Practical oriented and I am happy that I have learned Big-Data from this and got the cloudera certification. Thank you very much And Appreciated your effort. Thanks Anvar Roy S
By Rammohan Pabba on 1/8/2018
Gadiraju explained each and every topic very much in details. Every command/function been explained using a use case which really helps to understand its applicability. It is really very good training and providing me utmost satisfactory feeling. Here we have only 5 start ratings. I would like to give more than 5 but here the limitation is 5 only.
By Raghavendra H S on 11/5/2017
Thanks Durga sir for well structured course, concepts are explained clearly with examples and easy to understand. I strongly recommend this course for anyone who have strong desire to learn Spark and big data. Thanks again for the great course. I have cleared my CCA175 exam last week. This course helped me to learn the exam related topics in details and helped me to clear the certification.
Quality Score
No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.
Overall Score : 88 / 100












