Data Engineering on Google Cloud platform (Udemy.com)

End to end batch processing,data orchestration and real time streaming analytics on GCP

( 5 Reviews )

Created by: Siddharth Raghunath

Produced in 2021

What you will learn

Pyspark for ETL/Batch Processing on GCP using Bigquery as data warehousing component
Automate and orchestrate SparkSql batch jobs using Apache Airflow and Google Workflows
Sqoop for Data ingestion from CloudSql and using Airflow to automate the batch ETL
Difference between Event-time data transformations and process-time transformations
Pyspark Structured Streaming - Real Time Data streaming and transformations
Save real time streaming raw data as external hive tables on Dataproc and perform ad-hoc queries using HiveSql
Run Hive-SparkSql jobs on Dataproc and automate micro-batching and transformations using Airflow
Pyspark Structured Streaming - Handling Late Data using watermarking and Event-time data processing
Using different file formats - AVRO and Parquet . Different scenarios in which to use the file formats

Quality Score

Content Quality

Video Quality

Qualified Instructor

Course Pace

Course Depth & Coverage

Overall Score : 72 / 100

Course Description

Google Cloud platform is catching up and a lot of companies have already started moving their infrastructure to GCP . This course provides the most practical solutions to real world use cases in terms of data engineering on Cloud . This course is designed keeping in mind end to end lifecycle of a typical Big data ETL project both batch processing and real time streaming and analytics .
Considering the most important components of any batch processing or streaming jobs , this course covers

Writing ETL jobs using Pyspark from scratch,
Storage components on GCP (GCS & Dataproc HDFS)
Loading Data into Data-warehousing tool on GCP (BigQuery)
Handling/Writing Data Orchestration and dependencies using Apache Airflow(Google Composer) in Python from scratch
Batch Data ingestion using Sqoop , CloudSql and Apache Airflow
Real Time data streaming and analytics using the latest API , Spark Structured Streaming with Python
Micro batching using PySpark streaming & Hive on Dataproc

The coding tutorials and the problem statements in this course are extremely comprehensive and will surely give one enough confidence to take up new challenges in the Big Data / Hadoop Ecosystem on cloud and start approaching problem statements & job interviews without inhibition .
Most importantly , this course makes use of Linux Ubuntu 18.02 as a local operating system.Though most of the codes are run and triggered on Cloud , this course expects one to be experienced enough to be able to set up Google SDKs , python and a GCP Account by themselves on their local machines because the local operating system does not matter in order to succeed in this course .Who this course is for:

Any techie who needs hands on project expertise on end to end batch data processing & real time streaming
Aspiring Data Engineers who find it hard to setup and work practically on distributed processing
Any Techie who is preparing for an interview for a Data engineering position and wants hands on expertise

Time: 9.5h

Instructor Details

Siddharth Raghunath

3.6 Rating
5 Reviews

I am a Data Engineer with a vast experience in the field of Software Development,Distributed processing and data engineering on cloud . I have worked on different cloud platforms such as AWS & GCP and also with on-prem hadoop clusters. I also give seminars on Distributed processing using Spark , real time streaming and analytics and best practices for ETL and data governance.I am also a passionate coder ,love writing and building optimal data pipelines for robust data processing and streaming solutions .

Practical Guide to Google Analytics Mastery

Practical Guide to Google Analytics Mastery (2018)
4.8 (28 Reviews)

- Provider: Udemy
- Time: 3.5h

$11.99

Google Analytics Made Easy - 2021 Success Guide

Google Analytics Made Easy - 2021 Success Guide (2021)
4.8 (25 Reviews)

- Provider: Udemy
- Time: 2h

$11.99

Foundations of Business Strategy
4.7 (251 Reviews)

- Provider: Coursera
- Time: 15h

Free

Google Tag Manager (GTM) Advanced Applications

Google Tag Manager (GTM) Advanced Applications (2021)
4.7 (22 Reviews)

- Provider: Udemy
- Time: 2h

$11.99

Google Analytics - Fundamentals (2018)
4.7 (20 Reviews)

- Provider: Fiverr
- Time: 4.1h

$20

Advanced Google Analytics course + 77 practical questions

Advanced Google Analytics course + 77 practical questions (2021)
4.6 (82 Reviews)

- Provider: Udemy
- Time: 4h

$11.99

All google analytics courses

Reviews

3.6

5 total reviews

5 star 4 star 3 star 2 star 1 star

Write a Review

By Syed Jaffery

Audio is too low

By Sanjay Shroff

he is reading while doing at the same time, makes the course extremely slow

By Louie Bafford

Straightforward content, easy to follow, covers very relevant stuff

By ring bell

Love this course! Look into more GCP practical courses like this Sidd. Like maybe use another tech, cloud bigtable, kubernetes, load balancing. I love the way you structured the courses not just the theoretical but the practical one!

By Nattakan Kodyord

Please launch new update for data engineering more because I would like to improve myself to be a data engineer the best in my country (Thailand).

Thank you