Data Engineering using AWS Data Analytics (Udemy.com)

Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using AWS Data Analytics Stack. It includes services such as Glue, Elastic Map Reduce (EMR), Lambda Functions, Athena, EMR, Kinesis, and many more.

Here are the high-level steps which you will follow as part of the course.

• Setup Development Environment

• Getting Started with AWS

• Storage - All about AWS s3 (Simple Storage Service)

• User Level Security - Managing Users, Roles, and Policies using IAM

• Infrastructure - AWS EC2 (Elastic Cloud Compute)

• Data Ingestion using AWS Lambda Functions

• Overview of AWS Glue Components

• Setup Spark History Server for AWS Glue Jobs

• Deep Dive into AWS Glue Catalog

• Exploring AWS Glue Job APIs

• AWS Glue Job Bookmarks

• Development Life Cycle of Pyspark

• Getting Started with AWS EMR

• Deploying Spark Applications using AWS EMR

• Streaming Pipeline using AWS Kinesis

• Consuming Data from AWS s3 using boto3 ingested using AWS Kinesis

• Populating GitHub Data to AWS Dynamodb

• Overview of Amazon AWS Athena

• Amazon AWS Athena using AWS CLI

• Amazon AWS Athena using Python boto3

• Getting Started with Amazon AWS Redshift

• Copy Data from AWS s3 into AWS Redshift Tables

• Develop Applications using AWS Redshift Cluster

• AWS Redshift Tables with Distkeys and Sortkeys

• AWS Redshift Federated Queries and Spectrum

Here are the details about what you will be learning as part of this course. We will cover most of the commonly used services with hands-on practice which are available under AWS Data Analytics.

Getting Started with AWS

As part of this section, you will be going through the details related to getting started with AWS.

• Introduction - AWS Getting Started

• Create s3 Bucket

• Create AWS IAM Group and AWS IAM User to have required access on s3 Bucket and other services

• Overview of AWS IAM Roles

• Create and Attach Custom AWS IAM Policy to both AWS IAM Groups as well as Users

• Configure and Validate AWS CLI to access AWS Services using AWS CLI Commands

Storage - All about AWS s3 (Simple Storage Service)

AWS s3 is one of the most prominent fully managed AWS services. All IT professionals who would like to work on AWS should be familiar with it. We will get into quite a few common features related to AWS s3 in this section.

• Getting Started with AWS S3

• Setup Data Set locally to upload to AWS s3

• Adding AWS S3 Buckets and Managing Objects (files and folders) in AWS s3 buckets

• Version Control for AWS S3 Buckets

• Cross-Region Replication for AWS S3 Buckets

• Overview of AWS S3 Storage Classes

• Overview of AWS S3 Glacier

• Managing AWS S3 using AWS CLI Commands

• Managing Objects in AWS S3 using CLI - Lab

User Level Security - Managing Users, Roles, and Policies using IAM

Once you start working on AWS, you need to understand the permissions you have as a non-admin user. As part of this section, you will understand the details related to AWS IAM users, groups, roles as well as policies.

• Creating AWS IAM Users

• Logging into AWS Management Console using AWS IAM User

• Validate Programmatic Access to AWS IAM User

• AWS IAM Identity-based Policies

• Managing AWS IAM Groups

• Managing AWS IAM Roles

• Overview of Custom AWS IAM Policies

• Managing AWS IAM users, groups, roles as well as policies using AWS CLI Commands

Infrastructure - AWS EC2 (Elastic Cloud Compute) Basics

AWS EC2 Instances are nothing but virtual machines on AWS. As part of this section, we will go through some of the basics related to AWS EC2 Basics.

• Getting Started with AWS EC2

• Create AWS EC2 Key Pair

• Launch AWS EC2 Instance

• Connecting to AWS EC2 Instance

• AWS EC2 Security Groups Basics

• AWS EC2 Public and Private IP Addresses

• AWS EC2 Life Cycle

• Allocating and Assigning AWS Elastic IP Address

• Managing AWS EC2 Using AWS CLI

• Upgrade or Downgrade AWS EC2 Instances

Infrastructure - AWS EC2 Advanced

In this section, we will continue with AWS EC2 to understand how we can manage EC2 instances using AWS Commands and also how to install additional OS modules leveraging bootstrap scripts.

• Getting Started with AWS EC2

• Understanding AWS EC2 Metadata

• Querying on AWS EC2 Metadata

• Fitering on AWS EC2 Metadata

• Using Bootstrapping Scripts with AWS EC2 Instances to install additional softwares on AWS EC2 instances

• Create an AWS AMI using AWS EC2 Instances

• Validate AWS AMI - Lab

Data Ingestion using Lambda Functions

AWS Lambda functions are nothing but serverless functions. In this section, we will understand how we can develop and deploy Lambda functions using Python as a programming language. We will also see how to maintain a bookmark or checkpoint using s3.

• Hello World using AWS Lambda

• Setup Project for local development of AWS Lambda Functions

• Deploy Project to AWS Lambda console

• Develop download functionality using requests for AWS Lambda Functions

• Using 3rd party libraries in AWS Lambda Functions

• Validating AWS s3 access for local development of AWS Lambda Functions

• Develop upload functionality to s3 using AWS Lambda Functions

• Validating AWS Lambda Functions using AWS Lambda Console

• Run AWS Lambda Functions using AWS Lambda Console

• Validating files incrementally downloaded using AWS Lambda Functions

• Reading and Writing Bookmark to s3 using AWS Lambda Functions

• Maintaining Bookmark on s3 using AWS Lambda Functions

• Review the incremental upload logic developed using AWS Lambda Functions

• Deploying AWS Lambda Functions

• Schedule AWS Lambda Functions using AWS Event Bridge

Overview of AWS Glue Components