Skip to content
UdemySoftware DevelopmentPaid courseAll LevelsCertificate

Scrapy Masterclass: Learn Web Scraping With Scrapy Framework (Udemy.com)

Master Web Scraping using Scrapy Python Framework and this Step-By-Step & In-Depth Web Scraping Guide For Beginners

Created by: Rahul Mula

Last updated March 2025

icon
What you will learn

  • Define the Steps Involved in Web Scraping and Creating Web Crawlers
  • Install and Setup Scrapy in Windows, Mac OS, Ubuntu (Linux) & Anaconda Environments
  • Send Request to a URL to Scrape Websites Using Scrapy Spider
  • Get the HTML Response From URL and Parse it for Web Scraping
  • Select Desired Data From Websites Using Scrapy Selector, CSS Selectors & XPath
  • Scrapy Crawl Spiders to Get Data From Websites And Extract it to JSON, CSV, XLSX ( Excel ) and XML Files
  • Use Scrapy Shell Commands to Test & Verify CSS Selectors or XPath
  • Export and Save Scraped Data to Online Databases Like MonogoDB Using Scrapy Item Pipelines
  • Define Scrapy Items to Organize Scraped Data And Load Items Using Scrapy Itemloaders with Input & Output Processors
  • Scrape Data From Multiple Web Pages Using Scrapy Pagination And Extract Data From HTML Tables
  • Login Into Websites Using Scrapy FormRequest With CSRF Tokens

How it compares

Against the other 518 rated Udemy courses in Software Development on CourseDuck:

  • Rating4.3 out of 5higher than 17% of them
  • Reviews175#362 of 519 by number of reviews
  • Length7.6 hourslonger than 58% of them
  • Last updatedMarch 2025earlier than 60% of them

icon
Course Description

Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy framework with a step-by-step and in-depth guide.


A Step-By-Step Guide

Assuming that you know nothing about web scraping, web crawling, scrapy framework, web scraping, or even web scrapping, we will start from the complete basics. In the first section, you'll learn about the web scraping process step-by-step (with infographics - no code), how to scrape data from websites and how to use scrapy for the same (i.e. scrapy meaning).


After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with bite-sized lessons. We'll take it slow so that it's easier for you to understand every step involved in scraping and extracting data from websites.


Web Scraping & Scrapy Essentials

Having built an actual web scraper, you'll get an idea of how web scraping works first-hand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.

  • CSS Selectors to select web elements

  • XPath to select web elements.

  • Scrapy Shell to test & verify selectors.

  • Items to organise extracted data

  • Load Items with ItemLoaders with input & output Processors

  • Export data to JSON, CSV, XLSX (Excel) & XML file formats

  • Save extracted data to online databases like MongoDB using ItemPipelines.


Master Web Scraping In-Depth

Learning how to scrape websites and the essentials already makes you a complete web scraper, but we'll take this even further and learn the advanced web scraping techniques to become an expert!

  • Follow links in a webpage to another page.

  • Crawl multiple pages and extract data, i.e. Pagination.

  • Scrape data using Regular Expressions (RegEx)

  • Extract Data From HTML Tables

  • Login Into Websites Using Scrapy FormRequest

  • Bypass CSRF-protected Login forms.

  • Scrape Dynamic or JavaScript Rendered Websites using Scrapy Playwright.

    • Interact with web elements like fill forms, click buttons, etc.

    • Handle Infinite Scroll websites.

    • Wait For Elements when contents/data take time to load

    • Take Screenshots of websites.

    • Save websites as PDFs.

  • Identify API calls from websites and scrape data from APIs

  • Use middleware in a scrapy project.

  • Configure settings in a scrapy project

  • Use and Rotate User-Agents & Proxies

  • Web scraping Best Practices


Real-World Projects

After mastering web scraping and web crawling, we need projects to start! That's why you'll perform three projects as well:

  • Champions League Table [ ESPN ]

  • Product Tracker [ Amazon ]

  • Scraper Application [ GUI ]


Join us in this in-depth course, where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to learn how web scraping works! See you there~

icon
Udemy Discount

The discount is applied through our link. Open the course from here and Udemy's current promotional price is applied at checkout on most courses, no code to type.

Some courses are excluded from Udemy's promotions. If the price does not drop, clear your browser cookies and use the button again.

icon
Instructor Details

Rahul Mula

Hi, my name is Rahul Mula and I'm a developer and instructor. I have authored books and instructed courses on python programming, web scraping and automation, APIs and thousands of students have joined so far.


I also have a knack for animation and that's why you'll find illustrative guides in the courses instead of boring slideshows. I've also developed Keyviz - the free and open-source keypress visualizer.

icon
More courses by Rahul Mula

$19.99

icon
More Software Development courses

$10.99

$109.99

$19.99

$19.99

$94.99

$94.99

icon
Reviews

4.3

175 ratings on Udemy

Select a bar to show only those reviews.Select the bar again to show every rating.

By Bhavya Lodaya on 4/1/2025

Poor, There are lots of issue while installing ubuntu and playwright on windows, you need to add a tutorial for doing the steps.

By Hamdi Özdemir on 11/2/2024

It is a good course and quite compherensive. However, It would be better to give more details about selectors, if it is a masterclass.

By Brett on 3/19/2024

I'm only a few videos in and I already love it. The videos get straight to the point, no time wasted. No random tangents, no fluff & no theory just diving right in and building. Love it.

By Fahmi Hamzah on 12/23/2023

This course is amazing! Rahul's responsiveness and clear explanations make learning web scraping with Scrapy a breeze. The course content is well-structured, and Rahul's quick responses to queries enhance the overall learning experience. Highly recommended!

By Avram on 9/11/2023

I am really enjoying this course, but I still have some hesitations about whether I will be able to use them in a web-scraping project or not.

By Pratipalsinh Jadeja on 7/22/2023

I have been scraping websites since 2019 using Python and I always preferred the "Requests" library over the Scrapy framework as it appeared to be more complex. Recently I thought of giving Scrapy a try and bought this course to deep dive and understand the concepts like items, pipelines, and middlewares. Although the instructor explains these concepts using practical examples but the websites he used for this are very basic and generic (quotes to scrape dot com, books to scrape dot com), provided by the developer of the Scrapy framework itself i.e., Zyte. I expected the scraping of some real-world websites with lots of data since Scrapy is mostly used when there's a huge amount of data to be scraped. For that reason, I am leaving 3 stars. So if you already have some experience scraping with Scrapy then this course is not for you but if you are just starting out then I believe this course is best suited for you. Thanks

By A. Rehman on 6/6/2023

A brilliant introduction to Python Scrapy and how web scraping works. Instructor Rahul is very knowledgeable in his field, clear in teaching and easy to follow along with this tutorial. Thank you for the course and crisp presentation.

By Thomas Jereczek on 4/17/2023

Of course, it's almost impossible to create a course that is comprehensive enough to cover all web scraping topics, but this course will give you a good start. You'll get more than just the basics, you'll dive deeper into web scraping right from the start. Everything is shown with examples used on real websites. Even for beginners, this course is easy to understand because of its step-by-step approach. If you want to learn how to use "scrapy", take this course.

By Joshua Ribeiro on 1/29/2023

Fantastic so far, very easy to follow. The one recommendation I would add is setting up iPython in Pycharm, as well as using Pycharm's built in Python c=Console. I understand VS Code is popular, bu tso is Pycharm and the Python Console in PyCharm has a built in object explorer which is useful for exploring objects

By Norma Ochoa on 11/5/2022

Good step-by-step instructions. I was able to follow up the exercises. The setup for the environment needs to be updated since Scrapy now requires a newer version of python. Once the environment was setup, I was able to follow up the exercises without a problem.

Showing all 10 reviews on CourseDuck

Read all 175 reviews on Udemy

icon
Quality Score

No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.

Content Quality
/
Video Quality
/
Qualified Instructor
/
Course Pace
/
Course Depth & Coverage
/

Overall Score : 86 / 100