Scrapy Masterclass: Learn Web Scraping With Scrapy Framework (Udemy.com)
Master Web Scraping using Scrapy Python Framework and this Step-By-Step & In-Depth Web Scraping Guide For Beginners
Created by: Rahul Mula
Last updated March 2025
What you will learn
- Define the Steps Involved in Web Scraping and Creating Web Crawlers
- Install and Setup Scrapy in Windows, Mac OS, Ubuntu (Linux) & Anaconda Environments
- Send Request to a URL to Scrape Websites Using Scrapy Spider
- Get the HTML Response From URL and Parse it for Web Scraping
- Select Desired Data From Websites Using Scrapy Selector, CSS Selectors & XPath
- Scrapy Crawl Spiders to Get Data From Websites And Extract it to JSON, CSV, XLSX ( Excel ) and XML Files
- Use Scrapy Shell Commands to Test & Verify CSS Selectors or XPath
- Export and Save Scraped Data to Online Databases Like MonogoDB Using Scrapy Item Pipelines
- Define Scrapy Items to Organize Scraped Data And Load Items Using Scrapy Itemloaders with Input & Output Processors
- Scrape Data From Multiple Web Pages Using Scrapy Pagination And Extract Data From HTML Tables
- Login Into Websites Using Scrapy FormRequest With CSRF Tokens
How it compares
Against the other 518 rated Udemy courses in Software Development on CourseDuck:
- Rating4.3 out of 5higher than 17% of them
- Reviews175#362 of 519 by number of reviews
- Length7.6 hourslonger than 58% of them
- Last updatedMarch 2025earlier than 60% of them
Course Description
Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy framework with a step-by-step and in-depth guide.
A Step-By-Step Guide
Assuming that you know nothing about web scraping, web crawling, scrapy framework, web scraping, or even web scrapping, we will start from the complete basics. In the first section, you'll learn about the web scraping process step-by-step (with infographics - no code), how to scrape data from websites and how to use scrapy for the same (i.e. scrapy meaning).
After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with bite-sized lessons. We'll take it slow so that it's easier for you to understand every step involved in scraping and extracting data from websites.
Web Scraping & Scrapy Essentials
Having built an actual web scraper, you'll get an idea of how web scraping works first-hand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.
CSS Selectors to select web elements
XPath to select web elements.
Scrapy Shell to test & verify selectors.
Items to organise extracted data
Load Items with ItemLoaders with input & output Processors
Export data to JSON, CSV, XLSX (Excel) & XML file formats
Save extracted data to online databases like MongoDB using ItemPipelines.
Master Web Scraping In-Depth
Learning how to scrape websites and the essentials already makes you a complete web scraper, but we'll take this even further and learn the advanced web scraping techniques to become an expert!
Follow links in a webpage to another page.
Crawl multiple pages and extract data, i.e. Pagination.
Scrape data using Regular Expressions (RegEx)
Extract Data From HTML Tables
Login Into Websites Using Scrapy FormRequest
Bypass CSRF-protected Login forms.
Scrape Dynamic or JavaScript Rendered Websites using Scrapy Playwright.
Interact with web elements like fill forms, click buttons, etc.
Handle Infinite Scroll websites.
Wait For Elements when contents/data take time to load
Take Screenshots of websites.
Save websites as PDFs.
Identify API calls from websites and scrape data from APIs
Use middleware in a scrapy project.
Configure settings in a scrapy project
Use and Rotate User-Agents & Proxies
Web scraping Best Practices
Real-World Projects
After mastering web scraping and web crawling, we need projects to start! That's why you'll perform three projects as well:
Champions League Table [ ESPN ]
Product Tracker [ Amazon ]
Scraper Application [ GUI ]
Join us in this in-depth course, where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to learn how web scraping works! See you there~
Instructor Details
- 4.3 Rating
175 Reviews
Rahul Mula
Hi, my name is Rahul Mula and I'm a developer and instructor. I have authored books and instructed courses on python programming, web scraping and automation, APIs and thousands of students have joined so far.
I also have a knack for animation and that's why you'll find illustrative guides in the courses instead of boring slideshows. I've also developed Keyviz - the free and open-source keypress visualizer.
More courses by Rahul Mula
Ultimate Guide to FastAPI and Backend Development (2026)
4.4 (475 Reviews)
Provider: Udemy
Time: 22.8h
$19.99
More Software Development courses
MATLAB Programming Simulink: Beginner to Pro with ChatGPT (2026)
5.0 (930 Reviews)
Provider: Udemy
Time: 9.6h
$10.99
Build Web Apps with Nuxt.js 3: Master TypeScript & API [New] (2026)
4.9 (635 Reviews)
Provider: Udemy
Time: 6.7h
$19.99
Terraform 101: The Ultimate Hands-On Guide [Azure Edition] (2025)
4.8 (1,975 Reviews)
Provider: Udemy
Time: 11.9h
$94.99
Reviews
By Bhavya Lodaya on 4/1/2025
Poor, There are lots of issue while installing ubuntu and playwright on windows, you need to add a tutorial for doing the steps.
By Hamdi Özdemir on 11/2/2024
It is a good course and quite compherensive. However, It would be better to give more details about selectors, if it is a masterclass.
By Brett on 3/19/2024
I'm only a few videos in and I already love it. The videos get straight to the point, no time wasted. No random tangents, no fluff & no theory just diving right in and building. Love it.
By Fahmi Hamzah on 12/23/2023
This course is amazing! Rahul's responsiveness and clear explanations make learning web scraping with Scrapy a breeze. The course content is well-structured, and Rahul's quick responses to queries enhance the overall learning experience. Highly recommended!
By Avram on 9/11/2023
I am really enjoying this course, but I still have some hesitations about whether I will be able to use them in a web-scraping project or not.
By Pratipalsinh Jadeja on 7/22/2023
I have been scraping websites since 2019 using Python and I always preferred the "Requests" library over the Scrapy framework as it appeared to be more complex. Recently I thought of giving Scrapy a try and bought this course to deep dive and understand the concepts like items, pipelines, and middlewares. Although the instructor explains these concepts using practical examples but the websites he used for this are very basic and generic (quotes to scrape dot com, books to scrape dot com), provided by the developer of the Scrapy framework itself i.e., Zyte. I expected the scraping of some real-world websites with lots of data since Scrapy is mostly used when there's a huge amount of data to be scraped. For that reason, I am leaving 3 stars. So if you already have some experience scraping with Scrapy then this course is not for you but if you are just starting out then I believe this course is best suited for you. Thanks
By A. Rehman on 6/6/2023
A brilliant introduction to Python Scrapy and how web scraping works. Instructor Rahul is very knowledgeable in his field, clear in teaching and easy to follow along with this tutorial. Thank you for the course and crisp presentation.
By Thomas Jereczek on 4/17/2023
Of course, it's almost impossible to create a course that is comprehensive enough to cover all web scraping topics, but this course will give you a good start. You'll get more than just the basics, you'll dive deeper into web scraping right from the start. Everything is shown with examples used on real websites. Even for beginners, this course is easy to understand because of its step-by-step approach. If you want to learn how to use "scrapy", take this course.
By Joshua Ribeiro on 1/29/2023
Fantastic so far, very easy to follow. The one recommendation I would add is setting up iPython in Pycharm, as well as using Pycharm's built in Python c=Console. I understand VS Code is popular, bu tso is Pycharm and the Python Console in PyCharm has a built in object explorer which is useful for exploring objects
By Norma Ochoa on 11/5/2022
Good step-by-step instructions. I was able to follow up the exercises. The setup for the environment needs to be updated since Scrapy now requires a newer version of python. Once the environment was setup, I was able to follow up the exercises without a problem.
Quality Score
No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.
Overall Score : 86 / 100




![Build Web Apps with Nuxt.js 3: Master TypeScript & API [New]](/programming/software-development/images/6457785_5adc_4.jpg?v=1789443959)

![Terraform 101: The Ultimate Hands-On Guide [Azure Edition]](/programming/software-development/images/6208847_f60a_7.jpg?v=1791193600)
