Skip to content
UdemyData SciencePaid courseExpert levelCertificate

Advanced Web Scraping with Python using Scrapy & Splash (Udemy.com)

The most advanced web scraping & crawling course using Scrapy & Splash! Take your web scraping skills to the next level.

Created by: Ahmed Rafik

Last updated August 2020

icon
What you will learn

  • Advanced web scraping techniques
  • Best techniques to analyse a website before scraping it
  • Write clean spiders
  • Optimize Splash scripts
  • Bypass 504 HTTP errors
  • Build Splash Cluster
  • Bypass Google ReCaptcha (not solving it)
  • Build Desktop apps for Scrapy Spiders (Tkinter)
  • ScrapyRT
  • Showcase scraped data using ScrapyRT & Flask
  • Heavy data processing
  • Input & Output processors

How it compares

Against the other 461 rated Udemy courses in Data Science on CourseDuck:

  • Rating4.5 out of 5higher than 46% of them
  • Reviews499#234 of 462 by number of reviews
  • Length5.6 hoursshorter than 64% of them
  • Last updatedAugust 2020earlier than 90% of them

icon
Course Description

Hi there & welcome to the most advanced online resource on Web Scraping with Python using Scrapy & Splash. This course is fully project-based means pretty much on each section we gonna scrape a different website & tackle a different web scraping dilemma also rather than focusing on the basics of Scrapy & Splash we gonna dive straight forward into real-world projects, this also means that this course is absolutely not suitable for beginners with no background on web scraping, Scrapy, Splash & XPath expressions.

---This courses covers a variety of topics such as:---

  1. Requests chaining, like how the requests must be sent in a certain order otherwise they won't be fulfilled at all.

  2. How to analyze a website before scraping it, this is an important step to do since it helps a lot in choosing the right tools to scrape a website & it literally has a huge impact on the performance of your final product.

  3. How to optimize Splash scripts by reducing/aborting all the unnecessary requests that have nothing to do with the data points you're going to scrape, this is an important thing to do if you care about the performance of Splash as it is the key to bypass 504 Gateway Timeout HTTP errors in Splash.

  4. We gonna also cover how to build a Cluster of Splash instances with a load balancer(HAProxy) rather than having one fully overloaded Splash instance this also helps in bypassing 504 Gateway Timeout errors.

  5. Heavy data processing, you'll understand how Input & Output processors work so you'll be able to use them in order to clean the scraped data points as this will ensure the quality of your feeds.

  6. We'll use ScrapyRT (Scrapy RealTime) to build spiders that can fetch data in real-time.

  7. Showcase the scraped data points in a minimalist web app using ScrapyRT & Flask, this is extremely helpful for web scraping freelancers.

  8. Bypass Google ReCaptcha, please don't get me wrong on this point, I don't mean that we will solve it using Scrapy, instead, I'm gonna show you a technique that I use frequently to fool websites and let them think that the request is sent using a browser & was performed by a human being!

  9. Build clean & well-structured spiders

  10. Finally, we gonna build a Desktop app using Tkinter, the app will fetch & execute all the available spiders in your Scrapy project, you can also choose the feed type, feed location & name, this is also extremely helpful & important if you're a web scraping freelancer, it is always a good idea to deliver to your client a desktop app rather than installing Scrapy on his machine & stuff like that.


This course is straight to the point, there's no "foobar" or "quotes to toscrape dot com" as other courses do so make sure you have a good level of focus & lot of determination & motivation.

By the end of this course, you'll sharpen your skills in web scraping using Scrapy & Splash, you'll be able to write clean & high performing spiders that differentiate you from others, this also means if you're a web scraping freelancer you'll get more offers since you can deliver "User-Friendly" spiders with a Graphical User Interface(GUI) or web apps that fetch data in real-time.

So join me on this course & let's harvest the web together!

icon
Udemy Discount

The discount is applied through our link. Open the course from here and Udemy's current promotional price is applied at checkout on most courses, no code to type.

Some courses are excluded from Udemy's promotions. If the price does not drop, clear your browser cookies and use the button again.

icon
Instructor Details

Ahmed Rafik

Who I am?

I’m Ahmed Rafik, I'm a self-taught developer & an online teacher on Udemy. I've helped thousands of people learning web scraping with Python using different tools such as Scrapy, Splash & Selenium.

As a self-taught developer, I found myself jumping between different tutorials and (e-)books trying to understand how things can work out together, I was literally wasting so much time trying to connect the dots rather than learning how to code and over time this has become so overwhelming with lots of things to learn and to connect.

I believe coding should be easy for everyone but this also requires you to choose the right instructor with the right knowledge.

In my courses, I'm gonna teach you the skills you need to start your web scraping career from the get-go with no fluff. I try as much as possible to avoid the boring theoretical explanations unless it's necessary, I always keep my courses up to date and that's what helped me to have the highest-rated and the best selling web scraping courses on Udemy.

I can't wait to see you enrolled in one of my courses, I'll make sure to be there for you on every step you make and answer any questions you have.

icon
More courses by Ahmed Rafik

$79.99

$12.99

icon
More Data Science courses

Free

Free

Free

Free

$124.99

Free

icon
Reviews

4.5

499 ratings on Udemy

Select a bar to show only those reviews.Select the bar again to show every rating.

By Ruben Avalos on 5/23/2025

Incredible course, each section is better than the one before. I've done both of Ahmed courses and I've passed from knowing zero about scraping to being confident enough to takle any professional project. I just feel so powerful! Thank you Ahmed, I wish you the best!

By Daiane Klein on 6/6/2023

The course is interesting, especially the projects. You can learn many advanced and unique concepts. The biggest problem is that most of the websites have changed since this course was recorded, so you can't follow the majority of the code presented. The instructor is no longer replying to the Q&A, so you either waste a lot of time searching for a possible solution or you give up on the project.

By Linda F on 12/5/2022

Bought this course 2 years ago, and now I have time to really learn this with all my concentration, and AMAZING, this course exceeds my expectation, even if it's outdated, the knowledge and the tricks are still working, he is really a NINJA in web scraping world and he is not holding his knowledge, THANK YOU SO MUCH, I bought all of your courses, please make more and more, also, can you please include the new PLAYWRIGHT course as an alternative to Selenium?

By Carlos Campos on 11/22/2022

Quite brief but really insightful. The three cases/ projects presented summarize some of the hardest challenges a developer might find while scraping complex websites. For me, adding two more projects will make this course a 5 Stars' one. Anyway, if you are going serious about Web Scraping you must take this course.

By Carl Parry on 11/3/2022

So I spent money on this course because I was hoping to learn about web scraping in a short time. As it turned out, this course is quite out of date and many of the sites have updated and so have the scripts and the lessons and code just don't work any more. There was enough for me to figure out how to do scraping using Google and with the help of stackoverflow. If only the lecturer would update the course material; it would get a 5 from me.

By Brendan Lydon on 4/5/2022

I do a lot of web scraping projects through sites like Upwork and primarily use Selenium to get around Javascript. This method works, but can feel clunky because of it actually opening the browser and running. I started this because I wanted a better alternative when handing the code off to clients to run themselves and I am already seeing the benefits of using Scrapy and Splash! Some info is outdated like the tagging, so having experience ahead of time is definitely needed to adjust things on the fly from the video but he is a great teacher!

By Khurram Chaudhry on 4/29/2021

What an amazing course. Full of great content. Ahmed is knowledgeable and approachable. He has also built a great little community of scrapers on facebook, which can be very helpful to learn from others issues as well share your own issue. I came from zero knowledge of scraping, (my initial issues were related to installation to put things in perspective) and now I’m comfortable using scrapy, splash and selenium to scrap literally any websites. If you are newbie like me and this is your first attempt at learning scraping, then purchase his other course to establish fundamental clarity and then move on to this course. However, if you already know things, then this course will surprise you with lots of new knowledge. I had no knowledge of scraping, now I find myself messing around with APIs on different websites. So much fun! I particularly enjoy how Ahmed writes code in real time and then correct errors plus explains different possibilities. Don’t worry about content being outdated he regularly updates it to keep it relevant. Thank you Ahmed for your hard work!

By LeamonLee on 9/28/2020

The course content is okay. But I was kinda hoping that the instructor can teach something advanced about anti-scraping stuffs like how to combine the OCR, the image recognition or SMS validation techniques, or scrape the website using GraphQL instead of the RESTful API something like that. But overall the course content is good.

By Dhaval Savalia on 8/16/2020

I am enrolled in both of Ahmed's courses. His courses are well worth thousands of dollars each. I completed the Modern Web Scraping course and got an immense amount of value out of it. The way he teaches and explains the topic is easy to grasp and understand. He also covers best practices to tackle the specific pitfalls you might come across. All in all, these are one of the best courses that you take on Udemy to learn web scraping in Python. I will also update this review once I finish this course.

By Swapnil Mane on 8/5/2020

Best Web scraping course in the market, Period! I have also been doing other courses on Udemy, but none of them help with the personal projects, but here Ahmed help's us even in our own web scraping projects through Q n A and this has helped me start as a freelancer on Upwork and have completed 11 jobs of web scrapping with very happy clients...All the credits go to Ahmed and I can never thank him enough...He is the best!

Showing all 10 reviews on CourseDuck

Read all 499 reviews on Udemy

icon
Quality Score

No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.

Content Quality
/
Video Quality
/
Qualified Instructor
/
Course Pace
/
Course Depth & Coverage
/

Overall Score : 90 / 100