Skip to content
UdemyWeb DevelopmentPaid courseAll LevelsCertificate

Modern Web Scraping with Python using Scrapy Splash Selenium (Udemy.com)

Become an expert in web scraping and web crawling using Python 3, Scrapy, Splash and Selenium 2nd EDITION (2021)

Created by: Ahmed Rafik

Last updated May 2021

icon
What you will learn

  • Understand the fundamentals of Web Scraping
  • Scrape websites using Scrapy
  • Understand Xpath & CSS Selectors
  • Build a complete Spider from A to Z
  • Store the extracted Data in MongoDb & SQLite3
  • Scrape JavaScript websites using Splash & Selenium
  • Build a CrawlSpider
  • Understand the Crawling behavior
  • Build a custom Middleware
  • Web Scraping best practices
  • Avoid getting banned while scraping websites
  • Bypass cloudflare
  • Scrape APIs
  • Scrape infinite scroll websites
  • Working with Cookies
  • Deploy spiders locally and to the cloud
  • Run spiders periodically
  • Prevent storing duplicated data
  • Build datasets
  • Login to websites using Scrapy
  • Download images and files using Scrapy

How it compares

Against the other 815 rated Udemy courses in Web Development on CourseDuck:

  • Rating4.4 out of 5higher than 33% of them
  • Reviews3,833#61 of 816 by number of reviews
  • Length8.8 hourslonger than 54% of them
  • Last updatedMay 2021earlier than 80% of them

icon
Course Description

Web Scraping nowadays has become one of the hottest topics, there are plenty of paid tools out there in the market that don't show you anything how things are done as you will be always limited to their functionalities as a consumer.

In this course you won't be a consumer anymore, i'll teach you how you can build your own scraping tool ( spider ) using Scrapy.

You will learn:

  1. The fundamentals of Web Scraping

  2. How to build a complete spider

  3. The fundamentals of XPath & CSS Selectors

  4. How to locate content/nodes from the DOM using XPath & CSS

  5. How to store the data in JSON, CSV... and even to an external database(MongoDb & SQLite3)

  6. How to write your own custom Pipeline

  7. Fundamentals of Splash

  8. How to scrape Javascript websites using Scrapy Splash & Selenium

  9. The Crawling behavior

  10. How to build a CrawlSpider

  11. How to avoid getting banned while scraping websites

  12. How to build a custom Middleware

  13. Web Scraping best practices

  14. How to scrape APIs

  15. How to use Request Cookies

  16. How to scrape infinite scroll websites

  17. Host spiders in Heroku for free

  18. Run spiders periodically with a custom script

  19. Prevent storing duplicated data

  20. Deploy Splash to Heroku

  21. Write data to Excel files

  22. Login to websites using Scrapy

  23. Download Files & Images using Scrapy

  24. Use Proxies with Scrapy Spider

  25. Use Crawlera with Scrapy & Splash

  26. Use Proxies with CrawlSpider


What makes this course different from the others, and why you should enroll ?

  • First, this is the most updated course. You will be using Python 3.7, Scrapy 1.6 and Splash 3.0

  • You will have an in-depth step by step guide on how to become a professional web scraper.


  • You will learn how to use Splash & Selenium to scrape JavaScript websites and I can assure you, you won't find any tutorials out there that teaches how to really use Splash like I'll be doing in this course.

  • You will learn how to host spiders in Heroku as well as Splash(Exclusive).

  • You will learn how to create a custom script so spiders can run periodically without any intervention from you.

  • 30 days money back guarantee by Udemy

So whether you are a data analyst who wants to add web scraping to his tool set or someone else who wants to learn how to extract unstructured data from unstructured HTML web pages and then store back that data in a structured way to apply some data analysis on it then you are welcome to join this course.

**STUDENTS THOUGHTS ABOUT THIS COURSE **

"I was particularly looking for web scraping using XPATHs and this course is addressing that. It also covers dynamic paging. A proper mix of theory and practical. A must-have for those who wants to do web scraping . GREAT learning experience !!! ". By Hiran Kumar

"90% of what I was searching for!!! Great job!! Clear explanations and great communication with Ahmed". By Raylyson Estanista 

"Admed’s Web scraping course is awesome . His approach using Python with scrapy and splash works well with all websites especially those that make heavy use of JavaScript. Ahmed is a gifted educator: expert communicator, passionate, conscientious and accessible to his students. I highly recommend this course and any of Ahmed Rafik’s Udemy courses. ". By Richard Blackmon

"Great course, and a nice introduction to Scrapy (I'm someone with no Python experience whatsoever).". By I S

"Excellent course. Quick and thorough at the same time. Ahmed is incredibly responsive to the students and often replies to questions within minutes! Highest recommendation." By Robert Nolte

"That course is very good and explanation is crystal clear! The instructor is very supportive in case of questions. Highly recommended." By Shubina Ekaterina

"I like the course. Clear explanations and good comunication with Ahmed. All topics is interesting and full of information. I improved my skils in Scrapy. Author update course content by new videos. It's a big bonus) Explained more advance topics I never see in other courses. Thank you, Ahmed. Waiting for new videos)". By Ruslan Romanenko



icon
Udemy Discount

The discount is applied through our link. Open the course from here and Udemy's current promotional price is applied at checkout on most courses, no code to type.

Some courses are excluded from Udemy's promotions. If the price does not drop, clear your browser cookies and use the button again.

icon
Instructor Details

Ahmed Rafik

Who I am?

I’m Ahmed Rafik, I'm a self-taught developer & an online teacher on Udemy. I've helped thousands of people learning web scraping with Python using different tools such as Scrapy, Splash & Selenium.

As a self-taught developer, I found myself jumping between different tutorials and (e-)books trying to understand how things can work out together, I was literally wasting so much time trying to connect the dots rather than learning how to code and over time this has become so overwhelming with lots of things to learn and to connect.

I believe coding should be easy for everyone but this also requires you to choose the right instructor with the right knowledge.

In my courses, I'm gonna teach you the skills you need to start your web scraping career from the get-go with no fluff. I try as much as possible to avoid the boring theoretical explanations unless it's necessary, I always keep my courses up to date and that's what helped me to have the highest-rated and the best selling web scraping courses on Udemy.

I can't wait to see you enrolled in one of my courses, I'll make sure to be there for you on every step you make and answer any questions you have.

icon
More courses by Ahmed Rafik

$10.99

$12.99

icon
More Web Development courses

$19.99

$89.99

$89.99

$79.99

$10.99

$84.99

icon
Reviews

4.4

3,833 ratings on Udemy

Select a bar to show only those reviews.Select the bar again to show every rating.

By Manish on 5/27/2026

Its a total scam no output driven inclusion

By Kishore Senapathi on 10/18/2023

1.The explanation is part is just OK, 2.Topics wise you can dig more into each topic rather than completing on the surface level 3.When you have lot to explain it's better to build a good foundation in the students minds

By Chris Jordan on 3/3/2021

What a fantastic course Ahmed has made for us. Not only does he teach the scrapy framework, but he also teaches web scraping practices and tools that help with scraping! I also love the exercises that he provides which are great learning assignments! The best part of the course was his readiness to answer everyone's questions in a matter of hours. Every problem I have faced in his two courses have been helped through to completion. Thank you for this course, I am so grateful for the work you put in and you really do deserve the 5 star rating

By Andy Clark on 7/27/2020

This course does enable someone who wants to learn how to scrape websites to gain that skill. I personally took another webscraping class from coursera, but at the conclusion of that specific class, I still could not scrape an actual webpage. This course does clearly show how to scrape real web pages and covers many issues encountered in webscraping. As I went through the course, I was able to scrape websites, applying what I learned in the course. The instructor, Ahmed, is also very responsive and also helped with questions and issues I had scraping some example web pages. It's a great course, I highly recommend it.

By Mario Dudas on 6/19/2020

Outstanding course. It's an amazing course for anybody who wants to start learning web scraping. Ahmed thoroughly explains everything from the fundamentals so even if you've never encountered this topic you won't have any trouble. Everything will be clearly explained to you. What I've found particularly valuable in this course is that after fundamentals you dive immediately into real-life type projects. And in a case you need help, you can be certain that in this course you find it. Ahmed is fantastic in doing this job and always trying to help. There is also a great community so you will find many other interesting projects, ideas and insights. And after completing this course there's also advance course from Ahmed so if you will want to expand your knowledge, it's there. I'm really looking forward to it.

By Lucas Fernando Guidi da Silva on 4/18/2020

The course is excellent, BUY! WORTH THE INVESTMENT! It handles scraping the web in a practical way. Even without knowing much about the python language, it was possible to understand and apply it to other personal projects. Fastest teacher on the platform, answers all questions quickly and always with the greatest willingness. I recommend it to everyone who wants to learn something new! Remember that the method and the teacher make all the difference in this type of course. Thank you Ahmed!

By David Atherton on 12/21/2019

Ahmed is amazing! Thoroughly goes through installing Python, Scrapy and Anaconda (bonus for Windows users). Starts off with the basics and builds upon them as the course moves on. Spends a fair bit of time on XPATHs because they are the primary way to extract data. Also shows you how to deal with websites that make heavy use of JavaScript. Also shows some advanced techniques like scraping REST APIs. He also answers EVERY question with completeness - giving you what you need to UNDERSTAND the problem and ALSO the solution. Fantastic teacher and course!

By Eranga Wijebandara on 10/23/2019

Actually this is the first time I had to do a web scraping module and i was looking for tutorial to learn about web scraping.I could not found any good resource like this. I am very glad that i found this course because the way this course has arranged is amazing. I got everything what i wanted to learn .Also the way he teaches using real world application and the complete project helps anybody to develop a professional scraping module. Simply this is amazing course and thank you Ahmed for publishing this course.

By Ketan R on 3/3/2019

This is an excellent course that goes in depth into many advanced topics of web scraping. I’ve taken an few other Udemy courses, and I must say this course is more hands on and project oriented. Ahmed is clearly a passionate instructor with many years of experience in web scraping, and his knowledge shows in the very in-depth project he builds up from scratch. He is also super responsive and willing to help students build their own project. I highly recommend this course for students who have some knowledge of Python and OOP.

By Anonymized User on 12/25/2018

I came to this course just because of our private project. Basically the instructor is super familiar with webcrawling and without spending much time on basic details, he cover lots of scrapy and splash's tricky but practical topics. Most importantly, I asked two questions and the instructor answered them within 24 hours. and one of the question really buged me for quite a few days, I almost gave up. the instructor show his patience and quickly help me solve the problem. I'm really glad I took this course.

Showing all 10 reviews on CourseDuck

Read all 3,833 reviews on Udemy

icon
Quality Score

No CourseDuck member has rated this course yet. Taken it? Give each part a thumbs up or down.

Content Quality
/
Video Quality
/
Qualified Instructor
/
Course Pace
/
Course Depth & Coverage
/

Overall Score : 88 / 100