What I learned from scraping over 15,000 web pages

In the grand scheme of the web, 15,000 web pages is a drop in the bucket. However, you can learn a lot by sourcing and scraping that many from across a diverse set of sites. In building a Python app to find and Tweet out interesting data science content, I had to gather a lot of potential articles, videos and blog posts to work with and then scrape those to learn more about them.

Heres some of what I learned along the way.Finding pages to scrape is a task in itselfI talked a little bit about how I sourced URLs to go after in the post I linked above, but getting started was harder than I initially thought. One of my main sources were links people posted on Twitter.

Its a good start but there are downsides.People often post irrelevant URLs or none at all in their Tweets. Twitter is used for more than sharing links after all.

Some topics and keywords in particular lead to spam links. For example, search Twitter for things like entrepreneurship and remote work and youll find a lot of get rich quick schemes. In my case those links are of no value.

A great number of people tweet out the same URLs, so getting distinct ones is a challenge when you want to gather a lot in a short amount of time. There are plenty of other ways to find good URLs to go after. It seems a bit recursive, but you can do so by scraping some common sites, grabbing URLs and then scraping those URLs.

YouTube, Reddit, and your favorite blogs and RSS feeds will do nicely. Youll find the limits of your scraper and piss off some people along the wayI wrote my scraper in Python and relied most on the requests and beautifulsoup packages. Neither give you exactly what you want out of the box, but theyre excellent building blocks.

They give you the power to request any URL, grab the resulting HTML and pull what you want from it. Sounds easy, right? Well, here are a few mistakes and poor assumptions I made along the way.

I didnt anticipate people would share links to large files We all share blog posts and articles on Twitter and Reddit but what about ZIP files, Google Docs and massive PDFs? Well, I dont but apparently its a thing. If you just go out and try to load all that data into a Beautiful Soup object youll pay the price.

This was the first (of a few) times my hosting service sent me a friendly email telling me they killed one of my jobs and I should be more careful. Side note Its better to move from a shared server to a private one :)Heres a quick and dirty way to avoid massive files, but you may still fall into a trap if the content-length property doesnt come back in the headers or its value isnt accurate (it happens).Some sites do their best to reject scrapers, but not many I wrote an article a while back titled Ethics in Web Scraping in which I discussed some best practices in respecting site owners when youre scraping as well as respecting scrapers as an owner.

To my surprise, it turns out that the vast majority (over 99%) of site owners are very reasonable and allow scraping as long as youre not slamming their site. There are however some that block you unless you fake your headers to look like a human, but I still prefer to be honest and identify myself truthfully. In scraping these 15K pages I didnt get a single contact from a site owner despite sharing my email in the request headers.

There are a lot of sites out thereOut of the 15,000 URLs I went through, there were over 3,000 domains. I was surprised there wasnt more dominance from the well known sites. Its actually a bit refreshing to see the results.

Keep in mind this is for data science related content, so even some big sites like the Wall Street Journal are sparse in this context, but none the less I wasnt expecting such a variety of personal blogs and other sites. Heres a look at the distribution.I wish more pages supported Open Graph, and that Open Graph was enoughIf youre looking to get common meta-data from a web page, Open Graph is a blessing.

If you havent heard of Open Graph, its what makes sharing URLs on social networks so easy and consistent. Want the title, author and image for an article when you share it on Twitter? Just grab those properties as defined in the Open Graph protocol from the HTML and be on your way.

The bad news?Only about 80% of pages I scraped supported Open GraphYou cant get all that you need from Open Graph properties if youre doing deeper analysis. Title, the date published, and share images are great, but if youre a data geek like me youll need to extract more from the HTML to get the raw materials you need.

Thats a topic for another day, but a massive challenge given a lack of standards.Dont worry about paywallsI had originally considered skipping over URLs from well-known paywalled sites, but you can at least get basic meta-data (often Open Graph properties) from such pages as well as summary and keyword information. And dont worry, the site owners are usually willing to let you scrape what they provide publicly without any pushback.

Review and revise your scraping logic oftenWhen I first built my scraper and fed it URLs to go after, I worked out a few bugs and then let it go for a few days unattended unless it blew up. What I didnt realize is that I was missing out on some valuable data and had to go back and re-scrape as I made improvements.For example, some pages dont support Open Graph but there are common fallback elements that you can grab if needed.

A good example is the title of the content. Id prefer to get it from theog:title property defined in Open Graph, but if its not there