Web Scraping

A basic introduction to web scraping

Let’s assume that you need to extract the meta titles out of your competeitor’s blog posts.

You can go to every web site individually, verify the HTML, find the title tag, then copy/paste that knowledge to wherever you wanted it (e.g. a spreadsheet).

view source https ahrefs com blog asking for tweets

However, this may be very time-consuming and boring.

That’s why it’s a lot simpler to scrape the info we would like utilizing a pc utility (i.e. internet scraper).

Basically, there are two methods to “scrape” the info you’re on the lookout for:

  1. Utilizing a path-based system (e.g. XPath/CSS selectors);
  2. Utilizing a search sample (e.g. Regex)

XPath/CSS (i.e. path-based system) is the easiest way to scrape most kinds of knowledge.

For instance, let’s assume that we needed to scrape the h1 tag from this doc:


We are able to see that the h1 is nested within the physique tag, which is nested below the html tag—right here’s easy methods to write this as XPath/CSS:

  • XPath: /html/physique/h1
  • CSS selector: html > physique > h1

As a result of there is just one h1 tag within the doc, we don’t really want to present the total path. As a substitute, we are able to simply inform the scraper to search out all situations of h1 all through the doc with “//h1” for XPath, and easily “h1” for CSS.

However what if we needed to scrape the checklist of fruit as a substitute?

html fruit

You would possibly guess one thing like: //ul/li (XPath), or ul > li (CSS), proper?

Certain, this may work. However as a result of there are literally two unordered lists (ul) within the doc, this may scrape each the checklist of fruit AND all checklist objects within the second checklist.

Nonetheless, we are able to reference the class of the ul to seize solely what we would like:

  • XPath: //ul[@class=’fruit’]/li
  • CSS selector: ul.fruit > li

Regex, then again, makes use of search patterns (quite than paths) to search out each matching occasion inside a doc.

That is helpful every time path-based searches received’t lower the mustard.

For instance, let’s assume that we needed to scrape the phrases “first’, “second,” and “third” from the opposite unordered checklist in our doc.

html regex

There’s no strategy to seize simply these phrases utilizing path-based queries, however we may use this regex sample to match what we’d like:

<li>That is the (.*) merchandise within the checklist<\/li>

This may search the doc for checklist objects (li) containing “That is the [ANY WORD] merchandise within the checklist” AND extract solely [ANY WORD] from that phrase.


As a result of regex doesn’t use the structured nature of HTML/XML recordsdata, outcomes are sometimes much less correct than they’re with CSS/XPath. It’s best to solely use Regex when XPath/CSS isn’t a viable choice.

Listed below are a number of helpful XPath/CSS/Regex assets:

And scraping instruments:

OK, let’s get began with a number of internet scraping hacks!

1. Discover “evangelists” who could also be thinking about studying your new content material by scraping current web site feedback

Most individuals who touch upon WordPress blogs will achieve this utilizing their title and web site.

wordpress comment name website

You may spot these in any feedback part as they’re the hyperlinked feedback.

hyperlinked comment

However what use is that this?

Properly, let’s assume that you simply’ve simply revealed a publish about X and also you’re on the lookout for individuals who could be thinking about studying it.

Right here’s a easy strategy to discover them (that includes a little bit of scraping):

  1. Discover a related publish in your web site (e.g. in case your new publish is about hyperlink constructing, discover a earlier publish you wrote about search engine optimisation/hyperlink constructing—simply be sure it has an honest quantity of feedback.);
  2. Scrape the names + web sites of all commenters;
  3. Attain out and inform them about your new content material.

This works nicely as a result of these persons are (a) current followers of your work, and (b) beloved considered one of your earlier posts on the subject a lot that they left a remark. So, whereas that is nonetheless “chilly” pitching, the probability of them being thinking about your content material is way greater compared to pitching on to strangers.

Right here’s easy methods to scrape them:

Go to the feedback part then right-click any top-level remark and choose “Scrape related…” (notice: you’ll need to put in the Scraper Chrome Extension for this).

scrape similar comments

This could carry up a neat scraped checklist of commenters names + web sites.

scrape similar done

Make a replica of this Google Sheet, then hit “Copy to clipboard,” and paste them into the tab labeled “1. START HERE”.


When you have a number of pages of feedback, you’ll should repeat this course of for every.

Go to the tab labeled “2. NAMES + WEBSITES” and use the Google Sheets hunter.io add-on to search out the e-mail addresses on your prospects.

email addresses


Hunter.io received’t succeed with all of your prospects so here are more actionable ways to find email addresses

You may then attain out to those individuals and inform them about your new/up to date publish.

IMPORTANT: We advise being very cautious with this technique. Keep in mind, these individuals might have left a remark, however they didn’t decide into your e-mail checklist. That would have been for a variety of causes, however chances are high they have been solely actually on this publish. We, subsequently, advocate utilizing this technique solely to inform commenters concerning the updates to the publish and/or different new posts which are related. In different phrases, don’t e-mail individuals about stuff they’re unlikely to care about!

Here’s the spreadsheet with sample data.

2. Discover individuals prepared to contribute to your posts by scraping current “knowledgeable roundups”

“Skilled” roundups are WAY overdone.

However, this doesn’t imply that together with recommendation/insights/quotes from educated business figures inside your content material is a foul thought; it can add a variety of worth.

The truth is, we did precisely this in our recent guide to learning SEO.

how to learn seo in 2017 experts

However, whereas it’s straightforward to search out “specialists” you might need to attain out to, it’s vital to keep in mind that not everybody responds positively to such requests. Some persons are too busy, whereas others merely despise all types of “chilly” outreach.

So, quite than guessing who could be thinking about offering a quote/opinion/and so forth on your upcoming publish, let’s as a substitute attain out to these with a monitor report of responding positively to such requests by:

  1. Discovering current “knowledgeable roundups” (or any publish containing “knowledgeable” recommendation/opinions/and so forth) in your business;
  2. Scraping the names + web sites of all contributors;
  3. Constructing a listing of people who find themselves most probably to answer your request.

Let’s give it a shot with this expert roundup post from Nikolay Stoyanov.

First, we have to perceive the construction/format of the info we need to scrape. On this occasion, it seems to be full title adopted by a hyperlinked web site.

tim soulo expert roundup

HTML-wise, that is all wrapped in a <sturdy> tag.

html inspect chrome


You may examine the HTML for any on-page factor by right-clicking on it and hitting “Examine” in Chrome.

As a result of we would like each the names (i.e. textual content) and web site (i.e. hyperlink) from inside this <sturdy> tag, we’re going to make use of the Scraper extension to scrape for the “textual content()” and “a/@href” utilizing XPath, like this:

strong scraper

Don’t fear in case your knowledge is just a little messy (as it’s above); this may get cleaned up mechanically in a second.


For these unfamiliar with XPath syntax, I like to recommend utilizing this cheat sheet. Assuming you may have fundamental HTML information, this must be sufficient that will help you perceive easy methods to extract the info you need from an online web page

Subsequent, make a replica of this Google Sheet, hit “Copy to clipboard,” then paste the uncooked knowledge into the primary tab (i.e. “1. START HERE”).

raw data from scraper

Repeat this course of for as many roundup posts as you want.

Lastly, navigate to the second tab within the Google Sheet (i.e. “2. NAMES + DOMAINS”) and also you’ll see a neat checklist of all contributors ordered by # of occurrences.

roundup scraping final tab

Listed below are 9 ways to find the email addresses for everyone on your list.

IMPORTANT: All the time analysis any prospects earlier than reaching out with questions/requests. And DON’T spam them!


[WooZoneProducts asin=”1482014092″][/WooZoneProducts]