Business listing scraper with review collection

This web scraping project began with a client launching a new business directory. Since the directory was new, with no users yet, he wanted a fast way to fill the site with quality business data and customer reviews.

Google Maps scraping

We quickly landed on Google Maps as a solid data source. So, just hook up the API and pull the data? Unfortunately… Although Google Maps has an API for retrieving business data, it is quite limited. You can only get a handful of details, such as name, address and average rating.

Scraping to the rescue. We built the client a reliable web scraper to collect all the information on Google Maps. This includes:

• Standard information such as name, address, opening hours.
• Photos of the business: Street View, Maps and photos uploaded by users.
• Reviews of the business: including text, username and any images.

Bypassing anti-scraping measures

Google naturally prefers to keep its user data to itself. This means they have built in a number of defenses against automated data collection. Fortunately we found a few clever tricks to get around this, including captcha-solving and rotating IP addresses.

The tech

We built this web scraper in Python, with a customized version of Selenium WebDriver, BeautifulSoup and Pandas. This modified scraper carries special “flags” that make it look like a regular Chrome driver. We also use rotating IP addresses so our server IP is never shown to the web server.

Automated captcha solving

What if a captcha shows up anyway? No problem, we route the session to a company that specializes in solving captchas. In essence, they have a pool of flexible workers in low-wage countries who solve the captchas manually through a remote session. Since we only pass along the captcha information in that remote session, any security risk is avoided.

Need help?

The result is a clean database with all the business data and reviews, ready to be integrated into the client’s business directory.

Working on a similar project, or need to collect other information from the web? Schedule a meeting and tell us about your project!