Tag: web scraping

Categories
Development

Choosing the Best Proxies for Web Scraping

From eCommerce and market research to competitive analysis and more, web scraping has become an integral part of data collection. And for some, it’s the secret sauce for success.

But with great scraping power comes great responsibility. 

Web scraping can result in IP bans and other harsh restrictions. To avoid these issues, many turn to proxies, which act as intermediaries between your requests and the target website. In this article, we’ll explore the top 3 proxy types for web scraping and focus on the key benefits of each proxy. Let’s go!

Continue reading “Choosing the Best Proxies for Web Scraping”
Categories
Challenge Development

Python, Selenium for custom browser automation scraper

Recently we’ve got the tricky website, its data being of dynamic nature. Yet we’ve applied the modern day scraping tools to fetch data. We’ve develop an effective Python scraper using Selenium library for browser automation.

About the project

We were asked to have a look at a retailer website.

And our task was to gather data on 210 products’ availability in 945 shops. The scrape resulted in about 200K data entries in a CSV format. Moreover, every line contained information about name, link, brand, store and the availability of a product. Below you can familiarise yourself with a small data sample we were able to gather.

Continue reading “Python, Selenium for custom browser automation scraper”
Categories
Challenge

Web Scraping: 5 pros and cons

Web scraping, also known as data mining or web harvesting, is the process of extracting data from websites automatically. The extracted data can be used for various purposes, such as market research, price monitoring, sentiment analysis, and many more. However, web scraping has both advantages and disadvantages. In this article, we will discuss the five main pros and cons of web scraping.

Continue reading “Web Scraping: 5 pros and cons”
Categories
Challenge Development

AirTable scrape challenge

I need to get info from AirTable, see a table example.

The problem is that data are loaded highly dynamically. HTML contains only the information that you currently see on the browser screen.

enter image description here

If there are a lot of records then it is difficult to collect such a table. One of the possible ways is to calculate the size of the screen and rows in the table. Then using the browser automation and use to make a script that will scroll through it bit by bit and collect data.

Is there any other feasible way to get data of a table? For example there is a HTTP requests coding way to get dynamic data.

JS infinite scroll does not work for AirTable either.

Please comment down here if having some tips, hints.

Categories
Development

Google Sheets or MS Excel to scrape business directories ?

We’ve already stated some Tips and Tricks of scraping business directories or data aggregators sites. Yet recently someone has asked us to do aggregators’ scraping in the context of Google Sheets and/or MS Excel.

Continue reading “Google Sheets or MS Excel to scrape business directories ?”
Categories
Challenge Development

Yelp scraping for high quality B2B leads

Recently we’ve performed the Yelp business directory scrape for acquiring high quality B2B leads (company + CEO info). This forced us to apply many techniques like proxying, external company site scrape, email verification and more.

Continue reading “Yelp scraping for high quality B2B leads”
Categories
Development Web Scraping Software

My experience of choosing web scraping platform for company critical data feed

Recently we engaged with the online e-commerce startup for the need of gov. tenders/RFP scraping. Since the project size is immense , we have to switch from the hand made scripting extractors to a enterprise grade scraping platform. Below I share my experience of the scraping platforms as a feature table.

   Service   ResidentialCost/monthTraffic/month$ per GBRotatingIP whitelistingPerformance and moreNotes
MarsProxies
N/AN/A3.5yesyes 500K+ IPs, 190+ locations
Test results		SOCKS5 supported
Proxy grey zone restrictions
Oxylabs.io
N/A25 GB9 - 12

"pay-as-you-go" - 15		yesyes100M+ IPs, 192 countries

- 30K requests
- 1.3 GB of data
- 5K pages crawled		Not allowing to scrape some of grey zone targets, incl. Linkedin.
Smartproxy
Link to the price page

N/A5.2 - 7

"pay-as-you-go" - 8.5		yesyes65M+ IPs, 195+ countriesFree Trial
Not allowing to scrape some of grey zone targets, incl. Linkedin.
Infatica.io
N/AN/A3 - 6.5

"pay-as-you-go" - 8		yesyesOver 95% success
*Bans from Cloudflare are also few, less than 5%.		Black list of sites —> proxies do not work with those.
  • 1000 ports for one Proxy List
  • Up to 20 Proxy Lists at a time
  • Using via API Tool
  • ISP-level targeting
  • Rotation time selection
Mango Proxy
N/A1-50 GB3-8"pay-as-you-go" - 8yesyes90M+ IPs, 240+ countries
IPRoyal
N/AN/A$4.55yesyes32M+ IPs, 195 countriesNot allowing to scrape some of grey zone targets, incl. Facebook. List of bloked sites.
Rainproxy.io yes$ 4from 1 GB4yes
BrightDatayes15
ScrapeOps Proxy AggregatoryesAPI Credits per monthN/A N/AyesAllows multithreading, the service provides browsers at its servers. It allows to run N [cloud] browsers from a local machine.
The number of threads depends on the subscription: min 5 threads. 		The All-In-One Proxy API that allows to use over 20+ proxy providers from a single API
Lunaproxy.comyesfrom $15x Gb per 90 days0.85 - 5 Each plan allows certain traffic amount for 90 days limit.
LiveProxies.ioyesfrom $454-50 GB5 - 12yesyesEg. 200 IPs with 4 GB for $70.00, for 30 days limit.
Charity Engine -docsyes-- starting from 3.6
Additionally:
CPU computing
- from $0.01 per avg CPU core-hour
- from $0.10 per GPU-hour - source.		failed to connect so far
proxy-sale.comyesfrom $17N/A
3 - 6

"pay-as-you-go" - 7		yesyes10M+ IPs, 210+ countries30 days limit for a single proxy batch
Tabproxy.comyesfrom $15N/A
0.8 - 3
(lowest price is for a chunk of 1000 GB)		yesyes 200M+ IPs, 195 countries,30-180 days limit for a single proxy batch (eg. 5 GB)
proxy-seller.comyesN/A
N/A
4.5 - 6

"pay-as-you-go" - 7		yesyes15M+ IPs, 220 countries- Generation up to 1000 proxy ports in each proxy list
- HTTP / Socks5 support
- One will be able to generate an infinite number of proxies by assigning unique parameters to each list
Categories
Challenge SaaS

Web Scraper IDE to scrape tough websites

Recently we encountered a new powerful scraping service called Web Scraper IDE [of Bright Data]. The life-test and thorough drill-in are coming soon. Yet now we want to highlight its main features that has badly (in positive sense, strongly) impressed us.

Continue reading “Web Scraper IDE to scrape tough websites”
Categories
Development Guest posting Web Scraping Software

Octoparse Alternatives

Let me tell you what you already know! Octoparse is a great web scraping tool! But like every great tool, it’s got its limitations. At times, you may wonder if there are any alternatives to Octoparse. We wondered the same and put together this blog to provide you a short list of Octoparse alternatives along with their features and distinguishing factors. Let’s get started!

Continue reading “Octoparse Alternatives”
Categories
Development

Selenium Web Scraping in simple words

Question: What is Selenium web scraping?

Answer: A picture is better than 1000 words:selenium main diagram

So, you make a program with Python, PHP, JAVA, Ruby and whatever language you use in order to browse(), select(), click(), submit(), save(), etc.,  target web pages.

Continue reading “Selenium Web Scraping in simple words”