Categories
Review

SquidProxies review

Today we want to share with you about SquidProxies. It is a service offering anonymous HTTP/HTTPS proxies.

SquidProxies offers 2 types of data-center proxy packages, private proxies and shared proxies. The proxies are designated for just about any legal use, and work great to surf to every website. The proxies’ main use are web scraping/web crawling and SEO tools.

Categories
Development

Proxy speed and performance test

I want to test a proxy [gateway] service. What would be the simplest script to check the proxy’s IP speed and  performance? See the following script.

Categories
Development

Selenium using proxy gateway, how?

I develop a web scraping project using Selenium. Since I need rotating proxies [in mass quantities] to be utilized in the project, I’ve turned to the proxy gateways (nohodo.com, charityengine.com and some others). The problem is how to incorporate those proxy gateways into Selenium for surfing web?

Categories
Development

Charles CA certificate with OpenSSL in Windows

Today I needed to enable a Charles proxy on my Windows PC. Later I have managed the Genymotion virtual device to be monitored by the Charles proxy.

Categories
Uncategorized

Reliable rotating proxies for business directories scrape

We’ve already written about suitable proxy servers for web scraping. Now we want to focus our readers on those for the huge/mass quantities data records scrape, particulary from the business directories. When scraping business directories, their web servers can identify repetitive requesting and put you on hold by looking at the IP address that is used for frequent http requests. Proxy rotation web service is the means for repeatedly changing IP address. Thus, target web server can only see the random IP addresses from rotating proxies pool at each request.

Categories
Development Web Scraping Software

Content Grabber with free proxy account integration for business directories scrape

Professional data extraction requires adequate proxying to keep anonymity of scraping robots. When attempting to extract large data sets (over 1M records, ex. business directories) reliable and fast proxy service is needed.

Sequentum has released the Nohodo proxy service integration for Content Grabber. Nohodo provides a free account for Content Grabber users (up to 5000 requests monthly for free). The feature is available for both trial users and regular customers. Here’s how it works…

Categories
Development

How to connect Content Grabber with Proxy-connect

Consistent web scraping requires the use of multiple rotating proxies to prevent blocking and throttling by your target website. Let’s take the Content Grabber – a visual scraper with the Proxy-Connect rotating proxy server service for an example scrape.

Categories
Web Scraping Software

Using ProxyMesh with Visual Web Ripper

ProxyMesh is another rotating anonymous proxy server service that lets users stay anonymous with the help of a network of continuously rotated IP proxy servers. This service requires no software to be downloaded but it can be easily used in conjunction with Visual Web Ripper software.

Categories
Development

How to change WebDriver’s IP address

I have already written several articles on how to use WebDriver for web scraping, but I have never touched on the topic of changing WebDriver’s IP address. Nevertheless, this topic is quite crucial when you come to web scraping, and here I’d like to show you an example of using proxies with WebDriver in Python (and you can easily convert it into your language API).

Categories
Web Scraping Software

Anonymous Scraping with Visual Web Ripper

It’s very common to use proxy servers for web data extraction. If you want to stay undetected when you scrape a website, you have to change your IP address periodically. Otherwise it is very easy to detect unusual activity by observing a large number of requests from a single IP address. Visual Web Ripper has a built-in support of proxy servers called Private Proxy Switch.