Consistent web scraping requires the use of multiple rotating proxies to prevent blocking and throttling by your target website. Let’s take the Content Grabber – a visual scraper with the Proxy-Connect rotating proxy server service for an example scrape.

Proxy-Connect rotating proxy service

Proxy-Connect.com is a back-connect proxy service. This service uses a single gateway IP and designated port for each proxy protocol assined to each client at signup. Clients connect their scraper software or scraping service to this gateway IP and the backend IP’s addresses are automatically rotated every few seconds.

The Proxy-Connect provides both HTTP, HTTPS and SOCKS5 proxy service protocols. The service offers a port for each proxy protocol, each port supporting unlimited threads.

Setup

Proxy-Connect setup

To get a gateway IP you need to signup to the Proxy-Connect service and enter/provide your static IP. Upon placing an order for their proxy service, you will be sent the secure gateway IP and the proxy ports (via email). However, if you have a dynamic IP, you will be given a web based panel that allows you to update the IP anytime you wish.

Note: the Gateway IP must always be used with your authenticated IP address.

Content Grabber agent setup

Let’s compose a light Content Grabber agent (download link) that will request ipchiken.com to fetch proxy IPs. Follow the steps to configure the proxy for an individual agent (if you wish to set up a proxy for the whole Content Grabber app just do the same from the Application Settings tab):

1. Go to the Agent Settings tab
2. For the Proxy Source  options choose 3rd Party.
3. For the 3rd Party options choose Proxy Switch.
4. Press the Proxy Switch Settings area to input the gateway IP and port:
5. Now as we run the agent we get results:

Note: For debugging purpose before running agent you might tick up “View browser” box in order that a browser window would display current proxy ip. 

With Content Grabber I’ve used a simple HTTP connection, but a SOCKS5 connection would be faster. Yet if your scraper software doesn’t support Socks5, then using both HTTP ports (5008 and 5003) at Proxy-connect will boost both the speed and efficiency of scraping.

I should also mention that these proxies are not private ones, they are public and shared proxies, so you cannot expect a 100% success rate, you may get the occasional gateway errors and invalid responses. With my testing the fail rate is 10-16%. Read more here on this proxy service.

Special offer

For our readers theyíre offering a discount price for the monthly proxy subscription plan: only $39/month ($49).