Web Scraping Software

How to use proxy in Visual Web Ripper

As we mentioned before, it’s often necessary to use proxy server when you gather infromation from the web. In this tutorial I’ll show you how to tune Visual Web Ripper to run the web requests through proxy servers.

1. Creating a test project

Let’s create a test project that scrapes a page from our testing ground, extracts the IP address and saves it as in the resulting table:

Visual Web Ripper test Project

2. Configuring proxy servers

Then we need to add proxies and let our scrape go through them. To do this, click Project>Project Options in the menu. Then in the opened window select the Proxies tab. Although Visual Web Ripper offers a large set of options here, let’s select Proxy List and Rotate per page load:

Visual Web Ripper Project Options

Now it’s time to add several proxies to our list. You can get a list of free open proxies at for instance, but if you need reliable private proxies you can get them at BestProxyAndVPN service for a little payment. To add the proxies click Add/Edit Proxy List button:

Visual Web Ripper Proxy List

Besides all other options, here the program allows you to test your proxies right away. You can do so by pressing the Test All Proxies button. After the test, all good proxies will be marked with a green box and bad ones, with a red box (also you’ll be asked for bad proxy removal).

3. Running the project

As soon as we have configured our project, let’s run it and see how it works. To run, click Project>Run Project in the menu. In my case I got the following execution log:

Visual Web Ripper Execution Log

As is seen from this log, first Visual Web Ripper switched the proxy and then requested the web page. Let’s see now what IP address has been scraped from the testing page (click View Data button):

Visual Web Ripper Execution Result

This IP address should differ from your machine IP address. If it is, then the proxy works!

Happy scraping!

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.