I have already written several articles on how to use WebDriver for web scraping, but I have never touched on the topic of changing WebDriver’s IP address. Nevertheless, this topic is quite crucial when you come to web scraping, and here I’d like to show you an example of using proxies with WebDriver in Python (and you can easily convert it into your language API).
First of all you need a proxy server. You can use any proxy service you like, but if you don’t have one, I would recommend that you consider BestProxyAndVPN (if you’re ready to pay for proxy services). I have tried many proxy services and ended up using this one as the most stable.
Now let’s look at two pieces of code displaying how to make WebDriver send its requests through a proxy.
Using Proxy with Chrome Driver
If you use Selenium WebDriver with the Chrome browser, you can tune it to use a proxy in the following way:
options = Options()
options.add_argument('--proxy-server=46.102.106.37:13228')
browser = webdriver.Chrome(executable_path='ChromeDriverPath', chrome_options=options)
Don’t forget to change the proxy server address and ChromeDriverPath to your values.
Using Proxy with Firefox WebDriver
If you prefer scraping with the Firefox browser, you can cause it to use proxy with the following code:
profile = FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", '46.102.106.59')
profile.set_preference("network.proxy.http_port", '13228')
profile.set_preference("network.proxy.ssl", '46.102.106.59')
profile.set_preference("network.proxy.ssl_port", '13228')
browser = webdriver.Firefox(profile)
And again, don’t forget to change the proxy address and port to your values.
EDIT: Here is another way to specify proxy servers for Firefox webdriver (thanks to Curro Pavon):
my_proxy = Proxy({‘proxyType’: ProxyType.MANUAL,
‘httpProxy’: ’127.0.0.1:8118′,
‘sslProxy’: ’127.0.0.1:8118′,
‘noProxy’: ‘www.google-analytics.com, ajax.googleapis.com, apis.google.com’
})
browser = webdriver.Firefox(proxy=my_proxy)
16 replies on “How to change WebDriver’s IP address”
Actually, the Firefox driver constructor has an argument in order to set the proxy settings:
webdriver.Firefox(firefox_profile=firefoxProfile, proxy=my_proxy)
And the proxy definition will be a pretty easy dictionary:
my_proxy = Proxy({‘proxyType’: ProxyType.MANUAL,
‘httpProxy’: ‘127.0.0.1:8118’,
‘sslProxy’: ‘127.0.0.1:8118’,
‘noProxy’: ‘www.google-analytics.com, ajax.googleapis.com, apis.google.com’
})
In my opinion… clearer than the firefoxProfile stuff.
(I don’t know if the Chrome driver support this proxy definition too)
Thanks! Good job!
Thank you! I have update the article.
Hi,
Thank you for this proposition but it’s a bit weird for me to create Proxy without importing it ?
Can you explain more how to use it
Thank you again
HI Michael,
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(‘–proxy-server=78.157.215.87:3128’)
chrome = webdriver.Chrome(chrome_options=op)
chrome.get(“http://www.google.com”)
This code was working yesterday But it does not works now. I dont know what is the issue I am new to selenium please help me.
What is the error? May be your proxy ceased working?
you never define op
If you use Curro Pavon’s method for allow google analitycs to know your ipSRC, will it be like to use nothing? Am I wrong?
I have a question about Selenium and Webdrivers.
I read in your article about using the webdriver with a proxy, but I have a question about using that proxy or VPN if it’s a paid one? How and where do I enter the username and password in order for the proxy to authenticate?
Usually, proxy service is a paid one. You might use free proxies, yet the quality is not that good.
As far as the authentication, you’d need to check with a proxy provider. Usually it’s like this:
options.add_argument('--proxy-server=46.102.106.37:13228::') Yes, but in order to access that proxy, I think you need to authenticate. So you need to input somehow the username and password. One solution is to create a Chrome extension that does that when you launch the webdriver. But it’s not the most elegant one.
In a coming post we’ll review some of proxy services.
Hey, really informative and interesting introduction!
Since I am not a computer science student, I am wondering your setting with proxy server in the web-driver(–proxy-server=46.102.106.37:13228) will create new ip address everytime you run the script? or 46.102.106.37:13228 is already an ip address and everytime this ip address will be created?
maybe it is a stupid question but I will appreciate your reply!
Thanks!
Is there a way to find out the IP my webdriver is using to access the website?
Your web-driver IP is your server IP. Unless you use a selenium grid within a distributed network.
how to set proxy for webdriver on the fly?
I don’t want to create a webdriver, just for using a different proxy.
Explain more: “I don’t want to create a webdriver, just for using a different proxy.”