Categories
Development

Selenium using proxy gateway, how?

I develop a web scraping project using Selenium. Since I need rotating proxies [in mass quantities] to be utilized in the project, I’ve turned to the proxy gateways (nohodo.com, charityengine.com and some others). The problem is how to incorporate those proxy gateways into Selenium for surfing web? What I already am aware of: Selenium using a […]

I develop a web scraping project using Selenium. Since I need rotating proxies [in mass quantities] to be utilized in the project, I’ve turned to the proxy gateways (nohodo.com, charityengine.com and some others). The problem is how to incorporate those proxy gateways into Selenium for surfing web?

What I already am aware of:

Selenium using a single proxy IP, yet at Remote WebDriver

PROXY = "249.265.13.07:8080"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
    "httpProxy":PROXY,
    "ftpProxy":PROXY,
    "sslProxy":PROXY,
    "noProxy":None,
    "proxyType":"MANUAL",
    "class":"org.openqa.selenium.Proxy",
    "autodetect":False
}
driver = webdriver.Remote("http://localhost:4444/wd/hub", webdriver.DesiredCapabilities.FIREFOX)

source

Python code to accomodate proxy gateway

import requests
proxies = { 'http': 'http://d49ab71e98fcb571eb7a1b092:@domain.site.com:2020',
'https': 'https://d49ab71e98fcb571eb7a1b092:@domain.site.com:2020'}
r = requests.get('https://www.ipinfo.io/', proxies=proxies, verify=False)
print(r.status_code)
print(r.text)

So, how to join both Selenium and a proxy gateway?

Please, share your thoughts or experinces in comments here.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.