I develop a web scraping project using Selenium. Since I need rotating proxies [in mass quantities] to be utilized in the project, I’ve turned to the proxy gateways (nohodo.com, charityengine.com and some others). The problem is how to incorporate those proxy gateways into Selenium for surfing web?
What I already am aware of:
Selenium using a single proxy IP, yet at Remote WebDriver
PROXY = "249.265.13.07:8080" webdriver.DesiredCapabilities.FIREFOX['proxy'] = { "httpProxy":PROXY, "ftpProxy":PROXY, "sslProxy":PROXY, "noProxy":None, "proxyType":"MANUAL", "class":"org.openqa.selenium.Proxy", "autodetect":False } driver = webdriver.Remote("http://localhost:4444/wd/hub", webdriver.DesiredCapabilities.FIREFOX)
Python code to accomodate proxy gateway
import requests proxies = { 'http': 'http://d49ab71e98fcb571eb7a1b092:@domain.site.com:2020', 'https': 'https://d49ab71e98fcb571eb7a1b092:@domain.site.com:2020'} r = requests.get('https://www.ipinfo.io/', proxies=proxies, verify=False) print(r.status_code) print(r.text)
So, how to join both Selenium and a proxy gateway?
Please, share your thoughts or experinces in comments here.