In this post I’d like to share my experience with the residential proxy of the Luminati proxy provider.

Residential proxies’ advantages

The traditional proxies’ disadvantage is that they are provided by data centers. Web services can easily recognize that those IPs originate from a dc and thus block it as a web robot (not a regular user) visit. Even with a decent proxy, websites can cloak their data or modify them when detecting a bot visit.

Residential IP is an IP provided to a home user by an Internet Service Provider (ISP). Users [of web apps] give their consent to allow access to their residential IPs while a device is idle, connected to the internet and not in use and has enough power (eg. luminati.io/sdk). Luminati does not collect any user data, rather it is interested only in IPs. To date, Luminati connects to over 30 million residential IPs, located across the world.

Luminati Proxy Manager

What does the proxy manager do? It is a [open-source] software for managing multiple proxies seamlessly via API and admin UI.
To see all the ways to install it, use Tools->Proxy Manager on the Luminati’s Dashboard left-side panel.

The advantages of using of the proxy manager:

• One entry point
• Concurrent connections
• Auto retry rules
• Real time statistics

Besides, proxy manager can be used with someone’s phone for scraping (link).
[box style=”note”]Note that residential proxies require special approval, so it took me 3 working ways till I could get that zone working.[/box] The personal Luminati manager (assistant) has been helpful in getting acquainted with LPM, creating zones and making it to work.

Test

We decided to test the Luminati service, particularly its residential proxies (residential, limited to 7 days only).

Setup

First of all we set up the Local Proxy Manager (LPM) and the residential proxies zone. Read more about LPM. Zones are a service’s custom configurations of parameters for [proxided] requests (zones’ board inside Luminati account).

[box style=”note”]Note that residential proxies require special approval, so it took me 3 working ways till I could get that zone working.[/box] The personal Luminati manager (assistant) has been helpful in getting acquainted with LPM, creating zones and making it to work.

We’ve set up 4 zones inside of LPM:

• data-center proxies zone, port 24000 (port number is assigned automatically or set manually)
• residential proxies zone, port 24001
• city asn proxies zone, port 24002
• gip proxies zone, port 24003

In the test code we were using port 24001, corresponding to the residential zone of my LPM, running at the address http://127.0.0.1:22999/. Basically the process looks like this:

Below you can see the proxy ports utilizing different proxy zones.

[box style=”note”]Note: There is also a mobile IPs option [proxies zone]. Mobile IPs are almost unblockable. Mobile proxies usage covers (1) website performance, (2) retail/travel: prices fetching, app promotions (Adverification). [/box]

So, we started to test YellowPages.com to gather links thru a simple GET request of hotel keyword and 2 letter state abbreviation (NY, CA, etc.)

https://www.yellowpages.com/search?search_terms={0}&geo_location_terms={1}&page={2}

We performed consequent requests to the YP site and extracted all US hotels rotating thru the 50 US states abbreviations array (as the scraper had not gotten new hotel items for a given state).

Test code

import requests, json
import re, time, sys, random

test_url="http://lumtest.com/myip.json"

proxies = {
  'http':  'http://lum-customer-scrapingpro-zone-gen:[luminati password]@zproxy.lum-superproxy.io:22225',
  }

def get_content(url='',
                proxies = {'http':"127.0.0.1:24001"},
                headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
                ):
    test_url="http://lumtest.com/myip.json"
    try:
        real_proxy = requests.get(test_url, proxies = proxies, headers = headers)
        real_proxy = json.loads(real_proxy.text)
    except: 
        real_proxy={'ip':'', 'country':''}
    try:
        r = requests.get( url, proxies = proxies, headers = headers)
        res_length=len(r.text)
        #print "Url: ", url, "\nGot html of "+str(res_length)+' bytes.'
        if '/recaptcha/' in r.text:
            captcha=True
        else:
            captcha=False
        return {'text': r.text,
            'params': { 'size':res_length, 'captcha':captcha,
                       'exit_node': {"ip":real_proxy['ip'], 'country':real_proxy['country']}
                       }
            }
    except:
        print 'Failure get html by url:', url
        return { 'text':{}, 'params':{'size':0, 'captcha':False, 'exit_node': real_proxy }  }
    
    #if res_length<10000:
        #print 'Result www.yellowpages.com.au.html\n', r.text
        
    
state_codes = ['AL','AK','AZ','AR','CA','CO','CT','DE','FL','GA','HI','ID','IL','IN','IA','KS','KY','LA',
             'ME','MD','MA','MI','MN','MS','MO','MT','NE','NV','NH','NJ','NM','NY','NC','ND','OH','OK',
             'OR','PA','RI','SC','SD','TN','TX','UT','VT','VA','WA','WV','WI','WY','AS','DC','FM','GU',
             'MH','MP','PW','PR','VI','AE','AA','AE','AE','AE','AP']
curr_state_code = 'CA'
state_codes_processed = []
p_type='residential'
proxy_type={'data-center':'0', 'residential':'1', 'city-asn':'2', 'gip':'3' }
print '****************************\n'+ p_type , 'proxies test:'
proxies ={
    'http':"127.0.0.1:2400" + proxy_type[p_type]
    }
#url = "https://www.yellowpages.com"
url = "https://www.yellowpages.com/search?search_terms={0}&geo_location_terms={1}&page={2}"
hotel_counter=0
start = time.time() 
total_links = set()
page_index=1
for i in range(1,5000): 
    res = get_content(url.format('hotel', curr_state_code, page_index), proxies,
    {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'},
                        )
     
    print '**************\nRequest', str(i), 'url: ' , url.format('hotel', curr_state_code, page_index) , '.\n', res['params']
    # write to file
    file_name= 'yellopages_search_{}_{}_page{}_links.html'.format('hotel', curr_state_code, page_index)
    all_sets = re.findall( r"<h2 class=\"n\">.*?<\/h2>", res['text'])
    all_links = re.findall( r'\"business-name\" href=\"(.*?)\"', res['text'])
    page_index+=1
    prev_total_links_amount=len(total_links)
    for link in all_links:
        total_links.add(link.split('?')[0])
    
    # check if links are the same
    if prev_total_links_amount >= len(total_links):
        state_codes_processed.append(curr_state_code)
        while curr_state_code in state_codes_processed:
            curr_state_code = random.choice(state_codes)
        page_index=1
        print "Processed state codes:", state_codes_processed
        print 'New state code:', curr_state_code
       
    print 'Found', len(all_links), 'links.'
    print 'All links amount:', len(total_links)
    print 'Total requests:', i
    print "Process time (seconds): " , round(time.time() - start, 1)
    #print ''.join(all_sets)
    with open('total_links.txt', "w" ) as text_file:
        text_file.write('\n'.join(total_links))

Results

All hotel links/items amount: 7147
Total requests: 263
Process time (seconds): 1267.1

We counted total links to assure that the aggregator does not expose the same hotel links in order to spoof a scrape-bot. The test result has shown us that for each request to a web page we got an average 7147/263 = 27 items per page (32 items on a page). The proxy extraction time was 1267/263 =~ 5 seconds per request.

Other service figures

Luminati has a 4-second timeout for DNS lookup.

Network uptime 99.9%, and it can be viewed live at https://luminati.io/cp/status

 

Conclusion

The Luminati proxy provider proved to be reliable in scraping from a challenging site aggregator. Its residential proxies proved to be high output proxies, and the scraper ran seamlessly using them.