Proxies are an integrated part of most major web scraping and data mining projects. Without them, data collection becomes sloppy and biased. This is why it’s essential to know how to find the best affordable proxies for any web scraping project.
One of the best proxy types you could use for scraping is residential proxies. In this post, you’ll learn what they are, how they are priced and what to look for before committing your project’s budget.
What are residential proxies
A residential proxy is a proxy server that uses as gateway IP addresses from consumer-level ISPs (Internet Service Providers). Essentially, each residential proxy’s IP address comes from a real device (smartphone, laptop, PC, Mac) connected to the Internet through consumer Internet Service Providers like AT&T, Vodafone, Comcast, etc.
Unlike datacenter proxies, which use datacenter IPs that are easier to identify and block, using residential proxies for web scraping, websites will see your project’s requests as originating from regular Internet connections. Just like those of their other visitors. Which in turn will help your scraper avoid cloakers and biased data.
To better understand how these proxies work, here’s a simple diagram of a residential proxy network.
All your requests will follow this path:
- Your PC or server sends requests through the proxy network
- Your requests reach the proxy provider’s residential master proxy (this is the server that will forward your requests, rotate and assign your IPs)
- Requests are then sent through the residential IP pool, reaching the Internet through a residential IP
- The website reached sees your requests as being started from the residential IP (your gateway proxy IP), not from your PC or server’s real IP address
As a side note, not all residential proxy networks are equal, not all services are the same, and indeed not all providers ask the same price.
Here’s what you should look for if you consider getting these proxy types for your project.
Pricing models
If by now, you used private proxies, you must be familiar with their per IP pricing model. If you buy 100 private proxies, you pay a set amount for each proxy (ranging from a few cents to a few dollars).
However, residential proxy networks use a different pricing model. In fact, they use two pricing models:
- Bandwidth-based pricing
- Ports-based pricing
Here’s how each one works.
Bandwidth pricing
- you pay for how much bandwidth you use
- unlimited access to a proxy network
- unlimited requests and no throttling
One provider offering bandwidth pricing is Smartproxy. Each package they offer comes with access to their whole IP pool and a bandwidth limit counted in GBs.
Ports-based pricing
- limits the number of threads/simultaneous connections per port
- unlimited bandwidth
- number of requests throttled
Among the providers offering ports-based pricing is Stormproxies, which offers packages with unlimited bandwidth but limits the number of ports (hence requests) you can send to their network at any given time.
Pay-as-you-go pricing model
Bright Data among its bandwidth based plans offers a Pay-as-you-go plan:
Which pricing model is better?
There isn’t a perfect, a one-size-fits-all model. Each model comes with its own advantages and disadvantages.
The important thing is to find which pricing model works best for you and your project.
If you run a small project, with a limited number of requests, ports-based pricing might be the most suited option for you. Whereas, if your project is request-intensive and you need to send multiple requests in a limited time frame, then bandwidth-based pricing is the right choice.
This decision, of choosing the best pricing model and the network with the most suited features for your project, leads to the next thing you should look for when selecting a provider.
Money-back guarantees and free trials
Always, ALWAYS look for providers offering either free trials, before purchasing, or money-back guarantees, after paying for access.
For example, Bright Data offers a 7 days free trial.
Whereas providers like Smartproxy offer a 3 days money-back guarantee.
At the end of the day, it doesn’t matter which option a given service has. All that should matter for you is to have to ability to access a residential IP pool and test it.
Why you need such a guarantee?
If it’s the first time you’re buying from a provider, you don’t know how good their proxy network is, or how slow or “bad” their IPs are.
Also, you might need some features that you missed checking before buying. Such as city-level geo-targeting. Because some residential proxy providers offer only country-level targeting.
What if you need to scrape two local versions of the same website, let’s say from New York and LA to compare the data?
In this case, a provider with country-level targeting won’t be enough and the scraped data will lack accuracy.
Unfortunately, if you already purchased your proxies, there is nothing you can do. However, if there is a trial or money-back guarantee, you can ask your provider for a refund because their service isn’t suited to your scraping needs.
IP Pool Size
Does the residential IP pool size matter? Yes, it does for a few reasons:
- IP history – a smaller IP pool means your requests will rotate between fewer IPs and you risk being allocated IPs with a bad history that will block your access to popular websites
- Not enough IPs from the city/metropolitan area where you need them (as in the above example, if you need IPs from New York and Los Angeles, a smaller IP pool might not have proxies in your chosen area)
- Faster scraping – more IPs means you can send more requests through multiple IPs without risking to rotate your requests back into already-used IPs
| Proxy Service | IP pool |
| Bright Data | 72+ million (see more here) |
| Smartproxy | 55 million (see more here) |
| Stormproxies | 40.000 |
When can you choose a service with a small IP pool?
While size does matter in the proxy world, you can also use services with smaller IP pools when:
- you don’t need many IPs for scraping
- your scraper doesn’t perform a lot of requests
But, the only differentiator, even when the IP pool comes into question, remains the price.
One provider with a smaller IP pool, like Stormproxies, offers access to their proxy pool, starting at $19 per month for one port.
Rotational factor
Not all residential proxy services are created equal. Thus, they don’t rotate their IPs the same.
Some residential services, like Bright Data, use a per-request rotation – each request you originate will rotate through a different IP.
While other providers, such as Smartproxy, also offer sticky sessions ranging from a few seconds to a few minutes – for a given timeframe, your requests will use the same IP address and then will rotate.
In other words, some providers offer requests-based rotations, while others provide time-based rotation.
Does the rotation factor matter?
Yes and no. Here, too, the decision to choose one type of rotation or another depends on your project’s needs.
Why residential IPs can’t be used for long? Why can’t I get longer rotations?
This is a valid question asked by a lot by developers before buying proxies. The answer to these two questions lies in how providers acquire their residential IPs.
Unlike data center private proxy providers who own their IPs, residential providers rent theirs from regular users through SDKs installed on various apps.
And for a user’s device to become part of the residential IP pool, it needs to comply with certain conditions.
One provider renting their proxies through an SDK is Bright Data. Here are some of the conditions under which a device is eligible to become part of their residential IP pool as a gateway server.
Geo-targeted IPs
Another thing to consider when choosing the best affordable residential proxies for web scraping is the ability to geo-target the gateway IPs.
Some services offer a country-level geo-targeting (you can select to use proxies only from one particular country). In contrast, other providers offer city-level targeting (you can choose to use proxies only from London or New York).
Preferably, a provider should offer you the feature to use proxies at a city level. But, such targeting is offered by providers with more expensive packages.
Therefore, if you don’t need a city-level targeting feature for your IPs, then you can choose any provider offering country-level targeting and cheaper packages.
Wrap up
These are the general things you should look for when searching for the best affordable residential proxies.
Here a checklist with all the things you should consider:
- Pricing:
- bandwidth
- ports
- Money-back guarantee or free trial offered?
- Yes/No
- IP Pool size
- Rotation type/factor:
- requests-based
- time-based
- Specific geo-targeting:
- country-level
- city-level
Based on your scraping project’s needs, you can add more checkpoints to this list.
As a final note: there isn’t a perfect (or ideal) proxy service out there. That’s why you should look only for the residential service most suited to your project’s features and needs.
