Presently (March 2024) anti-bots are actively applied for web data protection. Some of them with their characteristics & bypass methods might be seen here. If you are interested, take a look at some bot protected websites table. In this post we’ll share our real case experience with fighting CloudFlare proection.

The CloudFlare is a multy functional platform incl. CDN, Firewalls, Application Security, Bot Management and more. So, as we’ve found it at swappie.com & airwise.com we tried some ways to bypass with initial weak success.

Local execution bypass success in Headful mode

We’ve managed to bypass the Cloudflare protection page when we used residential proxy* and a browser in the headful mode (both Chrome headless at Selenium and Chromium at Playwright). But we considered browser working only in headless mode as Linux server.

*We applied Ultra Residential Proxies of MarsProxies.

Xvbf (X virtual framebuffer) to help

As we turned to community we’ve got help.

There’s a Linux tool called Xvfb**, which creates a virtual display so that you don’t have to use Chrome’s headless mode when running browser tests in it. Various frameworks have a option to use it.

**Xvbf stands for “X virtual framebuffer”.

We have installed Xvbf to the Linux server.
Now we’re able to apply browser scrape in headful mode with command:

xvfb-run [command]

where [command] is the orignal command to launch a scraper code.

Screen max-size emulation with Playwright

Later we managed to scrape data using Playwright with browser in Headless mode but with emulation of the screen size of 1920×1080 + residential proxy.

Read more on Choosing reliable residential proxies for web scraping