Tag: Python

Categories
Challenge Development

How to avoid IP bans with rotating proxies [with example code]

When scraping websites or automating online activities, IP bans can be a major obstacle. Many websites implement anti-scraping measures that block repeated requests from the same IP address. To bypass this, using rotating proxies is a common and effective strategy. Rotating proxies automatically switch your IP address with each request, making it harder for websites to detect and block your activity.

Why Use Rotating Proxies?

  • Avoid IP Bans: Changing IPs helps prevent your IP from being flagged or blocked.
  • Bypass Geo-restrictions: Access content restricted to certain regions by rotating through proxies in different locations.
  • Increase Success Rate: Improves the chances of successful requests by mimicking more natural browsing behavior.
Continue reading “How to avoid IP bans with rotating proxies [with example code]”
Categories
Development

Crawlee library for fast crawler composure

Crawlee is a free web scraping & browser
automation library fitting for composing Node.js (and Python) crawlers.

 
Continue reading “Crawlee library for fast crawler composure”
Categories
Challenge Development

Playwright Scraper Undetected: Strategies for Seamless Web Data Extraction

Web scraping has become an essential tool for many businesses seeking to gather data and insights from the web. As companies increasingly rely on this method for analytics and pricing strategies, the techniques used in scraping are evolving. It is crucial for scrapers to simulate human-like behaviors to avoid detection by sophisticated anti-bot measures implemented by various websites.

Understanding the importance of configuring scraping tools effectively can make a significant difference in acquiring the necessary data without interruptions. The growth in demand for such data has led to innovations in strategies and technology that assist scrapers in navigating these challenges. This article will explore recent developments in tools and libraries that help enhance the functionality of web scraping procedures.

Continue reading “Playwright Scraper Undetected: Strategies for Seamless Web Data Extraction”
Categories
Challenge Development

Oxylabs’ Web Scraper API – Experience & more

Experience

We’ve succesfully tested the Web-Scraper-API of Oxylabs. It did well to get data off the highly protected sites. One eg. is Zoro.com protected with Akamai, DataDome, CloudFlare and ReCaptcha! See the numerical results here.

Continue reading “Oxylabs’ Web Scraper API – Experience & more”
Categories
Challenge Development

Undetected ChromeDriver in Python Selenium

Selenium comes with a default WebDriver that often fails to bypass scraping anti-bots. Yet you can complement it with Undetected ChromeDriver, a third-party WebDriver tool that will do a better job.

In this tutorial, you’ll learn how to use Undetected ChromeDriver with Selenium in Python and solve the most common errors.

Continue reading “Undetected ChromeDriver in Python Selenium”
Categories
Challenge Development

Python, Selenium for custom browser automation scraper

Recently we’ve got the tricky website, its data being of dynamic nature. Yet we’ve applied the modern day scraping tools to fetch data. We’ve develop an effective Python scraper using Selenium library for browser automation.

About the project

We were asked to have a look at a retailer website.

And our task was to gather data on 210 products’ availability in 945 shops. The scrape resulted in about 200K data entries in a CSV format. Moreover, every line contained information about name, link, brand, store and the availability of a product. Below you can familiarise yourself with a small data sample we were able to gather.

Continue reading “Python, Selenium for custom browser automation scraper”
Categories
Challenge Development

Node.js, Python & Ruby Bots Zoo repo

Today, I got in touch with the Node.js [and Python] bots garden/zoo providing modern bots with different kinds of browsers (Firefox, Chrome, Headless/not headless) using different automation frameworks (Puppeteer, Selenium, Playwright) in several programming languages.

Continue reading “Node.js, Python & Ruby Bots Zoo repo”
Categories
Data Science Development

AI code-writing assistant that understands data content and generates code

Recently I’ve encountred a client that predicts “in 6 month AI will be able to do much coding instead of man”.

…in years you’ll be able to on the fly, ask the AI to purchase a server, or create a website with X website builder… and basically, I bet it will write code on the fly on your demand where it connects to these tool’s APIs to really make things happen. It could do this now for some easy stuff but it’s unreliable and will mess up.

Now we’ve ancountered a interesing public repo, called Sketch. It’s AI code-writing assistant for Pandas (Python) users.

Continue reading “AI code-writing assistant that understands data content and generates code”
Categories
Development

Scrape YouTube comments using Python & Selenium

Scraping youtube comments has become crucial if you are working on some sentiment analysis project. The comments section will give you an overview of the public sentiment toward any election or sports results, scams, wars, etc. Comments reflect an overall feeling of a person. What according to them is right and wrong is mentioned in the comments.

Continue reading “Scrape YouTube comments using Python & Selenium”
Categories
Development Guest posting

Scrape ‘Ticketmaster’ using Selenium with Python

We’ve got some code provided by Akash D. working on ticketmaster.co.uk. He automates browser (Chrome as well as Edge) using Selenium with Python. The rotating authenticated proxies are leveraged to keep undetected. Yet, the site is protected with Distil network.

Continue reading “Scrape ‘Ticketmaster’ using Selenium with Python”