Category: Challenge

Python, Selenium for custom browser automation scraper

Post author By admin
Post date 10.04.2023
No Comments on Python, Selenium for custom browser automation scraper

Recently we’ve got the tricky website, its data being of dynamic nature. Yet we’ve applied the modern day scraping tools to fetch data. We’ve develop an effective Python scraper using Selenium library for browser automation.

About the project

We were asked to have a look at a retailer website.

And our task was to gather data on 210 products’ availability in 945 shops. The scrape resulted in about 200K data entries in a CSV format. Moreover, every line contained information about name, link, brand, store and the availability of a product. Below you can familiarise yourself with a small data sample we were able to gather.

Tags browser-automation, Python, Selenium, web scraping

Challenge Development

Discord Bot to detect on-site anti-scrape & scrape-proof tools

Post author By admin
Post date 22.03.2023
No Comments on Discord Bot to detect on-site anti-scrape & scrape-proof tools

Today, I’ll share of a Dicord server 1 and server 2 that accomodate a bot able to detect multiple modern scrape-protection and scrape-detection means. The server’s channels with the bot are #antibot-test and #antibot-scan respectively

Tags anti-scrape, CloudFlare, scrape detection, scrape protection

Challenge

Web Scraping: 5 pros and cons

Post author By admin
Post date 17.03.2023
No Comments on Web Scraping: 5 pros and cons

Web scraping, also known as data mining or web harvesting, is the process of extracting data from websites automatically. The extracted data can be used for various purposes, such as market research, price monitoring, sentiment analysis, and many more. However, web scraping has both advantages and disadvantages. In this article, we will discuss the five main pros and cons of web scraping.

Tags web scraping

Challenge Development

TLS (SSL) Fingerprinting

In the post we share about web fingerprinting and particularly TLS fingerprinting. First let’s categorize the fingerprinting.

Tags fingerprinting

Challenge Development

Node.js, Python & Ruby Bots Zoo repo

Post author By admin
Post date 08.03.2023
No Comments on Node.js, Python & Ruby Bots Zoo repo

Today, I got in touch with the Node.js [and Python] bots garden/zoo providing modern bots with different kinds of browsers (Firefox, Chrome, Headless/not headless) using different automation frameworks (Puppeteer, Selenium, Playwright) in several programming languages.

Tags Node.js, Python, scrape detection

Challenge Development

CodeStar connection full working policy at AWS

Post author By admin
Post date 10.02.2023
No Comments on CodeStar connection full working policy at AWS

The original AWSCodeStarFullAccess policy is full [to provide access to AWS CodeStar via the AWS Management Console] yet it still does not grant enough access to create a Code connection for IAM at CodePipeline. So we had to manage to create a custom policy based on AWS tutorial suggestions.

Tags devops

Challenge

Bot protected websites

Tags anti-scrape, CloudFlare, scrape protection

Challenge Development

AirTable scrape challenge

The problem is that data are loaded highly dynamically. HTML contains only the information that you currently see on the browser screen.

If there are a lot of records then it is difficult to collect such a table. One of the possible ways is to calculate the size of the screen and rows in the table. Then using the browser automation and use to make a script that will scroll through it bit by bit and collect data.

Is there any other feasible way to get data of a table? For example there is a HTTP requests coding way to get dynamic data.

JS infinite scroll does not work for AirTable either.

Please comment down here if having some tips, hints.

Tags web scraping

Challenge Development

Yelp scraping for high quality B2B leads

Post author By Slava Mihaschenko
Post date 16.09.2022
No Comments on Yelp scraping for high quality B2B leads

Recently we’ve performed the Yelp business directory scrape for acquiring high quality B2B leads (company + CEO info). This forced us to apply many techniques like proxying, external company site scrape, email verification and more.