Categories
Challenge Development

Undetected ChromeDriver in Python Selenium

Selenium comes with a default WebDriver that often fails to bypass scraping anti-bots. Yet you can complement it with Undetected ChromeDriver, a third-party WebDriver tool that will do a better job.

In this tutorial, you’ll learn how to use Undetected ChromeDriver with Selenium in Python and solve the most common errors.

Categories
Challenge Development

Python, Selenium for custom browser automation scraper

Recently we’ve got the tricky website, its data being of dynamic nature. Yet we’ve applied the modern day scraping tools to fetch data. We’ve develop an effective Python scraper using Selenium library for browser automation.

About the project

We were asked to have a look at a retailer website.

And our task was to gather data on 210 products’ availability in 945 shops. The scrape resulted in about 200K data entries in a CSV format. Moreover, every line contained information about name, link, brand, store and the availability of a product. Below you can familiarise yourself with a small data sample we were able to gather.

Categories
Challenge Development

Node.js, Python & Ruby Bots Zoo repo

Today, I got in touch with the Node.js [and Python] bots garden/zoo providing modern bots with different kinds of browsers (Firefox, Chrome, Headless/not headless) using different automation frameworks (Puppeteer, Selenium, Playwright) in several programming languages.

Categories
Data Mining Development

AI code-writing assistant that understands data content and generates code

Recently I’ve encountred a client that predicts “in 6 month AI will be able to do much coding instead of man”.

…in years you’ll be able to on the fly, ask the AI to purchase a server, or create a website with X website builder… and basically, I bet it will write code on the fly on your demand where it connects to these tool’s APIs to really make things happen. It could do this now for some easy stuff but it’s unreliable and will mess up.

Now we’ve ancountered a interesing public repo, called Sketch. It’s AI code-writing assistant for Pandas (Python) users.

Categories
Development

Scrape YouTube comments using Python & Selenium

Scraping youtube comments has become crucial if you are working on some sentiment analysis project. The comments section will give you an overview of the public sentiment toward any election or sports results, scams, wars, etc. Comments reflect an overall feeling of a person. What according to them is right and wrong is mentioned in the comments.

Categories
Development Guest posting

Scrape ‘Ticketmaster’ using Selenium with Python

We’ve got some code provided by Akash D. working on ticketmaster.co.uk. He automates browser (Chrome as well as Edge) using Selenium with Python. The rotating authenticated proxies are leveraged to keep undetected. Yet, the site is protected with Distil network.

Categories
Development

Map(), lambda() functions for 2-d arrays

Suppose we’ve a following array:

arr = [[ 5.60241616e+02,  1.01946349e+03,  8.61527813e+01],
 [ 4.10969632e+02 , 9.77019409e+02 , -5.34489688e+01],
 [ 6.10031512e+02, 9.10689615e+01, 1.45066095e+02 ]]

How to print it with rounded elements using map() and lamba() functions?

l = list(map(lambda i: list(map(lambda j: round(j, 2), i)), arr))
print(l)

The result will be the following:

[[560.24, 1019.46, 86.15], 
 [410.97, 977.02, -53.45], 
 [610.03, 91.07, 145.07]]
Categories
Data Mining

Sklearn, Classification and Regression metrics

in the post will reviewed a number of metrics for evaluating classification and regression models. For that we use the functions we use of the sklearn library. We’ll learn how to generate model data and how to train linear models and evaluate their quality.

The code as an IPython notebook

Categories
Data Mining

Linear models, Sklearn.linear_model, Regression

In this post we’ll show how to build regression linear models using the sklearn.linear.model module.

See also the post on classification linear models using the sklearn.linear.model module.

The code as an IPython notebook

Categories
Data Mining Development

Linear models, Sklearn.linear_model, Classification

In this post we’ll show how to build classification linear models using the sklearn.linear.model module.

The code as an IPython notebook