Companies like Facebook owner Meta, e-commerce behemoth Amazon and ride-hailing firm Lyft have announced job cuts in recent weeks as the US tech industry deals with an uncertain economic climate.
DW, 01.12.2022

Data Set

So, we’ve got lists (as a data directory) of those laid off tech professionals (from open sources). If someone is interesting in the data, we would gladly share it for a cost (starting from $5 per entry). The filtering by role, expertise and location is available.

Sample

Take a look at a data sample (no emails) below.

2022-laid-off-employees-sample Download

Please enquire me at: igor [dot] savinkin [at] gmail [dot] com for a customized data set.

Post author By Igor Savinkin
Post date December 2, 2022

Challenge Development

Node.js & Privacy Pass application for Cloudflare scrape solution

Over 7.59 million of websites use Cloudflare protection, 26% of
them are among the top 100K website worldwide. As Cloudflare
establishes itself as the norm regarding service protection, chances are, the site you want to scrape is more likely to use it than not.

When it comes to scrapping websites, captchas and other type of
protections were always the main obstacle in providing reliable data collection solutions. And most often this would lead to consider bypass services which aren’t always free.

Post author By Igor Savinkin
Post date May 2, 2023

Development

MobaXterm, better Putty alternative

No Comments on MobaXterm, better Putty alternative

MobaXterm is a server connectivity software (Windows) and it’s much better than Putty. It’s branded as “Enhanced terminal for Windows with X11 server, tabbed SSH client, network tools and much more”.

Post author By Igor Savinkin
Post date April 21, 2023

Uncategorized

Github AI pair programmer

GitHub Copilot · Your AI pair programmer

The copilot is integratable into 4 major IDEs

Visual Studio

Neovim

VS Code

JetBrains IDEs

Post author By Igor Savinkin
Post date April 14, 2023

Challenge Development

Undetected ChromeDriver in Python Selenium

Selenium comes with a default WebDriver that often fails to bypass scraping anti-bots. Yet you can complement it with Undetected ChromeDriver, a third-party WebDriver tool that will do a better job.

In this tutorial, you’ll learn how to use Undetected ChromeDriver with Selenium in Python and solve the most common errors.

Post author By Igor Savinkin
Post date April 11, 2023

Challenge Development

How to bypass PerimeterX

You’ve found the website you need to scrape, set up your scraper and fired it, just to sadly realize PerimeterX has blocked you.

PerimeterX’s dynamically complex bot detection system relies on server-side and client-side checks to distinguish humans from bots. It deploys several layers of protection and, for the most part, manages to do its job without interrupting the user experience.

But don’t fall into despair! There are a couple of things you can try to bypass PerimeterX (called HUMAN now) before giving up on your goal of scraping that delicious data.

Post author By Igor Savinkin
Post date April 11, 2023

Challenge Development

Python, Selenium for custom browser automation scraper

Recently we’ve got the tricky website, its data being of dynamic nature. Yet we’ve applied the modern day scraping tools to fetch data. We’ve develop an effective Python scraper using Selenium library for browser automation.

About the project

We were asked to have a look at a retailer website.

And our task was to gather data on 210 products’ availability in 945 shops. The scrape resulted in about 200K data entries in a CSV format. Moreover, every line contained information about name, link, brand, store and the availability of a product. Below you can familiarise yourself with a small data sample we were able to gather.

Post author By Denis Soloviev
Post date April 10, 2023

Challenge Development

Discord Bot to detect on-site anti-scrape & scrape-proof tools

Today, I’ll share of a Dicord server that accomodates a bot able to detect multiple modern scrape-protection and scrape-detection means. The server name is Scraping Enthusiasts, channel with the bot being #antibot-test

Post author By Igor Savinkin
Post date March 22, 2023

Challenge

Web Scraping: 5 pros and cons

Web scraping, also known as data mining or web harvesting, is the process of extracting data from websites automatically. The extracted data can be used for various purposes, such as market research, price monitoring, sentiment analysis, and many more. However, web scraping has both advantages and disadvantages. In this article, we will discuss the five main pros and cons of web scraping.

Post author By Igor Savinkin
Post date March 17, 2023

Challenge Development

TLS (SSL) Fingerprinting

In the post we share about web fingerprinting and particularly TLS fingerprinting. First let’s categorize the fingerprinting.

Post author By Igor Savinkin
Post date March 15, 2023

Challenge Development

Node.js, Python & Ruby Bots Zoo repo

Today, I got in touch with the Node.js [and Python] bots garden/zoo providing modern bots with different kinds of browsers (Firefox, Chrome, Headless/not headless) using different automation frameworks (Puppeteer, Selenium, Playwright) in several programming languages.

Post author By Igor Savinkin
Post date March 8, 2023