Companies like Facebook owner Meta, e-commerce behemoth Amazon and ride-hailing firm Lyft have announced job cuts in recent weeks as the US tech industry deals with an uncertain economic climate.
DW, 01.12.2022

Data Set

So, we’ve got lists (as a data directory) of those laid off tech professionals (from open sources). If someone is interesting in the data, we would gladly share it for a cost (starting from $5 per entry). The filtering by role, expertise and location is available.

Sample

Take a look at a data sample (no emails) below.

2022-laid-off-employees-sample Download

Please enquire me at: igor [dot] savinkin [at] gmail [dot] com for a customized data set.

Post author By Igor Savinkin
Post date December 2, 2022

Challenge Development

Discord Bot to detect on-site anti-scrape & scrape-proof tools

Today, I’ll share of a Dicord server that accomodates a bot able to detect multiple modern scrape-protection and scrape-detection means. The server name is Scraping Enthusiasts, channel with the bot being #antibot-test

Post author By Igor Savinkin
Post date March 22, 2023

Challenge

Web Scraping: 5 pros and cons

Web scraping, also known as data mining or web harvesting, is the process of extracting data from websites automatically. The extracted data can be used for various purposes, such as market research, price monitoring, sentiment analysis, and many more. However, web scraping has both advantages and disadvantages. In this article, we will discuss the five main pros and cons of web scraping.

Post author By Igor Savinkin
Post date March 17, 2023

Challenge Development

TLS (SSL) Fingerprinting

In the post we share about web fingerprinting and particularly TLS fingerprinting. First let’s categorize the fingerprinting.

Post author By Igor Savinkin
Post date March 15, 2023

Challenge Development

Node.js, Python & Ruby Bots Zoo repo

Today, I got in touch with the Node.js [and Python] bots garden/zoo providing modern bots with different kinds of browsers (Firefox, Chrome, Headless/not headless) using different automation frameworks (Puppeteer, Selenium, Playwright) in several programming languages.

Post author By Igor Savinkin
Post date March 8, 2023

Development

Service for getting real SMS on virtual online numbers + API

With 365SMS service you can receive SMS to a virtual number and verify your accounts on Facebook, Instagram, Twitter, Amazon, Airbnb, etc.

Buying virtual numbers and receiving SMS in automatic mode.
A virtual phone number is issued for 20 minutes. During this time, you can receive an unlimited number of SMS absolutely free.
You pay only for the received SMS, prices are as low as 0.3 EUR per number. If the number did not accept SMS, the money will be returned to the balance.
Large selection of services (incl. Linkedin) and countries for SMS activation.
Convenient API for comfortable work with external software.
User-friendly interface. Search by country and service makes it easy to buy a virtual phone number.
Dicounts for numbers in bulk.

Note

Since the virtual number is not in the country it represents, so a created 3d party account might be blocked. Therefore it’s better to:

1. Use mobile proxies that match the geolocation of the region of the purchased virtual number;
2. Use a mobile user agent.

Often the problem is not in the number, but in the wrong registration method.

Free numbers

Another vistual number SMS service, incl. free numbers.

Post author By Igor Savinkin
Post date March 8, 2023

Data Mining Development

AI code-writing assistant that understands data content and generates code

Recently I’ve encountred a client that predicts “in 6 month AI will be able to do much coding instead of man”.

…in years you’ll be able to on the fly, ask the AI to purchase a server, or create a website with X website builder… and basically, I bet it will write code on the fly on your demand where it connects to these tool’s APIs to really make things happen. It could do this now for some easy stuff but it’s unreliable and will mess up.

Now we’ve ancountered a interesing public repo, called Sketch. It’s AI code-writing assistant for Pandas (Python) users.

Post author By Igor Savinkin
Post date March 6, 2023

Development

Scrape YouTube comments using Python & Selenium

Scraping youtube comments has become crucial if you are working on some sentiment analysis project. The comments section will give you an overview of the public sentiment toward any election or sports results, scams, wars, etc. Comments reflect an overall feeling of a person. What according to them is right and wrong is mentioned in the comments.

Post author By Igor Savinkin
Post date March 3, 2023

Challenge Development

CodeStar connection full working policy at AWS

The original AWSCodeStarFullAccess policy is full [to provide access to AWS CodeStar via the AWS Management Console] yet it still does not grant enough access to create a Code connection for IAM at CodePipeline. So we had to manage to create a custom policy based on AWS tutorial suggestions.

Post author By Igor Savinkin
Post date February 10, 2023

Development

Puppeteer async scraper with browsers number to be tuned based on CPU capacity

Recently we’ve got a tricky website of dynamic content to scrape. The data are loaded thru XHRs into each part of the DOM (HTML markup). So, the task was to develop an effective scraper that does async while using reasonable CPU recourses.

Post author By Igor Savinkin
Post date February 9, 2023

Development

Choosing reliable [rotating] residential proxies

	Residential	Cost/month	Traffic/month	$ per GB	Rotating	IP whitelisting	Performance and more	Notes
Oxylabs.io		$ 300	25 GB	9 - 12 "pay-as-you-go" - 15	yes	yes	- 30K requests - 1.3 GB of data - 5K pages crawled	Not allowing to scrape some of grey zone targets, incl. Linkedin.
Rainproxy.io	yes	$ 4	from 1 GB	4	yes
Infatica.io		from $ 96	from 10 GB	3.5 - 12	yes	yes	Over 95% success *Bans from Cloudflare are also few, less than 5%.	No "pay-as-you-go" plan
BrightData	yes			15
Charity Engine -docs	yes	-	-	starting from 3.6 Additionally: CPU computing - from $0.01 per avg CPU core-hour - from $0.10 per GPU-hour - source.			failed to connect so far
Smartproxy	yes			8-10 "pay-as-you-go" - 12.5		yes		Not allowing to scrape some of grey zone targets, incl. Linkedin.

See more exensive and detailed comparison in here.

Post author By Igor Savinkin
Post date January 18, 2023