Categories
Data Mining Development

AI code-writing assistant that understands data content and generates code

Recently I’ve encountred a client that predicts “in 6 month AI will be able to do much coding instead of man”.

…in years you’ll be able to on the fly, ask the AI to purchase a server, or create a website with X website builder… and basically, I bet it will write code on the fly on your demand where it connects to these tool’s APIs to really make things happen. It could do this now for some easy stuff but it’s unreliable and will mess up.

Now we’ve ancountered a interesing public repo, called Sketch. It’s AI code-writing assistant for Pandas (Python) users.

Continue reading “AI code-writing assistant that understands data content and generates code”
Categories
Development

Scrape YouTube comments using Python & Selenium

Scraping youtube comments has become crucial if you are working on some sentiment analysis project. The comments section will give you an overview of the public sentiment toward any election or sports results, scams, wars, etc. Comments reflect an overall feeling of a person. What according to them is right and wrong is mentioned in the comments.

Continue reading “Scrape YouTube comments using Python & Selenium”
Categories
Challenge Development

CodeStar connection full working policy at AWS

The original AWSCodeStarFullAccess policy is full [to provide access to  AWS CodeStar via the AWS Management Console] yet it still does not grant enough access to create a Code connection for IAM at CodePipeline. So we had to manage to create a custom policy based on AWS tutorial suggestions.

Continue reading “CodeStar connection full working policy at AWS”
Categories
Development

Puppeteer async scraper with browsers number to be tuned based on CPU capacity

Recently we’ve got a tricky website of dynamic content to scrape. The data are loaded thru XHRs into each part of the DOM (HTML markup). So, the task was to develop an effective scraper that does async while using reasonable CPU recourses.

Continue reading “Puppeteer async scraper with browsers number to be tuned based on CPU capacity”
Categories
Development

Choosing reliable [rotating] residential proxies

Our experience with the residential proxy services. The table data are the work in progress…

   Service   ResidentialCost/monthTraffic/month$ per GBRotatingIP whitelistingPerformance and moreNotes
MarsProxies
N/AN/A3.5yesyes 500K+ IPs, 190+ locations
Test results		SOCKS5 supported
Proxy grey zone restrictions
Oxylabs.io
N/A25 GB9 - 12

"pay-as-you-go" - 15		yesyes100M+ IPs, 192 countries

- 30K requests
- 1.3 GB of data
- 5K pages crawled		Not allowing to scrape some of grey zone targets, incl. Linkedin.
Smartproxy
Link to the price page

N/A5.2 - 7

"pay-as-you-go" - 8.5		yesyes65M+ IPs, 195+ countriesFree Trial
Not allowing to scrape some of grey zone targets, incl. Linkedin.
Infatica.io
N/AN/A3 - 6.5

"pay-as-you-go" - 8		yesyesOver 95% success
*Bans from Cloudflare are also few, less than 5%.		Black list of sites —> proxies do not work with those.
  • 1000 ports for one Proxy List
  • Up to 20 Proxy Lists at a time
  • Using via API Tool
  • ISP-level targeting
  • Rotation time selection
Mango Proxy
N/A1-50 GB3-8"pay-as-you-go" - 8yesyes90M+ IPs, 240+ countries
IPRoyal
N/AN/A$4.55yesyes32M+ IPs, 195 countriesNot allowing to scrape some of grey zone targets, incl. Facebook. List of bloked sites.
Rainproxy.io yes$ 4from 1 GB4yes
BrightDatayes15
ScrapeOps Proxy AggregatoryesAPI Credits per monthN/A N/AyesAllows multithreading, the service provides browsers at its servers. It allows to run N [cloud] browsers from a local machine.
The number of threads depends on the subscription: min 5 threads. 		The All-In-One Proxy API that allows to use over 20+ proxy providers from a single API
Lunaproxy.comyesfrom $15x Gb per 90 days0.85 - 5 Each plan allows certain traffic amount for 90 days only.
LiveProxies.ioyesfrom $454-50 GB5 - 12yesyesEg. 200 IPs with 4 GB for $70.00, for 30 Days limit
Charity Engine -docsyes-- starting from 3.6
Additionally:
CPU computing
- from $0.01 per avg CPU core-hour
- from $0.10 per GPU-hour - source.		failed to connect so far
proxy-sale.comyesfrom $17N/A
3 - 6

"pay-as-you-go" - 7		yesyes10M+ IPs, 210+ countries30 days limit for a single proxy batch
Tabproxy.comyesfrom $15N/A
0.8 - 3
(lowest price is for a chunk of 1000 GB)		yesyes 200M+ IPs, 195 countries30-180 days limit for a single proxy batch (eg. 5GB)

Additional

Another proxy comparison in here.

Undetectable’s proxy partners.

Categories
Challenge

Bot protected websites

We share here some bot-protected sites.

Continue reading “Bot protected websites”
Categories
Development

MERN Stack – Build a Film Hall Application

Handle Your Project in Details

What is MERN?

The MERN stack is a set of frameworks and tools used for developing a software product. They are very specifically chosen to work together in creating a well-functioning software (see a MERN app code at the post bottom).

Continue reading “MERN Stack – Build a Film Hall Application”
Categories
Development

A Full Stack Todo App

In this post we are presenting you a Full-Stack Todo App, which is built in React.js, Next.js and Sequelize Sqlite3, and it is responsively implemented according to different screen sizes (see the code at the post bottom).

Continue reading “A Full Stack Todo App”
Categories
Challenge Development

AirTable scrape challenge

I need to get info from AirTable, see a table example.

The problem is that data are loaded highly dynamically. HTML contains only the information that you currently see on the browser screen.

enter image description here

If there are a lot of records then it is difficult to collect such a table. One of the possible ways is to calculate the size of the screen and rows in the table. Then using the browser automation and use to make a script that will scroll through it bit by bit and collect data.

Is there any other feasible way to get data of a table? For example there is a HTTP requests coding way to get dynamic data.

JS infinite scroll does not work for AirTable either.

Please comment down here if having some tips, hints.

Categories
Featured Monetize

2022 laid off [tech] Specialists Data Set

Companies like Facebook owner Meta, e-commerce behemoth Amazon and ride-hailing firm Lyft have announced job cuts in recent weeks as the US tech industry deals with an uncertain economic climate.

DW, 01.12.2022

Data Set

So, we’ve got lists (as a data directory) of those laid off tech professionals (from open sources). If someone is interesting in the data, we would gladly share it for a cost (starting from $5 per entry). The filtering by role, expertise and location is available.

Sample

Take a look at a data sample (no emails) below.

2022-laid-off-employees-sampleDownload

Please enquire me at: igor [dot] savinkin [at] gmail [dot] com for a customized data set.