Categories
Miscellaneous

Huge JSON files view and search tool with excellent performance

Dadroit JSON Viewer LogoThe results of scraping activities are most often stored as json data, the latter having many advantages over .xml or .csv formats. Recently in one of my projects, I had to deal with JSON files of over 6Mb. Even though I managed them in Notepad++, still the proper search and count could have been better.

Categories
Miscellaneous

Endcaptcha now solving Recaptcha V2!

endcaptchaSo far the latest developments of the services that develop captchas  (google, nucaptcha, etc.) are no match for the captcha bypassers, and Endcaptcha is living proof of it.
Endcaptcha developers have been working hard to make this new feature possible – they’re finally releasing Recaptcha V2 support!

Categories
Uncategorized

Smartproxy Review

Getting precise and localized data is becoming difficult. Advanced proxy networks are the only thing that is keeping some companies running intense data gathering operations.

Residential proxies are in extremely high demand, and there are only a few networks available that can offer millions of IP addresses around the world. 

Smartproxy is one of those networks, rapidly growing to offer the best product in residential and data center proxies.

Categories
Legal

Linkedin lost in court to data analytic company that scrapes Linkedin’s public profiles info

On September 9th, 2019 the UNITED STATES COURT OF APPEALS 1 has affirmed the former district court’s determination that a certain [dataanalytic company is lawful to scrape [perform automated gathering] LinkedIn’s public profiles info. Now the historical event has happened in which a court is protecting a data extractor’s right for mass gathering openly presented business directory information.

Categories
Development

Scraping with free or paid proxies – what is the difference?

Anything free always sounds appealing. And we are often ready to go an extra mile to avoid expenses if we can. But is it a good idea to choose the free option when it comes to using proxies for data scraping? Or should you stick to the paid ones for better results?

Let’s weigh all the pros and cons to see why you should consider using residential IP providers like Infatica, Bright Data, NetNut, Geosurf and others.

Categories
Development

Using Modern Tools such as Node.js, Puppeteer, Apify for Web Scraping (Xing scrape)

I want to share with you the practical implementation of modern scraping tools for scraping JS-rendered websites (pages loaded dynamically by JavaScript). You can read more about scraping JS rendered content  here.

Categories
Development Guest posting

Captcha solving with Java and why you should avoid it

In this blog post we are going to show how you can solve [Re]captcha with Java and some third party APIs, and why you should probably avoid them in the first place.
For the Python code (+ captcha API) see that post.

The post author is Kevin Sahin from ScrapingNinja.co.

Captcha solving

“Completely Automated Public Turing test to tell Computers and Humans Apart” is what captcha stands for. Captchas are used to prevent bots from accessing and performing actions on websites or applications.

The last one is the most used captcha mechanism, Google ReCaptcha v2. That’s why we are going to see how to “break” these captchas.

Categories
Challenge

What are the best online resources to acquire data?

Recently I received this question: What are the best online resources to acquire data from?

The top sites for data scrape are data aggregators. Why are they top in data extraction?
They are top because they provide the fullest, most comprehensive data [sets]. The data in them are highly categorized. Therefore you do not need to crawl and fetch other resources and then combine multiple-resource data.

Those sites fall into 2 categories:

  1. Goods and services aggregators. Eg. AliExpress, Amazon, Craiglist.
  2. Personal data and companies data aggregators. Eg. Linkedin, Xing, YellowPages. For such aggregators another name is business directories.

The first category of sites and services is quite wide-spread. These sites and services promote their goods with the goal of being well-known online, to have as many backlinks as possible to them.

The second category, the business directories, does not tend to reveal its data to the public. These directories rather promote their brand and give scraping bots minimum opportunity for data acquiring*.

Consider the following picture where a company’s data aggregator gives to the user only 2 input fields: what and where.

You can find more of how to scrape data aggregators in this post.

————–
*You have to adhere to the ToS of each particular website/web service when you perform its data scraping.

Categories
Guest posting

How to increase your security while shopping online

VPNReasons - online payment security

As fraudsters and hackers are polishing their techniques, identity theft and online shopping fraud cases are rising every year. Most online shoppers are unaware of these threats and of the simple rules that can make online shopping safe. If you want to protect your money and your identity, you need to take certain precautionary measures.

Categories
SEO and Growth Hacking

Strategies on how to protect your data from cyber theft

cyber-theft-protection-strategies

Cyber-attacks are becoming a real threat to businesses both small and large. The damage they bring into people’s lives is more severe than people presume. In 2019, hundreds of billions of dollars went  down this tunnel, and the crime is yet to stop. With the evolvement of threat landscapes, attacks are becoming more and more sophisticated. It has also become clear that big companies need to understand that they cannot be 100% secure from such breaches. The real question is, if hackers manage to attack the big companies, how long would it take them to steal your data? The only way to handle this menace is if you understand these basic security strategies and implement them.