JavaScript rendering library for scraping javascript sites

logo-js-rendering-libraryCan you imagine how many scraping instruments are at our service? Though it has a long history, scraping has at last become a multi-lingual and simple approach. Unfortunately, there is a list of non-trivial tasks which can’t be resolved in a snap.

One of these tasks is scraping javascript sites, those that output data using JavaScript. Facing this task, classic scrapers (not all of them though) ignore JS-data and continue their own life-cycle. However, when this little defect becomes a big trouble, developers all over the world take measures. And they did it! Today we consider one of the most awesome tools which scrapes JS-generated data – Splash. addon functionality (images & geocoding)

dexi-io-geo-maps-logoThe web scraping service has remade its functionality by adding [paid plan] addons. Through addons, more features are made available to customers, e.g. more step types/pipe actions. Those features also allow the integration of scrape results to data stores and endpoints like PostgreSQL, MySQL, Amazon S3 and other. 

Octoparse 7.0 – a free web scraping tool for non-developers


octoparse has recently launched a brand new version 7.0, which has turned out to be the most revolutionary upgrade in the past two years, with not only a more user-friendly UI, but also some of the advanced features make web scraping even easier. In this post, I will walk through some of the new features/changes made available in this new version, with respect to how a beginner, even one without any coding background, can approach this web scraping tool.

SquidProxies review

Today we want to share with you about SquidProxies. It is a service offering anonymous HTTP/HTTPS proxies.

SquidProxies offers 2 types of data-center proxy packages, private proxies and shared proxies. The proxies are designated for just about any legal use, and work great to surf to every website. The proxies’ main use are web scraping/web crawling and SEO tools.

New European e-communication regulations and web scraping

GDPR-eu-rulesGeneral Data Protection Regulation or GDPR: enforcement date – 25 May 2018. The GDPR covers the matter of online user data privacy rules for electronic communication and data protection. The regulation includes modern communication messengers and services, eg. Skype, Viber, Gmail, etc., that have not been previously mentioned in the former EU e-communication directives.  

“Privacy is guaranteed for content of communication as well as metadata (e.g. time of a call and location) which have a high privacy component and need to be anonymised or deleted if users did not give their consent, unless the data is needed for billing.”

See the infographic on GDPR or the main elements of GDPR in EU.

Design patterns for hierarchical data storage and effective processing

The hierarchical data storage problem is a non-trivial task in relational database context. For example, your online shop has goods of different categories and subcategories creating tree spans for 5 levels. How should they be stored in a database?

Luckily, there are several approaches (design patterns) that will help the developer to design database structure without both odd tables and code. As a result, the site will work faster and any changes, even on database layer, won’t cause troubles. We will study these approaches below.

US court stated scraping, even when against TOS, is legal

Last month a legal case took place in a US court where four professors plus a media organization sued the US Government. The District Court for the District of Columbia conclusion stated that moderate scraping, even when against ToS, is legal.

A district court in Washington, D.C. has ruled that using automated tools to access publicly available information on the open web is not a computer crime — even when a website bans automated access in its terms of service (document). The court ruled that the notoriously vague and outdated Computer Fraud and Abuse Act (CFAA) — a 1986 statute meant to target malicious computer break-ins — does not make it a crime to access information in a manner that the website doesn’t like if you are otherwise entitled to access that same information.

Before we put down all the legal statement fine details, we better give you a conclusion derived from the law case.
If you do not read all the legal statement fine points, jump right to the conclusion.

US court stated scraping, even when against TOS, is legal

court_smallLast month a legal case took place in a US court where four professors plus a media organization sued the US Government. The District Court for the District of Columbia conclusion stated that moderate scraping, even when against ToS, is legal.

Big Data Basics

big-data-worldPower of Big Data: capabilities and perspectives

As everyone knows, technological development is evolving flash like. At the same time software requirements, approaches and algorithms are growing with equal speed.  In particular, relatively recently, developers have faced the problem of huge data volume processing – making it necessary to create a new, effective approach, a new paradigm of data storage. The solution was not long in coming – in 2011 huge companies all over the world started using the Big Data concept. In this article we will talk about this engaging approach.

SQL-injection: how to use them and how to defend against them

sql-injection-logoSQL (Structured Query Language) is a powerful language for working with relational databases, but quite a few people are in fact ignorant of the dark side of this language, which is called SQL-injection. Anyone who knows this language well enough can extract the needed data from your site by means of SQL – unless developers build defenses against SQL-injection, of course. Let’s discuss how to hack data and how to secure your web resource from these kinds of data leaks!