Tag: web scraping

Categories
Review

DataFlowKit review

data-flow-kit-logoRecently we encountered a new service that helps users to scrape the modern web 2.0. It’s a simple, comfortable, easy to learn service – https://dataflowkit.com
Let’s first highlight some of its outstanding features:

  1. Visual online scraper tool: point, click and extract.
  2. Javascript rendering; any interactive site scrape by headless Chrome run in the cloud
  3. Open-source back-end
  4. Scrape a website behind a login form
  5. Web page interactions: Input, Click, Wait, Scroll, etc.
  6. Proxy support, incl. Geo-target proxying
  7. Scraper API
  8. Follow the direction of robots.txt
  9. Export results to Google drive, DropBox, MS OneDrive.
Continue reading “DataFlowKit review”
Categories
Guest posting Review

Octoparse 8 vs Octoparse 7 comparison – what’s new in 8.1

Our brand new version Octoparse 8 (OP 8) just came out a few weeks ago. To help you get a better understanding of what the differences between OP 8 and 7 are, we have included all the updates in this article.

Continue reading “Octoparse 8 vs Octoparse 7 comparison – what’s new in 8.1”
Categories
Legal Monetize

What is legal: scrape, or scrape & sell, or code a scraper

Which of the following is illegal:
(1) Scrape emails from a site and send one email to each address.
(2) Scrape emails from a website and sell them.
(3) Make a scraping script and sell it without using it.
Note: The target website Terms of Use (ToU) state that no one can crawl/scrape it.

Continue reading “What is legal: scrape, or scrape & sell, or code a scraper”
Categories
Review

ScrapingBee, an API for web scraping

scrapingbee_smallThe web is becoming increasingly difficult to scrape. There are more and more websites using single page application frameworks like Vue.js / Angular.js / React.js and you need to use headless browsers to extract data from those websites.

Using headless Chrome on your local computer is easy. But scaling to dozens of Chrome instances in production is a difficult task. There are many problems, you need powerful servers with plenty of RAM, you’ll get into random crashes, zombie processes…

Continue reading “ScrapingBee, an API for web scraping”
Categories
Review

Software for Web Scraping

There are many web data extraction applications and some cloud services available and they vary widely in cost and features. Here weíve summarized them to help you to make your choice. All of these programs and services have been either tested by us or have been in general use for web ripping. We hope these brief overviews and the following reviews will help you to choose a best web scraper for your purposes.

Continue reading “Software for Web Scraping”

Categories
Web Scraping Software

The present trends in web scraping tools

 

Recently I got a question from one of the blog readers. After I replied to it, I decided to share it with a wider audience.
Question:

Hi,

I found your [web]scraping.pro site and found it very helpful, then realized the web scraper solutions rating was from 2014.  What is the best solution for today?   I have lots of sites I need to scrape, mainly search then drill-down sites.   I would like to be able to schedule the scraping to run on a daily basis.  Is there a direction you could point me?  I’m a seasoned developer by trade but am seeing all these point and click solutions (e.g. import.io) and am wondering if I should stick with Node.JS or .NET or if I should investigate some of these GUI scrapers of today.
Continue reading “The present trends in web scraping tools”
Categories
Development

Scraping with free or paid proxies – what is the difference?

Anything free always sounds appealing. And we are often ready to go an extra mile to avoid expenses if we can. But is it a good idea to choose the free option when it comes to using proxies for data scraping? Or should you stick to the paid ones for better results?

Let’s weigh all the pros and cons to see why you should consider using residential IP providers like Infatica, Bright Data, NetNut, Geosurf and others.

Continue reading “Scraping with free or paid proxies – what is the difference?”
Categories
Guest posting Web Scraping Software

A revolutionary web scraping software to boost your business

If you were an Amazon seller, would you want to know the listing price of a product of all competitors? Since you don’t have direct access to the Amazon database, you are out of luck and have to browse and click through every listing in order to construct a table of sellers and prices. A web scraping tool comes in handy. It automatically downloads your desired information such as product name, seller’s name, price, etc. However, web scraping that requires coding skill can be painful for professionals in IT, SEO, marketing, e-commerce, real estate, hospitality, etc.

It seems beyond one’s job description if he/she needs to learn how to code in order to obtain certain useful data from the web. For example, I have a friend who graduated in Mass Communication and works as a content marketer. She wants to scrape some data from the web, so she decided to learn Python herself. It took her two weeks to come up with a page of messy codes. Not only did she waste time on learning Python, but she also lost the time she could have used for doing her real work.

Continue reading “A revolutionary web scraping software to boost your business”
Categories
Development

Make crawling easy with Real Time Crawler of Oxylabs.io

logo-oxylabs-ioNowadays, it’s hard to imagine our life without search systems. “If you don’t know something, google it!” –  is one of the most popular maxims in our life. But how many people use Google in an optimal way? A lot of developers use google commands to get needed answers as fast as it possible.

Even this is not enough today! Large and small companies need terabytes of data to make their business profitable. It’s necessary to automate the search process and make it reliable to satisfy the user with fresh news, updates or posts. In today’s article we will consider a very helpful tool – Real-Time Crawler (RTC) for the collection of fresh data. Let’s start!

Continue reading “Make crawling easy with Real Time Crawler of Oxylabs.io”
Categories
Guest posting

Web scraping and why you should learn It

Why should you learn web scraping and who is doing web scraping out there? We are going to address this question by looking into the different industries and jobs that require web scraping skills. To do this, we’ve compiled and analyzed the data extracted from job sites, including Indeed, Glassdoor and LinkedIn. Followings are our findings to share with you.

Continue reading “Web scraping and why you should learn It”