Octoparse review

Octoparse is a new modern visual web data extraction software. It provides users a point-&-click UI to develop extraction patterns, so that scrapers can apply these patterns to structured websites. Both experienced and inexperienced users find it easy to use Octoparse to bulk extract information from websites – for most of scraping tasks no coding needed! 

Overview

Octoparse, being both Windows and Mac application, is designed to harvest data from both static and dynamic websites (including those whose web pages that use ajax). The software simulates human operation to interact with web pages. To make data extraction easier, Octoparse features filling out forms, entering a search term into the text box, etc. You can run your extraction project either on your own local machine (Local Extraction) or in the cloud (Cloud Extraction). Octoparse’s cloud service, being available only in paid editions though, works well for harvesting large amounts of data to meet large-scale extraction needs. There are various export formats of your choice like CSV, Excel formats, HTML, TXT, and database (MySQL, SQL Server, and Oracle).             

Let’s describe some of Octoparse advantages

1. High speed

Getting data faster. This is self-evident and may be the core reason people resort to web scraping. Compared to manually doing this, a web scraper can execute your commands automatically, according to the workflow you have built for it. Each step of work that would have taken up your time will be done by the scraper. 

Once you set Octoparse agent/acrper up, it will run for you relentlessly, getting all kinds of web data fast from different websites. If you wanna try how fast a scraper can be, I recommend you to try our scraper templates. You may try an Amazon scraper (link to it ??) to gather product details or product reviews and see how a scraper can get you hundreds of well structured data lines in just a minute.

Welcome to download Octoparse to witness the speed of web scraping.

2. Cost-effectiveness

Web scraping is widely accepted by not only big companies but also SMBs. It just saves money.

First of all, hiring a development team would cost a lot. Looking for the right talents, dealing with leadership and management stuff to put them together and work effectively, human resources issues, all of these are time-consuming (and sometimes psyche-consuming). 

Web scraping takes repetitive work without the need of a coffee break. Unless you have a long-term development plan to carry out (or you got a big budget to squander), web scraping is worth a try.

The returns of setting up a scraper (or a set of scrapers) can be considerable. 

For example, if you want a series of product prices information and get it updated daily for a year, you may spend a few days or even weeks to configure the scrapers, test and fine-tune it. Once it is well built, it can work for you as long as you need it. Fresh price data will be delivered to you every morning then, more punctual than the employees in your office. 

Well it is not a once-for-all work. You may spend some time maintaining the scraper now and then, but it saves you a grand number compared to the cost of data delivery for 365 sets of price data a year.

3. Compatibility and flexibility

The flexibility of web scraping enables you to get data exactly in the form of how you need it. For example, regular expression is one of the ways to get your data cleaned. You can set commands with regex to refine the strings of data by adding a prefix, replacing A by B, cutting certain bytes of data, etc. 

If you are to scrape San Francisco local business phone numbers from Yelp to import it into your tele-marketing system, while your system recognizes only local numbers without the area code, regex can help you eliminate the string (415) of every number. Data can be cleaned as the scraper processes.

File formats also make things easier. Octoparse scrapes web data and export it into different formats like XLSX, CSV, HTML and JSON. These files are compatible with most apps and systems of data management, data analysis and visualization. 

One of Octoparse users runs a price comparison website. He set the tasks to run daily so that the prices are fresh. And the data scraped is automatically exported to a database which is connected to his website. 

A web scraper can be a data pipeline that extracts data from the web, clean it, organize it into the right format, copy it to your database and which could be then uploaded to your websites/systems. You can’t imagine how an individual could run and maintain a website by himself, getting the data on it refreshed everyday without automatic web scraping.  

That’s how web scraping can perfectly fit into your workflow and extremely improve productivity. 

4. Getting data that API can’t fetch

API is short for Application Programming Interface. An API gives people access to the data of an application or system which is granted by the owner.

According to Wikipedia:

“An API is a connection between computers or between computer programs. It is a type of software interface, offering a service to other pieces of software. One purpose of APIs is to hide the internal details of how a system works, exposing only those parts a programmer will find useful and keeping them consistent even if the internal details later change.”

Hence what data you can get through API highly depends on which part of the information is open to the public. In most cases, applications and services would offer only limited access to the public free of charge or sometimes at a cost.

And the specification to build an API connection varies from apps to apps. If you are looking for data from only one or a few sources, and luckily the data you need is all granted, you can study the API specification and take advantage of it. Honestly, to maintain an API connection could be easier than to maintain a web scraper.

Octoparse can become a data console where you gather data from different websites with a series of scrapers and become a data pipeline when you connect the tool to your database. Thus an Octoparse scraper can ignore the limit of API and extract what data you can see on the browser. Therefore, the scraping is more customizable, and when you need data from multiple sources, more powerful as an aggregator.

5. Gentle learning curve

Web scraping tasks can be a lot easier with a no-code web scraping tool like Octoparse since a non-coder does not have to learn python, R or PHP from scratch and can take advantage of the intuitive UI and the guide panel. 

However, even a web scraping tool takes time to get familiar with because you have to understand the basic idea of how a web scraper works so that you can command the tool to help you build the scraper.

For example, you shall know how the data is weaved into a HTML file with different HTML tags and structures. You may learn the basics of Xpath to locate the data you need. This is preferable to learn throughout your journey with Octoparse.

6. Avoiding IP blocking

Web scraping is about frequently visiting a website or a set of web pages, sometimes clicking and sending many requests in a short period of time (so as to capture the required data on the pages). Because of the abnormal frequency, your device or IP may be detected as a suspicious robot that is executing malicious attacks to the service.

IP blocking is likely to happen when you scrape websites that are protected with strict anti-scraping techniques such as LinkedIn and Facebook. The tracking and detection is mainly based on the IP footprint. Therefore, IP rotation or using IP proxies is an important technique for anti-blocking.

This is inevitable due to the speed of web scraping, while solutions like IP proxies and Octoparse cloud services offer a way out. See more here:

7. Cloud Service for scrape

Scraping the web on a large scale simultaneously, based on distributed computing, is the modern tendency, and Octoparse provides the feature also. To use it, you first have to switch from the free edition to any of the paid editions. After you upload your configuration project to the cloud, you can perform the extraction concurrently through Octoparse’s cloud servers. If you need to scrape thousands of web pages within a short time, Octoparse cloud service is ideal. Standard Edition limits you with only 4 concurrent threads (10 in Professional Edition). Extraction scheduling also is offered.

I’ve tried cloud extraction. The speed of simple link extraction has impressed me: over 3000 links in 1.5 min.

8. API

The Octoparse API makes it easy to connect your system to your scraped data in real time. You can either import the Octoparse data into your own DB, or use our API to require access to your account’s data. Just configure the rule for your task, run it in cloud, and Octoparse cloud servers will do the rest. API request data are returned as XML.

The Octorparse’s API allows the user to extract data on a timely basis: from a datetime till a datetime with max interval being 1 hour. Not that convenient. Insert datetime markers into the link parameters as follows:

9. Proxying

Does it ever drive you crazy that your IP address has been banned and you cannot access a website if you scrape a website frequently?  Yeah, it always happens especially when you extract data from business directories, which apply strict ban based on recurring IP(s). However, Octoparse enables you to scrape these websites by rotating anonymous HTTP proxy servers. In the cloud extraction mode, Octoparse applies more than 500 3rd party proxies for automatic IP rotation.

For local extraction, you have to add a list of external proxy addresses manually and configure them for automatic rotation. To learn how to include IP rotation into scraping project, please refer to here.

IPs are rotated with a interval of time that you set. In this way, you can extract data from the website without the risk of getting the IP address banned – in case you do not overload site’s bandwidth.

10. Support

The customer support is responsive and provides equal assistance both to paying plan users and free plan users. Support is accessible via Phone, Email, Skype (no limit for free users).

Octoparse in experience (trial & errors)

One user wanted to scrape Youtube KOL channel data for marketing purposes. Since she could not write a line of code (except for some HTML basics). It doesn’t matter – Octoparse works.

So she picked the software up and for two weeks she learned the basics of HTML and Xpath, and got familiar with the software as she hasn’t really used it for long. She started to build her own scrapers. You may be interested in the story of how a marketer made use of a no-code web scraping in work.

If you are looking for a way to get data with web scraping and want to start easy, a no-code tool would be a nice pick even though you are new to it (you can’t be more new to a programming language). 

Ask yourself two questions before downloading a web scraping tool:

• What data are you looking for?
• Where to obtain it?

When you have the answers, you will get a list of web pages and the exact data you need to download from them in mind. Next step, you can enter the URL into Octoparse and start point and click at the built-in browser and build yourself a web scraper.

Tips: Need to scrape some data from the web and not sure if it is feasible with the web scraping tool? Feel free to consult us through support@octoparse.com.

She plans to make a checklist of what one shall learn before building yourself a web scraper. She herself got the basics of HTML and Xpath along the way when she built scrapers with Octoparse through trial and error.

So if you have a web page URL and the target data you want to pull down, just download Octoparse and play with it.

Conclusion

Octoparse is a feature rich visual scraping application. It offers good point-&-click interface, though tasks handling sometimes lag. It’s easy to master in short time (with help of the good tutorials). Software is able to handle modern dynamic sites (in advanced mode). What impressed me is Octoparse’s cloud service – extract data in the cloud in a short time – not free though. In my opinion, it’s worth a try if you are collecting a large amount of data.