What is a Web Crawler and how does it work?

Nowadays, when having some questions, it almost comes naturally for us to just type it in a search bar and get helpful answers. But we rarely wonder how all that information is available and how it appears as soon as we start typing. Search engines provide easy access to information, but web crawling/scraping tools who are not so much known players have a crucial role in wrapping up online content. Over the years, these tools have become a true game-changer in many businesses including e-commerce. So, if you are still unfamiliar with it, keep reading to learn more.

Price2Spy Web Crawling Service for the Competitors Price Scraping

The number of companies which use web crawler is growing rapidly due to the current competitive market conditions. Because of that, the number of companies that offer this service is growing day by day. Since the use purpose of web crawler varies on a case to case basis, here is a more detailed explanation of the way that Price2Spy works.

Download a file from a link in Python

I recently got a question and it looked like this : how to download a file from a link in Python?

“I need to go to every link which will open a website and that would have the download file “Export offers to XML”. This link is javascript enabled.”

Let us consider how to get a file from a JS-driven weblink using Python :

Using DOMXPath for parsing page content in PHP

The DOMXPath class is a convenient and popular means to parse HTML content with XPath.
After I’ve done a simple PHP/cURL scraper using Regex some have reasonably mentioned a request for a more efficient scrape with XPath. So, instead of parsing the content with Regex, I used DOMXPath class methods.

Is there any way to skip CAPTCHA?

JavaScript powered CAPTCHA

Most of the answers to the question in internet forums are given by services that automatically solve captchas. They provide services to solve CAPTCHA rather than to fully skip it.

Octoparse 7.0 – a free web scraping tool for non-developers

octoparseOctoparse has recently launched a brand new version 7.0, which has turned out to be the most revolutionary upgrade in the past two years, with not only a more user-friendly UI, but also some of the advanced features make web scraping even easier. In this post, I will walk through some of the new features/changes made available in this new version, with respect to how a beginner, even one without any coding background, can approach this web scraping tool.