– how to improve performance


Some may argue that extracting 3 records per minute is not fast enough for an automated scraper (see my last post on Dexi multi-threaded jobs). However, you should realize that Dexi extractor robots behave like a full-blown modern browser and fetch all the resources that crawled pages load (CSS, JS, fonts, etc.).
In terms of performance, an extractor robot might not be as fast as a pure HTTP scraping script, but its advantage is the ability to extract data from dynamic websites which require running JavaScript code in order to generate a user-facing content. It will also be harder for anti-bot mechanisms to detect and block it.

Fiddler web sniffer Review

The Fiddler web sniffer software is one of several well-known HTTP debugging proxies. It works well, and the proxy nature of the sniffer makes it stand out among other web sniffing tools.

HTTP Scoop Web Sniffer Review

The HTTP Scoop sniffer by Tuffcode is a Mac OS web sniffer doing multiple HTTP watches. This tools stands in a row of other HTTP protocol sniffing tools.

Inductor and Condencer Types Web Sniffers

This post is on the distinctions between specific web sniffers: those of induction nature, and those of condenser or proxy nature.

CharlesProxy HTTP Sniffer Review

The Charlesproxy website sniffer is the subject of this post. This sniffing/monitoring application works with Windows, Mac and Linux OS. It rather differs from other web traffic sniffing tools.

Firebug for HTTP activity sniffing

Firebug is an all-inclusive web development tool. One of its features is to monitor network activity. Firebug is excellent in HTTP as well as HTTPS data catch and display. Other protocol analyzers are viewable here.

WireShark as HTTP Sniffer Review

Wireshark logoWireShark is an all-inclusive network protocol analyzer. It works to display all the protocol layers including application layer protocols (HTTP and SSL). Though it is well able to capture a multitude of protocols, we focus on the HTTP, which is vital to Web Scraping. Other traffic analyzers are reviewed here.

HttpFox – HTTP analyzer add-on for Firefox Review

HttpFox is an HTTP sniffer add-on for Firefox by Martin Theimer. The other free and paid HTTP headers sniffing tools are here.

LiveHTTPHeaders Sniffer Review

This post deals with LiveHTTPHeaders, the FireFox add-on, by Daniel Savard. This built-in tool is designed to show live HTTP headers and traffic content, too. This tool stands among other simple and multy-functional HTTP traffic analysers.

EffeTech HTTP Sniffer Review

EffeTech Sniffer is an HTTP standalone sniffing application for the Windows operating system. The other HTTP traffic analyzers are available here.