Webscraping.proI recently got a question and it looked like this : how to download a file from a link in Python?
“I need to go to every link which will open a website and that would have the download file “Export offers to XML”. This link is javascript enabled.”
We’ve done the Linkedin scraper that downloades the free study courses. They include text data, exercise files and 720HD videos. The code does not represent the pure Linkedin scraper, a business directory data extractor. Yet, you might grasp the main thoughts and useful techniques for your Linkedin scraper development. 
Recently, I was challenged to do bulk submits through an authenticated form. The website required a login. While there are plenty of examples of how to use POST and GET in Python, I want to share with you how I handled the session along with a cookie and authenticity token (CSRF-like protection).
In the post, we are going to cover the crucial techniques needed in the scripting web scraping:
I often receive requests asking about email crawling. It is evident that this topic is quite interesting for those who want to scrape contact information from the web (like direct marketers), and previously we have already mentioned GSA Email Spider as an off-the-shelf solution for email crawling. In this article I want to demonstrate how easy it is to build a simple email crawler in Python. This crawler is simple, but you can learn many things from this example (especially if you’re new to scraping in Python).
test.py
import MySQLdb, db_config
class Test:
def connect(self):
self.conn = MySQLdb.connect(host=config.db_credentials["mysql"]["host"],
user=config.db_credentials["mysql"]["user"],
passwd=config.db_credentials["mysql"]["pass"],
db=config.db_credentials["mysql"]["name"])
self.conn.autocommit(True)
return self.conn
def insert_parametrized(self, test_value="L'le-Perrot"):
cur = self.connect().cursor()
cur.execute("INSERT INTO a_table (name, city) VALUES (%s,%s)", ('temp', test_value))
# run it
t=Test().insert_parametrized("test city'; DROP TABLE a_table;")
db_config.py (place it in the same directory as the test.py file)
db_credentials = {
"mysql": {
"name": "db_name",
"host": "db_host", # eg. '127.0.0.1'
"user": "xxxx",
"pass": "xxxxxxxx",
}
}
The Python requests library is a useful library having tons of advantages compared to other similar libraries. However, as I was trying to retrieve the Wikipedia page, requests.get() retrieved it only partially:
Recently I decided to work with pythonanywhere.com for running python scripts on JS stuffed websites.
Originally I tried to leverage the dryscrape library, but I failed to do it, and a nice support explained to me: “…unfortunately dryscrape depends on WebKit, and WebKit doesn’t work with our virtualisation system.”
In this post we want to show you the code for an automatic connection to 2captcha service for solving google reCaptcha v2.0. Not long ago, google drastically complicated the user-behavior reCaptcha (v2.0). This online service provides a method for solving it.
Let’s suppose you want to extract a price with a currency sign from a web page (eg. £220.00), but its HTML code is this:
which is obviously encoded HTML.
Some of you may be wondering if it’s possible to extract a web browser’s local storage by web scraping?