Extract browser’s Local Storage with Python

Some of you may be wondering if it’s possible to extract a web browser’s local storage by web scraping?

Local storage in a nutshell is when a website stores data on your machine instead of having to call the server for it every time. Local storage is more secure than cookies, and large amounts of data can be stored locally, without affecting website performance. It’s accessible through browser scripting, for  example JavaScript.

Why JavaScript to fetch local storage, is there another way to get it?

Here’s some from the Wikipedia’s definition of local storage:

So in my view the local storage is data stored by web browser (ex. Opera) somewhere on your hard drive (or cloud machine) where browser is run. So to fetch them you need to locally hack Opera’s data files, which is much harder. I think the simplest way is to apply the client-scripting, namely JavaScript.

Python and Selenium

None of the high level programming languages invoke a browser instance, they request and extract pure HTML only. So if we want to access the browser’s local storage when scraping a page, we need to invoke both a browser instance and leverage a JavaScript interpreter to read the local storage. For my money, Selenium is the best solution.

Here’s how to leverage custom scripting through Selenium’s framework upon a web browser instance.

A possible replacement for Selenium is PhantomJS, running a headless browser.

JaveScript to iterate over localStorage browser object

for (var i = 0; i < localStorage.length; i++){
    key=localStorage.key(i); 
    console.log(key+': '+localStorage.getItem(key));
}

Advanced script

As mentioned here a HTML5 featured browser should also implement Array.prototype.map. So script would be:

Array.apply(0, new Array(localStorage.length)).map(function (o, i) 
   { return localStorage.key(i)+':'+localStorage.getItem(localStorage.key(i)); }
)

Python with Selenium script for setting up and scraping local storage

from selenium import webdriver
driver = webdriver.Firefox() 
url='http://www.w3schools.com/'
driver.get(url)
scriptArray="""localStorage.setItem("key1", 'new item');
               localStorage.setItem("key2", 'second item'); 
				return Array.apply(0, new Array(localStorage.length)).map(function (o, i) { return localStorage.getItem(localStorage.key(i)); }
				)""" 	
result = driver.execute_script(scriptArray)
print(result)

Python bindings alternative to Python+Selenium

Some might argue Selenium is inefficient for only local storage extracting. If you think Selenium is too bulky, you might want to try a Python binding with a development framework for desktop, ex. PyQt. Something I might touch on in a later post.