Problem scraping javascript site – help needed


I am trying to scrape the page and I have installed the docker and followed all the required steps given in the post. Splash works well, but the spyder does not and I don’t know why. The IP of the splash_url is correct but I can’t see in the response object when I write scrapy shell “webpage” the complete page, ie, the page has not rendered correctly.


I have followed the steps for installing and start with the docker in this link:
My code is very simple. I create in pyCharm a project and I have only changed two files: and the

BOT_NAME = 'MercadonaINE'
SPIDER_MODULES = ['MercadonaINE.spiders']
NEWSPIDER_MODULE = 'MercadonaINE.spiders'
USER_AGENT = 'Mozilla/5.0 (compatible; Googlebot/2.1; +'

    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

import scrapy
from ..items import MercadonaineItem
#from scrapy.utils.response import open_in_browser
from scrapy_splash import SplashRequest

class MercadonaSpider(scrapy.Spider):
    name = 'mercadona'
    start_urls = [

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 3},)

    def parse(self, response):
        title = response.css('title::text').extract()
        yield {'title': title}

In a browser  I check with I see that the code loads the page with the initial form and the products and I can see the html code of all the components. The page loads the content dynamically with javascript. (see the following screen captures).Captura-de-pantallaCaptura-de-pantalla

The problem is when I run the spider. The response object does not contain the complete html elements (see a screen capture below). In the html code there are not the first form of the page and there are not any product with the prices.Captura-de-pantalla

I think this example is the most simple one. I have checked some examples that also work with javascript for loading the main page and they work well (for example the quotes example that is in the web), but the case I described above is impossible for me.

Please, advise any way out thru posting in comments. I appreciate any help.

Julio César

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.