I’ve described my initial experience with Zyte AI spiders leveraging Zype API and Scrapy Cloud Units. You might find it here. Now I’d share more sobering report of what happened with the data aggregator scrape.
As I’ve got more credit for Zyte API I decided to test the service once more (to scrape the same ozon.ru aggregator). Now the results turned out to be much poorer. First I had to switch from pure HTTP requests to browser automation (cost significant increase đ ). Then I’ve found out the free Scrapy Cloud Unit allows spider runs in cloud (Zyte-hosted) for 1 hour only. Even after I paid for a Cloud Unit, the spider again encountered some execution issues. The spider even stopped for inexpected reasons…
Till now I could not get a seamless spider run with over 5K products albeit following all the support instructions.
