Lately I needed to scrape some data that are dynamically loaded by “Load more” button. A website JavaScript invokes XHR (or Ajax request) to fetch a next data portion. So, the need was to re-run those XHR with some POST parameters as variables.
![](http://game.zhichkinroman.ru/wp-content/uploads/2021/06/dev-tools-1.jpg)
So, how to make it in Node.js?
1. Replay XHR and fetch it from Dev. tools
We copy the XHR from Developer tools (F12) as a Node.js fetch() function call:
![](http://game.zhichkinroman.ru/wp-content/uploads/2021/06/copy-xhr-for-node-js.png)
The result will be something like the following:
fetch("https://www.xxx.com/graphql", { "headers": { "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.9,ru;q=0.8", "content-type": "application/json", "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"", "sec-ch-ua-mobile": "?0", "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-original-url": "https://www.xxx.com/search/page/2?q=yyy", "cookie": "wlw_client_id=rBEAAmC2BOKcHADzAxVeAg==;..." }, "referrer": "https://www.xxx.com/search/page/2?q=B%C3%BCrom%C3%B6bel&supplierTypes=Hersteller", "referrerPolicy": "origin-when-cross-origin", "body": "{\"query\":\"query( $ufsSessionId: String, $searchMethod: String, $q: String, $conceptSlug: String, $citySlug: String, $filters: [FilterInput!], $page: Int, $sort: String, $tracking: Boolean, $disableCityExtraction: Boolean, $cityExtractionRadius: String } }\",\"variables\":{\"ufsSessionId\":\"9dcdece22e1f0a694730ea638e4996bb\",\"searchMethod\":\"direct\",\"locale\":\"de\",\"q\":\"Büromöbel\",\"filters\":[{\"name\":\"Lieferantentyp\",\"selected\":[\"Hersteller\"]}],\"page\":2,\"sort\":\"best\",\"tracking\":true,\"disableCityExtraction\":false,\"cityExtractionRadius\":\"50km\"}}", "method": "POST", "mode": "cors" });
2. The full code
Wrap up that fetch() call with an asynchronous function. We also change the parameter value using the wrapper function input variable, named i
. The data is written into a file using a file stream.
async function getdata(i){ fetch("https://www.xxx.com/graphql", { "headers": { "accept": "application/json, text/plain, */*" }, "variables ":{ "page":"+ i + "}", "method": "POST", "mode": "cors" }).then(response => response.json()) .then((value) => { let companies = value.data.search.companies; //console.log( companies ); let names=[]; // we fetch names only console.log( '\nPage: ', i); for(c of companies){ console.log( c['name']); names.push( c['name']); } names.forEach( function (item, index) { stream.write(item + "\n"); }); }); }
The whole code you might load below.