The problem is that data are loaded highly dynamically. HTML contains only the information that you currently see on the browser screen.
If there are a lot of records then it is difficult to collect such a table. One of the possible ways is to calculate the size of the screen and rows in the table. Then using the browser automation and use to make a script that will scroll through it bit by bit and collect data.
Is there any other feasible way to get data of a table? For example there is a HTTP requests coding way to get dynamic data.
JS infinite scroll does not work for AirTable either.
Please comment down here if having some tips, hints.
One reply on “AirTable scrape challenge”
@IgorSavinkin, Look at the requests like this and that JSON.
You can find them in the Network tab of Developer Tools (F12).
Then you scroll down the table, find the requests and extract the pattern of them.
Then you can find such requests when you scrolling down the table:
https://airtable.com/icons/icon_definitions.svg?v=350094b58ca651c8c1337a5da69b93bc
It returns data for the icons you mentioned in the question of that post and then it takes all the images one by one.