Categories
Development

Map(), lambda() functions for 2-d arrays

Suppose we’ve a following array:

arr = [[ 5.60241616e+02,  1.01946349e+03,  8.61527813e+01],
 [ 4.10969632e+02 , 9.77019409e+02 , -5.34489688e+01],
 [ 6.10031512e+02, 9.10689615e+01, 1.45066095e+02 ]]

How to print it with rounded elements using map() and lamba() functions?

l = list(map(lambda i: list(map(lambda j: round(j, 2), i)), arr))
print(l)

The result will be the following:

[[560.24, 1019.46, 86.15], 
 [410.97, 977.02, -53.45], 
 [610.03, 91.07, 145.07]]
Categories
Development Featured Review Web Scraping Software

Sequentum Enterprise review

Sequentum Enterprise is a powerful, multi-featured enterprise data pipeline platform and web data extraction solution. Sequentum’s CEO Sarah Mckenna doesn’t like to call it web scraping because, in its description, the web scraping refers to many different types of unmanaged and non-compliant techniques for obtaining web-based datasets. 

Categories
Development

How to print out requestQueue info (Apify) at run time

The docs on requestQueue.getInfo().

After some unsuccessful tries I could have managed to get the requestQueue info output. Note, we run the function inside the Apify runtime environment:

Apify.main(async () => { ... }

Solution 1

We make the function async and add await to the getInfo() Promise call:

async function printRequestQueue (requestQueue){
   var { totalRequestCount, handledRequestCount, pendingRequestCount } = await requestQueue.getInfo();
   console.log(`Request Queue info:` );
   console.log(' - handled :', handledRequestCount);
   console.log(' - pending :', pendingRequestCount);
   console.log(' - total:'  , totalRequestCount); 
}

with the following result:

Request Queue info:
 - handled : 479
 - pending : 312
 - total: 791

Solution 2, using then/catch

In this case we do not need to make our function async since we catch the the getInfo() promise result thru .then(response).

function printRequestQueue (requestQueue){ 
  requestQueue.getInfo().then((response)=> { 
    console.log('total:', response.totalRequestCount); 
    console.log('handled:', response.handledRequestCount);
    console.log('pending:', response.pendingRequestCount);  
    console.log('\nFull response:\n', response); })
 .catch( (error) => console.log(error)); 
}

with the following result:

total: 791
handled: 479
pending: 312

Full response:
 { id: 'queue-name',
  name: 'queue-name',
  userId: null,
  createdAt: 2021-02-26T11:57:00.453Z,
  modifiedAt: 2021-02-26T11:58:47.988Z,
  accessedAt: 2021-02-26T11:58:47.989Z,
  totalRequestCount: 791,
  handledRequestCount: 479,
  pendingRequestCount: 312 
}
Categories
Development

Node.js Cheerio scraper, replace element

let table = $('table');
if ($(table).has('br')) {  				     
    $("br").replaceWith(" ");
}
Categories
Development

DOM selector excluding certain elements

Often we need to select certain html DOM elements excluding ones with certain names/ attributes/ attribute values. Let’s show how to do that.

Categories
Data Mining Development

Linear models, Sklearn.linear_model, Classification

In this post we’ll show how to build classification linear models using the sklearn.linear.model module.

Categories
Data Mining Development

Work with inbuilt datasets of Sklearn and Seaborn libraries

In the post we will show how to generate model data and load standard datasets using the sklearn datasets module. We use sklearn.datasets in the Python 3.

Categories
Development

Puppeteer Stealth to prevent detection

In the previous post we shared how to disguise Selenium Chrome automation against Fingerprint checks. In this post we share the Puppeteer-extra with Stealth plugin to do the same. The test results are available as html files and screenshots.

Categories
Challenge Development

Human-operated and automated Browser Fingerprints testing and needed parameters

In a previous post we’ve considered the ways to disguise an automated Chrome browser by spoofing some of its parameters – Headless Chrome detection and anti-detection. Here we’ll share the practical results of Fingerprints testing against a benchmark for both human-operated and automated Chrome browsers.

Categories
Development

Headless Chrome detection and anti-detection

In the post we summarize how to detect the headless Chrome browser and how to bypass the detection. The headless browser testing should be a very important part of todays web 2.0. If we look at some of the site’s JS, we find them to checking on many fields of a browser. They are similar to those collected by fingerprintjs2.

So in this post we consider most of them and show both how to detect the headless browser by those attributes and how to bypass that detection by spoofing them.

See the test results of disguising the browser automation for both Selenium and Puppeteer extra.