In the post we summarize how to detect the headless Chrome browser and how to bypass the detection. The headless browser testing should be a very important part of todays web 2.0. If we look at some of the site’s JS, we find them to checking on many fields of a browser. They are similar to those collected by fingerprintjs2.
So in this post we consider most of them and show both how to detect the headless browser by those attributes and how to bypass that detection by spoofing them.
See the test results of disguising the browser automation for both Selenium and Puppeteer extra.
- userAgent
UA can be said to be the most basic of the field to spoof in a headless browser. Yet simply modifying and spoofing the UA will not work. You need both UA and some of the fields such as the navigator.platform to modify too. Of course, a simple UA tries to mimic the desktop browser.
Detection
Through the navigator.userAgent one gets UA to identify whether browser is a crawler.
Bypass detection
You can modify the properties by the following:
Object.defineProperty(navigator, 'userAgent', {
get: () => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'
});
- webdriver
The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation
Detection
The desktop Chrome browser returns navigator.webdriver as undefined. One can be detect by: !!navigator.webdriver
Bypass detection
Object.defineProperty(navigator, 'webdriver', {
get: () => false
});
Or by executing:
delete navigator.proto.webdriver
Instead of the monitoring whether navigator.webdriver is undefined, one may simply check it through !!navigator.webdriver.
- language
Different browsers support for the language property is different; Chrome is only for navigator.language reference
Detection
navigator.language || navigator.userLanguage || navigator.browserLanguage || navigator.systemLanguage if there is one.
Bypass detection
Object.defineProperty(navigator, 'language', {
get: () => "zh-CN",
});
- colorDepth
window.screen.colorDepth of the screen color depth, is how to collect fingerprints when used. Both headless browser and a desktop browser have generally the same value in this field.
- deviceMemory
The read-only attribute, it returns the approximate machine memory in gigabytes. This value is an approximation of a power of 2 divided by 1024, rounding off to the decimal point. Yet this is just an experimental feature, not all browsers support it. It will be used during fingerprint collection, just in case you can set a value.
navigator.deviceMemory
Bypass detection
Object.defineProperty(navigator, 'deviceMemory', {
get: () => 8
});
- hardwareConcurrency
The browser environment with the number of CPU cores, to check:
navigator.hardwareConcurrency
Bypass detection
Object.defineProperty(navigator, 'hardwareConcurrency', {
get: () => 8
});
- screenresolution
Check the following to fingerprint:
window.screen.width and window.screen.height
- availableScreenResolution
Check the following to fingerprint: window.screen.availHeight and window.screen.availWidth
- timezoneOffset
It sets world time UTC with respect to the current time zone time difference value, in units of minutes. Set it up the following for bypassing:
new Date().getTimezoneOffset()
- timezone
new window.Intl.DateTimeFormat().resolvedOptions().timeZone- sessionStorage *
First you determine whether there is sessionStorage, then:
!!window.sessionStorage
- localStorage *
!!window.localStorage
- indexedDb
!!window.indexedDB
- addBehavior
This field is unique to IE
!!(document.body && document.body.addBehavior)
- openDatabase *
It will return a database where one may execute sql. The fingerprint only judges whether there is this method.
!!window.openDatabase
- cpuClass
It seems that Chrome browser returns undefined.
navigator.cpuClass
- platform *
navigator.platform
Detection
if (navigator.platform === "Linux x86_64")
Yet only a small share of browers has Linux x86_64. Better you use MacIntel or Win32.
Bypass Detection
Object.defineProperty(navigator, 'platform', {
get: () => 'MacIntel'
});
- plugins
navigator.plugins
The plugin object has name, filename, description and version attributes. You can use the plugin[0] to get the MimeType object.
- mimeTypes
navigator.mimeTypes
The mimeTypes objects need and plugin object corresponds to a type, suffixes, description, enabledPlugin property.
- canvas
In order to identify a browser we get canvas fingerprint reference. Then we identify a browser by the fingerprint, and then identify what browser it is based on some characteristics of the fingerprint.
Canvas based detection
(function (name, context, definition) {
if (typeof module !== 'undefined' && module.exports) {
module.exports = definition();
}
else if (typeof define === 'function' && define.amd) {
define(definition);
}
else {
context[name] = definition();
}
})('Fingerprint', this, function () {
'use strict';
var Fingerprint = function (options) {
var nativeForEach, nativeMap;
nativeForEach = Array.prototype.forEach;
nativeMap = Array.prototype.map;
this.each = function (obj, iterator, context) {
if (obj === null) {
return;
}
if (nativeForEach && obj.forEach === nativeForEach) {
obj.forEach(iterator, context);
} else if (obj.length === +obj.length) {
for (var i = 0, l = obj.length; i < l; i++) {
if (iterator.call(context, obj[i], i, obj) === {}) return;
}
} else {
for (var key in obj) {
if (obj.hasOwnProperty(key)) {
if (iterator.call(context, obj[key], key, obj) === {}) return;
}
}
}
};
this.map = function (obj, iterator, context) {
var results = [];
// Not using strict equality so that this acts as a
// shortcut to checking for `null` and `undefined`.
if (obj == null) return results;
if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
this.each(obj, function (value, index, list) {
results[results.length] = iterator.call(context, value, index, list);
});
return results;
};
if (typeof options == 'object') {
this.hasher = options.hasher;
this.screen_resolution = options.screen_resolution;
this.screen_orientation = options.screen_orientation;
this.canvas = options.canvas;
this.ie_activex = options.ie_activex;
} else if (typeof options == 'function') {
this.hasher = options;
}
};
Fingerprint.prototype = {
get: function () {
var keys = [];
keys.push(navigator.userAgent);
keys.push(navigator.language);
keys.push(screen.colorDepth);
if (this.screen_resolution) {
var resolution = this.getScreenResolution();
if (typeof resolution !== 'undefined') { // headless browsers, such as phantomjs
keys.push(resolution.join('x'));
}
}
keys.push(new Date().getTimezoneOffset());
keys.push(this.hasSessionStorage());
keys.push(this.hasLocalStorage());
keys.push(this.hasIndexDb());
//body might not be defined at this point or removed programmatically
if (document.body) {
keys.push(typeof(document.body.addBehavior));
} else {
keys.push(typeof undefined);
}
keys.push(typeof(window.openDatabase));
keys.push(navigator.cpuClass);
keys.push(navigator.platform);
keys.push(navigator.doNotTrack);
keys.push(this.getPluginsString());
if (this.canvas && this.isCanvasSupported()) {
keys.push(this.getCanvasFingerprint());
}
if (this.hasher) {
return this.hasher(keys.join('###'), 31);
} else {
return this.murmurhash3_32_gc(keys.join('###'), 31);
}
},
/**
* JS Implementation of MurmurHash3 (r136) (as of May 20, 2011)
*
* @author Gary Court
* @see http://github.com/garycourt/murmurhash-js
* @author Austin Appleby
* @see http://sites.google.com/site/murmurhash/
*
* @param {string} key ASCII only
* @param {number} seed Positive integer only
* @return {number} 32-bit positive integer hash
*/
murmurhash3_32_gc: function (key, seed) {
var remainder, bytes, h1, h1b, c1, c2, k1, i;
remainder = key.length & 3; // key.length % 4
bytes = key.length - remainder;
h1 = seed;
c1 = 0xcc9e2d51;
c2 = 0x1b873593;
i = 0;
while (i < bytes) {
k1 =
((key.charCodeAt(i) & 0xff)) |
((key.charCodeAt(++i) & 0xff) << 8) |
((key.charCodeAt(++i) & 0xff) << 16) |
((key.charCodeAt(++i) & 0xff) << 24);
++i;
k1 = ((((k1 & 0xffff) * c1) + ((((k1 >>> 16) * c1) & 0xffff) << 16))) & 0xffffffff;
k1 = (k1 << 15) | (k1 >>> 17);
k1 = ((((k1 & 0xffff) * c2) + ((((k1 >>> 16) * c2) & 0xffff) << 16))) & 0xffffffff;
h1 ^= k1;
h1 = (h1 << 13) | (h1 >>> 19);
h1b = ((((h1 & 0xffff) * 5) + ((((h1 >>> 16) * 5) & 0xffff) << 16))) & 0xffffffff;
h1 = (((h1b & 0xffff) + 0x6b64) + ((((h1b >>> 16) + 0xe654) & 0xffff) << 16));
}
k1 = 0;
switch (remainder) {
case 3:
k1 ^= (key.charCodeAt(i + 2) & 0xff) << 16;
case 2:
k1 ^= (key.charCodeAt(i + 1) & 0xff) << 8;
case 1:
k1 ^= (key.charCodeAt(i) & 0xff);
k1 = (((k1 & 0xffff) * c1) + ((((k1 >>> 16) * c1) & 0xffff) << 16)) & 0xffffffff;
k1 = (k1 << 15) | (k1 >>> 17);
k1 = (((k1 & 0xffff) * c2) + ((((k1 >>> 16) * c2) & 0xffff) << 16)) & 0xffffffff;
h1 ^= k1;
}
h1 ^= key.length;
h1 ^= h1 >>> 16;
h1 = (((h1 & 0xffff) * 0x85ebca6b) + ((((h1 >>> 16) * 0x85ebca6b) & 0xffff) << 16)) & 0xffffffff;
h1 ^= h1 >>> 13;
h1 = ((((h1 & 0xffff) * 0xc2b2ae35) + ((((h1 >>> 16) * 0xc2b2ae35) & 0xffff) << 16))) & 0xffffffff;
h1 ^= h1 >>> 16;
return h1 >>> 0;
},
// https://bugzilla.mozilla.org/show_bug.cgi?id=781447
hasLocalStorage: function () {
try {
return !!window.localStorage;
} catch (e) {
return true; // SecurityError when referencing it means it exists
}
},
hasSessionStorage: function () {
try {
return !!window.sessionStorage;
} catch (e) {
return true; // SecurityError when referencing it means it exists
}
},
hasIndexDb: function () {
try {
return !!window.indexedDB;
} catch (e) {
return true; // SecurityError when referencing it means it exists
}
},
isCanvasSupported: function () {
var elem = document.createElement('canvas');
return !!(elem.getContext && elem.getContext('2d'));
},
isIE: function () {
if (navigator.appName === 'Microsoft Internet Explorer') {
return true;
} else if (navigator.appName === 'Netscape' && /Trident/.test(navigator.userAgent)) {// IE 11
return true;
}
return false;
},
getPluginsString: function () {
if (this.isIE() && this.ie_activex) {
return this.getIEPluginsString();
} else {
return this.getRegularPluginsString();
}
},
getRegularPluginsString: function () {
return this.map(navigator.plugins, function (p) {
var mimeTypes = this.map(p, function (mt) {
return [mt.type, mt.suffixes].join('~');
}).join(',');
return [p.name, p.description, mimeTypes].join('::');
}, this).join(';');
},
getIEPluginsString: function () {
if (window.ActiveXObject) {
var names = ['ShockwaveFlash.ShockwaveFlash',//flash plugin
'AcroPDF.PDF', // Adobe PDF reader 7+
'PDF.PdfCtrl', // Adobe PDF reader 6 and earlier, brrr
'QuickTime.QuickTime', // QuickTime
// 5 versions of real players
'rmocx.RealPlayer G2 Control',
'rmocx.RealPlayer G2 Control.1',
'RealPlayer.RealPlayer(tm) ActiveX Control (32-bit)',
'RealVideo.RealVideo(tm) ActiveX Control (32-bit)',
'RealPlayer',
'SWCtl.SWCtl', // ShockWave player
'WMPlayer.OCX', // Windows media player
'AgControl.AgControl', // Silverlight
'Skype.Detection'];
// starting to detect plugins in IE
return this.map(names, function (name) {
try {
new ActiveXObject(name);
return name;
} catch (e) {
return null;
}
}).join(';');
} else {
return ""; // behavior prior version 0.5.0, not breaking backwards compat.
}
},
getScreenResolution: function () {
var resolution;
if (this.screen_orientation) {
resolution = (screen.height > screen.width) ? [screen.height, screen.width] : [screen.width, screen.height];
} else {
resolution = [screen.height, screen.width];
}
return resolution;
},
getCanvasFingerprint: function () {
var canvas = document.createElement('canvas');
var ctx = canvas.getContext('2d');
// https://www.browserleaks.com/canvas#how-does-it-work
var txt = 'http://valve.github.io';
ctx.textBaseline = "top";
ctx.font = "14px 'Arial'";
ctx.textBaseline = "alphabetic";
ctx.fillStyle = "#f60";
ctx.fillRect(125, 1, 62, 20);
ctx.fillStyle = "#069";
ctx.fillText(txt, 2, 15);
ctx.fillStyle = "rgba(102, 204, 0, 0.7)";
ctx.fillText(txt, 4, 17);
return canvas.toDataURL();
}
};
return Fingerprint;
});
new Fingerprint({canvas: true}).get();
Bypass canvas based detection
By rewriting the toBlob and toDataURL method to solve this.
var inject = function () {
var overwrite = function (name) {
const OLD = HTMLCanvasElement.prototype[name];
Object.defineProperty(HTMLCanvasElement.prototype, name, {
"value": function () {
var shift = {
'r': Math.floor(Math.random() * 10) - 5,
'g': Math.floor(Math.random() * 10) - 5,
'b': Math.floor(Math.random() * 10) - 5,
'a': Math.floor(Math.random() * 10) - 5
};
var width = this.width, height = this.height, context = this.getContext("2d");
var imageData = context.getImageData(0, 0, width, height);
for (var i = 0; i < height; i++) {
for (var j = 0; j < width; j++) {
var n = ((i * (width * 4)) + (j * 4));
imageData.data[n + 0] = imageData.data[n + 0] + shift.r;
imageData.data[n + 1] = imageData.data[n + 1] + shift.g;
imageData.data[n + 2] = imageData.data[n + 2] + shift.b;
imageData.data[n + 3] = imageData.data[n + 3] + shift.a;
}
}
context.putImageData(imageData, 0, 0);
return OLD.apply(this, arguments);
}
});
};
overwrite('toBlob');
overwrite('toDataURL');
};
inject();
- webglVendorAndRenderer
The return will be Card model related information
- adBlock
Detection of adBlock plugin exists
- hasLiedLanguages
Determine by whether navigator.language is the first in the array of navigator.languages
- hasLiedResolution
window.screen.width < window. screen.availWidth || window.screen.height < window.screen.availHeight
- hasLiedOs
Detection
Determine the connection between UA and Platform.
var getHasLiedOs = function () {
var userAgent = navigator.userAgent.toLowerCase()
var oscpu = navigator.oscpu
var platform = navigator.platform.toLowerCase()
var os
// We extract the OS from the user agent (respect the order of the if else if statement)
if (userAgent.indexOf('windows phone') >= 0) {
os = 'Windows Phone'
} else if (userAgent.indexOf('win') >= 0) {
os = 'Windows'
} else if (userAgent.indexOf('android') >= 0) {
os = 'Android'
} else if (userAgent.indexOf('linux') >= 0 || userAgent.indexOf('cros') >= 0) {
os = 'Linux'
} else if (userAgent.indexOf('iphone') >= 0 || userAgent.indexOf('ipad') >= 0) {
os = 'iOS'
} else if (userAgent.indexOf('mac') >= 0) {
os = 'Mac'
} else {
os = 'Other'
}
// We detect if the person uses a mobile device
var mobileDevice = (('ontouchstart' in window) ||
(navigator.maxTouchPoints > 0) ||
(navigator.msMaxTouchPoints > 0))
if (mobileDevice && os !== 'Windows Phone' && os !== 'Android' && os !== 'iOS' && os !== 'Other') {
return true
}
console.log(oscpu);
// We compare oscpu with the OS extracted from the UA
if (typeof oscpu !== 'undefined') {
oscpu = oscpu.toLowerCase()
if (oscpu.indexOf('win') >= 0 && os !== 'Windows' && os !== 'Windows Phone') {
return true
} else if (oscpu.indexOf('linux') >= 0 && os !== 'Linux' && os !== 'Android') {
return true
} else if (oscpu.indexOf('mac') >= 0 && os !== 'Mac' && os !== 'iOS') {
return true
} else if ((oscpu.indexOf('win') === -1 && oscpu.indexOf('linux') === -1 && oscpu.indexOf('mac') === -1) !== (os === 'Other')) {
return true
}
}
// We compare platform with the OS extracted from the UA
if (platform.indexOf('win') >= 0 && os !== 'Windows' && os !== 'Windows Phone') {
return true
} else if ((platform.indexOf('linux') >= 0 || platform.indexOf('android') >= 0 || platform.indexOf('pike') >= 0) && os !== 'Linux' && os !== 'Android') {
return true
} else if ((platform.indexOf('mac') >= 0 || platform.indexOf('ipad') >= 0 || platform.indexOf('ipod') >= 0 || platform.indexOf('iphone') >= 0) && os !== 'Mac' && os !== 'iOS') {
return true
} else {
var platformIsOther = platform.indexOf('win') < 0 &&
platform.indexOf('linux') < 0 &&
platform.indexOf('mac') < 0 &&
platform.indexOf('iphone') < 0 &&
platform.indexOf('ipad') < 0
if (platformIsOther !== (os === 'Other')) {
return true
}
}
return typeof navigator.plugins === 'undefined' && os !== 'Windows' && os !== 'Windows Phone'
}
Bypass detection
To ensure there is a relationship among the UA and Platform.
- hasLiedBrowser
Detection
It determines the browser based on the UA in accordance with some of the magical characteristics to check.
var getHasLiedBrowser = function () {
var userAgent = navigator.userAgent.toLowerCase()
var productSub = navigator.productSub
// we extract the browser from the user agent (respect the order of the tests)
var browser
if (userAgent.indexOf('firefox') >= 0) {
browser = 'Firefox'
} else if (userAgent.indexOf('opera') >= 0 || userAgent.indexOf('opr') >= 0) {
browser = 'Opera'
} else if (userAgent.indexOf('chrome') >= 0) {
browser = 'Chrome'
} else if (userAgent.indexOf('safari') >= 0) {
browser = 'Safari'
} else if (userAgent.indexOf('trident') >= 0) {
browser = 'Internet Explorer'
} else {
browser = 'Other'
}
if ((browser === 'Chrome' || browser === 'Safari' || browser === 'Opera') && productSub !== '20030107') {
return true
}
// eslint-disable-next-line no-eval
var tempRes = eval.toString().length
if (tempRes === 37 && browser !== 'Safari' && browser !== 'Firefox' && browser !== 'Other') {
return true
} else if (tempRes === 39 && browser !== 'Internet Explorer' && browser !== 'Other') {
return true
} else if (tempRes === 33 && browser !== 'Chrome' && browser !== 'Opera' && browser !== 'Other') {
return true
}
// We create an error to see how it is handled
var errFirefox
try {
// eslint-disable-next-line no-throw-literal
throw 'a'
} catch (err) {
try {
err.toSource()
errFirefox = true
} catch (errOfErr) {
errFirefox = false
}
}
return errFirefox && browser !== 'Firefox' && browser !== 'Other'
}
- touchSupport
var getTouchSupport = function () {
var maxTouchPoints = 0
var touchEvent
if (typeof navigator.maxTouchPoints !== 'undefined') {
maxTouchPoints = navigator.maxTouchPoints
} else if (typeof navigator.msMaxTouchPoints !== 'undefined') {
maxTouchPoints = navigator.msMaxTouchPoints
}
try {
document.createEvent('TouchEvent')
touchEvent = true
} catch (_) {
touchEvent = false
}
var touchStart = 'ontouchstart' in window
// [0, false, false]
return [maxTouchPoints, touchEvent, touchStart]
}
- fonts
Returns a font size
Sites where one may check the browser fingerprints
- https://bot.sannysoft.com
- https://fingerprintjs.github.io/fingerprintjs/
- https://antoinevastel.com/bots/