Categories
Development

Headless Chrome detection and anti-detection

In the post we summarize how to detect the headless Chrome browser and how to bypass the detection. The headless browser testing should be a very important part of todays web 2.0. If we look at some of the site’s JS, we find them to checking on many fields of a browser. They are similar to those collected by fingerprintjs2.

So in this post we consider most of them and show both how to detect the headless browser by those attributes and how to bypass that detection by spoofing them.

See the test results of disguising the browser automation for both Selenium and Puppeteer extra.

  • userAgent

UA can be said to be the most basic of the field to spoof in a headless browser. Yet simply modifying and spoofing the UA will not work. You need both UA and some of the fields such as the navigator.platform to modify too. Of course, a simple UA tries to mimic the desktop browser.

Detection

Through the navigator.userAgent one gets UA to identify whether browser is a crawler.

Bypass detection

You can modify the properties by the following:

Object.defineProperty(navigator, 'userAgent', {
    get: () => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'
});
  • webdriver

The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation

Detection

The desktop Chrome browser returns navigator.webdriver as undefined. One can be detect by: !!navigator.webdriver

Bypass detection

Object.defineProperty(navigator, 'webdriver', {
    get: () => false
});

Or by executing:

delete navigator.proto.webdriver
Instead of the monitoring whether navigator.webdriver is undefined, one may simply check it through !!navigator.webdriver.

  • language

Different browsers support for the language property is different; Chrome is only for navigator.language reference

Detection

navigator.language || navigator.userLanguage || navigator.browserLanguage || navigator.systemLanguage if there is one.

Bypass detection

Object.defineProperty(navigator, 'language', {
    get: () => "zh-CN",
});
  • colorDepth

window.screen.colorDepth of the screen color depth, is how to collect fingerprints when used. Both headless browser and a desktop browser have generally the same value in this field.

  • deviceMemory

The read-only attribute, it returns the approximate machine memory in gigabytes. This value is an approximation of a power of 2 divided by 1024, rounding off to the decimal point. Yet this is just an experimental feature, not all browsers support it. It will be used during fingerprint collection, just in case you can set a value.

navigator.deviceMemory

Bypass detection

Object.defineProperty(navigator, 'deviceMemory', {
    get: () => 8
});
  • hardwareConcurrency

The browser environment with the number of CPU cores, to check:

navigator.hardwareConcurrency

Bypass detection

Object.defineProperty(navigator, 'hardwareConcurrency', {
    get: () => 8
});
  • screenresolution

Check the following to fingerprint:

window.screen.width and window.screen.height

  • availableScreenResolution

Check the following to fingerprint: window.screen.availHeight and window.screen.availWidth

  • timezoneOffset

It sets world time UTC with respect to the current time zone time difference value, in units of minutes. Set it up the following for bypassing:

new Date().getTimezoneOffset()

  • timezone
new window.Intl.DateTimeFormat().resolvedOptions().timeZone
  • sessionStorage *

First you determine whether there is sessionStorage, then:

!!window.sessionStorage

  • localStorage *

!!window.localStorage

  • indexedDb

!!window.indexedDB

  • addBehavior

This field is unique to IE

!!(document.body && document.body.addBehavior)

  • openDatabase *

It will return a database where one may execute sql. The fingerprint only judges whether there is this method.

!!window.openDatabase

  • cpuClass

It seems that Chrome browser returns undefined.

navigator.cpuClass

  • platform *

navigator.platform

Detection

if (navigator.platform === "Linux x86_64")

Yet only a small share of browers has Linux x86_64. Better you use MacIntel or Win32.

Bypass Detection

Object.defineProperty(navigator, 'platform', {
    get: () => 'MacIntel'
});
  • plugins

navigator.plugins

The plugin object has name, filename, description and version attributes. You can use the plugin[0] to get the MimeType object.

  • mimeTypes

navigator.mimeTypes

The mimeTypes objects need and plugin object corresponds to a type, suffixes, description, enabledPlugin property.

  • canvas

In order to identify a browser we get canvas fingerprint reference. Then we identify a browser by the fingerprint, and then identify what browser it is based on some characteristics of the fingerprint.

Canvas based detection

(function (name, context, definition) {
        if (typeof module !== 'undefined' && module.exports) {
            module.exports = definition();
        }
        else if (typeof define === 'function' && define.amd) {
            define(definition);
        }
        else {
            context[name] = definition();
        }
    })('Fingerprint', this, function () {
        'use strict';

        var Fingerprint = function (options) {
            var nativeForEach, nativeMap;
            nativeForEach = Array.prototype.forEach;
            nativeMap = Array.prototype.map;

            this.each = function (obj, iterator, context) {
                if (obj === null) {
                    return;
                }
                if (nativeForEach && obj.forEach === nativeForEach) {
                    obj.forEach(iterator, context);
                } else if (obj.length === +obj.length) {
                    for (var i = 0, l = obj.length; i < l; i++) {
                        if (iterator.call(context, obj[i], i, obj) === {}) return;
                    }
                } else {
                    for (var key in obj) {
                        if (obj.hasOwnProperty(key)) {
                            if (iterator.call(context, obj[key], key, obj) === {}) return;
                        }
                    }
                }
            };

            this.map = function (obj, iterator, context) {
                var results = [];
                // Not using strict equality so that this acts as a
                // shortcut to checking for `null` and `undefined`.
                if (obj == null) return results;
                if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
                this.each(obj, function (value, index, list) {
                    results[results.length] = iterator.call(context, value, index, list);
                });
                return results;
            };

            if (typeof options == 'object') {
                this.hasher = options.hasher;
                this.screen_resolution = options.screen_resolution;
                this.screen_orientation = options.screen_orientation;
                this.canvas = options.canvas;
                this.ie_activex = options.ie_activex;
            } else if (typeof options == 'function') {
                this.hasher = options;
            }
        };

        Fingerprint.prototype = {
            get: function () {
                var keys = [];
                keys.push(navigator.userAgent);
                keys.push(navigator.language);
                keys.push(screen.colorDepth);
                if (this.screen_resolution) {
                    var resolution = this.getScreenResolution();
                    if (typeof resolution !== 'undefined') { // headless browsers, such as phantomjs
                        keys.push(resolution.join('x'));
                    }
                }
                keys.push(new Date().getTimezoneOffset());
                keys.push(this.hasSessionStorage());
                keys.push(this.hasLocalStorage());
                keys.push(this.hasIndexDb());
                //body might not be defined at this point or removed programmatically
                if (document.body) {
                    keys.push(typeof(document.body.addBehavior));
                } else {
                    keys.push(typeof undefined);
                }
                keys.push(typeof(window.openDatabase));
                keys.push(navigator.cpuClass);
                keys.push(navigator.platform);
                keys.push(navigator.doNotTrack);
                keys.push(this.getPluginsString());
                if (this.canvas && this.isCanvasSupported()) {
                    keys.push(this.getCanvasFingerprint());
                }
                if (this.hasher) {
                    return this.hasher(keys.join('###'), 31);
                } else {
                    return this.murmurhash3_32_gc(keys.join('###'), 31);
                }
            },

            /**
             * JS Implementation of MurmurHash3 (r136) (as of May 20, 2011)
             *
             * @author Gary Court
             * @see http://github.com/garycourt/murmurhash-js
             * @author Austin Appleby
             * @see http://sites.google.com/site/murmurhash/
             *
             * @param {string} key ASCII only
             * @param {number} seed Positive integer only
             * @return {number} 32-bit positive integer hash
             */

            murmurhash3_32_gc: function (key, seed) {
                var remainder, bytes, h1, h1b, c1, c2, k1, i;

                remainder = key.length & 3; // key.length % 4
                bytes = key.length - remainder;
                h1 = seed;
                c1 = 0xcc9e2d51;
                c2 = 0x1b873593;
                i = 0;

                while (i < bytes) {
                    k1 =
                        ((key.charCodeAt(i) & 0xff)) |
                        ((key.charCodeAt(++i) & 0xff) << 8) |
                        ((key.charCodeAt(++i) & 0xff) << 16) |
                        ((key.charCodeAt(++i) & 0xff) << 24);
                    ++i;

                    k1 = ((((k1 & 0xffff) * c1) + ((((k1 >>> 16) * c1) & 0xffff) << 16))) & 0xffffffff;
                    k1 = (k1 << 15) | (k1 >>> 17);
                    k1 = ((((k1 & 0xffff) * c2) + ((((k1 >>> 16) * c2) & 0xffff) << 16))) & 0xffffffff;

                    h1 ^= k1;
                    h1 = (h1 << 13) | (h1 >>> 19);
                    h1b = ((((h1 & 0xffff) * 5) + ((((h1 >>> 16) * 5) & 0xffff) << 16))) & 0xffffffff;
                    h1 = (((h1b & 0xffff) + 0x6b64) + ((((h1b >>> 16) + 0xe654) & 0xffff) << 16));
                }

                k1 = 0;

                switch (remainder) {
                    case 3:
                        k1 ^= (key.charCodeAt(i + 2) & 0xff) << 16;
                    case 2:
                        k1 ^= (key.charCodeAt(i + 1) & 0xff) << 8;
                    case 1:
                        k1 ^= (key.charCodeAt(i) & 0xff);

                        k1 = (((k1 & 0xffff) * c1) + ((((k1 >>> 16) * c1) & 0xffff) << 16)) & 0xffffffff;
                        k1 = (k1 << 15) | (k1 >>> 17);
                        k1 = (((k1 & 0xffff) * c2) + ((((k1 >>> 16) * c2) & 0xffff) << 16)) & 0xffffffff;
                        h1 ^= k1;
                }

                h1 ^= key.length;

                h1 ^= h1 >>> 16;
                h1 = (((h1 & 0xffff) * 0x85ebca6b) + ((((h1 >>> 16) * 0x85ebca6b) & 0xffff) << 16)) & 0xffffffff;
                h1 ^= h1 >>> 13;
                h1 = ((((h1 & 0xffff) * 0xc2b2ae35) + ((((h1 >>> 16) * 0xc2b2ae35) & 0xffff) << 16))) & 0xffffffff;
                h1 ^= h1 >>> 16;

                return h1 >>> 0;
            },

            // https://bugzilla.mozilla.org/show_bug.cgi?id=781447
            hasLocalStorage: function () {
                try {
                    return !!window.localStorage;
                } catch (e) {
                    return true; // SecurityError when referencing it means it exists
                }
            },

            hasSessionStorage: function () {
                try {
                    return !!window.sessionStorage;
                } catch (e) {
                    return true; // SecurityError when referencing it means it exists
                }
            },

            hasIndexDb: function () {
                try {
                    return !!window.indexedDB;
                } catch (e) {
                    return true; // SecurityError when referencing it means it exists
                }
            },

            isCanvasSupported: function () {
                var elem = document.createElement('canvas');
                return !!(elem.getContext && elem.getContext('2d'));
            },

            isIE: function () {
                if (navigator.appName === 'Microsoft Internet Explorer') {
                    return true;
                } else if (navigator.appName === 'Netscape' && /Trident/.test(navigator.userAgent)) {// IE 11
                    return true;
                }
                return false;
            },

            getPluginsString: function () {
                if (this.isIE() && this.ie_activex) {
                    return this.getIEPluginsString();
                } else {
                    return this.getRegularPluginsString();
                }
            },

            getRegularPluginsString: function () {
                return this.map(navigator.plugins, function (p) {
                    var mimeTypes = this.map(p, function (mt) {
                        return [mt.type, mt.suffixes].join('~');
                    }).join(',');
                    return [p.name, p.description, mimeTypes].join('::');
                }, this).join(';');
            },

            getIEPluginsString: function () {
                if (window.ActiveXObject) {
                    var names = ['ShockwaveFlash.ShockwaveFlash',//flash plugin
                        'AcroPDF.PDF', // Adobe PDF reader 7+
                        'PDF.PdfCtrl', // Adobe PDF reader 6 and earlier, brrr
                        'QuickTime.QuickTime', // QuickTime
                        // 5 versions of real players
                        'rmocx.RealPlayer G2 Control',
                        'rmocx.RealPlayer G2 Control.1',
                        'RealPlayer.RealPlayer(tm) ActiveX Control (32-bit)',
                        'RealVideo.RealVideo(tm) ActiveX Control (32-bit)',
                        'RealPlayer',
                        'SWCtl.SWCtl', // ShockWave player
                        'WMPlayer.OCX', // Windows media player
                        'AgControl.AgControl', // Silverlight
                        'Skype.Detection'];

                    // starting to detect plugins in IE
                    return this.map(names, function (name) {
                        try {
                            new ActiveXObject(name);
                            return name;
                        } catch (e) {
                            return null;
                        }
                    }).join(';');
                } else {
                    return ""; // behavior prior version 0.5.0, not breaking backwards compat.
                }
            },

            getScreenResolution: function () {
                var resolution;
                if (this.screen_orientation) {
                    resolution = (screen.height > screen.width) ? [screen.height, screen.width] : [screen.width, screen.height];
                } else {
                    resolution = [screen.height, screen.width];
                }
                return resolution;
            },

            getCanvasFingerprint: function () {
                var canvas = document.createElement('canvas');
                var ctx = canvas.getContext('2d');
                // https://www.browserleaks.com/canvas#how-does-it-work
                var txt = 'http://valve.github.io';
                ctx.textBaseline = "top";
                ctx.font = "14px 'Arial'";
                ctx.textBaseline = "alphabetic";
                ctx.fillStyle = "#f60";
                ctx.fillRect(125, 1, 62, 20);
                ctx.fillStyle = "#069";
                ctx.fillText(txt, 2, 15);
                ctx.fillStyle = "rgba(102, 204, 0, 0.7)";
                ctx.fillText(txt, 4, 17);
                return canvas.toDataURL();
            }
        };
        return Fingerprint;
    });

    new Fingerprint({canvas: true}).get();

Bypass canvas based detection

By rewriting the toBlob and toDataURL method to solve this.

  var inject = function () {
    var overwrite = function (name) {
      const OLD = HTMLCanvasElement.prototype[name];
      Object.defineProperty(HTMLCanvasElement.prototype, name, {
        "value": function () {
          var shift = {
            'r': Math.floor(Math.random() * 10) - 5,
            'g': Math.floor(Math.random() * 10) - 5,
            'b': Math.floor(Math.random() * 10) - 5,
            'a': Math.floor(Math.random() * 10) - 5
          };
          var width = this.width, height = this.height, context = this.getContext("2d");
          var imageData = context.getImageData(0, 0, width, height);
          for (var i = 0; i < height; i++) {
            for (var j = 0; j < width; j++) {
              var n = ((i * (width * 4)) + (j * 4));
              imageData.data[n + 0] = imageData.data[n + 0] + shift.r;
              imageData.data[n + 1] = imageData.data[n + 1] + shift.g;
              imageData.data[n + 2] = imageData.data[n + 2] + shift.b;
              imageData.data[n + 3] = imageData.data[n + 3] + shift.a;
            }
          }
          context.putImageData(imageData, 0, 0);
          return OLD.apply(this, arguments);
        }
      });
    };
    overwrite('toBlob');
    overwrite('toDataURL');
  };
  inject();
  • webglVendorAndRenderer

The return will be Card model related information

  • adBlock

Detection of adBlock plugin exists

  • hasLiedLanguages

Determine by whether navigator.language is the first in the array of navigator.languages

  • hasLiedResolution

window.screen.width < window. screen.availWidth || window.screen.height < window.screen.availHeight

  • hasLiedOs

Detection

Determine the connection between UA and Platform.

var getHasLiedOs = function () {
    var userAgent = navigator.userAgent.toLowerCase()
    var oscpu = navigator.oscpu
    var platform = navigator.platform.toLowerCase()
    var os
    // We extract the OS from the user agent (respect the order of the if else if statement)
    if (userAgent.indexOf('windows phone') >= 0) {
      os = 'Windows Phone'
    } else if (userAgent.indexOf('win') >= 0) {
      os = 'Windows'
    } else if (userAgent.indexOf('android') >= 0) {
      os = 'Android'
    } else if (userAgent.indexOf('linux') >= 0 || userAgent.indexOf('cros') >= 0) {
      os = 'Linux'
    } else if (userAgent.indexOf('iphone') >= 0 || userAgent.indexOf('ipad') >= 0) {
      os = 'iOS'
    } else if (userAgent.indexOf('mac') >= 0) {
      os = 'Mac'
    } else {
      os = 'Other'
    }
    // We detect if the person uses a mobile device
    var mobileDevice = (('ontouchstart' in window) ||
      (navigator.maxTouchPoints > 0) ||
      (navigator.msMaxTouchPoints > 0))

    if (mobileDevice && os !== 'Windows Phone' && os !== 'Android' && os !== 'iOS' && os !== 'Other') {
      return true
    }

    console.log(oscpu);
    // We compare oscpu with the OS extracted from the UA
    if (typeof oscpu !== 'undefined') {
      oscpu = oscpu.toLowerCase()
      if (oscpu.indexOf('win') >= 0 && os !== 'Windows' && os !== 'Windows Phone') {
        return true
      } else if (oscpu.indexOf('linux') >= 0 && os !== 'Linux' && os !== 'Android') {
        return true
      } else if (oscpu.indexOf('mac') >= 0 && os !== 'Mac' && os !== 'iOS') {
        return true
      } else if ((oscpu.indexOf('win') === -1 && oscpu.indexOf('linux') === -1 && oscpu.indexOf('mac') === -1) !== (os === 'Other')) {
        return true
      }
    }

    // We compare platform with the OS extracted from the UA
    if (platform.indexOf('win') >= 0 && os !== 'Windows' && os !== 'Windows Phone') {
      return true
    } else if ((platform.indexOf('linux') >= 0 || platform.indexOf('android') >= 0 || platform.indexOf('pike') >= 0) && os !== 'Linux' && os !== 'Android') {
      return true
    } else if ((platform.indexOf('mac') >= 0 || platform.indexOf('ipad') >= 0 || platform.indexOf('ipod') >= 0 || platform.indexOf('iphone') >= 0) && os !== 'Mac' && os !== 'iOS') {
      return true
    } else {
      var platformIsOther = platform.indexOf('win') < 0 &&
        platform.indexOf('linux') < 0 &&
        platform.indexOf('mac') < 0 &&
        platform.indexOf('iphone') < 0 &&
        platform.indexOf('ipad') < 0
      if (platformIsOther !== (os === 'Other')) {
        return true
      }
    }

    return typeof navigator.plugins === 'undefined' && os !== 'Windows' && os !== 'Windows Phone'
  }

Bypass detection

To ensure there is a relationship among the UA and Platform.

  • hasLiedBrowser

Detection

It determines the browser based on the UA in accordance with some of the magical characteristics to check.

var getHasLiedBrowser = function () {
    var userAgent = navigator.userAgent.toLowerCase()
    var productSub = navigator.productSub

    // we extract the browser from the user agent (respect the order of the tests)
    var browser
    if (userAgent.indexOf('firefox') >= 0) {
      browser = 'Firefox'
    } else if (userAgent.indexOf('opera') >= 0 || userAgent.indexOf('opr') >= 0) {
      browser = 'Opera'
    } else if (userAgent.indexOf('chrome') >= 0) {
      browser = 'Chrome'
    } else if (userAgent.indexOf('safari') >= 0) {
      browser = 'Safari'
    } else if (userAgent.indexOf('trident') >= 0) {
      browser = 'Internet Explorer'
    } else {
      browser = 'Other'
    }

    if ((browser === 'Chrome' || browser === 'Safari' || browser === 'Opera') && productSub !== '20030107') {
      return true
    }

    // eslint-disable-next-line no-eval
    var tempRes = eval.toString().length
    if (tempRes === 37 && browser !== 'Safari' && browser !== 'Firefox' && browser !== 'Other') {
      return true
    } else if (tempRes === 39 && browser !== 'Internet Explorer' && browser !== 'Other') {
      return true
    } else if (tempRes === 33 && browser !== 'Chrome' && browser !== 'Opera' && browser !== 'Other') {
      return true
    }

    // We create an error to see how it is handled
    var errFirefox
    try {
      // eslint-disable-next-line no-throw-literal
      throw 'a'
    } catch (err) {
      try {
        err.toSource()
        errFirefox = true
      } catch (errOfErr) {
        errFirefox = false
      }
    }
    return errFirefox && browser !== 'Firefox' && browser !== 'Other'
  }
  • touchSupport
  var getTouchSupport = function () {
    var maxTouchPoints = 0
    var touchEvent
    if (typeof navigator.maxTouchPoints !== 'undefined') {
      maxTouchPoints = navigator.maxTouchPoints
    } else if (typeof navigator.msMaxTouchPoints !== 'undefined') {
      maxTouchPoints = navigator.msMaxTouchPoints
    }
    try {
      document.createEvent('TouchEvent')
      touchEvent = true
    } catch (_) {
      touchEvent = false
    }
    var touchStart = 'ontouchstart' in window
    // [0, false, false]
    return [maxTouchPoints, touchEvent, touchStart]
  }
  • fonts

Returns a font size

Sites where one may check the browser fingerprints

  • https://bot.sannysoft.com
  • https://fingerprintjs.github.io/fingerprintjs/
  • https://antoinevastel.com/bots/

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.