When we use Selenium or Node.js + Puppeteer to run [headless] Chrome/Chromium we might need to add some extra functionality/conditions to launch browsers with. Below you’ll find all kinds of Conditions and their explanations.
The Chromium Team has made a page on which they briefly explain how to use these switches.
Recently I noticed the question about extracting emails, phones, links(urls) from text fragments and immediately I decided to write this short post.
Each of the following: email, phones, link, form a category that falls under/matches a certain text pattern. What are the text patterns ? These are regexes, aka regex patterns, short for regular expressions. Eg. most emails fit into the following regex pattern:
^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$
The post will share with you the difference between the production and development builds of PWA. If you are not familiar to PWA (Progressive Web Application) please visit that blog post.
I’ve met with the challenge that composer failed to load guzzle library file:
https://packagist.org/p/guzzlehttp/guzzle%241f150aaa79afd8bc5d6f08f730634a0d60 f5dfcd1dd4a6fc5263fb4b1cefeb16.json" file could not be downloaded (HTTP/1.1 404 Not Found)
The solution has been the following:
composer clear-cache
Let me tell you what you already know! Octoparse is a great web scraping tool! But like every great tool, it’s got its limitations. At times, you may wonder if there are any alternatives to Octoparse. We wondered the same and put together this blog to provide you a short list of Octoparse alternatives along with their features and distinguishing factors. Let’s get started!
…we use your personal data so we can provide the best service, tell you about products and services you may be interested in…
These or similar statements are often “tiny printed” at most of modern sites as part of Terms of Service (ToS). Below we share with you what particular data are collected from web users or app users.
Question: What is Selenium web scraping?
Answer: A picture is better than 1000 words:
So, you make a program with Python, PHP, JAVA, Ruby and whatever language you use in order to browse(), select(), click(), submit(), save(), etc., target web pages.
The LinkedIn crawl success rate is low; one request that a bot makes might require several retries to be successful. So, here we share the crucial Linkedin scraping guide lines.
import sys
import requests
URL = 'https://portal.bitcasa.com/login'
client = requests.session()
# Retrieve the CSRF token first
client.get(URL) # sets cookie
if 'csrftoken' in client.cookies:
# Django 1.6 and up
csrftoken = client.cookies['csrftoken']
else:
# older versions
csrftoken = client.cookies['csrf']
# Pass CSRF token both in login parameters (csrfmiddlewaretoken)
# and in the session cookies (csrf in client.cookies)
login_data = dict(username=EMAIL, password=PASSWORD, csrfmiddlewaretoken=csrftoken, next='/')
r = client.post(URL, data=login_data, headers=dict(Referer=URL))
My goal was to retrieve data from a web business directory.
Since the business directories scrape is the most challenging task (beside SERP scrape) there are some basic questions for me to answer: