Google can use machine learning in crawling


Google employee Martin Splitt said that a search engine can use machine learning in crawling. It is not yet involved in rendering.

In scanning, machine learning is used for two purposes:

  • To determine the quality of the URL
  • To determine the freshness of content

Definition of quality

“I know we are using machine learning to determine or predict what we will get from a scan in terms of quality,” Splitt said.

This allows Google to schedule crawls more efficiently, he said.

Machine learning helps a search engine know if the resource is worth spending on URL crawling based on quality.

Determination of freshness

Google can also use machine learning to figure out if a site needs to be crawled every day or less, based on the frequency of content updates.

At the same time, Splitt noted that he could not say for sure what the use of ML in scanning is - an active functional or an experiment.

You can listen to Splitt's question and answer to it starting at 19:58 in the video below:

This issue was raised during a regular JavaScript SEO hangout for webmasters.